69 lines
3.6 KiB
Markdown
69 lines
3.6 KiB
Markdown
# Incident Response Plan
|
|
|
|
## 1. Incident Severity Levels
|
|
|
|
* **P0 (Critical):** Production down, data loss, security breach.
|
|
* **Response Time:** Within 15 minutes
|
|
* **P1 (High):** Degraded service, single server down.
|
|
* **Response Time:** Within 1 hour
|
|
* **P2 (Medium):** Non-critical service issue.
|
|
* **Response Time:** Within 4 hours
|
|
* **P3 (Low):** Cosmetic, minor issue.
|
|
* **Response Time:** Within 24 hours
|
|
|
|
## 2. Alert Sources
|
|
|
|
Alerts are aggregated from the following monitoring systems:
|
|
|
|
* **Hermes `service-health-check.sh`:** Runs every 5 minutes, alerts routed to Telegram.
|
|
* **Hermes Standby Watchdog:** Monitors primary system health, alerts to Telegram and email upon failover.
|
|
* **Uptime Kuma (Docker Box):** External endpoint and service monitoring.
|
|
* **VPS Threshold Checks:** Alerts on CPU, Memory, Disk, and Network utilization limits.
|
|
* **Backup Audit Watchdog:** Daily verification of backup integrity and completion.
|
|
|
|
## 3. Response Team
|
|
|
|
* **CEO:** Germaine — Final decision authority, executive communications.
|
|
* **Lead Sysadmin:** Sho'Nuff — First responder, triage, root cause analysis, and resolution.
|
|
* **Operations Manager:** Anita — Internal and external communications, client notification.
|
|
* **Developer:** Tony — Application-level troubleshooting and code-level fixes.
|
|
|
|
## 4. Incident Response Steps
|
|
|
|
1. **Detect:** Alert received via Telegram, email, or client report.
|
|
2. **Triage:** Determine severity level (P0-P3), impact scope, and assign an Incident Owner.
|
|
3. **Contain:** Stop the bleeding. Immediate actions may include failing over to standby, blocking malicious IPs, or stopping affected services to prevent further damage or data loss.
|
|
4. **Investigate:** Identify the root cause, determine affected systems, and assess any data exposure.
|
|
5. **Resolve:** Apply the necessary fix and verify that the issue is mitigated.
|
|
6. **Recover:** Restore systems from backup if needed, and comprehensively verify all services are operating normally.
|
|
7. **Post-Mortem:** Document the root cause, establish a timeline of events, detail the applied fix, and define prevention measures.
|
|
|
|
## 5. Communication Plan
|
|
|
|
* **Internal Communication:**
|
|
* **P0/P1:** Immediate communication via dedicated Telegram incident channel.
|
|
* **P2/P3:** Communication via internal ticketing system and email.
|
|
* **Client Notification:** Notify affected clients within **1 hour** of a confirmed outage.
|
|
* **Update Frequency (During Active Incident):**
|
|
* **P0:** Updates every 30 minutes.
|
|
* **P1:** Updates every 2 hours.
|
|
|
|
## 6. Escalation Procedures
|
|
|
|
* **Initial Escalation:** If the first responder (Lead Sysadmin) does not acknowledge a P0/P1 alert within **15 minutes**, the alert automatically escalates to the **CEO**.
|
|
* **Vendor Escalation:** If the issue remains unresolved after **2 hours** of active investigation, escalate to external vendor support.
|
|
|
|
## 7. Key Contacts
|
|
|
|
* **netcup Support:** [Refer to `.env` for current contact details / PIN]
|
|
* **Hetzner Support:** [via Cloud Console / Support Ticket]
|
|
* **Wasabi Support:** [via Wasabi Management Console]
|
|
* **Cloudflare Support:** [via Cloudflare Dashboard]
|
|
* **MXroute Support:** [via MXroute Billing Portal / Support Desk]
|
|
|
|
## 8. Post-Incident Actions
|
|
|
|
* **Log Update:** Update the Disaster Recovery (DR) issue log with incident details.
|
|
* **Runbook/Skill Update:** If a new procedure, workaround, or troubleshooting step was discovered, document it and add it to the Hermes skills/runbooks.
|
|
* **Follow-up:** Schedule and track follow-up tasks to permanently fix the root cause and prevent recurrence.
|