# Incident Response Plan

## 1. Incident Severity Levels

*   **P0 (Critical):** Production down, data loss, security breach.
    *   **Response Time:** Within 15 minutes
*   **P1 (High):** Degraded service, single server down.
    *   **Response Time:** Within 1 hour
*   **P2 (Medium):** Non-critical service issue.
    *   **Response Time:** Within 4 hours
*   **P3 (Low):** Cosmetic, minor issue.
    *   **Response Time:** Within 24 hours

## 2. Alert Sources

Alerts are aggregated from the following monitoring systems:

*   **Hermes `service-health-check.sh`:** Runs every 5 minutes, alerts routed to Telegram.
*   **Hermes Standby Watchdog:** Monitors primary system health, alerts to Telegram and email upon failover.
*   **Uptime Kuma (Docker Box):** External endpoint and service monitoring.
*   **VPS Threshold Checks:** Alerts on CPU, Memory, Disk, and Network utilization limits.
*   **Backup Audit Watchdog:** Daily verification of backup integrity and completion.

## 3. Response Team

*   **CEO:** Germaine — Final decision authority, executive communications.
*   **Lead Sysadmin:** Sho'Nuff — First responder, triage, root cause analysis, and resolution.
*   **Operations Manager:** Anita — Internal and external communications, client notification.
*   **Developer:** Tony — Application-level troubleshooting and code-level fixes.

## 4. Incident Response Steps

1.  **Detect:** Alert received via Telegram, email, or client report.
2.  **Triage:** Determine severity level (P0-P3), impact scope, and assign an Incident Owner.
3.  **Contain:** Stop the bleeding. Immediate actions may include failing over to standby, blocking malicious IPs, or stopping affected services to prevent further damage or data loss.
4.  **Investigate:** Identify the root cause, determine affected systems, and assess any data exposure.
5.  **Resolve:** Apply the necessary fix and verify that the issue is mitigated.
6.  **Recover:** Restore systems from backup if needed, and comprehensively verify all services are operating normally.
7.  **Post-Mortem:** Document the root cause, establish a timeline of events, detail the applied fix, and define prevention measures.

## 5. Communication Plan

*   **Internal Communication:**
    *   **P0/P1:** Immediate communication via dedicated Telegram incident channel.
    *   **P2/P3:** Communication via internal ticketing system and email.
*   **Client Notification:** Notify affected clients within **1 hour** of a confirmed outage.
*   **Update Frequency (During Active Incident):**
    *   **P0:** Updates every 30 minutes.
    *   **P1:** Updates every 2 hours.

## 6. Escalation Procedures

*   **Initial Escalation:** If the first responder (Lead Sysadmin) does not acknowledge a P0/P1 alert within **15 minutes**, the alert automatically escalates to the **CEO**.
*   **Vendor Escalation:** If the issue remains unresolved after **2 hours** of active investigation, escalate to external vendor support.

## 7. Key Contacts

*   **netcup Support:** [Refer to `.env` for current contact details / PIN]
*   **Hetzner Support:** [via Cloud Console / Support Ticket]
*   **Wasabi Support:** [via Wasabi Management Console]
*   **Cloudflare Support:** [via Cloudflare Dashboard]
*   **MXroute Support:** [via MXroute Billing Portal / Support Desk]

## 8. Post-Incident Actions

*   **Log Update:** Update the Disaster Recovery (DR) issue log with incident details.
*   **Runbook/Skill Update:** If a new procedure, workaround, or troubleshooting step was discovered, document it and add it to the Hermes skills/runbooks.
*   **Follow-up:** Schedule and track follow-up tasks to permanently fix the root cause and prevent recurrence.