Initial resurrection kit — 81 scripts, 62 references, configs, systemd units, crons, Docker compose files, Caddy config, master README

references/incident-response-plan.md

@@ -0,0 +1,68 @@
+# Incident Response Plan
+
+## 1. Incident Severity Levels
+
+*   **P0 (Critical):** Production down, data loss, security breach.
+    *   **Response Time:** Within 15 minutes
+*   **P1 (High):** Degraded service, single server down.
+    *   **Response Time:** Within 1 hour
+*   **P2 (Medium):** Non-critical service issue.
+    *   **Response Time:** Within 4 hours
+*   **P3 (Low):** Cosmetic, minor issue.
+    *   **Response Time:** Within 24 hours
+
+## 2. Alert Sources
+
+Alerts are aggregated from the following monitoring systems:
+
+*   **Hermes `service-health-check.sh`:** Runs every 5 minutes, alerts routed to Telegram.
+*   **Hermes Standby Watchdog:** Monitors primary system health, alerts to Telegram and email upon failover.
+*   **Uptime Kuma (Docker Box):** External endpoint and service monitoring.
+*   **VPS Threshold Checks:** Alerts on CPU, Memory, Disk, and Network utilization limits.
+*   **Backup Audit Watchdog:** Daily verification of backup integrity and completion.
+
+## 3. Response Team
+
+*   **CEO:** Germaine — Final decision authority, executive communications.
+*   **Lead Sysadmin:** Sho'Nuff — First responder, triage, root cause analysis, and resolution.
+*   **Operations Manager:** Anita — Internal and external communications, client notification.
+*   **Developer:** Tony — Application-level troubleshooting and code-level fixes.
+
+## 4. Incident Response Steps
+
+1.  **Detect:** Alert received via Telegram, email, or client report.
+2.  **Triage:** Determine severity level (P0-P3), impact scope, and assign an Incident Owner.
+3.  **Contain:** Stop the bleeding. Immediate actions may include failing over to standby, blocking malicious IPs, or stopping affected services to prevent further damage or data loss.
+4.  **Investigate:** Identify the root cause, determine affected systems, and assess any data exposure.
+5.  **Resolve:** Apply the necessary fix and verify that the issue is mitigated.
+6.  **Recover:** Restore systems from backup if needed, and comprehensively verify all services are operating normally.
+7.  **Post-Mortem:** Document the root cause, establish a timeline of events, detail the applied fix, and define prevention measures.
+
+## 5. Communication Plan
+
+*   **Internal Communication:**
+    *   **P0/P1:** Immediate communication via dedicated Telegram incident channel.
+    *   **P2/P3:** Communication via internal ticketing system and email.
+*   **Client Notification:** Notify affected clients within **1 hour** of a confirmed outage.
+*   **Update Frequency (During Active Incident):**
+    *   **P0:** Updates every 30 minutes.
+    *   **P1:** Updates every 2 hours.
+
+## 6. Escalation Procedures
+
+*   **Initial Escalation:** If the first responder (Lead Sysadmin) does not acknowledge a P0/P1 alert within **15 minutes**, the alert automatically escalates to the **CEO**.
+*   **Vendor Escalation:** If the issue remains unresolved after **2 hours** of active investigation, escalate to external vendor support.
+
+## 7. Key Contacts
+
+*   **netcup Support:** [Refer to `.env` for current contact details / PIN]
+*   **Hetzner Support:** [via Cloud Console / Support Ticket]
+*   **Wasabi Support:** [via Wasabi Management Console]
+*   **Cloudflare Support:** [via Cloudflare Dashboard]
+*   **MXroute Support:** [via MXroute Billing Portal / Support Desk]
+
+## 8. Post-Incident Actions
+
+*   **Log Update:** Update the Disaster Recovery (DR) issue log with incident details.
+*   **Runbook/Skill Update:** If a new procedure, workaround, or troubleshooting step was discovered, document it and add it to the Hermes skills/runbooks.
+*   **Follow-up:** Schedule and track follow-up tasks to permanently fix the root cause and prevent recurrence.