hermes-recovery/references/incident-response-plan.md

Incident Response Plan

1. Incident Severity Levels

• P0 (Critical): Production down, data loss, security breach.
  • Response Time: Within 15 minutes
• P1 (High): Degraded service, single server down.
  • Response Time: Within 1 hour
• P2 (Medium): Non-critical service issue.
  • Response Time: Within 4 hours
• P3 (Low): Cosmetic, minor issue.
  • Response Time: Within 24 hours

2. Alert Sources

Alerts are aggregated from the following monitoring systems:

• Hermes service-health-check.sh: Runs every 5 minutes, alerts routed to Telegram.
• Hermes Standby Watchdog: Monitors primary system health, alerts to Telegram and email upon failover.
• Uptime Kuma (Docker Box): External endpoint and service monitoring.
• VPS Threshold Checks: Alerts on CPU, Memory, Disk, and Network utilization limits.
• Backup Audit Watchdog: Daily verification of backup integrity and completion.

3. Response Team

• CEO: Germaine — Final decision authority, executive communications.
• Lead Sysadmin: Sho'Nuff — First responder, triage, root cause analysis, and resolution.
• Operations Manager: Anita — Internal and external communications, client notification.
• Developer: Tony — Application-level troubleshooting and code-level fixes.

4. Incident Response Steps

1. Detect: Alert received via Telegram, email, or client report.
2. Triage: Determine severity level (P0-P3), impact scope, and assign an Incident Owner.
3. Contain: Stop the bleeding. Immediate actions may include failing over to standby, blocking malicious IPs, or stopping affected services to prevent further damage or data loss.
4. Investigate: Identify the root cause, determine affected systems, and assess any data exposure.
5. Resolve: Apply the necessary fix and verify that the issue is mitigated.
6. Recover: Restore systems from backup if needed, and comprehensively verify all services are operating normally.
7. Post-Mortem: Document the root cause, establish a timeline of events, detail the applied fix, and define prevention measures.

5. Communication Plan

• Internal Communication:
  • P0/P1: Immediate communication via dedicated Telegram incident channel.
  • P2/P3: Communication via internal ticketing system and email.
• Client Notification: Notify affected clients within 1 hour of a confirmed outage.
• Update Frequency (During Active Incident):
  • P0: Updates every 30 minutes.
  • P1: Updates every 2 hours.

6. Escalation Procedures

• Initial Escalation: If the first responder (Lead Sysadmin) does not acknowledge a P0/P1 alert within 15 minutes, the alert automatically escalates to the CEO.
• Vendor Escalation: If the issue remains unresolved after 2 hours of active investigation, escalate to external vendor support.

7. Key Contacts

• netcup Support: [Refer to .env for current contact details / PIN]
• Hetzner Support: [via Cloud Console / Support Ticket]
• Wasabi Support: [via Wasabi Management Console]
• Cloudflare Support: [via Cloudflare Dashboard]
• MXroute Support: [via MXroute Billing Portal / Support Desk]

8. Post-Incident Actions

• Log Update: Update the Disaster Recovery (DR) issue log with incident details.
• Runbook/Skill Update: If a new procedure, workaround, or troubleshooting step was discovered, document it and add it to the Hermes skills/runbooks.
• Follow-up: Schedule and track follow-up tasks to permanently fix the root cause and prevent recurrence.