109 lines
6.8 KiB
Markdown
109 lines
6.8 KiB
Markdown
IT PRO PARTNER — DOCUMENT 1 OF 4
|
||
Disaster Recovery Master Plan
|
||
v2.0 — Firm RTO/RPO targets, failover design, and failback procedure
|
||
Field
|
||
Value
|
||
Document status
|
||
ACTIVE — supersedes v1 draft and all prior DR notes
|
||
Date
|
||
July 9, 2026
|
||
Priority directive
|
||
Uptime is critical. Full-service restorability takes priority over cost or convenience.
|
||
Owner
|
||
Network Services Team
|
||
Companion documents
|
||
Backup & Security Hardening Standard; Per-Server Runbooks; DR Testing & Validation Schedule
|
||
|
||
1. Priority Directive
|
||
This plan is written under one governing rule: uptime and full-service restorability are the top priority, ahead of infrastructure cost, engineering convenience, or process overhead. Every target below is set to that standard — they are commitments, not estimates, and they drive the hardening work in the companion Backup & Security Hardening Standard.
|
||
2. RTO / RPO Targets — Committed
|
||
These are the recovery targets ITPP is designing and testing against. They replace the “not yet defined” placeholders from the v1 draft.
|
||
Tier / System
|
||
RTO (target)
|
||
RPO (target)
|
||
Basis
|
||
Core (Hermes)
|
||
5 minutes
|
||
15 minutes
|
||
Warm standby (app1-bu) + 15-min Hermes state sync
|
||
app1-bu (standby)
|
||
N/A — is the recovery path for Core
|
||
≤ 15 minutes (sync lag)
|
||
Continuous local state sync from Core
|
||
app1 / app2 / app3 (Standard tier)
|
||
60 minutes
|
||
24 hours
|
||
Rebuild from S3 + docker compose up, daily backups
|
||
wphost02 (WordPress)
|
||
2 hours
|
||
24 hours
|
||
Rebuild + DB restore, daily dumps
|
||
fleettracker360
|
||
2 hours
|
||
24 hours
|
||
Rebuild + DB restore, daily dumps
|
||
Legacy Hetzner hosts
|
||
4 hours
|
||
24 hours
|
||
Rebuild via rescue mode, daily backups
|
||
|
||
Why 5 minutes for Core, not less: A 2-minute RTO leaves no margin for S3 sync, service startup, DNS/cache propagation, or transient network issues. 5 minutes is achievable with the failover design in Section 3 and still leaves real margin. Do not compress this further without re-testing.
|
||
Compliance rule: These targets are only real once they are tested (see DR Testing & Validation Schedule). Until a target has passed a live test, treat it as “target, unverified,” not “achieved.”
|
||
3. Core Failover Design — app1 / app1-bu
|
||
3.1 Detection and trigger
|
||
Parameter
|
||
Value
|
||
Health check interval
|
||
30 seconds
|
||
Consecutive failed checks to trigger failover
|
||
4 (→ ~2 minutes detection time)
|
||
Check sources required to agree
|
||
app1-bu local check AND one external monitor
|
||
Startup buffer (service start + validation)
|
||
45–90 seconds
|
||
Resulting effective RTO
|
||
~3–3.5 minutes actual, 5 minutes committed target
|
||
|
||
A single missed check must never trigger failover — it has to clear 4 consecutive failures from two independent vantage points before app1-bu is allowed to act. This is what prevents a transient network blip on one path from causing an unnecessary takeover.
|
||
3.2 Split-brain prevention — required before go-live
|
||
Mechanism: an active/standby lock file maintained in S3, checked before any failover.
|
||
s3://hermes-vps-backups/live/active-lock.json
|
||
{ "active_node": "core", "heartbeat": "<timestamp>", "lock_holder": "core" }
|
||
Core writes a heartbeat to this object every 30 seconds while healthy. Failover sequence:
|
||
Confirm Core unreachable for 4 consecutive checks (local).
|
||
Confirm external monitor independently reports Core down.
|
||
Confirm the active-lock heartbeat is stale (older than 90 seconds).
|
||
app1-bu writes itself in as lock_holder and active_node.
|
||
app1-bu starts the Hermes gateway.
|
||
Alert the on-call operator immediately, success or failure.
|
||
Fencing: Where reachable, app1-bu should also attempt to firewall-isolate or force-stop Core before going active, to eliminate any chance both nodes serve traffic at once. If Core cannot be reached to fence, proceed on the lock file alone but flag the incident as high severity for manual review.
|
||
3.3 Failback procedure — authoritative steps
|
||
Failback, not failover, is where most DR programs actually lose data — typically because the restored primary starts serving before it has the standby's newer state. This sequence is mandatory and may not be shortcut during a live incident.
|
||
Confirm app1-bu is currently the authoritative active node (check active-lock.json).
|
||
Keep Core services stopped after it is repaired/restored — do not allow auto-start.
|
||
Restore Core's OS and application stack per its runbook.
|
||
Sync the latest state from app1-bu (or the authoritative S3 state) into Core.
|
||
Validate the synced state's timestamp is newer than Core's last known state before the outage.
|
||
Start Core in passive/maintenance mode only — not yet serving traffic.
|
||
Stop the Hermes gateway on app1-bu.
|
||
Update active-lock.json: active_node = core.
|
||
Start the Hermes gateway on Core.
|
||
Confirm Core health checks pass for at least 5 consecutive checks.
|
||
Release the standby lock; return app1-bu to passive standby mode.
|
||
Monitor both nodes for 30 minutes before closing the incident.
|
||
Rollback: If Core fails health checks after step 9, immediately revert: set active_node back to app1-bu in the lock file, restart the Hermes gateway on app1-bu, and keep Core in maintenance mode until root cause is found. Do not retry failback more than once without a human decision to proceed.
|
||
4. Standby Capacity Check
|
||
app1-bu runs on the Standby tier (Hetzner CPX11 — 2 vCPU / 2GB RAM / 40GB). Under the failover design above, it must be able to run the Hermes gateway plus health-check/lock tooling concurrently within that footprint.
|
||
Action required: Load-test app1-bu with the Hermes gateway running under representative traffic before relying on the 5-minute RTO commitment. If headroom is marginal, upgrade the standby tier rather than compress the failover design to fit — uptime takes priority over the ~$8/mo savings of staying on CPX11.
|
||
5. Restore Path Priority Order
|
||
During an incident, follow this order to protect the uptime priority:
|
||
If Core is down but app1-bu is healthy → failover (Section 3), not restore. This is the fastest path back to service.
|
||
If a non-Core server is down and S3 is reachable → Standard Restore (see Per-Server Runbooks).
|
||
If S3 is unreachable → Emergency Restore using the secondary backup destination defined in the Backup & Security Hardening Standard, before falling back to Hudu-documented manual rebuild.
|
||
If both backup destinations and the standby are unavailable → declare a major incident and engage the full DR tabletop response, not a routine runbook.
|
||
6. Document Control
|
||
This is the live, authoritative DR plan. Any older document describing different failover timing (e.g., 5-minute polling) is superseded as of this version and should be marked:
|
||
SUPERSEDED: Replaced by Disaster Recovery Master Plan v2.0 (July 9, 2026). Do not use for active DR procedures.
|
||
Companion documents — Backup & Security Hardening Standard, Per-Server Runbooks, and the DR Testing & Validation Schedule — are equally authoritative for their respective scope and are maintained alongside this plan.
|
||
|
||
Next: see the DR Testing & Validation Schedule for the cadence that converts these targets from committed to verified. |