9.6 KiB
IT PRO PARTNER — DOCUMENT 3 OF 4 Per-Server Runbooks Exact restore steps, dependencies, and validation checks for every host Field Value Document status ACTIVE — fill in bracketed [ ] fields with live DNS/port/secret values Date July 9, 2026 Companion documents DR Master Plan v2; Backup & Security Hardening Standard; DR Testing & Validation Schedule
Each runbook below follows the same structure so responders can find what they need under pressure without hunting. Restore commands use the standard base-install and S3 pull process from the Server Provisioning Standard. Fields in [brackets] are site-specific values to confirm and fill in — they are structural placeholders, not unknowns to guess at.
Core Hermes orchestration hub — highest priority, protected by warm standby (app1-bu) Field Value Tier Light — Hetzner/netcup RS 2000 G12, 8C/16GB/512GB, Debian 13 RTO / RPO target 5 min / 15 min (see DR Master Plan v2, Section 2) Dependencies app1-bu (standby), S3 backup bucket, [DNS provider], [external monitor service] DNS records [list A/CNAME records pointing to Core, e.g. hermes.itpp.internal] Firewall ports 22 (SSH, key-only), [Hermes gateway port], [any inter-service ports] Backup path s3://hermes-vps-backups/live/ (15-min), s3://hermes-vps-backups/core/ (daily) Required secrets [.env for Hermes gateway], [DB credentials if applicable], itpp-infra SSH key
Restore Procedure If app1-bu is healthy, use failover (DR Master Plan v2, Section 3) instead of a full restore — it is faster and is the primary recovery path for Core. If a full rebuild is required: provision a Light-tier server, run base install (hostname, updates, timezone, fail2ban, node-exporter, ufw). Deploy itpp-infra SSH key; disable password auth and root login. Pull latest Hermes state: aws s3 cp s3://hermes-vps-backups/live/latest/ /opt/restore/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive Pull latest daily config/backup set from s3://hermes-vps-backups/core/latest/. Restore Docker stacks: cd /opt/ && docker compose up -d Do not accept live traffic until health checks below pass. Health & Validation Checks Hermes gateway responds on its health endpoint MEMORY.md loads without truncation or parse errors Last Hermes state timestamp is within 15 minutes of restore time active-lock.json reflects Core as active_node before releasing standby Failback Follow the full failback sequence in DR Master Plan v2, Section 3.3 — do not improvise during an incident. Known Failure Modes Core restored from a stale backup while app1-bu holds newer state — mitigated by the mandatory sync-before-start step in failback Both Core and app1-bu active simultaneously — mitigated by the active-lock file and fencing MEMORY.md over-pruned or corrupted on restore — cross-check against the priority-tagging scheme before resuming automation
app1-bu (Warm Standby) Standby tier, activates the Hermes gateway when Core is confirmed down Field Value Tier Standby — Hetzner CPX11, 2C/2G/40GB, Debian 12 Role Passive by default; becomes active only via the failover sequence in DR Master Plan v2, Section 3 Dependencies Local state cache, S3 (secondary path), external monitor for quorum check DNS records [failover target / low-TTL record used to redirect traffic on activation] Firewall ports 22 (SSH, key-only), [Hermes gateway port, opened only on activation] Backup path N/A — consumes Core's backups; does not independently back up production data Required secrets Same Hermes gateway .env as Core, kept in sync but inactive until failover
Restore Procedure app1-bu itself is restored like any Standby-tier host if it fails: provision replacement CPX11, base install, deploy itpp-infra key. Re-establish the local state cache sync from Core (Backup & Security Hardening Standard, Section 4). Confirm standby readiness checklist below before trusting it as a failover target again. Health & Validation Checks Can reach primary and secondary S3 backup destinations Local state cache is less than 15 minutes stale Hermes gateway package/image present and startable Required secrets present and match Core's current .env Disk usage under 80% active-lock.json reachable and readable Known Failure Modes Standby capacity marginal under real load — see DR Master Plan v2, Section 4 load-test action item Local cache silently falls stale without alerting — mitigated by the sync-freshness alert in the Backup & Security Hardening Standard
app1 / app2 / app3 Standard tier application hosts Field Value Tier Standard — RS 4000 G12, 12C/32GB/1TB NVMe, Debian 13 RTO / RPO target 60 min / 24 hours Dependencies [list per-app: DB, reverse proxy, upstream APIs] DNS records [A/CNAME records per app] Firewall ports 22 (SSH), 80/443 (Caddy, if serving web) Backup path s3://hermes-vps-backups// Required secrets /opt//.env (mode 600) per stack
Restore Procedure Provision a Standard-tier server (same tier or better). Run base install (Section 2 of DR Master Plan / provisioning standard). Deploy itpp-infra SSH key; disable password auth and root login. aws s3 cp s3://hermes-vps-backups//latest/ /opt/restore/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive cd /opt/ && docker compose up -d for each stack. Point DNS/traffic at the restored host only after health checks pass. Docker Volume Restore docker volume ls — confirm all expected volumes are present after restore. docker inspect — confirm mount paths match the pre-incident layout. Spot-check restored volume contents against the last known-good file or record. Health & Validation Checks App-specific health endpoint returns 200 Docker Compose stack reports all containers healthy Restored data timestamp is within the 24-hour RPO window Known Failure Modes Docker volume not actually included in the backup job — verify against the inventory table in the Backup & Security Hardening Standard before assuming restore is complete
wphost02 WordPress host — MariaDB-backed Field Value Tier [confirm tier — Standard or Legacy] RTO / RPO target 2 hours / 24 hours Dependencies MariaDB, [WordPress plugins/uploads directory], Caddy DNS records [WordPress site domain(s)] Firewall ports 22, 80/443 Backup path s3://hermes-vps-backups/wphost02/ Required secrets WordPress DB credentials, [any API keys in wp-config.php]
Restore Procedure Provision replacement host matching current tier. Run base install; deploy itpp-infra SSH key. aws s3 cp s3://hermes-vps-backups/wphost02/latest/ /opt/restore/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive Restore Docker/Caddy stack: cd /opt/wordpress && docker compose up -d Database Restore Confirm the latest MariaDB dump exists and is non-empty in the restored backup set. Import the dump into a fresh MariaDB instance before pointing WordPress at it. Verify wp-config.php DB credentials match the restored instance. Spot-check a known page/post to confirm data integrity. Health & Validation Checks Site loads over HTTPS with valid cert WordPress admin login succeeds Latest DB dump timestamp is within 24 hours (RPO compliance) Known Failure Modes Scheduled DB dump not actually running — flagged as a known gap; verify before trusting this runbook's RPO claim Note: Automated MariaDB dump scheduling for this host needs verification — see Backup & Security Hardening Standard, Section 6.
fleettracker360 Fleet tracking application — MariaDB-backed Field Value Tier [confirm tier — Standard or Legacy] RTO / RPO target 2 hours / 24 hours Dependencies MariaDB, [application-specific services] DNS records [fleettracker360 domain(s)] Firewall ports 22, [app port(s)] Backup path s3://hermes-vps-backups/fleettracker360/ Required secrets DB credentials, [any third-party API keys]
Restore Procedure Provision replacement host matching current tier. Run base install; deploy itpp-infra SSH key. aws s3 cp s3://hermes-vps-backups/fleettracker360/latest/ /opt/restore/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive Restore Docker stack: cd /opt/fleettracker360 && docker compose up -d Database Restore Confirm latest MariaDB dump exists and is non-empty. Import into a fresh instance before pointing the app at it. Validate against a known vehicle/record ID. Health & Validation Checks Application health endpoint returns 200 Latest DB dump timestamp is within 24 hours Known Failure Modes Scheduled DB dump not actually running — same known gap as wphost02; verify before trusting this runbook's RPO claim Note: Automated MariaDB dump scheduling for this host needs verification — see Backup & Security Hardening Standard, Section 6.
Legacy Hetzner Hosts CPX21 / CPX41 boxes not yet migrated to the current standard Field Value Tier Legacy — CPX21 or CPX41, Debian 12 RTO / RPO target 4 hours / 24 hours Dependencies [list per host] DNS records [list per host] Firewall ports [list per host] Backup path s3://hermes-vps-backups// Required secrets [per host]
Restore Procedure If S3 is reachable: provision replacement, base install, restore from S3 per the Standard Restore process. If S3 is unreachable: use Hetzner rescue mode to reinstall OS, redeploy itpp-infra key, and rebuild from Hudu-documented configs (Emergency Restore). Plan to migrate to Debian 13 and current tier standards on next rebuild rather than restoring like-for-like. Health & Validation Checks Service-specific health check for whatever this host runs Confirm restored config matches the most recent Hudu documentation Known Failure Modes Emergency Restore path depends entirely on Hudu documentation being complete and current — audit this quarterly
Fill in all [bracketed] fields before this document is relied upon in a live incident. Track completion in the DR Testing & Validation Schedule.