Initial resurrection kit — 81 scripts, 62 references, configs, systemd units, crons, Docker compose files, Caddy config, master README

references/3-Per-Server-Runbooks.md

@@ -0,0 +1,227 @@
+IT PRO PARTNER — DOCUMENT 3 OF 4
+Per-Server Runbooks
+Exact restore steps, dependencies, and validation checks for every host
+Field
+Value
+Document status
+ACTIVE — fill in bracketed [ ] fields with live DNS/port/secret values
+Date
+July 9, 2026
+Companion documents
+DR Master Plan v2; Backup & Security Hardening Standard; DR Testing & Validation Schedule
+
+Each runbook below follows the same structure so responders can find what they need under pressure without hunting. Restore commands use the standard base-install and S3 pull process from the Server Provisioning Standard. Fields in [brackets] are site-specific values to confirm and fill in — they are structural placeholders, not unknowns to guess at.
+
+Core
+Hermes orchestration hub — highest priority, protected by warm standby (app1-bu)
+Field
+Value
+Tier
+Light — Hetzner/netcup RS 2000 G12, 8C/16GB/512GB, Debian 13
+RTO / RPO target
+5 min / 15 min (see DR Master Plan v2, Section 2)
+Dependencies
+app1-bu (standby), S3 backup bucket, [DNS provider], [external monitor service]
+DNS records
+[list A/CNAME records pointing to Core, e.g. hermes.itpp.internal]
+Firewall ports
+22 (SSH, key-only), [Hermes gateway port], [any inter-service ports]
+Backup path
+s3://hermes-vps-backups/live/ (15-min), s3://hermes-vps-backups/core/ (daily)
+Required secrets
+[.env for Hermes gateway], [DB credentials if applicable], itpp-infra SSH key
+
+Restore Procedure
+If app1-bu is healthy, use failover (DR Master Plan v2, Section 3) instead of a full restore — it is faster and is the primary recovery path for Core.
+If a full rebuild is required: provision a Light-tier server, run base install (hostname, updates, timezone, fail2ban, node-exporter, ufw).
+Deploy itpp-infra SSH key; disable password auth and root login.
+Pull latest Hermes state: aws s3 cp s3://hermes-vps-backups/live/latest/ /opt/restore/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive
+Pull latest daily config/backup set from s3://hermes-vps-backups/core/latest/.
+Restore Docker stacks: cd /opt/<service> && docker compose up -d
+Do not accept live traffic until health checks below pass.
+Health & Validation Checks
+Hermes gateway responds on its health endpoint
+MEMORY.md loads without truncation or parse errors
+Last Hermes state timestamp is within 15 minutes of restore time
+active-lock.json reflects Core as active_node before releasing standby
+Failback
+Follow the full failback sequence in DR Master Plan v2, Section 3.3 — do not improvise during an incident.
+Known Failure Modes
+Core restored from a stale backup while app1-bu holds newer state — mitigated by the mandatory sync-before-start step in failback
+Both Core and app1-bu active simultaneously — mitigated by the active-lock file and fencing
+MEMORY.md over-pruned or corrupted on restore — cross-check against the priority-tagging scheme before resuming automation
+
+app1-bu (Warm Standby)
+Standby tier, activates the Hermes gateway when Core is confirmed down
+Field
+Value
+Tier
+Standby — Hetzner CPX11, 2C/2G/40GB, Debian 12
+Role
+Passive by default; becomes active only via the failover sequence in DR Master Plan v2, Section 3
+Dependencies
+Local state cache, S3 (secondary path), external monitor for quorum check
+DNS records
+[failover target / low-TTL record used to redirect traffic on activation]
+Firewall ports
+22 (SSH, key-only), [Hermes gateway port, opened only on activation]
+Backup path
+N/A — consumes Core's backups; does not independently back up production data
+Required secrets
+Same Hermes gateway .env as Core, kept in sync but inactive until failover
+
+Restore Procedure
+app1-bu itself is restored like any Standby-tier host if it fails: provision replacement CPX11, base install, deploy itpp-infra key.
+Re-establish the local state cache sync from Core (Backup & Security Hardening Standard, Section 4).
+Confirm standby readiness checklist below before trusting it as a failover target again.
+Health & Validation Checks
+Can reach primary and secondary S3 backup destinations
+Local state cache is less than 15 minutes stale
+Hermes gateway package/image present and startable
+Required secrets present and match Core's current .env
+Disk usage under 80%
+active-lock.json reachable and readable
+Known Failure Modes
+Standby capacity marginal under real load — see DR Master Plan v2, Section 4 load-test action item
+Local cache silently falls stale without alerting — mitigated by the sync-freshness alert in the Backup & Security Hardening Standard
+
+app1 / app2 / app3
+Standard tier application hosts
+Field
+Value
+Tier
+Standard — RS 4000 G12, 12C/32GB/1TB NVMe, Debian 13
+RTO / RPO target
+60 min / 24 hours
+Dependencies
+[list per-app: DB, reverse proxy, upstream APIs]
+DNS records
+[A/CNAME records per app]
+Firewall ports
+22 (SSH), 80/443 (Caddy, if serving web)
+Backup path
+s3://hermes-vps-backups/<server>/
+Required secrets
+/opt/<service>/.env (mode 600) per stack
+
+Restore Procedure
+Provision a Standard-tier server (same tier or better).
+Run base install (Section 2 of DR Master Plan / provisioning standard).
+Deploy itpp-infra SSH key; disable password auth and root login.
+aws s3 cp s3://hermes-vps-backups/<server>/latest/ /opt/restore/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive
+cd /opt/<service> && docker compose up -d for each stack.
+Point DNS/traffic at the restored host only after health checks pass.
+Docker Volume Restore
+docker volume ls — confirm all expected volumes are present after restore.
+docker inspect <volume> — confirm mount paths match the pre-incident layout.
+Spot-check restored volume contents against the last known-good file or record.
+Health & Validation Checks
+App-specific health endpoint returns 200
+Docker Compose stack reports all containers healthy
+Restored data timestamp is within the 24-hour RPO window
+Known Failure Modes
+Docker volume not actually included in the backup job — verify against the inventory table in the Backup & Security Hardening Standard before assuming restore is complete
+
+wphost02
+WordPress host — MariaDB-backed
+Field
+Value
+Tier
+[confirm tier — Standard or Legacy]
+RTO / RPO target
+2 hours / 24 hours
+Dependencies
+MariaDB, [WordPress plugins/uploads directory], Caddy
+DNS records
+[WordPress site domain(s)]
+Firewall ports
+22, 80/443
+Backup path
+s3://hermes-vps-backups/wphost02/
+Required secrets
+WordPress DB credentials, [any API keys in wp-config.php]
+
+Restore Procedure
+Provision replacement host matching current tier.
+Run base install; deploy itpp-infra SSH key.
+aws s3 cp s3://hermes-vps-backups/wphost02/latest/ /opt/restore/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive
+Restore Docker/Caddy stack: cd /opt/wordpress && docker compose up -d
+Database Restore
+Confirm the latest MariaDB dump exists and is non-empty in the restored backup set.
+Import the dump into a fresh MariaDB instance before pointing WordPress at it.
+Verify wp-config.php DB credentials match the restored instance.
+Spot-check a known page/post to confirm data integrity.
+Health & Validation Checks
+Site loads over HTTPS with valid cert
+WordPress admin login succeeds
+Latest DB dump timestamp is within 24 hours (RPO compliance)
+Known Failure Modes
+Scheduled DB dump not actually running — flagged as a known gap; verify before trusting this runbook's RPO claim
+Note: Automated MariaDB dump scheduling for this host needs verification — see Backup & Security Hardening Standard, Section 6.
+
+fleettracker360
+Fleet tracking application — MariaDB-backed
+Field
+Value
+Tier
+[confirm tier — Standard or Legacy]
+RTO / RPO target
+2 hours / 24 hours
+Dependencies
+MariaDB, [application-specific services]
+DNS records
+[fleettracker360 domain(s)]
+Firewall ports
+22, [app port(s)]
+Backup path
+s3://hermes-vps-backups/fleettracker360/
+Required secrets
+DB credentials, [any third-party API keys]
+
+Restore Procedure
+Provision replacement host matching current tier.
+Run base install; deploy itpp-infra SSH key.
+aws s3 cp s3://hermes-vps-backups/fleettracker360/latest/ /opt/restore/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive
+Restore Docker stack: cd /opt/fleettracker360 && docker compose up -d
+Database Restore
+Confirm latest MariaDB dump exists and is non-empty.
+Import into a fresh instance before pointing the app at it.
+Validate against a known vehicle/record ID.
+Health & Validation Checks
+Application health endpoint returns 200
+Latest DB dump timestamp is within 24 hours
+Known Failure Modes
+Scheduled DB dump not actually running — same known gap as wphost02; verify before trusting this runbook's RPO claim
+Note: Automated MariaDB dump scheduling for this host needs verification — see Backup & Security Hardening Standard, Section 6.
+
+Legacy Hetzner Hosts
+CPX21 / CPX41 boxes not yet migrated to the current standard
+Field
+Value
+Tier
+Legacy — CPX21 or CPX41, Debian 12
+RTO / RPO target
+4 hours / 24 hours
+Dependencies
+[list per host]
+DNS records
+[list per host]
+Firewall ports
+[list per host]
+Backup path
+s3://hermes-vps-backups/<server>/
+Required secrets
+[per host]
+
+Restore Procedure
+If S3 is reachable: provision replacement, base install, restore from S3 per the Standard Restore process.
+If S3 is unreachable: use Hetzner rescue mode to reinstall OS, redeploy itpp-infra key, and rebuild from Hudu-documented configs (Emergency Restore).
+Plan to migrate to Debian 13 and current tier standards on next rebuild rather than restoring like-for-like.
+Health & Validation Checks
+Service-specific health check for whatever this host runs
+Confirm restored config matches the most recent Hudu documentation
+Known Failure Modes
+Emergency Restore path depends entirely on Hudu documentation being complete and current — audit this quarterly
+
+
+Fill in all [bracketed] fields before this document is relied upon in a live incident. Track completion in the DR Testing & Validation Schedule.