227 lines
9.6 KiB
Markdown
227 lines
9.6 KiB
Markdown
IT PRO PARTNER — DOCUMENT 3 OF 4
|
|
Per-Server Runbooks
|
|
Exact restore steps, dependencies, and validation checks for every host
|
|
Field
|
|
Value
|
|
Document status
|
|
ACTIVE — fill in bracketed [ ] fields with live DNS/port/secret values
|
|
Date
|
|
July 9, 2026
|
|
Companion documents
|
|
DR Master Plan v2; Backup & Security Hardening Standard; DR Testing & Validation Schedule
|
|
|
|
Each runbook below follows the same structure so responders can find what they need under pressure without hunting. Restore commands use the standard base-install and S3 pull process from the Server Provisioning Standard. Fields in [brackets] are site-specific values to confirm and fill in — they are structural placeholders, not unknowns to guess at.
|
|
|
|
Core
|
|
Hermes orchestration hub — highest priority, protected by warm standby (app1-bu)
|
|
Field
|
|
Value
|
|
Tier
|
|
Light — Hetzner/netcup RS 2000 G12, 8C/16GB/512GB, Debian 13
|
|
RTO / RPO target
|
|
5 min / 15 min (see DR Master Plan v2, Section 2)
|
|
Dependencies
|
|
app1-bu (standby), S3 backup bucket, [DNS provider], [external monitor service]
|
|
DNS records
|
|
[list A/CNAME records pointing to Core, e.g. hermes.itpp.internal]
|
|
Firewall ports
|
|
22 (SSH, key-only), [Hermes gateway port], [any inter-service ports]
|
|
Backup path
|
|
s3://hermes-vps-backups/live/ (15-min), s3://hermes-vps-backups/core/ (daily)
|
|
Required secrets
|
|
[.env for Hermes gateway], [DB credentials if applicable], itpp-infra SSH key
|
|
|
|
Restore Procedure
|
|
If app1-bu is healthy, use failover (DR Master Plan v2, Section 3) instead of a full restore — it is faster and is the primary recovery path for Core.
|
|
If a full rebuild is required: provision a Light-tier server, run base install (hostname, updates, timezone, fail2ban, node-exporter, ufw).
|
|
Deploy itpp-infra SSH key; disable password auth and root login.
|
|
Pull latest Hermes state: aws s3 cp s3://hermes-vps-backups/live/latest/ /opt/restore/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive
|
|
Pull latest daily config/backup set from s3://hermes-vps-backups/core/latest/.
|
|
Restore Docker stacks: cd /opt/<service> && docker compose up -d
|
|
Do not accept live traffic until health checks below pass.
|
|
Health & Validation Checks
|
|
Hermes gateway responds on its health endpoint
|
|
MEMORY.md loads without truncation or parse errors
|
|
Last Hermes state timestamp is within 15 minutes of restore time
|
|
active-lock.json reflects Core as active_node before releasing standby
|
|
Failback
|
|
Follow the full failback sequence in DR Master Plan v2, Section 3.3 — do not improvise during an incident.
|
|
Known Failure Modes
|
|
Core restored from a stale backup while app1-bu holds newer state — mitigated by the mandatory sync-before-start step in failback
|
|
Both Core and app1-bu active simultaneously — mitigated by the active-lock file and fencing
|
|
MEMORY.md over-pruned or corrupted on restore — cross-check against the priority-tagging scheme before resuming automation
|
|
|
|
app1-bu (Warm Standby)
|
|
Standby tier, activates the Hermes gateway when Core is confirmed down
|
|
Field
|
|
Value
|
|
Tier
|
|
Standby — Hetzner CPX11, 2C/2G/40GB, Debian 12
|
|
Role
|
|
Passive by default; becomes active only via the failover sequence in DR Master Plan v2, Section 3
|
|
Dependencies
|
|
Local state cache, S3 (secondary path), external monitor for quorum check
|
|
DNS records
|
|
[failover target / low-TTL record used to redirect traffic on activation]
|
|
Firewall ports
|
|
22 (SSH, key-only), [Hermes gateway port, opened only on activation]
|
|
Backup path
|
|
N/A — consumes Core's backups; does not independently back up production data
|
|
Required secrets
|
|
Same Hermes gateway .env as Core, kept in sync but inactive until failover
|
|
|
|
Restore Procedure
|
|
app1-bu itself is restored like any Standby-tier host if it fails: provision replacement CPX11, base install, deploy itpp-infra key.
|
|
Re-establish the local state cache sync from Core (Backup & Security Hardening Standard, Section 4).
|
|
Confirm standby readiness checklist below before trusting it as a failover target again.
|
|
Health & Validation Checks
|
|
Can reach primary and secondary S3 backup destinations
|
|
Local state cache is less than 15 minutes stale
|
|
Hermes gateway package/image present and startable
|
|
Required secrets present and match Core's current .env
|
|
Disk usage under 80%
|
|
active-lock.json reachable and readable
|
|
Known Failure Modes
|
|
Standby capacity marginal under real load — see DR Master Plan v2, Section 4 load-test action item
|
|
Local cache silently falls stale without alerting — mitigated by the sync-freshness alert in the Backup & Security Hardening Standard
|
|
|
|
app1 / app2 / app3
|
|
Standard tier application hosts
|
|
Field
|
|
Value
|
|
Tier
|
|
Standard — RS 4000 G12, 12C/32GB/1TB NVMe, Debian 13
|
|
RTO / RPO target
|
|
60 min / 24 hours
|
|
Dependencies
|
|
[list per-app: DB, reverse proxy, upstream APIs]
|
|
DNS records
|
|
[A/CNAME records per app]
|
|
Firewall ports
|
|
22 (SSH), 80/443 (Caddy, if serving web)
|
|
Backup path
|
|
s3://hermes-vps-backups/<server>/
|
|
Required secrets
|
|
/opt/<service>/.env (mode 600) per stack
|
|
|
|
Restore Procedure
|
|
Provision a Standard-tier server (same tier or better).
|
|
Run base install (Section 2 of DR Master Plan / provisioning standard).
|
|
Deploy itpp-infra SSH key; disable password auth and root login.
|
|
aws s3 cp s3://hermes-vps-backups/<server>/latest/ /opt/restore/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive
|
|
cd /opt/<service> && docker compose up -d for each stack.
|
|
Point DNS/traffic at the restored host only after health checks pass.
|
|
Docker Volume Restore
|
|
docker volume ls — confirm all expected volumes are present after restore.
|
|
docker inspect <volume> — confirm mount paths match the pre-incident layout.
|
|
Spot-check restored volume contents against the last known-good file or record.
|
|
Health & Validation Checks
|
|
App-specific health endpoint returns 200
|
|
Docker Compose stack reports all containers healthy
|
|
Restored data timestamp is within the 24-hour RPO window
|
|
Known Failure Modes
|
|
Docker volume not actually included in the backup job — verify against the inventory table in the Backup & Security Hardening Standard before assuming restore is complete
|
|
|
|
wphost02
|
|
WordPress host — MariaDB-backed
|
|
Field
|
|
Value
|
|
Tier
|
|
[confirm tier — Standard or Legacy]
|
|
RTO / RPO target
|
|
2 hours / 24 hours
|
|
Dependencies
|
|
MariaDB, [WordPress plugins/uploads directory], Caddy
|
|
DNS records
|
|
[WordPress site domain(s)]
|
|
Firewall ports
|
|
22, 80/443
|
|
Backup path
|
|
s3://hermes-vps-backups/wphost02/
|
|
Required secrets
|
|
WordPress DB credentials, [any API keys in wp-config.php]
|
|
|
|
Restore Procedure
|
|
Provision replacement host matching current tier.
|
|
Run base install; deploy itpp-infra SSH key.
|
|
aws s3 cp s3://hermes-vps-backups/wphost02/latest/ /opt/restore/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive
|
|
Restore Docker/Caddy stack: cd /opt/wordpress && docker compose up -d
|
|
Database Restore
|
|
Confirm the latest MariaDB dump exists and is non-empty in the restored backup set.
|
|
Import the dump into a fresh MariaDB instance before pointing WordPress at it.
|
|
Verify wp-config.php DB credentials match the restored instance.
|
|
Spot-check a known page/post to confirm data integrity.
|
|
Health & Validation Checks
|
|
Site loads over HTTPS with valid cert
|
|
WordPress admin login succeeds
|
|
Latest DB dump timestamp is within 24 hours (RPO compliance)
|
|
Known Failure Modes
|
|
Scheduled DB dump not actually running — flagged as a known gap; verify before trusting this runbook's RPO claim
|
|
Note: Automated MariaDB dump scheduling for this host needs verification — see Backup & Security Hardening Standard, Section 6.
|
|
|
|
fleettracker360
|
|
Fleet tracking application — MariaDB-backed
|
|
Field
|
|
Value
|
|
Tier
|
|
[confirm tier — Standard or Legacy]
|
|
RTO / RPO target
|
|
2 hours / 24 hours
|
|
Dependencies
|
|
MariaDB, [application-specific services]
|
|
DNS records
|
|
[fleettracker360 domain(s)]
|
|
Firewall ports
|
|
22, [app port(s)]
|
|
Backup path
|
|
s3://hermes-vps-backups/fleettracker360/
|
|
Required secrets
|
|
DB credentials, [any third-party API keys]
|
|
|
|
Restore Procedure
|
|
Provision replacement host matching current tier.
|
|
Run base install; deploy itpp-infra SSH key.
|
|
aws s3 cp s3://hermes-vps-backups/fleettracker360/latest/ /opt/restore/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive
|
|
Restore Docker stack: cd /opt/fleettracker360 && docker compose up -d
|
|
Database Restore
|
|
Confirm latest MariaDB dump exists and is non-empty.
|
|
Import into a fresh instance before pointing the app at it.
|
|
Validate against a known vehicle/record ID.
|
|
Health & Validation Checks
|
|
Application health endpoint returns 200
|
|
Latest DB dump timestamp is within 24 hours
|
|
Known Failure Modes
|
|
Scheduled DB dump not actually running — same known gap as wphost02; verify before trusting this runbook's RPO claim
|
|
Note: Automated MariaDB dump scheduling for this host needs verification — see Backup & Security Hardening Standard, Section 6.
|
|
|
|
Legacy Hetzner Hosts
|
|
CPX21 / CPX41 boxes not yet migrated to the current standard
|
|
Field
|
|
Value
|
|
Tier
|
|
Legacy — CPX21 or CPX41, Debian 12
|
|
RTO / RPO target
|
|
4 hours / 24 hours
|
|
Dependencies
|
|
[list per host]
|
|
DNS records
|
|
[list per host]
|
|
Firewall ports
|
|
[list per host]
|
|
Backup path
|
|
s3://hermes-vps-backups/<server>/
|
|
Required secrets
|
|
[per host]
|
|
|
|
Restore Procedure
|
|
If S3 is reachable: provision replacement, base install, restore from S3 per the Standard Restore process.
|
|
If S3 is unreachable: use Hetzner rescue mode to reinstall OS, redeploy itpp-infra key, and rebuild from Hudu-documented configs (Emergency Restore).
|
|
Plan to migrate to Debian 13 and current tier standards on next rebuild rather than restoring like-for-like.
|
|
Health & Validation Checks
|
|
Service-specific health check for whatever this host runs
|
|
Confirm restored config matches the most recent Hudu documentation
|
|
Known Failure Modes
|
|
Emergency Restore path depends entirely on Hudu documentation being complete and current — audit this quarterly
|
|
|
|
|
|
Fill in all [bracketed] fields before this document is relied upon in a live incident. Track completion in the DR Testing & Validation Schedule. |