Infrastructure Audit

Use this umbrella for periodic infrastructure health checks and disaster recovery verification. Covers S3 backups, warm standby failover readiness, config file coverage, credential security, cron job health, and system integrity.

Audit Checklist (run in order)

1. S3 Backup Verification

Check all Wasabi buckets for recency, file counts, and versioning:

hermes-vps-backups — Hermes config/sessions/profiles (check live/, standby/, hermes-full-backup/)
itpropartner-backups — portal files, public data
mikrotik-ccr-backups — router configs

What to check per bucket:

Exists and is accessible
Versioning enabled (MFADelete should be disabled)
Most recent upload timestamp — flag anything >48h stale
File counts and total size seem reasonable
Zero-byte files are expected WAL artifacts only

2. Warm Standby Verification

Check app1-bu (5.161.114.8) via SSH with itpp-infra key:

Server power state (should be running — warm standby)
Hermes gateway status (should be inactive/dormant)
AWS CLI installed (/opt/awscli-venv/bin/activate)
S3 access to hermes-vps-backups bucket
Cron jobs active: watchdog every 5 min, sync every 10 min
Local data freshness (state.db, config.yaml should be recent from sync)
Network to Core (ping 152.53.192.33)
Disk space (should have >20% free)
Latest sync log shows recent activity

3. Config & Password Audit

Password files should be chmod 600:

File	Path
Germaine email	`~/.config/himalaya/g-germainebrown.pass`
Sho'Nuff email	`~/.config/himalaya/shonuff.pass`
iCloud CalDAV	`~/.config/himalaya/g-germainebrown-icloud-calendar.pass`
AWS/Wasabi	`~/.aws/credentials`
SSH keys	`~/.ssh/*`
DRE temp passwords	`~/.hermes/references/dre-temp-passwords.txt`
Migration creds	`~/.hermes/migration-creds.txt`

Config files to check are in backup pipeline:

/root/.hermes/config.yaml — Hermes config
/etc/caddy/Caddyfile — Caddy reverse proxy
/root/.hermes/.env — Environment variables
/etc/systemd/system/*.service — All custom systemd services

4. Cron Job Health

List all Hermes cron jobs with cronjob(action='list') and verify:

All critical jobs have last_status: ok
Full backup job exists (schedule: 0 5 * * *)
Live sync job exists (every 15m)
Service health check job exists (every 5m)
Standby sync job exists on app1-bu (every 10m)
Router backups are running

5. Documentation

All findings must be logged in /root/.hermes/references/dr-issue-log.md.

Documentation format preference (Germaine, Jul 8 2026):

Do NOT use markdown pipe tables in the summary — they're unreadable on mobile.

Use bullet lists grouped by status instead. Example:

**Fixed** ✅
- **DR-001** 🔴 Short description → fix summary

**Resolved** 🟢
- **DR-009** Short description

**Investigating** 🔄
- **DR-006** 🔴 Short description

Entry structure — use Problem → Root Cause → Fix → Verification blocks with --- separators between entries.

Reference files

File	Purpose
`/root/.hermes/references/dr-issue-log.md`	Permanent issue log, updated on each audit

Pitfalls

netcup REST API authentication is complex and poorly documented. The API uses Keycloak OAuth at servercontrolpanel.de, not the CCP API from customercontrolpanel.de. There are two separate authentication domains: (1) CCP API key (Master Data → API) for domain/account management, (2) SCP REST API (per-server) for provisioning — requires Keycloak, separate credentials. The CCP key cannot authenticate against the SCP API.
Netcup API requires three credentials: customer number, API key, and API password (both generated separately from the Master Data page).
When provisioning fails via API, fall back to: order through the web interface, then I configure via SSH.
Caddyfile is OUTSIDE the Hermes home dir — it lives at /etc/caddy/Caddyfile and is NOT included in aws s3 sync of ~/.hermes/. Must be explicitly added to backup scripts.
Systemd services live in /etc/systemd/system/ NOT ~/.config/systemd/user/. Backup scripts often target the wrong path.
Hetzner API token may exist as a shell env var but not as ~/.hermes/scripts/.hetzner_token file (the recovery bundle references the file). Check both sources during audit.
Full backup cron may not exist — it's a separate cron from the live sync. If missing, create it.
Home-router backup can fail silently when the MikroTik export produces a .in_progress file. Check for stuck export files on the router.
MikroTik CCR backup requires paramiko installed on the backup server. Missing Python deps are a common failure cause.
APP1-bu AWS CLI venv may be missing on a fresh install. python3 -m venv fails without python3-venv package. Test with: source /opt/awscli-venv/bin/activate && aws s3 ls
Hetzner pricing is inflated for new servers (Jul 2026) — the API returns current prices which are significantly higher than historical rates. netcup root servers offer better value: RS 1000 (4C/8G/256GB) at €10.74/mo vs Hetzner CPX32 (4C/8G) at ~€35/mo. Strategy: consolidate Hetzner workloads onto netcup rather than provisioning new Hetzner servers.
DR issue log formatting: This user reads on mobile. Pipe tables render poorly. Use grouped bullet lists by status (Fix/Resolved/Investigating), not markdown pipe tables.
Fault is common, not exceptional — when multiple systems fail in the same audit (stale backups, wrong IPs, missing packages, stuck files), log each as a separate DR issue. Don't treat the cluster of failures as a single incident.

6.0 KiB Raw Blame History