---
name: infrastructure-audit
description: "Umbrella for infrastructure audits: S3 backup verification, warm standby failover testing, config file coverage, password audit, systemd service backup, cron health check, and documentation of findings."
version: 1.0.0
author: ShoNuff
tags: [devops, audit, backup, dr, failover, compliance]
---

# Infrastructure Audit

Use this umbrella for periodic infrastructure health checks and disaster recovery verification. Covers S3 backups, warm standby failover readiness, config file coverage, credential security, cron job health, and system integrity.

## Audit Checklist (run in order)

### 1. S3 Backup Verification
Check all Wasabi buckets for recency, file counts, and versioning:
- `hermes-vps-backups` — Hermes config/sessions/profiles (check `live/`, `standby/`, `hermes-full-backup/`)
- `itpropartner-backups` — portal files, public data
- `mikrotik-ccr-backups` — router configs

**What to check per bucket:**
- Exists and is accessible
- Versioning enabled (MFADelete should be disabled)
- Most recent upload timestamp — flag anything >48h stale
- File counts and total size seem reasonable
- Zero-byte files are expected WAL artifacts only

### 2. Warm Standby Verification
Check app1-bu (5.161.114.8) via SSH with `itpp-infra` key:
- Server power state (should be running — warm standby)
- Hermes gateway status (should be inactive/dormant)
- AWS CLI installed (`/opt/awscli-venv/bin/activate`)
- S3 access to `hermes-vps-backups` bucket
- Cron jobs active: watchdog every 5 min, sync every 10 min
- Local data freshness (state.db, config.yaml should be recent from sync)
- Network to Core (ping 152.53.192.33)
- Disk space (should have >20% free)
- Latest sync log shows recent activity

### 3. Config & Password Audit

**Password files** should be `chmod 600`:
| File | Path |
|------|------|
| Germaine email | `~/.config/himalaya/g-germainebrown.pass` |
| Sho'Nuff email | `~/.config/himalaya/shonuff.pass` |
| iCloud CalDAV | `~/.config/himalaya/g-germainebrown-icloud-calendar.pass` |
| AWS/Wasabi | `~/.aws/credentials` |
| SSH keys | `~/.ssh/*` |
| DRE temp passwords | `~/.hermes/references/dre-temp-passwords.txt` |
| Migration creds | `~/.hermes/migration-creds.txt` |

**Config files** to check are in backup pipeline:
- `/root/.hermes/config.yaml` — Hermes config
- `/etc/caddy/Caddyfile` — Caddy reverse proxy
- `/root/.hermes/.env` — Environment variables
- `/etc/systemd/system/*.service` — All custom systemd services

### 4. Cron Job Health
List all Hermes cron jobs with `cronjob(action='list')` and verify:
- All critical jobs have `last_status: ok`
- Full backup job exists (schedule: 0 5 * * *)
- Live sync job exists (every 15m)
- Service health check job exists (every 5m)
- Standby sync job exists on app1-bu (every 10m)
- Router backups are running

### 5. Documentation
All findings must be logged in `/root/.hermes/references/dr-issue-log.md`.

**Documentation format preference (Germaine, Jul 8 2026):**
- **Do NOT use markdown pipe tables in the summary** — they're unreadable on mobile.
- **Use bullet lists grouped by status** instead. Example:
  ```
  **Fixed** ✅
  - **DR-001** 🔴 Short description → fix summary

  **Resolved** 🟢
  - **DR-009** Short description

  **Investigating** 🔄
  - **DR-006** 🔴 Short description
  ```
- **Entry structure** — use `Problem → Root Cause → Fix → Verification` blocks with `---` separators between entries.

## Reference files
| File | Purpose |
|------|---------|
| `/root/.hermes/references/dr-issue-log.md` | Permanent issue log, updated on each audit |

## Pitfalls
- **netcup REST API authentication is complex and poorly documented.** The API uses Keycloak OAuth at `servercontrolpanel.de`, not the CCP API from `customercontrolpanel.de`. There are two separate authentication domains: (1) CCP API key (Master Data → API) for domain/account management, (2) SCP REST API (per-server) for provisioning — requires Keycloak, separate credentials. The CCP key cannot authenticate against the SCP API.
- **Netcup API requires three credentials:** customer number, API key, and API password (both generated separately from the Master Data page).
- **When provisioning fails via API,** fall back to: order through the web interface, then I configure via SSH.
- **Caddyfile is OUTSIDE the Hermes home dir** — it lives at `/etc/caddy/Caddyfile` and is NOT included in `aws s3 sync` of `~/.hermes/`. Must be explicitly added to backup scripts.
- **Systemd services live in `/etc/systemd/system/`** NOT `~/.config/systemd/user/`. Backup scripts often target the wrong path.
- **Hetzner API token** may exist as a shell env var but not as `~/.hermes/scripts/.hetzner_token` file (the recovery bundle references the file). Check both sources during audit.
- **Full backup cron** may not exist — it's a separate cron from the live sync. If missing, create it.
- **Home-router backup** can fail silently when the MikroTik export produces a `.in_progress` file. Check for stuck export files on the router.
- **MikroTik CCR backup** requires `paramiko` installed on the backup server. Missing Python deps are a common failure cause.
- **APP1-bu AWS CLI venv** may be missing on a fresh install. `python3 -m venv` fails without `python3-venv` package. Test with: `source /opt/awscli-venv/bin/activate && aws s3 ls`
- **Hetzner pricing is inflated for new servers (Jul 2026)** — the API returns current prices which are significantly higher than historical rates. netcup root servers offer better value: RS 1000 (4C/8G/256GB) at €10.74/mo vs Hetzner CPX32 (4C/8G) at ~€35/mo. Strategy: consolidate Hetzner workloads onto netcup rather than provisioning new Hetzner servers.
- **DR issue log formatting:** This user reads on mobile. Pipe tables render poorly. Use grouped bullet lists by status (Fix/Resolved/Investigating), not markdown pipe tables.
- **Fault is common, not exceptional** — when multiple systems fail in the same audit (stale backups, wrong IPs, missing packages, stuck files), log each as a separate DR issue. Don't treat the cluster of failures as a single incident.
