105 lines
6.0 KiB
Markdown
105 lines
6.0 KiB
Markdown
---
|
|
name: infrastructure-audit
|
|
description: "Umbrella for infrastructure audits: S3 backup verification, warm standby failover testing, config file coverage, password audit, systemd service backup, cron health check, and documentation of findings."
|
|
version: 1.0.0
|
|
author: ShoNuff
|
|
tags: [devops, audit, backup, dr, failover, compliance]
|
|
---
|
|
|
|
# Infrastructure Audit
|
|
|
|
Use this umbrella for periodic infrastructure health checks and disaster recovery verification. Covers S3 backups, warm standby failover readiness, config file coverage, credential security, cron job health, and system integrity.
|
|
|
|
## Audit Checklist (run in order)
|
|
|
|
### 1. S3 Backup Verification
|
|
Check all Wasabi buckets for recency, file counts, and versioning:
|
|
- `hermes-vps-backups` — Hermes config/sessions/profiles (check `live/`, `standby/`, `hermes-full-backup/`)
|
|
- `itpropartner-backups` — portal files, public data
|
|
- `mikrotik-ccr-backups` — router configs
|
|
|
|
**What to check per bucket:**
|
|
- Exists and is accessible
|
|
- Versioning enabled (MFADelete should be disabled)
|
|
- Most recent upload timestamp — flag anything >48h stale
|
|
- File counts and total size seem reasonable
|
|
- Zero-byte files are expected WAL artifacts only
|
|
|
|
### 2. Warm Standby Verification
|
|
Check app1-bu (5.161.114.8) via SSH with `itpp-infra` key:
|
|
- Server power state (should be running — warm standby)
|
|
- Hermes gateway status (should be inactive/dormant)
|
|
- AWS CLI installed (`/opt/awscli-venv/bin/activate`)
|
|
- S3 access to `hermes-vps-backups` bucket
|
|
- Cron jobs active: watchdog every 5 min, sync every 10 min
|
|
- Local data freshness (state.db, config.yaml should be recent from sync)
|
|
- Network to Core (ping 152.53.192.33)
|
|
- Disk space (should have >20% free)
|
|
- Latest sync log shows recent activity
|
|
|
|
### 3. Config & Password Audit
|
|
|
|
**Password files** should be `chmod 600`:
|
|
| File | Path |
|
|
|------|------|
|
|
| Germaine email | `~/.config/himalaya/g-germainebrown.pass` |
|
|
| Sho'Nuff email | `~/.config/himalaya/shonuff.pass` |
|
|
| iCloud CalDAV | `~/.config/himalaya/g-germainebrown-icloud-calendar.pass` |
|
|
| AWS/Wasabi | `~/.aws/credentials` |
|
|
| SSH keys | `~/.ssh/*` |
|
|
| DRE temp passwords | `~/.hermes/references/dre-temp-passwords.txt` |
|
|
| Migration creds | `~/.hermes/migration-creds.txt` |
|
|
|
|
**Config files** to check are in backup pipeline:
|
|
- `/root/.hermes/config.yaml` — Hermes config
|
|
- `/etc/caddy/Caddyfile` — Caddy reverse proxy
|
|
- `/root/.hermes/.env` — Environment variables
|
|
- `/etc/systemd/system/*.service` — All custom systemd services
|
|
|
|
### 4. Cron Job Health
|
|
List all Hermes cron jobs with `cronjob(action='list')` and verify:
|
|
- All critical jobs have `last_status: ok`
|
|
- Full backup job exists (schedule: 0 5 * * *)
|
|
- Live sync job exists (every 15m)
|
|
- Service health check job exists (every 5m)
|
|
- Standby sync job exists on app1-bu (every 10m)
|
|
- Router backups are running
|
|
|
|
### 5. Documentation
|
|
All findings must be logged in `/root/.hermes/references/dr-issue-log.md`.
|
|
|
|
**Documentation format preference (Germaine, Jul 8 2026):**
|
|
- **Do NOT use markdown pipe tables in the summary** — they're unreadable on mobile.
|
|
- **Use bullet lists grouped by status** instead. Example:
|
|
```
|
|
**Fixed** ✅
|
|
- **DR-001** 🔴 Short description → fix summary
|
|
|
|
**Resolved** 🟢
|
|
- **DR-009** Short description
|
|
|
|
**Investigating** 🔄
|
|
- **DR-006** 🔴 Short description
|
|
```
|
|
- **Entry structure** — use `Problem → Root Cause → Fix → Verification` blocks with `---` separators between entries.
|
|
|
|
## Reference files
|
|
| File | Purpose |
|
|
|------|---------|
|
|
| `/root/.hermes/references/dr-issue-log.md` | Permanent issue log, updated on each audit |
|
|
|
|
## Pitfalls
|
|
- **netcup REST API authentication is complex and poorly documented.** The API uses Keycloak OAuth at `servercontrolpanel.de`, not the CCP API from `customercontrolpanel.de`. There are two separate authentication domains: (1) CCP API key (Master Data → API) for domain/account management, (2) SCP REST API (per-server) for provisioning — requires Keycloak, separate credentials. The CCP key cannot authenticate against the SCP API.
|
|
- **Netcup API requires three credentials:** customer number, API key, and API password (both generated separately from the Master Data page).
|
|
- **When provisioning fails via API,** fall back to: order through the web interface, then I configure via SSH.
|
|
- **Caddyfile is OUTSIDE the Hermes home dir** — it lives at `/etc/caddy/Caddyfile` and is NOT included in `aws s3 sync` of `~/.hermes/`. Must be explicitly added to backup scripts.
|
|
- **Systemd services live in `/etc/systemd/system/`** NOT `~/.config/systemd/user/`. Backup scripts often target the wrong path.
|
|
- **Hetzner API token** may exist as a shell env var but not as `~/.hermes/scripts/.hetzner_token` file (the recovery bundle references the file). Check both sources during audit.
|
|
- **Full backup cron** may not exist — it's a separate cron from the live sync. If missing, create it.
|
|
- **Home-router backup** can fail silently when the MikroTik export produces a `.in_progress` file. Check for stuck export files on the router.
|
|
- **MikroTik CCR backup** requires `paramiko` installed on the backup server. Missing Python deps are a common failure cause.
|
|
- **APP1-bu AWS CLI venv** may be missing on a fresh install. `python3 -m venv` fails without `python3-venv` package. Test with: `source /opt/awscli-venv/bin/activate && aws s3 ls`
|
|
- **Hetzner pricing is inflated for new servers (Jul 2026)** — the API returns current prices which are significantly higher than historical rates. netcup root servers offer better value: RS 1000 (4C/8G/256GB) at €10.74/mo vs Hetzner CPX32 (4C/8G) at ~€35/mo. Strategy: consolidate Hetzner workloads onto netcup rather than provisioning new Hetzner servers.
|
|
- **DR issue log formatting:** This user reads on mobile. Pipe tables render poorly. Use grouped bullet lists by status (Fix/Resolved/Investigating), not markdown pipe tables.
|
|
- **Fault is common, not exceptional** — when multiple systems fail in the same audit (stale backups, wrong IPs, missing packages, stuck files), log each as a separate DR issue. Don't treat the cluster of failures as a single incident.
|