Initial skills documentation — 25 categories, all SKILL.md + references + scripts
This commit is contained in:
root
@@ -0,0 +1,104 @@
|
||||
---
|
||||
name: infrastructure-audit
|
||||
description: "Umbrella for infrastructure audits: S3 backup verification, warm standby failover testing, config file coverage, password audit, systemd service backup, cron health check, and documentation of findings."
|
||||
version: 1.0.0
|
||||
author: ShoNuff
|
||||
tags: [devops, audit, backup, dr, failover, compliance]
|
||||
---
|
||||
|
||||
# Infrastructure Audit
|
||||
|
||||
Use this umbrella for periodic infrastructure health checks and disaster recovery verification. Covers S3 backups, warm standby failover readiness, config file coverage, credential security, cron job health, and system integrity.
|
||||
|
||||
## Audit Checklist (run in order)
|
||||
|
||||
### 1. S3 Backup Verification
|
||||
Check all Wasabi buckets for recency, file counts, and versioning:
|
||||
- `hermes-vps-backups` — Hermes config/sessions/profiles (check `live/`, `standby/`, `hermes-full-backup/`)
|
||||
- `itpropartner-backups` — portal files, public data
|
||||
- `mikrotik-ccr-backups` — router configs
|
||||
|
||||
**What to check per bucket:**
|
||||
- Exists and is accessible
|
||||
- Versioning enabled (MFADelete should be disabled)
|
||||
- Most recent upload timestamp — flag anything >48h stale
|
||||
- File counts and total size seem reasonable
|
||||
- Zero-byte files are expected WAL artifacts only
|
||||
|
||||
### 2. Warm Standby Verification
|
||||
Check app1-bu (5.161.114.8) via SSH with `itpp-infra` key:
|
||||
- Server power state (should be running — warm standby)
|
||||
- Hermes gateway status (should be inactive/dormant)
|
||||
- AWS CLI installed (`/opt/awscli-venv/bin/activate`)
|
||||
- S3 access to `hermes-vps-backups` bucket
|
||||
- Cron jobs active: watchdog every 5 min, sync every 10 min
|
||||
- Local data freshness (state.db, config.yaml should be recent from sync)
|
||||
- Network to Core (ping 152.53.192.33)
|
||||
- Disk space (should have >20% free)
|
||||
- Latest sync log shows recent activity
|
||||
|
||||
### 3. Config & Password Audit
|
||||
|
||||
**Password files** should be `chmod 600`:
|
||||
| File | Path |
|
||||
|------|------|
|
||||
| Germaine email | `~/.config/himalaya/g-germainebrown.pass` |
|
||||
| Sho'Nuff email | `~/.config/himalaya/shonuff.pass` |
|
||||
| iCloud CalDAV | `~/.config/himalaya/g-germainebrown-icloud-calendar.pass` |
|
||||
| AWS/Wasabi | `~/.aws/credentials` |
|
||||
| SSH keys | `~/.ssh/*` |
|
||||
| DRE temp passwords | `~/.hermes/references/dre-temp-passwords.txt` |
|
||||
| Migration creds | `~/.hermes/migration-creds.txt` |
|
||||
|
||||
**Config files** to check are in backup pipeline:
|
||||
- `/root/.hermes/config.yaml` — Hermes config
|
||||
- `/etc/caddy/Caddyfile` — Caddy reverse proxy
|
||||
- `/root/.hermes/.env` — Environment variables
|
||||
- `/etc/systemd/system/*.service` — All custom systemd services
|
||||
|
||||
### 4. Cron Job Health
|
||||
List all Hermes cron jobs with `cronjob(action='list')` and verify:
|
||||
- All critical jobs have `last_status: ok`
|
||||
- Full backup job exists (schedule: 0 5 * * *)
|
||||
- Live sync job exists (every 15m)
|
||||
- Service health check job exists (every 5m)
|
||||
- Standby sync job exists on app1-bu (every 10m)
|
||||
- Router backups are running
|
||||
|
||||
### 5. Documentation
|
||||
All findings must be logged in `/root/.hermes/references/dr-issue-log.md`.
|
||||
|
||||
**Documentation format preference (Germaine, Jul 8 2026):**
|
||||
- **Do NOT use markdown pipe tables in the summary** — they're unreadable on mobile.
|
||||
- **Use bullet lists grouped by status** instead. Example:
|
||||
```
|
||||
**Fixed** ✅
|
||||
- **DR-001** 🔴 Short description → fix summary
|
||||
|
||||
**Resolved** 🟢
|
||||
- **DR-009** Short description
|
||||
|
||||
**Investigating** 🔄
|
||||
- **DR-006** 🔴 Short description
|
||||
```
|
||||
- **Entry structure** — use `Problem → Root Cause → Fix → Verification` blocks with `---` separators between entries.
|
||||
|
||||
## Reference files
|
||||
| File | Purpose |
|
||||
|------|---------|
|
||||
| `/root/.hermes/references/dr-issue-log.md` | Permanent issue log, updated on each audit |
|
||||
|
||||
## Pitfalls
|
||||
- **netcup REST API authentication is complex and poorly documented.** The API uses Keycloak OAuth at `servercontrolpanel.de`, not the CCP API from `customercontrolpanel.de`. There are two separate authentication domains: (1) CCP API key (Master Data → API) for domain/account management, (2) SCP REST API (per-server) for provisioning — requires Keycloak, separate credentials. The CCP key cannot authenticate against the SCP API.
|
||||
- **Netcup API requires three credentials:** customer number, API key, and API password (both generated separately from the Master Data page).
|
||||
- **When provisioning fails via API,** fall back to: order through the web interface, then I configure via SSH.
|
||||
- **Caddyfile is OUTSIDE the Hermes home dir** — it lives at `/etc/caddy/Caddyfile` and is NOT included in `aws s3 sync` of `~/.hermes/`. Must be explicitly added to backup scripts.
|
||||
- **Systemd services live in `/etc/systemd/system/`** NOT `~/.config/systemd/user/`. Backup scripts often target the wrong path.
|
||||
- **Hetzner API token** may exist as a shell env var but not as `~/.hermes/scripts/.hetzner_token` file (the recovery bundle references the file). Check both sources during audit.
|
||||
- **Full backup cron** may not exist — it's a separate cron from the live sync. If missing, create it.
|
||||
- **Home-router backup** can fail silently when the MikroTik export produces a `.in_progress` file. Check for stuck export files on the router.
|
||||
- **MikroTik CCR backup** requires `paramiko` installed on the backup server. Missing Python deps are a common failure cause.
|
||||
- **APP1-bu AWS CLI venv** may be missing on a fresh install. `python3 -m venv` fails without `python3-venv` package. Test with: `source /opt/awscli-venv/bin/activate && aws s3 ls`
|
||||
- **Hetzner pricing is inflated for new servers (Jul 2026)** — the API returns current prices which are significantly higher than historical rates. netcup root servers offer better value: RS 1000 (4C/8G/256GB) at €10.74/mo vs Hetzner CPX32 (4C/8G) at ~€35/mo. Strategy: consolidate Hetzner workloads onto netcup rather than provisioning new Hetzner servers.
|
||||
- **DR issue log formatting:** This user reads on mobile. Pipe tables render poorly. Use grouped bullet lists by status (Fix/Resolved/Investigating), not markdown pipe tables.
|
||||
- **Fault is common, not exceptional** — when multiple systems fail in the same audit (stale backups, wrong IPs, missing packages, stuck files), log each as a separate DR issue. Don't treat the cluster of failures as a single incident.
|
||||
Reference in New Issue
Block a user