ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

migration plan: corrected scope + waves, backup gap reconciliation, Wave1/Wave2 runbooks, core-bu standby package; backup-plan: docker-stack-defs + app-data jobs recorded (2026-09-15)

Browse Source
This commit is contained in:
ippadmin
2026-09-15 10:40:34 -04:00
parent cac3cde372
commit d0467758b0
5 changed files with 2115 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files
backup-plan.md
+18 -5
View File
@@ -19,7 +19,8 @@
| /root Essentials | `root-essentials-backup.sh` — dotfiles, keys, scripts | `s3://hermes-vps-backups/root-backup/` | 3:00 AM | 2026-07-28 |
| Grafana | `core-services-backup.sh` — SQLite DB dump | `s3://hermes-vps-backups/core/grafana/` | 1:30 AM | 2026-07-28 |
| Uptime Kuma | `core-services-backup.sh` — SQLite DB dump | `s3://hermes-vps-backups/core/uptime-kuma/` | 1:30 AM | 2026-07-28 |
| Docker Volumes | `core-services-backup.sh` — tar of key compose volumes | `s3://hermes-vps-backups/volumes/` | 1:30 AM | 2026-07-28 |
| Docker Stack Definitions | `docker-stack-defs-backup.sh` — compose, `.env`, configs, service code under /root/docker | `s3://hermes-vps-backups/docker-stack-defs/` | Core 1:40 AM, app4 5:10 AM | 2026-09-15 |
| Core App Data + Code | `app-data-backup.sh` — 10 SQLite DBs via `.backup`, `pg_dump hotnow`, app code | `s3://hermes-vps-backups/app-data/` | 2:15 AM, Sun 2:50 AM | 2026-09-15 |
| Prometheus | `core-services-backup.sh` — TSDB snapshot | `s3://hermes-vps-backups/core/prometheus/` | 1:30 AM | 2026-07-28 |
| Auth API | `auth-api-backup.sh` — SQLite .backup + .env | `s3://hermes-vps-backups/core/auth-api/` | 3:15 AM | 2026-08-08 |
@@ -96,7 +97,10 @@ wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has
|------|------|--------|--------|
| Every 15 min | Hermes session state | Hermes cron | `hermes-live-sync` |
| 1:00 AM | Full Hermes backup | crontab | `hermes-backup.sh` |
| 1:30 AM | Grafana, Uptime Kuma, Docker volumes, Prometheus | crontab | `core-services-backup.sh` |
| 1:30 AM | Grafana, Uptime Kuma, Prometheus | crontab | `core-services-backup.sh` |
| 1:40 AM | Docker stack definitions (/root/docker) | crontab | `docker-stack-defs-backup.sh` |
| 2:15 AM | Core app data: 10 SQLite DBs + hotnow pg_dump + app code | crontab | `app-data-backup.sh` |
| Sun 2:50 AM | Core app upload/attachment blobs (~195 MB) | crontab | `app-data-backup.sh --full` |
| 2:00 AM | Open WebUI, n8n, MCP configs (App1) + **UniFi sync** (App2) | crontab | `app1-backup.sh`, `unifi-backup-sync.sh` |
| 2:30 AM | Vaultwarden (App1) + Traccar (App2) | Hermes cron / crontab | `vaultwarden-backup.sh`, `app2-backup.sh` |
| 2:45 AM | Technitium DNS (App2) | Hermes cron | `technitium-backup.sh` |
@@ -127,7 +131,9 @@ wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has
| Script | Purpose | Runs |
|--------|---------|------|
| `hermes-backup.sh` | Full Hermes tar.gz to S3 | crontab 1:00 AM |
| `core-services-backup.sh` | Grafana, Uptime Kuma, volumes, Prometheus | crontab 1:30 AM |
| `core-services-backup.sh` | Grafana, Uptime Kuma, Prometheus | crontab 1:30 AM |
| `docker-stack-defs-backup.sh` | /root/docker stack definitions (compose, .env, configs, code) | crontab 1:40 AM (app4 5:10 AM) |
| `app-data-backup.sh` | /opt + /root app SQLite DBs, hotnow pg_dump, app code (+ blobs Sun) | crontab 2:15 AM / Sun 2:50 AM |
| `root-essentials-backup.sh` | /root keys, configs, scripts | crontab 3:00 AM |
| `backup-audit-check.sh` | Verify recent backup timestamps | crontab 2:00 AM |
| `vaultwarden-backup.sh` | Vaultwarden SQLite dump (SSH to App1) | Hermes cron 2:30 AM |
@@ -249,8 +255,15 @@ These paths contain data from services that migrated off Core (Jul 28, 2026) or
## Unbacked Services
These services are running in production with **zero backup coverage**:
| Unbacked | *(none)* | N/A | N/A | All services are backed up as of 2026-08-08 |
**Verified 2026-09-15 — two real gaps found and closed.**
**Gap 1: `/root/docker` had no coverage.** The plan previously claimed a "Docker Volumes" leg writing to `s3://hermes-vps-backups/volumes/`. That leg does not exist in `core-services-backup.sh` (which only covers Grafana, Uptime Kuma and Prometheus), and the two scripts that would do it — `docker-volume-sync.sh` and `hermes-docker-sync.sh` — are present on Core but **scheduled nowhere**. Both are orphans. Closed with `docker-stack-defs-backup.sh`: 557 files / 682 KB covering the definitions of all 9 stacks, deliberately excluding `venv/`, `node_modules/`, `data/`, `database/`, `packages/`, caches and anything ≥ 256 KiB, because `/root/docker` is 5.8 GB of mostly reproducible installs and volume state that has its own per-service backups. Restore proven 2026-09-15: `.env`, `settings.yml` and `prometheus.yml` all recovered.
**Gap 2: the non-Docker apps under `/opt` and `/root` had zero coverage.** Live databases with no backup: `ops-portal/ops.db`, `rally/data/rally.db`, `dre-portal/data/dre.db`, `seemytrip/data/seemytrip.db`, `verdicttank/users.db`, `pta-survey/survey.db`, `pta-registration/pta.db`, `mooresunnydaze/db.sqlite`, `intelsight-api/intelsight.db`, `projects/pipeline/pipeline.db`, plus the host PostgreSQL database `hotnow` and irreplaceable upload/attachment blobs (182 MB of SeeMyTrip uploads alone). Three of these apps run from `/root`, not `/opt`, and were missed by every earlier sweep: `hotnow-api`, `intelsight-api`, `pipeline`. Closed with `app-data-backup.sh` — daily DBs + Postgres + app code, weekly blobs. SQLite is snapshotted with the `.backup` API and `quick_check`ed, never copied raw. Restore proven 2026-09-15: 10/10 DBs `quick_check=ok`, `hotnow.dump` a valid `pg_restore` archive (52 TOC entries), 275 code files recovered with `ops-portal/server.py` byte-identical to live.
**Pitfall recorded:** GNU `find -size -1M` rounds **up**, so it matches only *empty* files — a backup built with that predicate archives 240 zero-byte files and none of the configs, while still reporting success. Size filters in backup scripts must use byte units (`-size -262144c`).
**Still open:** the `/opt` apps that are not git checkouts now have code, data and configs captured, but nothing captures the *deployment* state (systemd unit files) for them; and the app1-bu retirement means its backup set must be inherited by the replacement US box.
**Done 2026-09-11 (Anita migration closeout).** The frozen Core copy `/root/.hermes/profiles/anita` (8.0 GB, excluded from live sync after the 15:53 cutover) was archived to
`s3://hermes-vps-backups/decommissioned/anita-core-frozen-profile-20260911-1732.tar.gz` — 2,786,082,561 B, sha256 `b457fa7f…b3dc`, 21,206 entries — then deleted, reclaiming 8 GB (125 GB → 117 GB used).