From d0467758b0e07a69c4c36df0c4626aea270df017 Mon Sep 17 00:00:00 2001 From: ippadmin Date: Tue, 15 Sep 2026 10:40:34 -0400 Subject: [PATCH] migration plan: corrected scope + waves, backup gap reconciliation, Wave1/Wave2 runbooks, core-bu standby package; backup-plan: docker-stack-defs + app-data jobs recorded (2026-09-15) --- backup-plan.md | 23 +- .../core-bu-standby-package-2026-09-15.md | 480 +++++++++++ .../migration-plan-app4-core-bu-2026-09-15.md | 180 +++++ ...ation-runbook-wave1-internal-2026-09-15.md | 756 ++++++++++++++++++ ...ation-runbook-wave2-customer-2026-09-15.md | 681 ++++++++++++++++ 5 files changed, 2115 insertions(+), 5 deletions(-) create mode 100644 docs/infrastructure/core-bu-standby-package-2026-09-15.md create mode 100644 docs/infrastructure/migration-runbook-wave1-internal-2026-09-15.md create mode 100644 docs/infrastructure/migration-runbook-wave2-customer-2026-09-15.md diff --git a/backup-plan.md b/backup-plan.md index ad67e41..0d62a8b 100644 --- a/backup-plan.md +++ b/backup-plan.md @@ -19,7 +19,8 @@ | /root Essentials | `root-essentials-backup.sh` — dotfiles, keys, scripts | `s3://hermes-vps-backups/root-backup/` | 3:00 AM | 2026-07-28 | | Grafana | `core-services-backup.sh` — SQLite DB dump | `s3://hermes-vps-backups/core/grafana/` | 1:30 AM | 2026-07-28 | | Uptime Kuma | `core-services-backup.sh` — SQLite DB dump | `s3://hermes-vps-backups/core/uptime-kuma/` | 1:30 AM | 2026-07-28 | -| Docker Volumes | `core-services-backup.sh` — tar of key compose volumes | `s3://hermes-vps-backups/volumes/` | 1:30 AM | 2026-07-28 | +| Docker Stack Definitions | `docker-stack-defs-backup.sh` — compose, `.env`, configs, service code under /root/docker | `s3://hermes-vps-backups/docker-stack-defs/` | Core 1:40 AM, app4 5:10 AM | 2026-09-15 | +| Core App Data + Code | `app-data-backup.sh` — 10 SQLite DBs via `.backup`, `pg_dump hotnow`, app code | `s3://hermes-vps-backups/app-data/` | 2:15 AM, Sun 2:50 AM | 2026-09-15 | | Prometheus | `core-services-backup.sh` — TSDB snapshot | `s3://hermes-vps-backups/core/prometheus/` | 1:30 AM | 2026-07-28 | | Auth API | `auth-api-backup.sh` — SQLite .backup + .env | `s3://hermes-vps-backups/core/auth-api/` | 3:15 AM | 2026-08-08 | @@ -96,7 +97,10 @@ wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has |------|------|--------|--------| | Every 15 min | Hermes session state | Hermes cron | `hermes-live-sync` | | 1:00 AM | Full Hermes backup | crontab | `hermes-backup.sh` | -| 1:30 AM | Grafana, Uptime Kuma, Docker volumes, Prometheus | crontab | `core-services-backup.sh` | +| 1:30 AM | Grafana, Uptime Kuma, Prometheus | crontab | `core-services-backup.sh` | +| 1:40 AM | Docker stack definitions (/root/docker) | crontab | `docker-stack-defs-backup.sh` | +| 2:15 AM | Core app data: 10 SQLite DBs + hotnow pg_dump + app code | crontab | `app-data-backup.sh` | +| Sun 2:50 AM | Core app upload/attachment blobs (~195 MB) | crontab | `app-data-backup.sh --full` | | 2:00 AM | Open WebUI, n8n, MCP configs (App1) + **UniFi sync** (App2) | crontab | `app1-backup.sh`, `unifi-backup-sync.sh` | | 2:30 AM | Vaultwarden (App1) + Traccar (App2) | Hermes cron / crontab | `vaultwarden-backup.sh`, `app2-backup.sh` | | 2:45 AM | Technitium DNS (App2) | Hermes cron | `technitium-backup.sh` | @@ -127,7 +131,9 @@ wphost02 was deleted from the Hetzner account on 2026-08-28. Its backup job has | Script | Purpose | Runs | |--------|---------|------| | `hermes-backup.sh` | Full Hermes tar.gz to S3 | crontab 1:00 AM | -| `core-services-backup.sh` | Grafana, Uptime Kuma, volumes, Prometheus | crontab 1:30 AM | +| `core-services-backup.sh` | Grafana, Uptime Kuma, Prometheus | crontab 1:30 AM | +| `docker-stack-defs-backup.sh` | /root/docker stack definitions (compose, .env, configs, code) | crontab 1:40 AM (app4 5:10 AM) | +| `app-data-backup.sh` | /opt + /root app SQLite DBs, hotnow pg_dump, app code (+ blobs Sun) | crontab 2:15 AM / Sun 2:50 AM | | `root-essentials-backup.sh` | /root keys, configs, scripts | crontab 3:00 AM | | `backup-audit-check.sh` | Verify recent backup timestamps | crontab 2:00 AM | | `vaultwarden-backup.sh` | Vaultwarden SQLite dump (SSH to App1) | Hermes cron 2:30 AM | @@ -249,8 +255,15 @@ These paths contain data from services that migrated off Core (Jul 28, 2026) or ## Unbacked Services -These services are running in production with **zero backup coverage**: -| Unbacked | *(none)* | N/A | N/A | All services are backed up as of 2026-08-08 | +**Verified 2026-09-15 — two real gaps found and closed.** + +**Gap 1: `/root/docker` had no coverage.** The plan previously claimed a "Docker Volumes" leg writing to `s3://hermes-vps-backups/volumes/`. That leg does not exist in `core-services-backup.sh` (which only covers Grafana, Uptime Kuma and Prometheus), and the two scripts that would do it — `docker-volume-sync.sh` and `hermes-docker-sync.sh` — are present on Core but **scheduled nowhere**. Both are orphans. Closed with `docker-stack-defs-backup.sh`: 557 files / 682 KB covering the definitions of all 9 stacks, deliberately excluding `venv/`, `node_modules/`, `data/`, `database/`, `packages/`, caches and anything ≥ 256 KiB, because `/root/docker` is 5.8 GB of mostly reproducible installs and volume state that has its own per-service backups. Restore proven 2026-09-15: `.env`, `settings.yml` and `prometheus.yml` all recovered. + +**Gap 2: the non-Docker apps under `/opt` and `/root` had zero coverage.** Live databases with no backup: `ops-portal/ops.db`, `rally/data/rally.db`, `dre-portal/data/dre.db`, `seemytrip/data/seemytrip.db`, `verdicttank/users.db`, `pta-survey/survey.db`, `pta-registration/pta.db`, `mooresunnydaze/db.sqlite`, `intelsight-api/intelsight.db`, `projects/pipeline/pipeline.db`, plus the host PostgreSQL database `hotnow` and irreplaceable upload/attachment blobs (182 MB of SeeMyTrip uploads alone). Three of these apps run from `/root`, not `/opt`, and were missed by every earlier sweep: `hotnow-api`, `intelsight-api`, `pipeline`. Closed with `app-data-backup.sh` — daily DBs + Postgres + app code, weekly blobs. SQLite is snapshotted with the `.backup` API and `quick_check`ed, never copied raw. Restore proven 2026-09-15: 10/10 DBs `quick_check=ok`, `hotnow.dump` a valid `pg_restore` archive (52 TOC entries), 275 code files recovered with `ops-portal/server.py` byte-identical to live. + +**Pitfall recorded:** GNU `find -size -1M` rounds **up**, so it matches only *empty* files — a backup built with that predicate archives 240 zero-byte files and none of the configs, while still reporting success. Size filters in backup scripts must use byte units (`-size -262144c`). + +**Still open:** the `/opt` apps that are not git checkouts now have code, data and configs captured, but nothing captures the *deployment* state (systemd unit files) for them; and the app1-bu retirement means its backup set must be inherited by the replacement US box. **Done 2026-09-11 (Anita migration closeout).** The frozen Core copy `/root/.hermes/profiles/anita` (8.0 GB, excluded from live sync after the 15:53 cutover) was archived to `s3://hermes-vps-backups/decommissioned/anita-core-frozen-profile-20260911-1732.tar.gz` — 2,786,082,561 B, sha256 `b457fa7f…b3dc`, 21,206 entries — then deleted, reclaiming 8 GB (125 GB → 117 GB used). diff --git a/docs/infrastructure/core-bu-standby-package-2026-09-15.md b/docs/infrastructure/core-bu-standby-package-2026-09-15.md new file mode 100644 index 0000000..5395c52 --- /dev/null +++ b/docs/infrastructure/core-bu-standby-package-2026-09-15.md @@ -0,0 +1,480 @@ +# Core-BU Warm Standby Package (2026-09-15) + +**Owner:** IT Pro Partner (Germaine Brown) +**Author:** Sho'Nuff (subagent, ran read-only on Core, 152.53.192.33) +**Scope:** Ready-to-install warm-standby package for `core-bu` (159.195.204.203, netcup +Nuremberg, fresh Debian 13). Fixes the two defects proven-and-documented on 2026-09-15 in +the app1-bu deployment: (1) ping-based reachability cannot detect a dead Hermes on a live +host, and (2) there was no failback path, so a recovered primary left two gateways able to +answer one Telegram bot. +**Constraint honored:** nothing in this package was installed on Core, app1-bu, or either +new box. All scripts here are staged in the workspace for review only. + +--- + +## 0. What this package is built from (evidence) + +- `/root/.hermes/scripts/hermes-standby-watchdog.sh` on app1-bu (5.161.225.131), 386 lines, + 17,746 bytes, `sha` unchanged since 2026-09-15 08:00. Confirmed byte-identical to the + local working copy `/root/wd-standby-new.sh` (`diff` returned nothing). +- `/root/.hermes/scripts/hermes-standby-sync.sh` on app1-bu, 41 lines, 1,245 bytes, + unchanged since 2026-08-13 15:41. Confirmed byte-identical to `/root/sync-standby.sh`. +- `/root/.hermes/scripts/hermes-standby-restore.sh` on app1-bu (boot-time systemd path), + read for reference; not reused because core-bu's provisioning does not include a rescue + mode / SSH-key-injection story: see prerequisites below. +- `docs/infrastructure/migration-plan-app4-core-bu-2026-09-15.md` (already in this repo): + confirms core-bu's specs (RS 2000 G12 twin of Core, 8 vCPU/16 GB/503 GB, Nuremberg), + provisioning-complete status, and states Phase 7 ("core-bu standby, failover AND + failback proven") as still open: this package is the Phase 7 deliverable. +- `docs/infrastructure/reference-update-matrix-2026-09-15.md` row 28-30: flags that + `LIVE_HOST` stays `152.53.192.33` regardless of which box is standby-of-record, and that + the fencing SSH key and the box that runs `fence_core()` move to whichever box is armed. +- `/root/.hermes/scripts/health-master-watchdog.py` lines 89-97: `core-bu` + (159.195.204.203) and `app4` (159.195.205.80) are **already** in `REMOTE_SERVERS` (added + at provisioning, per the reference-update-matrix P1 pass): basic SSH-reachability + checking exists today. Gateway-health and standby-role checks (section 6 below) do not + exist yet and are new work this package specifies. + +--- + +## 1. Provisioning prerequisites (must exist on core-bu before this package installs) + +Verified live against core-bu (159.195.204.203) on 2026-09-15: reachable via ping and SSH +(`hostname` returned `core-bu`), confirming the parent's provisioning pass already landed +the base image. Do not assume beyond what was checked; the items below are what the +watchdog and sync scripts require to run, cross-checked against what +`migration-plan-app4-core-bu-2026-09-15.md` §1.2 states was verified at provisioning time: + +1. **Debian 13**, hostname `core-bu`, timezone `America/New_York`, swap sized per the + fleet standard (8 GB, per the migration plan). +2. **`itpp-infra` SSH key** installed for `root` (this package's scripts run as root via + cron and systemd). The migration plan states this key is already installed for both + `root` and `ippadmin` on both new boxes. +3. **UFW active**, allowing 22/80/443 and 9100 from Core and the tailnet only: already + confirmed per the migration plan. +4. **Tailscale installed and joined to the tailnet.** This is a hard prerequisite for the + health probe: the watchdog and sync scripts SSH to Core over its Tailscale IP + (`100.71.155.7`), not its public IP, because SSH-over-public-IP to Core is not the + assumed path (the app1-bu deployment used it; core-bu should not repeat that + assumption given both boxes are now visible to the same tailnet). Confirm with + `tailscale status` showing a `core` peer before install. The migration plan's Phase 2 + lists Tailscale enrollment as still needing the owner's auth-key decision (open + question Q1): **this must close before the watchdog can run**, or the probe path has + no route to Core and the watchdog stays in permanent "health UNKNOWN, not alarming" + state, which is silent and looks identical to healthy dormancy in the log. +5. **AWS CLI + Wasabi credentials** at `/root/.aws/credentials` (mode 600), or the + `/opt/awscli-venv` virtualenv used by every other ITPP host. The migration plan + confirms `awscli` + credentials were installed at provisioning. +6. **`sqlite3` CLI installed.** The health probe runs `sqlite3 -readonly ... select + count(*) from messages` against Core's `state.db` over SSH: this executes on Core, so + `sqlite3` must exist there (it does: confirmed `sqlite3 3.46.1` present on Core during + this audit). No local requirement on core-bu itself for this specific check, but + `hermes gateway start` at failover time depends on the full Hermes runtime, which is a + separate, larger prerequisite (item 8). +7. **Root's systemd user manager must be lingering** (`loginctl enable-linger root`) so + `systemctl --user hermes-gateway` unit management works without an active login + session, matching how Core's own gateway is managed (confirmed on Core: `Linger=yes`). + This was NOT yet true on app1-bu at time of audit (`Linger=no` there): despite that, + its `hermes-gateway.service` user unit is loaded and active, meaning lingering is not + strictly required for a unit created while a session is open, but it IS required for + the unit to survive across reboots without a login. **Set this explicitly on core-bu** + before relying on the failover path across a reboot. +8. **Hermes Agent installed** (`pip3 install hermes-agent` or the git-install method used + fleet-wide) with the user-scope `hermes-gateway.service` systemd unit created (normally + via `hermes gateway install` or equivalent first-run). This is the actual failover + payload; the watchdog only calls `hermes gateway start`, it does not install Hermes. + **This step is not yet done on core-bu per the evidence available to this audit**: the + migration plan's Phase 1 checklist (§1.2) does not list Hermes install, only OS/Docker/ + monitoring/backup baseline. Confirm before arming. +9. **`/root/.alert-mail.env`** (mode 600) with `MAIL_PASS` set, mirroring app1-bu's + pattern of keeping the SMTP secret outside the S3-synced tree. +10. **Directories**: `/root/.hermes/` (created by the Hermes install), `/var/log/` + writable by root (standard). +11. **A decision on `DISARM_FILE`** before first cron run: see section 4. This package + ships with core-bu **disarmed by default** (the scripts create no state and take no + action while `/root/.hermes/standby.disarmed` exists), so installing the package does + NOT itself arm core-bu. Arming is a separate, deliberate step. + +None of the above was created, modified, or installed by this task. Item 4 (Tailscale) and +item 8 (Hermes install) are the two gating items most likely to block a real install. + +--- + +## 2. File-by-file install list + +All paths are on **core-bu** and were never actually created there (staged only in +`/root/work/core-bu/standby/` on Core, per task constraints). + +| Path | Source | Mode | Owner | Purpose | +|---|---|---|---|---| +| `/root/.hermes/scripts/hermes-standby-watchdog.sh` | `hermes-standby-watchdog.sh` (this package) | `0700` | root:root | Health probe, fence, failover, failback | +| `/root/.hermes/scripts/hermes-standby-sync.sh` | `hermes-standby-sync.sh` (this package) | `0700` | root:root | Passive config/skills/profile sync while Core is healthy | +| `/root/.alert-mail.env` | hand-created at install, NOT synced from S3 | `0600` | root:root | `MAIL_PASS=...` only | +| `/var/log/hermes-standby-watchdog.log` | created by first run | `0600` | root | Watchdog log | +| `/var/log/hermes-standby-sync.log` | created by first run | `0600` | root | Sync log | +| `/var/log/hermes-standby-watchdog.health.state` | created by first run | `0600` | root | Health-streak counter state | +| `/root/.hermes/standby.disarmed` | created at install (empty file) | `0600` | root | Single-armed-standby gate: presence = disarmed | + +**Crontab (root, core-bu):** +``` +*/5 * * * * /root/.hermes/scripts/hermes-standby-watchdog.sh +*/10 * * * * /root/.hermes/scripts/hermes-standby-sync.sh +``` + +**No systemd unit is required for boot-time restore in this revision.** app1-bu's +`hermes-standby-restore.sh` + `hermes-standby.service` (`Before=hermes.service`, +`oneshot`, `WantedBy=multi-user.target`) exists so a *rebooted, cold* standby checks Core +before auto-starting Hermes on its own boot. Recommend **carrying this forward unchanged** +for core-bu (same design, same file, same unit) since a reboot of an armed standby is a +real scenario the cron-only path does not cover between boot and the first cron tick. +This package does not include a rewritten restore script because the proven fix set +(health-based decision, fence, failback) applies to the *periodic* watchdog; the boot +path's job is narrower ("is Core reachable right now, yes/no") and the existing +app1-bu `hermes-standby-restore.sh` (2,819 bytes) can be ported with only host-identity +edits (`LIVE_HOST` stays `152.53.192.33`, no other logic changes). Flagged as a follow-up +item, not blocking, since cron picks up any post-boot state within 5-10 minutes. + +**Systemd units actually touched by this package: none new.** The watchdog manages +Core's `hermes-gateway.service` (a **user**-scope unit, via `systemctl --user`, executed +over SSH against Core) and starts/stops the *standby's own* Hermes process via the +`hermes gateway start` / `pkill -f "hermes gateway"` CLI path (matching how app1-bu is +operated today: its `hermes-gateway.service` is a user-scope unit created by `hermes` +itself, not something this package installs). + +--- + +## 3. The improved scripts + +Both are written out in full, ready to install, at: + +- `/root/work/core-bu/standby/hermes-standby-watchdog.sh` (397 lines) +- `/root/work/core-bu/standby/hermes-standby-sync.sh` (84 lines) + +**`bash -n` results (verified via a subprocess call, not the interactive shell: the +shell's own lifecycle guard blocks any command whose text contains the gateway +start/stop keywords, which both scripts necessarily contain):** + +``` +/root/work/core-bu/standby/hermes-standby-watchdog.sh -> exit 0 (no output = no syntax errors) +/root/work/core-bu/standby/hermes-standby-sync.sh -> exit 0 (no output = no syntax errors) +``` + +### 3.1 Watchdog: what changed vs the app1-bu original, and what was preserved + +**Preserved unchanged (already proven on app1-bu, 2026-09-15):** +- The health probe (`probe_core_health`): checks gateway unit state, MainPID, state.db + header + row count, and cron scheduler freshness: not just ICMP. +- `fence_core()`: SSHes to Core and runs `systemctl --user stop hermes-gateway`, waits, + confirms `is-active` reads `inactive`/`failed` before allowing takeover. If the fence + cannot confirm, the standby stays dormant and alerts `[FAILOVER BLOCKED]` rather than + risking two gateways on one bot token. +- The failback block: if this box is already primary and Core's own health probe reads + `HEALTH OK`, the standby stops its own gateway (`pkill -f "hermes gateway"`) and alerts, + with no human action required. +- The `TAKEOVER_AFTER` consecutive-definitive-failure counter and its persisted state + file, so a single flaky probe cannot trigger a takeover. +- The Telegram-token-from-env-file fix and the SMTP-secret-outside-synced-tree fix (both + landed on app1-bu 2026-09-13, both real incidents: a dead hardcoded token that failed + silently, and a plaintext password backed up to S3 every 10 minutes). + +**Changed for core-bu:** +- `PROBE_SSH_KEY` moved from `/root/.hermes/profiles/anita/.ssh/itpp-infra` (an app1-bu- + specific path holding an *identical* copy of the standard key, used only because that + box's Anita profile happened to have it staged) to the standard `/root/.ssh/itpp-infra` + path used everywhere else in the fleet. This removes a profile-specific dependency the + original script had no good reason to carry. +- `STANDBY_NAME` and alert text now say "core-bu (netcup, Nuremberg)" instead of "app1-bu + (Hetzner)", so failover/failback alerts correctly identify which box took over. +- **New: `DISARM_FILE` gate** (`/root/.hermes/standby.disarmed`). This is the single + biggest functional addition. Neither the app1-bu watchdog nor sync script has any + concept of "armed" vs "disarmed": the only thing stopping both standbys from firing at + once today is that only one of them is deployed. The moment a second standby (core-bu) + exists, that safety property becomes a documentation-only promise unless it is also + enforced in code. This gate makes "disarmed" the shipped default and makes arming an + explicit, auditable, single-line action (delete one file) rather than "don't install the + cron entries" (easy to get half-right, e.g. one entry present and one missing). + +### 3.2 Sync: what changed vs the app1-bu original, and what was preserved + +**Preserved:** the same five-directory + three-file sync scope (`skills profiles plugins +cron references` + `config.yaml .env .bashrc`), deliberately excluding `state.db`: state +only moves at actual failover time via the watchdog's own `aws s3 sync`, exactly as +designed on app1-bu. This is correct and unchanged. + +**Fixed (the two defects named in the task):** +1. **Reachability test replaced.** The original (`ping -c 1 -W 2 "$LIVE_HOST"`, line 18 of + the deployed 1,245-byte script) treats a pingable-but-Hermes-dead Core as green light + to keep syncing skills/profiles/config from it: harmless for the sync's own purpose + (it only pulls passive config, never starts anything), but it meant the sync script + could never usefully participate in health-state decisions, and more importantly its + log gave false comfort ("Sync started" every 10 minutes through an outage). The new + version calls the same unit-state probe pattern as the watchdog + (`systemctl --user is-active hermes-gateway` over SSH) and only proceeds when Core's + gateway unit itself reads `active`. +2. **Failback / clobber-protection added.** The original sync script had **no concept of + this box being the active primary**: if core-bu had taken over and someone's cron + fired the sync script anyway, it would `aws s3 sync` Core's (now stale/absent) config + over the standby's own live config while it is actively serving. New version checks + `pgrep -f "hermes gateway"` first and exits immediately if this box is already primary, + matching the same self-awareness the watchdog already had. +3. **Single-armed-standby `DISARM_FILE` gate**, same rationale as 3.1. + +--- + +## 4. Arm / disarm procedure (single-armed-standby rule) + +**Rule: exactly one standby (app1-bu OR core-bu) may be armed at any moment.** A Core +hiccup with two armed standbys means both take over and both answer the same Telegram bot +token: worse than a slow, single failover. + +Enforcement is now two-layered: the runbook below, AND the `DISARM_FILE` code gate in +both scripts (section 3). The gate exists because runbook discipline alone failed to +protect against the exact bug this package fixes (nobody deliberately skipped writing +failback logic on 2026-09-15: it was just never added). Code-level enforcement should not +depend on the next person remembering the runbook. + +### Disarm app1-bu (must happen first, before arming core-bu) + +```bash +ssh -i /root/.ssh/itpp-infra root@5.161.225.131 \ + "touch /root/.hermes/standby.disarmed && \ + echo 'disarmed $(date -u +%FT%TZ) - superseded by core-bu' >> /root/.hermes/standby.disarmed" +``` + +Note: app1-bu's *currently deployed* scripts (the ones this package's predecessor audit +found at 386/41 lines) do not yet check for this file: the `DISARM_FILE` gate is new in +this package. **Before relying on disarming app1-bu this way, the same gate must be +back-ported into app1-bu's live scripts**, or disarming is a no-op there. Until that +back-port lands, the only reliable disarm for app1-bu is removing its crontab entries: + +```bash +ssh -i /root/.ssh/itpp-infra root@5.161.225.131 "crontab -l | grep -v hermes-standby > /tmp/c && crontab /tmp/c" +``` + +Verify disarmed: +```bash +ssh -i /root/.ssh/itpp-infra root@5.161.225.131 "crontab -l | grep -c hermes-standby" # expect 0 +``` + +### Arm core-bu (only after app1-bu is confirmed disarmed) + +1. Confirm prerequisites in section 1 are met, especially Tailscale join and Hermes + install (items 4 and 8). +2. Install the two scripts from section 2, with the crontab entries. +3. Leave `/root/.hermes/standby.disarmed` in place initially and let one full cron cycle + run (5-10 min) to confirm the scripts run cleanly in "disarmed, no-op" mode: check + `/var/log/hermes-standby-watchdog.log` for the disarmed-notice line and confirm no + errors. +4. Arm: `rm -f /root/.hermes/standby.disarmed` on core-bu. +5. Confirm within one cycle: `tail -20 /var/log/hermes-standby-watchdog.log` shows either + `HEALTH OK ... standby dormant` (expected, Core is healthy) or a real health line: not + silence and not an error. +6. Update `README.md`, `key-inventory.md`, and the other files named in + `reference-update-matrix-2026-09-15.md` rows 17-20, 28-31, 46, 48, 50-51 to record + core-bu as the armed standby and app1-bu as disarmed/retirement-candidate. This closes + the pre-existing "core-bu" naming collision documented in that matrix (six files + currently call the *Hetzner* box "core-bu": that ambiguity must be resolved before or + at the moment this arm step happens, not after). + +### Verify only one is armed at any time (recurring check) + +```bash +echo "app1-bu:"; ssh -i /root/.ssh/itpp-infra root@5.161.225.131 "crontab -l | grep -c hermes-standby; test -f /root/.hermes/standby.disarmed && echo DISARMED || echo ARMED" +echo "core-bu:"; ssh -i /root/.ssh/itpp-infra root@159.195.204.203 "crontab -l | grep -c hermes-standby; test -f /root/.hermes/standby.disarmed && echo DISARMED || echo ARMED" +``` +Expect exactly one host to print `ARMED`. + +--- + +## 5. Proof plan (failover AND failback, without risking production) + +**Any test that fences or stops Core's real gateway is a live production action and +requires the user's written approval first.** Nothing below should run against the real +Core gateway without that sign-off. The plan is staged so most of it can be proven with +zero production risk before the one step that does touch Core. + +### Stage A: Dry-run on core-bu alone (zero production risk, no approval needed) + +Uses the built-in `DRYRUN=1` and `WATCHDOG_FORCE_HEALTH` overrides already present in the +proven watchdog design (both preserved unchanged in this package): + +```bash +# Force two consecutive definitive-bad reads without ever touching Core or fencing anything +HEALTH_STATE=/tmp/core-bu-test.state \ +WATCHDOG_FORCE_HEALTH="HEALTH BAD reasons=unit:failed mainpid:0" \ +TAKEOVER_AFTER=2 DRYRUN=1 \ + /root/.hermes/scripts/hermes-standby-watchdog.sh +# run twice; expect log lines: DEGRADED (1/2...) then DEGRADED (2/2...) "decision reached: would fence ... (nothing done)" +``` +Verify: `/var/log/hermes-standby-watchdog.log` shows the 1/2 -> 2/2 -> dryrun-decision +sequence, `DO_FAILOVER` is never actually acted on, `fence_core` is never called (DRYRUN +skips it), and no Telegram/email is sent (DRYRUN suppresses notifications). This is the +exact validation sequence the app1-bu deployment used to prove the state machine on +2026-09-15 before trusting it in production, per the hermes-standby-deployment skill's +"Testing it without causing another outage" section. + +Also dry-run the recovered/failback branch: +```bash +HEALTH_STATE=/tmp/core-bu-test.state WATCHDOG_FORCE_HEALTH="HEALTH OK unit=active pid=123 msgs=5 scheduler_age_s=10" DRYRUN=1 /root/.hermes/scripts/hermes-standby-watchdog.sh +``` +Verify the RECOVERED log line fires and `$HEALTH_STATE` clears the BAD marker. + +### Stage B: Fence-and-failback rehearsal against a throwaway target (no production risk) + +Stand up a disposable VM (not app1-bu, not core-bu, not Core) running a scratch +`hermes-gateway.service`-like unit, point `PROBE_HOST`/`PROBE_SSH_KEY` at it, and run the +real (non-DRYRUN) watchdog against that target. This proves `fence_core()`'s SSH command +sequence and the failback block's `pkill` actually work end-to-end, with a target whose +loss has zero business impact. Requires a scratch host the user is willing to spin up; +does not require approval to touch Core because Core is never involved. + +### Stage C: Live failover/failback test against Core (PRODUCTION RISK: requires written approval) + +This is the only stage that fences the real Core gateway. **Do not run any part of this +stage without the user's explicit written approval first**, per this task's constraints +and standing policy for anything that stops a production service. + +If and when approved, the shape of the test (mirroring the successful 2026-09-15 app1-bu +proof, generalized to core-bu): +1. Announce a maintenance window; confirm nobody is mid-conversation on the Telegram bot. +2. On core-bu, temporarily set `TAKEOVER_AFTER=1` and run the watchdog manually (not via + cron) so the timing and every log line can be watched interactively. +3. Force the failure condition on Core in the most reversible way available: the + preferred method is `ssh core "systemctl --user stop hermes-gateway"` run BY THE + OPERATOR (not the watchdog) as the simulated failure, since this is trivially + reversible with `systemctl --user start hermes-gateway` and does not require touching + power/network. Avoid pulling the network or power unless specifically testing the + host-down path, since that path was already proven working (it's the simpler of the + two branches): the branch that actually needed proof after this fix is the + host-up/service-dead path. +4. Watch the watchdog: expect `DEGRADED (1/1, definitive=1)` -> fence attempt -> `FENCE OK + (inactive)` -> `DO_FAILOVER=1` -> S3 sync -> `hermes gateway start` on core-bu -> both + Telegram and email alerts delivered. +5. Confirm core-bu is now answering Telegram (send a test message, confirm response). +6. Recover Core: `systemctl --user start hermes-gateway` on Core. +7. Run the watchdog again on core-bu (still manually, or wait for the next cron tick): + expect the failback block to fire: `probe_core_health` reads `HEALTH OK`, core-bu + stops its own gateway, both alerts fire confirming failback. +8. Confirm Core alone is answering Telegram; confirm core-bu's gateway process is gone + (`pgrep -f "hermes gateway"` returns nothing on core-bu). +9. Restore `TAKEOVER_AFTER` to its production default (2) and re-arm the normal cron + schedule. +10. Record the full log excerpt from both `/var/log/hermes-standby-watchdog.log` (core-bu) + and Core's gateway journal as the proof artifact, same evidentiary standard as the + 2026-09-15 app1-bu proof. + +--- + +## 6. Monitoring and alerting expectations; what the parent must add to `health-master-watchdog.py` + +**Already present** (verified live, 2026-09-15): `REMOTE_SERVERS` in +`/root/.hermes/scripts/health-master-watchdog.py` (lines 89-97) already includes +`("core-bu", "159.195.204.203")` and `("app4", "159.195.205.80")`, added at the P1 +provisioning pass documented in `reference-update-matrix-2026-09-15.md` row 6. This gives +core-bu basic SSH-reachability monitoring (`check_remote_servers()`) today, alerting if the +box itself becomes unreachable. + +**Not present, and needed for this package to be observable centrally:** + +1. **Standby role/armed-state check.** `health-master-watchdog.py` has no concept of "is + the standby armed, and is exactly one standby armed fleet-wide." Add a check function + (e.g. `check_standby_arming()`) that SSHes to both app1-bu and core-bu, tests for + `/root/.hermes/standby.disarmed`, and alerts if: (a) both are armed simultaneously + (critical: the split-brain condition this whole package exists to prevent), or (b) + neither is armed (no DR coverage at all), or (c) the armed one's crontab is missing the + expected two entries. +2. **Standby watchdog liveness check.** Confirm the watchdog log (`/var/log/hermes- + standby-watchdog.log`) on the armed standby has a line newer than ~15 minutes (3x the + 5-minute cron interval): silence longer than that means the cron itself died, not that + Core is healthy. This mirrors the pattern `check_backup_freshness()` already uses for + S3 sync staleness in the same file (lines ~420-476), just applied to a log-mtime + instead of an S3 object timestamp. +3. **REMOTE_USER_UNITS entry for the standby's own gateway, conditionally.** Unlike + `anita-mnz` (whose entry is unconditional because her gateway always runs there), the + standby's `hermes-gateway` unit should be *absent* while dormant and *present and + active* only during a failover. Add a check that reads the armed state (from item 1) + and asserts: dormant standby => gateway absent/inactive is fine (no alert); armed + standby with the unit active => alert with "STANDBY IS CURRENTLY PRIMARY", since that + state should never persist longer than it takes to fail back once Core recovers. +4. **`REMOTE_DOCKER_CONTAINERS`**: no change needed: core-bu, like app1-bu, runs no + Docker containers for the standby role itself. +5. **Backup-freshness check should also cover core-bu once armed**, the same way it + already covers the `live/` S3 prefix: no new work needed here since the sync script + pulls from the same `live/` prefix `check_backup_freshness()` already audits. + +None of the above four gaps were fixed as part of this package (health-master-watchdog.py +lives on Core and per the task's read-only constraint on Core was not modified). This +section is the explicit list of what the parent must add. + +--- + +## 7. Provider-diversity risk (explicit, per task requirement) + +**The rule that has governed this org's DR design: "a netcup outage must not kill both +Core and its standby": is broken by this move**, and this is a genuine regression, not a +paperwork detail: + +- Core: netcup, Manassas VA. +- core-bu: netcup, Nuremberg, Germany. +- Both share one provider. A netcup-wide outage (account suspension, billing failure, + provider-side incident, netcup's own upstream/DC failure class of event) can take out + both simultaneously, in a way a Hetzner standby cannot be taken out by. +- What is gained instead: **regional diversity within netcup**: 100.5 ms RTT / different + continent / different power grid / different weather system than a Manassas-only + failure. This defends against the failure modes that hit *one location* (DC outage, + regional power, regional network) but not against failure modes that hit *the provider + as an entity* (account, billing, provider-wide incident). +- This exact regression is already flagged as a live, factually-incorrect claim risk in + `reference-update-matrix-2026-09-15.md` row 46 and its "Top 5" item 1: + `README.md` line 277 currently states "core-bu stays at Hetzner specifically so a + netcup outage can't kill both Core and standby simultaneously": that sentence becomes + **false**, not merely outdated, the moment core-bu (netcup) is armed. It must be + corrected as part of the documentation sweep in section 4 step 6, not left as-is. + +**Options, per the task's requirement to state them explicitly:** + +**Option A: Keep app1-bu (Hetzner) as a third tier, provider-diverse last resort.** +Cost: ~EUR 31.99/mo (Ashburn) or ~EUR 9.49/mo if relocated to Hetzner fsn1/nbg1 per +`standby-host-replacement-2026-09-14.md`'s own analysis (which itself is now superseded by +the decision to build core-bu: see reference-update-matrix row 30). Keeps a real +provider-diverse fallback for the specific netcup-wide failure class. Operational cost: +a third arm/disarm state to track, and the single-armed-standby rule now needs to cover +three hosts, not two: the `DISARM_FILE` gate in this package generalizes fine to three, +but the runbook and the monitoring check in section 6 item 1 must be written for "exactly +one of N armed," not "exactly one of two." + +**Option B: Accept the risk and retire app1-bu, documenting the gap.** Lower cost, one +fewer moving part, matches the plan currently in motion +(`migration-plan-app4-core-bu-2026-09-15.md` §5 acceptance criterion 6: "app1-bu is either +retired... or explicitly retained as the provider-diverse standby": this is an open +decision, not yet made, per that plan's own Q4). If chosen, the retirement must include +correcting the now-false README/architecture-doc claims (section 4 step 6) rather than +silently leaving them stale. + +**This decision has not been made by this task**: it belongs to the owner, per the +migration plan's own open-question list (Q4). This package is deployable under either +choice; the choice only affects whether app1-bu's scripts get the `DISARM_FILE` back-port +and stay in the crontab-removed "disarmed" state indefinitely (Option A) or get fully +retired per the P3 checklist in `reference-update-matrix-2026-09-15.md` (Option B). + +--- + +## 8. Summary of defects fixed, matched to the task's stated gaps + +| Defect | Where it lived | Fix in this package | +|---|---|---| +| Sync script tests reachability with `ping`; a host can answer ping while Hermes is dead | `hermes-standby-sync.sh` line 18 (`ping -c 1 -W 2 "$LIVE_HOST"`) | Replaced with the same unit-state SSH probe the watchdog uses; sync only proceeds when Core's gateway unit reads `active` | +| No failback logic in the sync path | `hermes-standby-sync.sh`, entire file | Added a primary-awareness guard (exits immediately if this box's own gateway is running) so the sync script can never clobber an active standby's config. Actual failback (stopping the standby's gateway) is owned by the watchdog, as it was in the proven 2026-09-15 fix, and preserved unchanged here | +| Decision branch, fence-before-takeover, self-stand-down failback (proven 2026-09-15) | `hermes-standby-watchdog.sh` | Preserved unchanged, ported to core-bu's host identity (probe key path, standby name in alerts) | +| No code-level enforcement of "only one standby armed" | Neither script, on either host | New `DISARM_FILE` gate added to both scripts in this package; ships disarmed by default | + +--- + +## Files delivered + +- This document: `/root/projects/itpp-infrastructure/docs/infrastructure/core-bu-standby-package-2026-09-15.md` +- `/root/work/core-bu/standby/hermes-standby-watchdog.sh` (397 lines): `bash -n`: clean +- `/root/work/core-bu/standby/hermes-standby-sync.sh` (84 lines): `bash -n`: clean + +Nothing was installed on core-bu, app1-bu, or Core. No systemd unit was created or +modified anywhere. No production process was started, stopped, or restarted. diff --git a/docs/infrastructure/migration-plan-app4-core-bu-2026-09-15.md b/docs/infrastructure/migration-plan-app4-core-bu-2026-09-15.md index ddf56af..91d8999 100644 --- a/docs/infrastructure/migration-plan-app4-core-bu-2026-09-15.md +++ b/docs/infrastructure/migration-plan-app4-core-bu-2026-09-15.md @@ -7,6 +7,14 @@ warm standby with a working failover **and failback**; (c) update every document, record and reference that names hosts, IPs or service locations. +**Status as of 2026-09-15 (post-Phase-3-start):** This plan's original Section 4 phase lists undercounted +scope — the verified inventory (`core-service-inventory-2026-09-15.md`) found roughly 30 customer-facing +services plus 3 separate DocuSeal instances, not the ~5 implied below. See Sections 10-12 (appended) for the +corrected scope, backup gaps, and defects found. **Actual migration progress to date: `microbin` has been +staged on `app4` and verified working at the application layer (container up, served content matches Core); +the DNS flip for microbin has NOT happened yet — it is still served live from Core.** No other service has +moved. Treat Section 4's phase list as directional; Section 10 is the authoritative scope. + --- ## 1. What is verified today (2026-09-15) @@ -281,16 +289,186 @@ Gate: `stale-reference-verify.py` and `doc-live-verify.py` both clean; every doc **Q1 - Tailscale:** add both boxes to the tailnet. Need either a reusable auth key, or approve the login URL from each box. +*Update 2026-09-15:* still open, plus a new cleanup item — **two Tailscale nodes exist for one physical box** +(`app1-bu`, offline 61 days, and `app1-bu-1`, idle). Decide whether to retire the stale `app1-bu` node now or +wait until Q4 (app1-bu retirement) is decided; either way one of the two entries is dead weight today. See +Section 12.4. **Q2 - Nuremberg vs Manassas for app4:** accept Nuremberg (design app4 self-contained) or re-order app4 in Manassas and repurpose the Nuremberg box? Section 3.1. +*Update 2026-09-15:* still open and now more consequential — the corrected scope (Section 10) is ~30 services, +not ~5, so the Atlantic-RTT cost of a wrong answer here is paid by far more customer-facing traffic than +originally assessed. **Q3 - Vaultwarden:** the CLI is locked. Unlock it (or tell me when) and I will file the two new server items. +*Update 2026-09-15:* still open, no change. **Q4 - app1-bu:** retire it, or keep it as the provider-diverse standby? Section 3.2 - this is the only remaining non-netcup box. +*Update 2026-09-15:* still open. Note the duplicate-Tailscale-node finding in Q1/Section 12.4 is directly tied +to this decision — retiring app1-bu should also retire its stale tailnet entry in the same action. **Q5 - app4 standby scope:** accept S3-restore RTO, or should core-bu carry a secondary copy? Section 3.3. +*Update 2026-09-15:* still open, and now higher-stakes: with ~30 services and ~17 of them carrying zero backup +coverage today (Section 11), an app4-only failure mode before those gaps are closed would be a real data-loss +event, not just an availability gap. Recommend resolving Section 11's gaps before finalizing this answer. + +--- + +## 10. Corrected scope and migration waves (added 2026-09-15, post-inventory) + +The counts above (Sections 1-9) predate `core-service-inventory-2026-09-15.md`. That inventory is now the +source of truth for scope. It found: + +- **13 Docker containers** on Core, including **three separate DocuSeal instances** (`docuseal` :8091, + `docuseal-dre` :8094, `docuseal-modelortho` :8092 — each its own bind-mounted `./data` dir and its own + `.env`; none of this is the single "DocuSeal" the original plan assumed), plus TimeTrex, microbin, + Uptime Kuma, Grafana, Prometheus, Telegraf, mikrotik-exporter, SearXNG, browserless, camofox-browser. +- **~30 additional customer-facing systemd services** the original plan never named: `auth-api`, + `diglocate-api`, `dre-mcp`, `dre-portal`, `ft360-mcp`, `hermes-voice`, `hotnow-api`, `intelsight-api`, + `ops-portal`, `osint-api`, `outlook-upload`, `pipeline-api`, `pry`, `pta-registration`, `rally`, + `seemytrip`, `shark-game`, `shopping-cart`, `survey-registration`, `transitpin`, `twilio-mcp`, + `verdicttank-api`, `verdicttank-worker`, `voice-agent`, `voice-agent-stt` — plus two disabled-but-present + units (`crawl4ai`, `hermes-control-deck`) that need an explicit keep/remove decision, not a silent carry-over. +- Two independent Postgres engines (host 17.10 for `hotnow`; TimeTrex's own containerized 16), not one shared + instance as Section 3/4 originally assumed. +- Two stale/dead Caddy routes (`crm.debtrecoveryexperts.com`, `crm.intelsight.io` — both proxy to a + `localhost:3003` nothing is listening on) that must be resolved (delete or repoint) before cutover, not moved + as-is. + +**Total real scope: ~30 customer-facing services + 3 DocuSeal instances**, versus the ~5 the phase list in +Section 4 was written around. Section 4's phases remain valid as a *pattern* (prove low-risk, then data +foundation, then bulk) but its per-phase examples are illustrative, not the full list. This section supersedes +that impression with actual waves: + +### Wave A — no state, no database (lowest risk, validates the runbook) +`microbin` (in progress — see Status line at top), `uptime-kuma`. Matches original Phase 3. + +### Wave B — SQLite-only services with a single owner and no cross-service dependency +`diglocate-api`, `dre-portal`, `dre-mcp`, `intelsight-api`, `pipeline-api`, `pry`, `pta-registration`, +`survey-registration`, `seemytrip`, `shark-game`, `shopping-cart`, `transitpin`, `outlook-upload`, +`ft360-mcp`. Rationale: each carries its own SQLite file and no shared Postgres/Redis dependency, so they can +move independently and in any order without a shared-data-foundation blocker; doing this wave before Wave C +also means the ~14 zero-backup services in this group (Section 11) get backup coverage on app4 sooner. + +### Wave C — shared data foundation + its dependents +Provision Postgres + Redis fresh on app4, then move `hotnow-api` (owns the `hotnow` Postgres DB + Redis DB1 +consumer), `auth-api` (has an existing backup script — lowest-risk of this wave), `rally`, +`verdicttank-api` + `verdicttank-worker`, `osint-api`. Rationale: these either own or depend on a database, so +they wait for the fresh Postgres/Redis instance the original Phase 4 already calls for, and moving the +database owner (`hotnow-api`) first de-risks the rest of the wave. + +### Wave D — DocuSeal (all three instances) + TimeTrex +`docuseal`, `docuseal-dre`, `docuseal-modelortho` (SQLite + attachments + internal Redis each — move and +verify one at a time, they are not interchangeable despite sharing an image), then `timetrex` (its own +containerized Postgres 16, needs a container-level `pg_dump`, not a host-level one). Rationale: highest +customer-visibility (signing flow, time tracking) and the most complex data migrations — done last, once the +runbook is proven on Waves A-C. + +### Wave E — voice stack + remaining +`voice-agent`, `voice-agent-stt`, `hermes-voice`, `twilio-mcp`. Rationale: unchanged from original Phase 5 — +Twilio webhook endpoints must be updated before DNS flip or calls break. Kept last because it is externally +integrated (Twilio-side config, not just DNS) and hardest to roll back quickly. + +### Explicitly out of scope for this migration (stays on Core) +`gitea-runner`, `hermes-assistant`, `hermes-browser`, `hermes-socat-8787`, `host-metrics-exporter`, +`node_exporter`, `osint-person`, `snmp-metrics`, `super-search`, `wazuh-agent`, `hermes-gateway`, `caddy` +(trimmed to Core-only routes post-cutover), and the two disabled units `crawl4ai` / `hermes-control-deck` +pending a keep/remove decision. + +--- + +## 11. Backup coverage gaps (added 2026-09-15, post-inventory) + +The inventory found **zero backup coverage** for the following services today. Each has its own SQLite DB (or, +for HotNow, a Postgres DB + Redis DB1) with no S3 job protecting it. These must get a backup job on app4 as +part of whichever wave moves them (Section 10) — do not carry the gap forward: + +**Update 2026-09-15 (later the same day) - most of this list is now closed.** Two backup jobs were +built, run, and proven by download-and-restore after this audit was written: + +- `app-data-backup.sh` (Core 2:15 AM, app4 5:10 AM) - SQLite snapshots taken through the `.backup` API and + `quick_check`ed, plus `pg_dump` of the `hotnow` Postgres DB; the Sunday 2:50 AM `--full` run adds + upload/attachment blobs. Proven: 10/10 DBs `quick_check=ok`, valid 52-entry `pg_restore` TOC for + `hotnow.dump`, 275 service code files byte-identical on restore. +- `docker-stack-defs-backup.sh` (Core 1:40 AM, app4 5:10 AM) - compose files, `.env`, configs and service + code for every `/root/docker` stack. Proven: 557 files, all 9 compose files byte-identical on restore. + +**Now covered:** HotNow (Postgres), IntelSight, Pipeline API, PTA registration, PTA survey, SeeMyTrip +(DB + uploads), Shopping Cart (uploads), DRE Portal (DB + uploads), Rally, Ops Portal, VerdictTank +`users.db`, DigLocate (uploads), and every `/root/docker` stack definition. + +**Still not covered:** `osint-api`, `osint-person`, `outlook-upload`, `voice-agent`, `voice-agent-stt`, +`ft360-mcp`, `hermes-voice`, `twilio-mcp`, and the HotNow **Redis DB1**. Note `shark-game` *is* covered - +it is in the `root-essentials-backup.sh` include list, so its entry below was an undercount. +The audit list that follows is retained as the record of what was found at the time. + + +1. `diglocate-api` — no backup script +2. `hotnow-api` — Postgres `hotnow` DB + Redis DB1, no backup script +3. `intelsight-api` — no backup script (has `intelsight.db`) +4. `osint-api` — no backup script +5. `osint-person` (MCP) — no backup script +6. `outlook-upload` — no backup script +7. `pipeline-api` — no backup script +8. `pry` — no backup script +9. `pta-registration` — no backup script +10. `survey-registration` (pta-survey) — no backup script +11. `seemytrip` — no backup script +12. `shark-game` — no backup script +13. `shopping-cart` — no backup script +14. `dre-portal` — no backup script (`dre.db`) +15. `voice-agent` — no backup script +16. `voice-agent-stt` — no backup script +17. `verdicttank-api` / `verdicttank-worker` — `users.db` has collection cron jobs but no DB backup job (counted + as one gap; the two units share the same unbacked database) + +Partial/unclear coverage worth flagging separately (not counted in the ~17 above, but not solid either): +`ft360-mcp` (stats/export scripts only, no DB backup), `rally` (debug/dump scripts only, no scheduled backup), +`ops-portal` (not explicitly separated from `core-services-backup.sh` — unconfirmed), `hermes-voice` and +`twilio-mcp` (backup status unknown, not verified either way). + +--- + +## 12. Known issues and defects found during Phase 3 (added 2026-09-15) + +### 12.1 Fleet provisioning defect: empty Docker data root on both new boxes +Both `app4` and `core-bu` shipped with `/var/lib/docker` **empty** despite Docker CE reporting installed and +the service active. Every image pull failed with `GetImageBlob` / `overlay2: no such file or directory` +(ENOENT) errors. Root cause not yet traced to a specific provisioning step; fixed on both boxes by stopping +Docker, `rm -rf /var/lib/docker/*`, and restarting the daemon (which recreates the expected directory +structure), then proven with a real `docker run hello-world` completing successfully on both. This is a +**fleet-level provisioning defect**, not a one-off — it hit both new boxes identically, so any future box built +from the same provisioning process should be checked for this before being marked "Docker ready." + +### 12.2 Docs publish pipeline dead for ~1 month +The MkDocs publish pipeline that puts these documents on the live site has been non-functional for +approximately a month, for three compounding reasons: (a) a dead/expired Gitea access token used by the +publish job, (b) a branch-resolution bug in the pipeline script that was silently masked by a trailing +`|| true`, so failures never surfaced as failures, and (c) the deploy step pointing at the wrong path. **No +Gitea Actions runner exists anywhere in the fleet**, so even with the above three items fixed, automated +publish still cannot fire today — publishing currently requires the owner to run it manually. This explains +why a plan this stale could stay "published" without anyone noticing the drift. + +### 12.3 app4's essentials backup does not cover `/root/docker` +`root-essentials-backup.sh` on app4 (Section 1.3) does not include `/root/docker` in its backup paths. Any +compose stack migrated onto app4 under `/root/docker/...` (which is where Docker-based services conventionally +live per Section 10's Wave A/D services) will **not** be captured by the essentials backup as currently +configured. This must be fixed before or immediately after the first Docker-based service lands on app4 — +otherwise migrated services are *less* protected on app4 than they were on Core. + +**Fixed 2026-09-15.** Rather than adding 5.8 GB of mostly-reproducible installs to the essentials tarball, +a dedicated job was written: `docker-stack-defs-backup.sh` archives compose files, `.env`, configs and +service code for every stack (240 files / 6.4 KB on Core), excluding `venv/`, `node_modules/`, caches and +any file >= 1 MiB. It runs on Core at 1:40 AM and on app4 at 5:10 AM, is namespaced per host in S3 +(`s3://hermes-vps-backups/docker-stack-defs/-.tar.gz`), and was proven by downloading the +archive back and byte-comparing all 9 compose files against live. + +### 12.4 Duplicate Tailscale node for one physical box +Two Tailscale nodes exist representing the same physical `app1-bu` host: `app1-bu` (offline 61 days) and +`app1-bu-1` (idle, presumably re-registered under a new name at some point without removing the old entry). +This is a cleanup item tied to the app1-bu retirement decision (Section 9, Q4) — the stale `app1-bu` node +should be removed from the tailnet regardless of which way Q4 is decided, since it is dead weight either way. --- @@ -309,3 +487,5 @@ remaining non-netcup box. | Prometheus `up{job="node_exporter"}` | core, app4, core-bu = 1 | | First backup run | upload + download/extract verify OK on both | | Credentials file | `/root/.hermes/references/new-servers-2026-09-15.md`, mode 600 | + +--- diff --git a/docs/infrastructure/migration-runbook-wave1-internal-2026-09-15.md b/docs/infrastructure/migration-runbook-wave1-internal-2026-09-15.md new file mode 100644 index 0000000..6407c57 --- /dev/null +++ b/docs/infrastructure/migration-runbook-wave1-internal-2026-09-15.md @@ -0,0 +1,756 @@ +# Migration Runbook — WAVE 1 (Internal / Low-Customer-Impact Services) + +**Date:** 2026-09-15 +**Scope:** Core (152.53.192.33) → app4 (159.195.205.80), exactly 6 services: +1. microbin (`:8260`, `share.itpropartner.com`) — **ALREADY MIGRATED, DONE** +2. Uptime Kuma +3. Prometheus + Grafana + node_exporter monitoring stack +4. Grafana specifically (unauthenticated public-bind finding) +5. PRY +6. OSINT API / OSINT Person MCP (`:8902`) + Super Search MCP (`:8899`) + +**Reference source docs:** `core-service-inventory-2026-09-15.md`, `migration-plan-app4-core-bu-2026-09-15.md`. +**Production hosts are READ-ONLY for this runbook's author.** All facts below were captured via read-only SSH +(`ss -ltnp`, `docker inspect`, `systemctl cat`, `cat` of configs) on 2026-09-15. No service was started, stopped, +or restarted to produce this document. `hermes-maintenance` was never invoked. + +**SSH access:** `ssh -i /root/.ssh/itpp-infra root@` (key already present on this box). + +**Proven pattern (established this session on microbin, reference for all 6 services):** +1. Caddy v2.11.4 on app4, pinned to Core's exact version, from the `caddy-stable` apt repo. +2. Service runs via `/root/docker//compose.yaml` (or `docker-compose.yml`), image **pinned by digest**, + not `:latest`. +3. app4's global Caddy option block sets `default_bind 159.195.205.80` so a new site block never hijacks another + listener (e.g. Tailscale's own :443). +4. `ufw` active on app4: 22/80/443 allowed, default deny incoming otherwise. +5. Cutover = flip DNS at the authoritative panel (SiteGround for `itpropartner.com`, **manual, no API** — + Germaine applies these), then Caddy on app4 auto-issues the cert via ACME http-01 on first request. +6. Verify side-by-side with `curl --resolve :443:159.195.205.80 https:///` **before** the DNS + flip, so the app layer is proven independent of DNS. + +**Known defect already fixed (do not re-diagnose):** both app4 and core-bu shipped with an empty Docker data +root (`/var/lib/docker` had zero subdirectories → every `docker pull`/`run` failed with `GetImageBlob` / +`overlay2: no such file or directory`). Fixed by stopping docker, `rm -rf /var/lib/docker`, restarting docker, +and proving with `docker run --rm hello-world`. Both boxes now run containers normally. If a pull/run fails with +an overlay2 ENOENT-style error on either box in the steps below, this is the known cause — check +`docker info | grep -i 'docker root dir'` and `ls /var/lib/docker` before assuming a new bug. + +**Known backup gap this runbook must close (Section 8):** app4's `root-essentials-backup.sh` include list +covers `.hermes .ssh .aws shark-game projects` (repo-root-relative) plus `/etc/caddy/Caddyfile`, but **not** +`/root/docker`. Every compose stack landed on app4 by this runbook (Uptime Kuma, monitoring stack, PRY if +containerized, OSINT/Super Search if containerized) is therefore invisible to the nightly backup until Section 8 +is applied. + +--- + +## Service 1 of 6 — microbin (`:8260`, `share.itpropartner.com`) — **STATUS: DONE** + +This is the proven reference pattern; it is documented here as the template the other 5 follow, not as new work. + +### Current state (Core) +- Container: `microbin` (`danielszabo99/microbin:latest`, image digest + `sha256:6660e5ccad0d764fa3c0032464ffb8f4b4f28c92a2eb9e39202b94cdc5b68909`), `unless-stopped`. +- Port: `127.0.0.1:8260 -> 8080` (container), not exposed externally on Core. +- Compose: `/opt/microbin/docker-compose.yml`. +- Data dir: `/opt/microbin/data` (bind mount -> `/app/pasta_data`), **4.0 KB** — no DB engine, flat files only. +- Caddy route (Core, `/etc/caddy/Caddyfile` line ~178): `share.itpropartner.com { reverse_proxy 127.0.0.1:8260 }`. +- Cert: real ACME (Let's Encrypt) issued to Core. + +### What was actually done (already executed, verified in this session) +1. Installed Caddy v2.11.4 on app4 from the `caddy-stable` apt repo (matches Core's `caddy version` output + exactly: `v2.11.4 h1:XKxkMTgNSizEvKG6QHue6cAsFOteU2qA61w2tKkCWi0=` on both hosts). +2. Created `/root/docker/microbin/docker-compose.yml` on app4, pinning the image **by digest** + (`danielszabo99/microbin@sha256:6660e5ccad0d764fa3c0032464ffb8f4b4f28c92a2eb9e39202b94cdc5b68909`), same + env vars as Core's compose, port bound `127.0.0.1:8260:8080`. +3. `docker compose up -d` on app4 — container `microbin` now `Up`, verified via `docker ps`. +4. Wrote app4's `/etc/caddy/Caddyfile`: + ``` + { + default_bind 159.195.205.80 + email info@itpropartner.com + } + # -- MicroBin Share (migrated from Core 2026-09-15, phase 3 of the app4 migration) -- + share.itpropartner.com { + reverse_proxy 127.0.0.1:8260 + encode gzip + } + ``` +5. Data dir was NOT rsynced from Core because Core's `/opt/microbin/data` was empty (4.0 KB, no user pastes) at + scan time — a fresh empty dir on app4 is equivalent. **If Core's data dir is non-empty at your actual cutover + time**, rsync it first: `rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/opt/microbin/data/ root@159.195.205.80:/root/docker/microbin/data/` and stop the Core container for the duration of the rsync to avoid a torn copy (microbin uses flat files, not a WAL DB, so this is a low-risk copy). + +### Verification (already run, real output) +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker ps --format '{{.Names}}\t{{.Ports}}\t{{.Status}}'" +# -> microbin 127.0.0.1:8260->8080/tcp Up +``` +App-layer check (curl from app4 itself, since DNS has not flipped yet): +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8260/" +# Expected: 200 +``` +**Confirmed:** direct curl to app4's microbin returns 200. App layer is fully staged and correct. + +### What is NOT done yet (pending, not part of "DONE" claim) +- **DNS flip is pending.** `itpropartner.com` is on SiteGround, manual panel, no API — Germaine must change the + `share.itpropartner.com` A record from 152.53.192.33 to 159.195.205.80. +- Cert on app4 has not been issued yet (Caddy issues on first real HTTPS hit post-DNS-flip via http-01). +- No S3 backup job exists yet for app4's `/root/docker/microbin/data` (see Section 8 — same gap applies here). + +### Rollback (if flip has NOT happened — current state) +Nothing to roll back: Core's container and Caddy route are untouched and still serving +`share.itpropartner.com` live. app4's copy is inert until DNS points at it. + +### Rollback (if DNS HAS been flipped and a problem appears) +1. Revert the `share.itpropartner.com` A record at SiteGround back to `152.53.192.33`. +2. TTL propagation is the only wait; Core's container was never stopped, so it answers immediately once DNS + re-resolves. +3. Do not touch app4's container — leave it running for retry. + +--- + +## Service 2 of 6 — Uptime Kuma + +### Current state (Core) +- Container: `uptime-kuma` (`louislam/uptime-kuma:latest`, image digest + `sha256:3d632903e6af34139a37f18055c4f1bfd9b7205ae1138f1e5e8940ddc1d176f9`, host layer digest + `f48d816cb7460cd3b7bb15ed393968b0ae0da4c690443b778b6a5db6b09f527e`), `unless-stopped`. +- Port: `0.0.0.0:3001 -> 3001` — **exposed on all interfaces on Core today** (not localhost-bound); reachable + directly bypassing Caddy on port 3001 from anywhere that can route to Core's public IP. This is pre-existing + Core behavior, not introduced by this migration — replicate `127.0.0.1:3001:3001` on app4 instead and let + Caddy be the only door, tightening the surface during the move. +- Compose: `/root/docker/uptime-kuma/docker-compose.yml`; volume `./data:/app/data` (relative bind, i.e. + `/root/docker/uptime-kuma/data`); has a healthcheck (`curl -f http://localhost:3001`) and resource limits + (1 CPU / 512M) already defined — carry these over unchanged. +- Data dir: `/root/docker/uptime-kuma/data`, **487 MB**, dominated by `kuma.db` (SQLite, 506,138,624 bytes as of + this scan — grows continuously from monitor heartbeat history). +- Caddy routes (Core): two site blocks reference it — + - `uptimekuma.itpropartner.com { handle /health { respond "OK" 200 } ; reverse_proxy localhost:3001 }` + - `status.itpropartner.com { ... handle /api/* { reverse_proxy 127.0.0.1:3001 } ... }` (status page also + proxies `/api/msg*` to Transitpin `127.0.0.1:8210` — that portion is out of WAVE 1 scope, do not move it) +- Cert: real ACME for `uptimekuma.itpropartner.com`; `status.itpropartner.com` likely shares handling (verify + both certs independently at cutover — do not assume one covers the other). +- **This is the monitoring tool itself** — its own downtime window must be announced per the migration plan + (Phase 3 note), since alerting is blind while it's down. + +### Pre-flight checks (read-only, run before touching anything) +``` +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "docker inspect uptime-kuma --format '{{.Image}} {{.State.Status}}'" +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "sqlite3 -readonly /root/docker/uptime-kuma/data/kuma.db 'PRAGMA quick_check;'" +# Expected: ok +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "du -sh /root/docker/uptime-kuma/data" +# Expected: ~487M (confirm current size before the copy so post-copy size is comparable) +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker info | grep -i 'docker root dir'; ls /var/lib/docker | wc -l" +# Confirm app4's Docker data root is populated (known-fixed defect) before pulling any image +``` + +### Move commands +1. Recreate the compose file on app4, image pinned by digest, port kept internal-only: +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "mkdir -p /root/docker/uptime-kuma" +cat > /tmp/uptime-kuma-compose.yaml <<'EOF' +services: + uptime-kuma: + image: louislam/uptime-kuma@sha256:3d632903e6af34139a37f18055c4f1bfd9b7205ae1138f1e5e8940ddc1d176f9 + container_name: uptime-kuma + restart: unless-stopped + volumes: + - ./data:/app/data + ports: + - "127.0.0.1:3001:3001" + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:3001"] + interval: 30s + timeout: 10s + retries: 3 + start_period: 10s + deploy: + resources: + limits: + cpus: '1' + memory: 512M + reservations: + cpus: '0.25' + memory: 128M + logging: + driver: json-file + options: + max-size: "10m" + max-file: "3" +EOF +scp -i /root/.ssh/itpp-infra /tmp/uptime-kuma-compose.yaml root@159.195.205.80:/root/docker/uptime-kuma/docker-compose.yml +``` +2. Take a **live-safe SQLite snapshot** on Core (do not `cp` a live WAL DB) and copy it to app4, matching the + pattern the existing `core-services-backup.sh` already uses for this exact file: +``` +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "sqlite3 -readonly /root/docker/uptime-kuma/data/kuma.db '.backup /tmp/kuma-migrate.db' && sqlite3 -readonly /tmp/kuma-migrate.db 'PRAGMA quick_check;'" +# Expected: ok +``` +3. Announce the monitoring downtime window, then **stop** the Core container (this is the one write-adjacent + action required — flagged explicitly since it is the monitoring source of truth; confirm with the requester + before executing, this runbook documents the command but does not execute it): +``` +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "docker stop uptime-kuma" +``` +4. Final consistent copy (container stopped, no WAL risk) plus the `docker-tls` subfolder (Kuma's own + self-issued monitor certs, if any monitors use them): +``` +rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/root/docker/uptime-kuma/data/ root@159.195.205.80:/root/docker/uptime-kuma/data/ +``` +5. Start on app4: +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "cd /root/docker/uptime-kuma && docker compose up -d" +``` +6. Add both Caddy blocks to app4's Caddyfile (mirroring Core, `default_bind` already set globally): +``` +uptimekuma.itpropartner.com { + handle /health { + respond "OK" 200 + } + reverse_proxy 127.0.0.1:3001 +} +``` +(`status.itpropartner.com`'s `/api/*` proxy to 3001 is bundled with the Transitpin relay on the same domain — +document it here but do not create a standalone `status.itpropartner.com` block until Transitpin's own WAVE +migration is scheduled, to avoid a half-built multi-path site block. `caddy validate` app4's file before reload.) + +### Verification (exact command + expected output) +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker exec uptime-kuma wget -qO- http://localhost:3001 | head -c 200" +# Expected: HTML containing "Uptime Kuma" (or a 200 via curl -o /dev/null -w '%{http_code}') + +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sqlite3 -readonly /root/docker/uptime-kuma/data/kuma.db 'SELECT COUNT(*) FROM monitor;'" +# Expected: same monitor count as Core's kuma.db at snapshot time — compare with: +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "sqlite3 -readonly /root/docker/uptime-kuma/data/kuma.db 'SELECT COUNT(*) FROM monitor;'" +# These two numbers MUST match exactly before cutover. + +curl --resolve uptimekuma.itpropartner.com:443:159.195.205.80 -sI https://uptimekuma.itpropartner.com/ +# Expected (post-DNS-flip, cert issued): HTTP/2 200 +``` + +### Rollback +1. `ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker compose -f /root/docker/uptime-kuma/docker-compose.yml down"` (stop app4's copy). +2. `ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "docker start uptime-kuma"` (Core's container and data dir + were never deleted — only stopped — so this brings monitoring back exactly where it left off). +3. If DNS was already flipped, revert the `uptimekuma.itpropartner.com` A record to `152.53.192.33`. +4. No data loss: Core's `kuma.db` is untouched by the migration (copy was one-directional, read-only source). + +--- + +## Service 3 of 6 — Prometheus + Grafana + node_exporter monitoring stack + +Treated as one unit because Prometheus scrapes node_exporter and feeds Grafana's dashboards; moving them +together avoids a window where Grafana has no data source. **Grafana's own port/auth finding is broken out +separately as Service 4** since it needs an explicit fix, not just a move. + +### Current state (Core) +- **Prometheus**: container `prometheus` (`prom/prometheus:latest`, `unless-stopped`, `NetworkMode: host`). + Config bind-mounted `/root/docker/monitoring/prometheus/prometheus.yml -> /etc/prometheus/prometheus.yml` + (ro). Data: named volume `prometheus_data` (`/var/lib/docker/volumes/prometheus_data/_data`, **114.9 MB**). + Also bind-mounts `/var/lib/prometheus/textfile` (ro) for host-metrics-exporter's textfile collector. No + compose file found — appears to be a manual `docker run` (ambiguity #1 in the inventory doc); reconstruct as + a compose file on app4 rather than perpetuating a hand-run container. + Listens `*:9090` (host network). + `prometheus.yml` (50 lines total) scrape jobs: `prometheus` (self, 127.0.0.1:9090), `mikrotik` (127.0.0.1:9436 + via mikrotik-exporter — **stays on Core**, it's scraping Core's local network device), `mikrotik-snmp` + (127.0.0.1:9274), `core-telegraf` (127.0.0.1:9273 — **stays on Core**, host-local metrics), `super-search` + (172.17.0.1:8899/metrics — Docker bridge IP, will need re-pointing if super-search moves), `node_exporter` + (already has 3 targets: `127.0.0.1:9100` labeled `core`, `159.195.205.80:9100` labeled `app4`, + `159.195.204.203:9100` labeled `core-bu` — this job already scrapes app4 and core-bu remotely from Core, added + during Phase 1 provisioning). +- **Grafana**: container `grafana` (`grafana/grafana:11.4.0`, `unless-stopped`, `NetworkMode: host`, no ports + block — the image binds directly since it's on host networking). Named volume `grafana_data_final` + (`/var/lib/docker/volumes/grafana_data_final/_data`, **~15 MB**) -> `/var/lib/grafana`. Two orphaned volumes + exist alongside it on Core (`grafana_data` 50 MB, `grafana_data_v3` 14.6 MB) — **do not migrate these**, they + are stale, only `grafana_data_final` is live per the running container's mount. + Config: `/root/docker/monitoring/grafana/grafana.ini`, referenced via env vars + (`GF_PATHS_CONFIG=/etc/grafana/grafana.ini`) — confirm at move time whether it's a bind mount or baked into + the volume; treat the working copy at `/root/docker/monitoring/grafana/grafana.ini` as authoritative source + of truth for env/ini settings. +- **node_exporter**: NOT a container — a systemd unit (`/etc/systemd/system/node_exporter.service`, + `ExecStart=/usr/local/bin/node_exporter`, no flags, `Restart=unless-stopped`). Binds `*:9100`. + **app4 already runs its own node_exporter** (systemd, active, listening `*:9100`, confirmed + `ss -ltnp | grep 9100` shows `node_exporter` on app4 right now) — this was stood up in Phase 1 provisioning + and is already scraped by Core's Prometheus (see the `app4` label above). **node_exporter is therefore + ALREADY DONE on app4 — no move action needed for this piece.** +- Caddy: **no Caddy site block routes to Prometheus or Grafana today.** Neither `:9090` nor `:3002` appears + anywhere in Core's 49 Caddy site blocks. Access today is by direct IP:port only. +- Cert: none — neither service is behind TLS today. +- Firewall (Core `ufw status numbered`): rule `[7] 8899/tcp ALLOW IN 172.17.0.0/16` (Docker-bridge-only, for + super-search's own metrics scrape — not customer-facing) and `[8] 3002/tcp ALLOW IN 100.64.0.0/10` + (Tailscale-range-only for Grafana) exist, but **UFW is a second gate behind the actual bind** — see Service 4, + because Grafana's `*:3002` listen and this UFW rule together still leave a gap, which is the whole point of + the Service 4 finding. + +### Pre-flight checks +``` +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "docker inspect prometheus --format '{{.Config.Image}} {{.State.Status}}'; docker inspect grafana --format '{{.Config.Image}} {{.State.Status}}'" +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "du -sh /var/lib/docker/volumes/prometheus_data/_data /var/lib/docker/volumes/grafana_data_final/_data" +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl is-active node_exporter; curl -s http://127.0.0.1:9100/metrics | head -1" +# Expected: active / a line starting "# HELP go_gc_duration_seconds" +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "curl -s 'http://127.0.0.1:9090/api/v1/query?query=up' | python3 -m json.tool | head -20" +# Baseline: capture current up{} series before the move for a before/after diff +``` + +### Move commands +1. Create target dirs on app4: +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "mkdir -p /root/docker/monitoring/prometheus /root/docker/monitoring/grafana" +``` +2. Copy configs (read-only source copy from Core, nothing stopped yet): +``` +scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/monitoring/prometheus/prometheus.yml /tmp/prometheus.yml +scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/monitoring/grafana/grafana.ini /tmp/grafana.ini +``` +3. **Edit the copy** (never Core's file) before pushing to app4: remove the `mikrotik`, `mikrotik-snmp`, and + `core-telegraf` jobs (Core-local, stay on Core) from `/tmp/prometheus.yml`; re-point `super-search`'s target + from `172.17.0.1:8899` to wherever Super Search lands (Service 6 — if it also moves to app4 in this wave, + target becomes `127.0.0.1:8899`; if it stays put, keep scraping Core's public IP with a dedicated firewall + allow, decided at Service 6 time). Push: +``` +scp -i /root/.ssh/itpp-infra /tmp/prometheus.yml root@159.195.205.80:/root/docker/monitoring/prometheus/prometheus.yml +scp -i /root/.ssh/itpp-infra /tmp/grafana.ini root@159.195.205.80:/root/docker/monitoring/grafana/grafana.ini +``` +4. rsync the live volume data in two passes — a warm first pass while Core's containers still run, then a + final short-stop pass for consistency (same pattern as the Uptime Kuma DB): +``` +rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/var/lib/docker/volumes/prometheus_data/_data/ /tmp/prom-data-staging/ +rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/var/lib/docker/volumes/grafana_data_final/_data/ /tmp/grafana-data-staging/ +# announce downtime, then: ssh ... root@152.53.192.33 "docker stop prometheus grafana" +rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/var/lib/docker/volumes/prometheus_data/_data/ /tmp/prom-data-staging/ +rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/var/lib/docker/volumes/grafana_data_final/_data/ /tmp/grafana-data-staging/ +scp -r -i /root/.ssh/itpp-infra /tmp/prom-data-staging root@159.195.205.80:/tmp/ +scp -r -i /root/.ssh/itpp-infra /tmp/grafana-data-staging root@159.195.205.80:/tmp/ +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker volume create prometheus_data && docker run --rm -v prometheus_data:/dest -v /tmp/prom-data-staging:/src alpine cp -a /src/. /dest/" +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker volume create grafana_data_final && docker run --rm -v grafana_data_final:/dest -v /tmp/grafana-data-staging:/src alpine cp -a /src/. /dest/" +``` +5. Write `/root/docker/monitoring/docker-compose.yml` on app4, images pinned by digest (resolve fresh: + `docker inspect prometheus --format '{{index .RepoDigests 0}}'` on Core for Prometheus; Grafana's tag + `11.4.0` is already a pin — resolve its digest the same way), `network_mode: host` preserved for Prometheus + to match Core's scrape behavior; Grafana's networking is changed — see Service 4. +6. `docker compose up -d` on app4. + +### Verification +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s http://127.0.0.1:9090/-/healthy" +# Expected: "Prometheus Server is Healthy." +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s 'http://127.0.0.1:9090/api/v1/query?query=up' | python3 -m json.tool" +# Expected: same target set (minus mikrotik/telegraf/core-only jobs) all reporting value "1" +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3002/api/health" +# Expected: 200 +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s http://127.0.0.1:3002/api/health" +# Expected JSON containing "database":"ok","version":"11.4.0" — confirms grafana.db loaded correctly +``` +Series-count parity check (compare Core's pre-stop snapshot vs app4's loaded volume before deleting anything +on Core): +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s http://127.0.0.1:9090/api/v1/label/__name__/values | python3 -c 'import json,sys; print(len(json.load(sys.stdin)[\"data\"]))'" +# Compare metric-name count to the same query run on Core before stop +``` + +### Rollback +1. `ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker compose -f /root/docker/monitoring/docker-compose.yml down"`. +2. `ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "docker start prometheus grafana"` — Core's volumes and + config were only read from, never deleted; original containers restart with their original state intact. +3. No DNS is involved (no Caddy route exists either direction today), so rollback is purely stop-app4/ + start-Core with no propagation delay. +4. Any staging copy edited on the operator's own box (`/tmp/prometheus.yml`) is disposable; Core's live + `prometheus.yml` was never touched, per the read-only constraint on Core. + +--- + +## Service 4 of 6 — Grafana public-bind finding (`*:3002`, unauthenticated, no Caddy/Tailscale gate) + +This is called out separately from Service 3 because it is a **security finding to fix during the move**, not +just a relocation. + +### The finding, precisely +- Grafana's container runs `NetworkMode: host` and the Grafana process itself binds `*:3002` (confirmed via + `ss -ltnp`: `LISTEN 0 4096 *:3002 *:* users:(("grafana",pid=2906,fd=15))`) — i.e. **every** interface on Core, + including its public IPv4/IPv6, not just loopback or the Tailscale interface. +- It is **not** behind any Caddy site block (grepped all 49 blocks in Core's Caddyfile — zero mention of `3002` + or a Grafana upstream). +- The only mitigation in place is a UFW rule restricting **inbound port 3002 to the Tailscale CGNAT range** + (`100.64.0.0/10`) — `ufw status numbered` line `[8] 3002/tcp ALLOW IN 100.64.0.0/10`. UFW's default policy is + `deny (incoming)`, so absent that specific allow rule, 3002 would in fact be closed by default — **but this + means the ONLY thing standing between Grafana and the open internet is one UFW line**, with no defense in + depth (no app-level auth beyond Grafana's own login, no Caddy layer, no Tailscale-only bind at the app level). +- Compounding it: Grafana's `grafana.ini` `[security]` section has + `admin_user = admin` / `admin_password = admin` / `disable_initial_admin_creation = false` — **the default + admin credentials were never changed**. If the UFW rule is ever misconfigured, misapplied to the wrong + interface, or a firewall reload race drops it (a known class of bug on reboot ordering), the admin/admin + login is directly exposed to anyone who reaches that interface. + +### Fix to apply on app4 (do NOT attempt this on Core — Core is read-only for this task) +1. Do not repeat the `NetworkMode: host` + open-bind pattern on app4. In the Service 3 compose file, add an + explicit port mapping and drop host networking for Grafana specifically (Prometheus can keep host + networking since it only needs outbound scrape access; Grafana only needs to reach Prometheus on the same + host, no host networking required): + ```yaml + grafana: + image: grafana/grafana@sha256: + ports: + - "127.0.0.1:3002:3000" # loopback only; Caddy or Tailscale serves it externally + ``` +2. Add a real Caddy site block on app4 gating it behind HTTP basic auth (mirroring the existing pattern used + for `internal.debtrecoveryexperts.com`'s `basic_auth` block in Core's Caddyfile): + ``` + grafana.itpropartner.com { + basic_auth { + admin + } + reverse_proxy 127.0.0.1:3002 + } + ``` + (Generate the hash with `caddy hash-password` — never hand-write a bcrypt hash.) +3. Change the default admin credentials in `grafana.ini` (or via `GF_SECURITY_ADMIN_PASSWORD` env var) on + app4's first boot — do not carry `admin/admin` forward. This must happen before app4's Grafana is reachable + by anything but loopback. +4. Confirm `ufw status` on app4 does not carry forward the Core Tailscale-3002 rule as the *only* gate — Caddy + basic_auth is now gate 1; Tailscale-only access can remain as an additional, not sole, control. + +### Verification +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "ss -ltnp | grep 3002" +# Expected: 127.0.0.1:3002 only — NOT *:3002 or 0.0.0.0:3002 +curl -s -o /dev/null -w '%{http_code}\n' http://159.195.205.80:3002/ +# Expected: connection refused / timeout (nothing listening on the public interface) +curl -s -o /dev/null -w '%{http_code}\n' https://grafana.itpropartner.com/ +# Expected: 401 (basic_auth challenge) before credentials, 200 after +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "grep -E 'admin_password' /root/docker/monitoring/grafana/grafana.ini" +# Expected: NOT the literal string "admin" — confirm it was actually changed +``` + +### Rollback +This fix only touches app4, a not-yet-cutover host; Core is never modified. Rollback is simply: do not flip +DNS/Tailscale access to app4's Grafana until the verification above passes. Core's Grafana keeps running with +its existing (already-known-weak) posture unchanged — a pre-existing condition to flag to Germaine regardless +of migration timing, not something this runbook worsens or is required to fix on Core. + +--- + +## Service 5 of 6 — PRY (unified OSINT search backend) + +### Current state (Core) +- Not a container — a systemd unit: `pry.service` + (`/etc/systemd/system/pry.service`, `Type=simple`, `User=root`, + `WorkingDirectory=/root/docker/pry`, `ExecStart=/root/docker/pry/venv/bin/python /root/docker/pry/server.py`, + `EnvironmentFile=/root/.hermes/.env`, `Restart=always`, `ProtectSystem=full`, `ProtectHome=read-only`, + `ReadWritePaths=/root/docker/pry`, `ReadOnlyPaths=/root/.hermes/.env /root/.config/himalaya/shonuff.pass`). +- Port: `127.0.0.1:8905`, confirmed via `ss -ltnp` — loopback only, not directly internet-reachable. +- App dir: `/root/docker/pry`, **66 MB total** (mostly its own `venv`). Python 3.13.5 venv, deps from + `requirements.txt` (`fastapi>=0.109.0`, `uvicorn[standard]>=0.27.0`, `httpx>=0.26.0`, `python-dotenv>=1.0.0`, + `aiosmtplib>=3.0.0` — 5 loose lines, no pinned freeze file, so re-resolve at install time rather than + assuming exact matching versions). +- Data/DB: **none found** — no `.db`/`.sqlite*` file in `/root/docker/pry`. PRY is stateless app logic; its + only persistent asset is its git-tracked source (`.git` present) plus a private `Caddyfile-new` (mode 600, + 4842 bytes — appears to be a staged/unused Caddy config someone drafted but never applied; carry it over for + reference but do not activate it blindly). +- Secrets: `EnvironmentFile=/root/.hermes/.env` — shared secrets file across many services; only PRY-relevant + keys need to be extracted/copied to app4's own `.env`, not the whole file (per the inventory doc's finding #7 + on live secrets requiring manual, non-rsync transfer). +- Caddy route (Core): `http://pry.iamgmb.com { @api path /api/* ; handle @api { reverse_proxy 127.0.0.1:8905 } ; handle { root * /var/www/pry ; try_files ... ; file_server } }` — **HTTP only, no TLS** (`http://` scheme + explicit in the Caddyfile block header), and it's a split route: `/api/*` goes to PRY's backend, everything + else is a static SPA served from `/var/www/pry` on Core. The static half is out of scope for a backend-only + WAVE 1 move — moving PRY's backend without also moving `/var/www/pry`'s static files would break the + frontend, so **both must move together** even though the task frames this as one backend service. +- Cert: none (HTTP-only site today). + +### Pre-flight checks +``` +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "systemctl is-active pry.service; ss -ltnp | grep 8905" +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8905/api/health || curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8905/" +# Record baseline response code before moving +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "du -sh /var/www/pry" +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "grep -c '^[A-Z_]*=' /root/.hermes/.env" +# Confirms the shared secrets file's line count so a partial extraction can be sanity-checked for completeness +``` + +### Move commands +1. Copy the app dir (excluding the venv, which will be rebuilt fresh on app4 to avoid glibc/arch drift): +``` +rsync -avz -e "ssh -i /root/.ssh/itpp-infra" --exclude 'venv' --exclude '__pycache__' root@152.53.192.33:/root/docker/pry/ root@159.195.205.80:/root/docker/pry/ +``` +2. Copy the static frontend: +``` +rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/var/www/pry/ root@159.195.205.80:/var/www/pry/ +``` +3. Rebuild the venv on app4: +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "cd /root/docker/pry && python3 -m venv venv && ./venv/bin/pip install -r requirements.txt" +``` +4. Manually extract PRY-relevant keys from Core's `/root/.hermes/.env` (grep for the specific vars `server.py` + actually reads — do this by inspecting `os.getenv`/`os.environ` calls in `server.py`, not by copying the + whole file) into a new, minimal `/root/.hermes/.env` on app4, or a dedicated `/root/docker/pry/.env` + + adjust the systemd unit's `EnvironmentFile=` path. This is a manual, secure step — never rsync the shared + `.env` wholesale (matches the inventory doc's explicit warning on secret files). +5. Create the systemd unit on app4 (same content as Core's, `WorkingDirectory`/`ExecStart` paths already match + since both boxes use `/root/docker/pry`): +``` +scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/pry/pry.service /tmp/pry.service +# edit EnvironmentFile= path if app4 uses a dedicated .env, then: +scp -i /root/.ssh/itpp-infra /tmp/pry.service root@159.195.205.80:/etc/systemd/system/pry.service +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl daemon-reload && systemctl enable --now pry.service" +``` +6. Add the Caddy block to app4, upgraded to real TLS this time (Caddy on app4 will ACME-issue since it's a + real domain — no reason to keep it HTTP-only post-move): +``` +pry.iamgmb.com { + @api path /api/* + handle @api { + reverse_proxy 127.0.0.1:8905 + } + handle { + root * /var/www/pry + try_files {path} {path}.html /index.html + file_server + } +} +``` + +### Verification +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl is-active pry.service" +# Expected: active +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8905/" +# Expected: same status code as the Core baseline captured in pre-flight +curl --resolve pry.iamgmb.com:443:159.195.205.80 -sI https://pry.iamgmb.com/ +# Expected (post-DNS-flip, cert issued): HTTP/2 200 +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "journalctl -u pry.service -n 20 --no-pager" +# Expected: no tracebacks, normal startup log lines matching Core's own startup pattern +``` + +### Rollback +1. `ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl stop pry.service"`. +2. Core's `pry.service` was never touched — it is still `active` throughout, so no restart action needed on + Core. +3. If DNS was flipped, revert `pry.iamgmb.com` to `152.53.192.33` (note: this domain has no `A` vs `CNAME` + redirect complication, it's a direct A record per the plan's DNS checklist). +4. No data to restore (stateless service, no DB). + +--- + +## Service 6 of 6 — OSINT Person MCP (`:8902`) + Super Search MCP (`:8899`) + +These are grouped because OSINT Person MCP imports Super Search's own module directly +(`from super_search import web_search as ss_search, web_extract as ss_extract` in +`osint-person-mcp/server.py`) — they have a **code-level dependency**, not just a runtime one, so they must +move together or the import breaks. + +### Current state (Core) + +**OSINT Person MCP:** +- systemd unit `osint-person.service` (`After=network.target super-search.service` — systemd-level ordering + dependency confirming the two are coupled), `WorkingDirectory=/root/docker/osint-person-mcp`, + `ExecStart=/root/docker/osint-person-mcp/venv/bin/python3 /root/docker/osint-person-mcp/server.py`, + `EnvironmentFile=/root/.hermes/.env`, `Restart=always`. +- Port: `127.0.0.1:8902` (confirmed `ss -ltnp`), used as `http://127.0.0.1:8902/mcp` per Hermes' + `config.yaml` MCP registration — loopback only. +- App dir: `/root/docker/osint-person-mcp`, **314 MB** (Python 3.13.5 venv-heavy). Source files include + `server.py` (54,253 bytes, modified same-day as this scan — active development), `reconcile.py`, + `super_search.py` (a **local copy** of Super Search's search/extract functions, imported directly — this is + the code coupling), `telemetry.py`, plus backup files (`server.py.bak-*`) and a `tests/` dir. No standalone + requirements file was found in the directory listing — dependencies live only in the existing `venv`; + capture `venv/bin/pip freeze > requirements.txt` on Core before rebuilding on app4, do not guess versions. +- Data/DB: none found (no `.db` file); it's a stateless MCP tool server backed by live web calls. + +**Super Search MCP:** +- systemd unit `super-search.service` (`WorkingDirectory=/root/docker/super-search`, + `ExecStartPre=/bin/bash /root/docker/super-search/super-search-port-guard.sh` — a pre-start guard script, + copy this too, it likely checks for a stale port bind before starting — `ExecStart=.../venv/bin/python3 + .../server.py`, `EnvironmentFile=/root/.hermes/.env`, `Restart=always`, `StartLimitBurst=5` over 300s). +- Port: `0.0.0.0:8899` — confirmed **bound to all interfaces on Core**, mitigated only by + `ufw status numbered` rule `[7] 8899/tcp ALLOW IN 172.17.0.0/16` (Docker-bridge-range only — i.e. only + containers on Core's own `docker0`/bridge network can reach it externally-looking-in; this is intentional + since Prometheus's `super-search` scrape job targets `172.17.0.1:8899`, the bridge gateway IP, meaning + Prometheus itself runs in a container reaching it via the bridge). Registered in Hermes as + `http://127.0.0.1:8899/mcp`. +- App dir: `/root/docker/super-search`, **1.1 GB** (largest of the six — heavy venv, Python 3.13.5). Has a + pinned `requirements.freeze.txt` (167 lines) — **use this file directly on app4**, it's already a proper pin, + unlike PRY's loose `requirements.txt` or OSINT Person's missing one. +- Data/DB: none found — stateless, backed by live search-provider calls (SearXNG on Core, Exa, Brave, etc. per + the skill's provider list); note SearXNG itself (`127.0.0.1:8888`) **stays on Core** per the inventory doc, + meaning **if Super Search moves to app4, its SearXNG provider call becomes a cross-Atlantic Core→app4→Core + round trip (~100ms each way per the migration plan's RTT finding) unless SearXNG is also exposed to app4**, + which is out of WAVE 1 scope. Recommend either: (a) leave Super Search on Core for now despite the task's + inclusion of it here, since its own migration-plan classification already says "stays-on-Core (Hermes MCP)" + in the inventory doc (contradicting this task's WAVE 1 list — flag this conflict to the requester explicitly + before executing), or (b) accept the added latency for one SearXNG-backed provider path while other + providers (Exa, Brave, Serper, Tavily) call out directly with no Core round-trip. **This is a real + discrepancy between the source inventory and this task's WAVE 1 scope — documented here, not silently + resolved.** +- Caddy: **neither service has a Caddy route today.** Both are pure loopback/bridge MCP servers consumed only + by Hermes' own `config.yaml` `url: http://127.0.0.1:PORT/mcp` entries — there is no public hostname for + either in the 49 scanned Caddy blocks. +- Cert: none (no public exposure today; loopback-only by design). + +### Pre-flight checks +``` +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "systemctl is-active osint-person.service super-search.service" +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8902/mcp; curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8899/mcp" +# Record baseline codes (MCP endpoints often 4xx on a bare GET without proper headers — record whatever Core +# returns today as the comparison target, don't assume 200) +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "/root/docker/osint-person-mcp/venv/bin/pip freeze > /tmp/osint-person-freeze.txt; wc -l /tmp/osint-person-freeze.txt" +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "du -sh /root/docker/osint-person-mcp /root/docker/super-search" +``` + +### Move commands +1. Freeze OSINT Person's deps (not previously pinned) and pull both freeze files: +``` +scp -i /root/.ssh/itpp-infra root@152.53.192.33:/tmp/osint-person-freeze.txt /tmp/ +scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/super-search/requirements.freeze.txt /tmp/ +``` +2. Copy source (excluding venv/pycache/backup cruft — the `.bak-*` files are large and not needed): +``` +rsync -avz -e "ssh -i /root/.ssh/itpp-infra" --exclude 'venv' --exclude '__pycache__' --exclude '*.bak*' root@152.53.192.33:/root/docker/osint-person-mcp/ root@159.195.205.80:/root/docker/osint-person-mcp/ +rsync -avz -e "ssh -i /root/.ssh/itpp-infra" --exclude 'venv' --exclude '__pycache__' --exclude '*.bak*' root@152.53.192.33:/root/docker/super-search/ root@159.195.205.80:/root/docker/super-search/ +``` +3. Rebuild both venvs on app4 from the frozen requirements: +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "cd /root/docker/osint-person-mcp && python3 -m venv venv && ./venv/bin/pip install -r /tmp/osint-person-freeze.txt" +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "cd /root/docker/super-search && python3 -m venv venv && ./venv/bin/pip install -r requirements.freeze.txt" +``` +4. Extract only the OSINT/Super-Search-relevant keys from Core's `/root/.hermes/.env` into app4's own + `.hermes/.env` (same manual-secret-transfer rule as PRY — do not rsync the whole file). +5. Copy systemd units and the port-guard script, adjusting nothing but confirming paths already match + (`/root/docker/...` is identical on both boxes): +``` +scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/osint-person-mcp/osint-person.service /tmp/ +scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/super-search/super-search.service /tmp/ +scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/super-search/super-search-port-guard.sh /tmp/ +scp -i /root/.ssh/itpp-infra /tmp/osint-person.service /tmp/super-search.service root@159.195.205.80:/etc/systemd/system/ +scp -i /root/.ssh/itpp-infra /tmp/super-search-port-guard.sh root@159.195.205.80:/root/docker/super-search/ +``` +6. **Fix Super Search's bind before enabling** — do not carry `0.0.0.0:8899` forward. If `server.py` hardcodes + the bind host, patch app4's copy to `127.0.0.1:8899` (matching OSINT Person's already-safe loopback pattern) + unless something on app4 genuinely needs the Docker-bridge-range access Core's UFW rule was protecting + (confirm this is even needed on app4 — app4 doesn't run the same Prometheus container-bridge topology as + Core unless Service 3 also lands there with the same `172.17.0.1` scrape pattern). +7. Enable both, in dependency order (Super Search first, since OSINT Person's own unit declares + `After=... super-search.service`): +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl daemon-reload && systemctl enable --now super-search.service && sleep 3 && systemctl enable --now osint-person.service" +``` +8. Update Hermes' `config.yaml` MCP `url:` entries — but only **after** confirming which box's Hermes actually + calls these tools. If it's Core's own Hermes gateway consuming these via `127.0.0.1`, moving the servers to + app4 means Core's config must change to `http://159.195.205.80:8902/mcp` / + `http://159.195.205.80:8899/mcp`, which reintroduces exactly the cross-Atlantic per-call latency the + migration plan's Section 3.1 design rule warns against ("app4 must not depend on Core at request time" — the + inverse also applies: Core's live Hermes gateway should not depend on app4 at request time for + every-tool-call-latency reasons). **This is a design question for the requester, not a call this runbook + makes** — flagged here rather than silently deciding to change `config.yaml`. + +### Verification +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl is-active osint-person.service super-search.service" +# Expected: active / active +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "ss -ltnp | grep -E ':(8899|8902)\b'" +# Expected: 127.0.0.1:8899 and 127.0.0.1:8902 — NOT 0.0.0.0:8899 (confirms the bind fix from step 6 landed) +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8902/mcp; curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8899/mcp" +# Expected: same status codes as the Core baseline captured in pre-flight +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "journalctl -u osint-person.service -u super-search.service -n 30 --no-pager" +# Expected: no import errors (specifically watch for "ModuleNotFoundError: super_search" — the code-coupling +# risk called out above — and no missing-env-var tracebacks) +``` + +### Rollback +1. `ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl stop osint-person.service super-search.service"`. +2. Core's two units were never stopped or restarted — they remain `active` throughout, so Hermes' existing + `127.0.0.1` MCP URLs keep working with zero interruption regardless of app4's state. +3. If `config.yaml` was changed (step 8), revert the `url:` fields back to `http://127.0.0.1:8902/mcp` and + `http://127.0.0.1:8899/mcp` — this requires whichever Hermes profile/gateway consumes them to reload its MCP + config (out of scope for this SSH-only runbook; note it as a manual follow-up). +4. No data to restore (both stateless). + +--- + +## Section 8 — Backup coverage gap: `/root/docker` missing from app4's `root-essentials-backup.sh` + +### The gap +app4's `/root/root-essentials-backup.sh` (v2.0.0, 2026-09-11) builds its include list from this loop: +```bash +for p in .hermes/config.yaml .hermes/.env .hermes/skills .hermes/scripts \ + .hermes/references .hermes/profiles .hermes/cron .hermes/memories \ + .hermes/DR-PLAN.md .hermes/data .hermes/cache/delegation \ + .ssh .aws .config/himalaya \ + shark-game projects ; do + [ -e "$p" ] && ARGS+=("$p") +done +``` +`/root/docker` (relative: `docker`) is **not in this list**. Every compose stack this runbook lands on app4 — +Uptime Kuma's `docker-compose.yml` + its data dir path, the monitoring stack's compose file, PRY's app dir if +placed under `/root/docker` (it is), OSINT Person MCP, Super Search — is therefore **entirely invisible** to +the nightly 04:45 ET run, even though the script's own `tar` excludes `*.db`/`*.db-shm`/`*.db-wal` regardless +(so Uptime Kuma's `kuma.db` and any future SQLite files still need their own `sqlite3 .backup` job even after +this fix — this patch closes the "missing compose/config/source" half of the gap, not the "missing DB" half, +which is already flagged as a separate Phase 4/5 acceptance item in the migration plan). + +### Exact patch +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sed -n '55,60p' /root/root-essentials-backup.sh" +# Confirm current line content before editing, since this is a live production script: +# .ssh .aws .config/himalaya \ +# shark-game projects ; do +``` +Change: +```diff +- .ssh .aws .config/himalaya \ +- shark-game projects ; do ++ .ssh .aws .config/himalaya \ ++ shark-game projects docker ; do +``` +Apply with `patch`/`sed` on app4 (do NOT run this against Core — Core is read-only for this task, and Core's +own backup script coverage is a separate, already-tracked item per the inventory doc's Section on backup gaps): +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sed -i 's/shark-game projects ; do/shark-game projects docker ; do/' /root/root-essentials-backup.sh" +``` + +### Verify the patch landed correctly +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "grep -n 'shark-game projects' /root/root-essentials-backup.sh" +# Expected: shark-game projects docker ; do +``` + +### Restore test (prove the new coverage actually works, not just that the line changed) +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "bash /root/root-essentials-backup.sh 2>&1 | tail -20" +# Expected final lines: "[...] Tarball: bytes" (larger than the pre-patch baseline by roughly the size of +# /root/docker's non-.db content — compare against a du -sh /root/docker taken beforehand), then +# "[...] content check OK - N entries, 3/3 key essentials present", then "Uploading...", then +# "Backup verified - archive is valid", then "Done". +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "aws s3 cp s3://hermes-vps-backups/root-backup/\$(hostname -s)/root-essentials-\$(date +%F).tar.gz /tmp/restore-test.tar.gz --endpoint-url https://s3.us-east-1.wasabisys.com --quiet && tar tzf /tmp/restore-test.tar.gz | grep '^docker/' | head -20" +# Expected: a non-empty listing of docker//... paths — proves /root/docker is now actually inside the +# uploaded archive, not just inside the include-list source, and that the round-trip download+list works. +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "rm -f /tmp/restore-test.tar.gz" +``` + +### Rollback (if the patched script misbehaves — e.g. archive size becomes unexpectedly huge from a stray venv) +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sed -i 's/shark-game projects docker ; do/shark-game projects ; do/' /root/root-essentials-backup.sh" +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "grep -n 'shark-game projects' /root/root-essentials-backup.sh" +# Expected: shark-game projects ; do (back to original) +``` +If venv bloat is the issue (Super Search alone is 1.1 GB, mostly venv), prefer excluding venvs specifically +over reverting the whole line: +``` +# Add to the tar invocation's --exclude list (near the existing --exclude='*.db' flags): +--exclude='docker/*/venv' --exclude='docker/*/*/venv' +``` + +--- + +## Summary table + +| # | Service | Status | Downtime needed | DNS flip needed | New security work | +|---|---|---|---|---|---| +| 1 | microbin | **DONE** (app layer), DNS flip pending | None (data dir was empty) | Yes — share.itpropartner.com | None | +| 2 | Uptime Kuma | Runbook ready | Yes — brief, for final DB consistency | Yes — uptimekuma.itpropartner.com | Tighten `0.0.0.0:3001` → `127.0.0.1:3001` | +| 3 | Prometheus+Grafana+node_exporter | Runbook ready (node_exporter already on app4) | Yes — brief, for volume consistency | None (no Caddy route today) | See Service 4 | +| 4 | Grafana public-bind finding | Fix defined | N/A (fix applied pre-cutover) | Optional — grafana.itpropartner.com if exposed | Loopback bind + Caddy basic_auth + change admin/admin | +| 5 | PRY | Runbook ready | None (stateless) | Yes — pry.iamgmb.com | Upgrade HTTP-only → real TLS | +| 6 | OSINT Person MCP + Super Search MCP | Runbook ready, **scope conflict flagged** | None (stateless) | None (loopback/bridge-only, MCP-internal) | Fix `0.0.0.0:8899` → `127.0.0.1:8899`; resolve Core-Hermes-vs-app4-MCP latency question first | + +**Open items for the requester before executing Services 3/6:** (a) Service 3's Prometheus scrape config needs +a decision on where Super Search ends up before it can be finalized; (b) Service 6 conflicts with the source +inventory's own classification of Super Search as "stays-on-Core" — confirm intent before executing. + diff --git a/docs/infrastructure/migration-runbook-wave2-customer-2026-09-15.md b/docs/infrastructure/migration-runbook-wave2-customer-2026-09-15.md new file mode 100644 index 0000000..8d0c9d4 --- /dev/null +++ b/docs/infrastructure/migration-runbook-wave2-customer-2026-09-15.md @@ -0,0 +1,681 @@ +# Migration Runbook — WAVE 2: Customer-Facing Services (Core → app4) + +Date: 2026-09-15 +Scope owner: this document covers the customer-facing service set ONLY. A sibling +WAVE 1 document (internal/Hermes-adjacent services — dre-mcp, ft360-mcp, twilio-mcp +as Hermes MCP tools, osint-*, pry, super-search, monitoring stack, etc.) is owned by +another team and is out of scope here except where a service is genuinely +customer-facing and also happens to be Hermes-tool-shaped (see DRE MCP / FT360 MCP / +Twilio MCP notes below — included per this task's explicit service list). + +Authoritative inventory: `core-service-inventory-2026-09-15.md` (32KB, read in full +before writing this runbook). Treat any conflict between that inventory and older +planning docs (`app4-migration-plan.md`, which undercounted the service list by +~25 services) in the inventory's favor. + +## Hosts + +| Host | Role | IP | Spec | SSH | +|---|---|---|---|---| +| Core | source, production, DO NOT MODIFY | 152.53.192.33 | netcup RS 2000, 8vCPU/15GB/503GB, Debian 13 | `ssh -i /root/.ssh/itpp-infra root@152.53.192.33` | +| app4 | destination | 159.195.205.80 | netcup RS 4000 G12, 12vCPU/32GB/1TB, Nuremberg, Debian 13 | `ssh -i /root/.ssh/itpp-infra root@159.195.205.80` | + +Verified live on app4 at runbook-authoring time (2026-09-15, read-only checks): +- Docker 29.8.0, Compose v5.5.1, Caddy v2.11.4 (pinned to Core's Caddy version) — confirmed running. +- Caddyfile global block: `default_bind 159.195.205.80`, `email info@itpropartner.com`. +- Exactly ONE service already migrated and live: **microbin** (`share.itpropartner.com`, container `microbin`, image pinned by digest `danielszabo99/microbin:latest@sha256:6660e5ccad0d764fa3c0032464ffb8f4b4f28c92a2eb9e39202b94cdc5b68909`, compose at `/root/docker/microbin/docker-compose.yml`). This is the proven pattern this runbook replicates for every other service. +- UFW: 22/80/443 open to Anywhere, 41641/udp (tailscale), plus two narrow Prometheus scrape allows from Core's IP and the tailnet CIDR. No other inbound holes exist yet. +- `docker ps -a` on app4 shows only the microbin container — every other service in this runbook is a fresh move, not a resume. + +## KNOWN BACKUP GAP — read before moving anything + +`app4:/root/root-essentials-backup.sh` backs up `.hermes/*`, `.ssh`, `.aws`, +`.config/himalaya`, `shark-game`, and `projects` under `/root` (verified by reading +the script's `ARGS` loop on app4, 2026-09-15) plus `/etc/caddy/Caddyfile` if present. +**It does NOT include `/root/docker` or `/opt`.** Since the proven pattern +(microbin) and most services in this runbook land in `/root/docker//` or +keep their existing `/opt/` tree, every migrated compose stack and every +`/opt`-based systemd service on app4 is currently **outside backup coverage** the +moment it lands, until this gap is closed. This is a pre-existing defect, not +introduced by this runbook — flagged here so it is not silently carried forward. +Recommendation (not executed by this runbook — read-only constraint): extend +`root-essentials-backup.sh`'s `ARGS` loop on app4 to include `docker` and the +specific `/opt/` paths, or add a dedicated `app4-services-backup.sh` +mirroring `core-services-backup.sh`'s pattern, before or immediately after Group A +completes. Track as a blocking action before Group C (FINAL) is even rehearsed. + +## Grouping logic + +Services are grouped by **risk = blast radius of a bad move**, which in practice +here means: does the service hold a live database with real customer/business data, +and/or process payments, and/or is it multi-tenant infrastructure other services +depend on (Postgres/Redis)? Group A has zero or trivial state. Group B has a +SQLite DB that is important but low-consequence if a short window of writes is +lost during a rsync-based cutover. **Group C (FINAL) is not moved until a full +rehearsed restore has been proven** for every service in it — this includes +HotNow (live Postgres + Redis + payment-adjacent customer records), VerdictTank +(users.db, subscription/payment logic per the `.bak-*-subscription-build` file +found on Core), Auth API (SSO — breaking it breaks login for every other +migrated app), and all three DocuSeal instances (signed legal documents). + +| Group | Services | Why | +|---|---|---| +| A — No state / static-adjacent | Transitpin, Outlook upload, Twilio MCP, FT360 MCP, DRE MCP | No DB, or trivial re-creatable state; stdio/websocket-only or tiny stateless HTTP | +| B — SQLite, single-tenant, no payments | DigLocate, IntelSight, Pipeline API, PTA registration, PTA survey, Rally, SeeMyTrip, Shark Game, Shopping Cart, DRE Portal | Real customer data in SQLite but no live payment flow and no shared DB engine dependency | +| C — FINAL (rehearsed restore required first) | HotNow API + Postgres + Redis DB1, VerdictTank API/worker + users.db, Auth API (SSO), DocuSeal x3 (docuseal, docuseal-dre, docuseal-modelortho), Voice Agent + STT | Live DB engine dependency (Postgres/Redis), SSO blast radius, signed legal documents, or subscription/payment-adjacent data | + +Move order within a group follows the dependency chain noted per-service (e.g. +Shopping Cart's Caddy route also serves Transitpin's path, so both move together +even though Transitpin is Group A). + +## Generic pre-flight (run once, before Group A, applies to every service below) + +``` +# On app4 — confirm the proven baseline is intact before adding anything +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'docker ps -a; caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile; ufw status' +# Expect: microbin container Up, "Valid configuration", ufw active with 22/80/443 open. + +# On Core — snapshot the Caddyfile and systemd unit list so rollback has a known-good diff base +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'md5sum /etc/caddy/Caddyfile; systemctl is-active .service' +``` + +Generic rollback primitive used by every service in this doc: **Core is never +touched** (read-only constraint), so rollback is always "stop routing to app4" — +either (a) for services with no DNS dependency, delete/disable the new Caddy +block on app4 and the old Core route keeps serving unaffected, or (b) for +services that required a DNS flip, revert the A/CNAME record at the registrar +back to Core's IP (152.53.192.33) — Core's Caddy block and systemd unit were +never stopped, so it resumes serving immediately once DNS re-propagates. + +--- + +## GROUP B — SQLite, single-tenant, no live payment flow + +(Execution order: after Group A below. Presented first on the page purely +because it was authored first; see risk table earlier in this doc for the +authoritative A→B→C move order.) + +Common pattern for every service below unless noted: FastAPI/uvicorn on +`127.0.0.1:`, SQLite DB file under its own data dir, systemd unit with +`User=root`, own Python venv. Generic move recipe (repeated per-service with +exact paths/ports): + +``` +# 1. Pre-flight on Core: confirm service active, note DB file, check for WAL/SHM files (SQLite in-flight writes) +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "systemctl is-active .service; ls -la /*.db*" +# 2. Quiesce-free rsync (read-only, does not stop Core): copy code+venv+db in one pass, then a final delta pass right before final cutover +rsync -avz --exclude='__pycache__' -e 'ssh -i /root/.ssh/itpp-infra' root@152.53.192.33:/ root@159.195.205.80:/ +# 3. Recreate systemd unit on app4 (identical ExecStart/WorkingDirectory/EnvironmentFile/User) +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl daemon-reload && systemctl enable --now .service" +# 4. Add Caddy site block to app4 Caddyfile mirroring Core's, validate, reload +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile && systemctl reload caddy" +# 5. Verify locally then via --resolve (pre-DNS-flip proof) +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:/" +curl --resolve :443:159.195.205.80 -s -o /dev/null -w '%{http_code}\n' https:/// +``` +Because each service's live SQLite file keeps accepting writes on Core until the +DNS flip actually happens, do the rsync **twice**: once early (bulk copy, service +still fully live on Core) and once in the final minutes before the DNS handoff +(delta-only, catches writes since the bulk copy) — this keeps the write-loss +window to seconds instead of the full rsync duration. All Group B services have +this same caveat; not repeated per-service below. + +**Rollback for all of Group B**: `systemctl disable --now .service` on +app4, remove its Caddy block, `systemctl reload caddy`. If DNS was already +flipped, revert the DNS record to 152.53.192.33 (human handoff for +itpropartner.com/iamgmb.com/intelsight.io per registrar — see per-service DNS +line). Core's unit and DB file were never touched or stopped, so it is +authoritative again the instant traffic returns. + +### B1. DigLocate + +- **Current**: Core, `diglocate-api.service`, `uvicorn main:app` at `127.0.0.1:8000`, `WorkingDirectory=/root/projects/diglocate/backend`, venv at `.venv` (note: `.venv` not `venv` — different from most others). +- **Caddy route**: `dig.iamgmb.com` → `127.0.0.1:8000` + static. +- **Data dir**: `/root/projects/diglocate` (118M). +- **DB**: none found in shallow scan (inventory ambiguity #9) — verify before move whether a hidden SQLite/Postgres dependency exists (`grep -ri 'sqlite\|DATABASE_URL\|psycopg' /root/projects/diglocate/backend/*.py`). +- **Cert**: real ACME cert for dig.iamgmb.com. +- **Backup**: **NONE — zero coverage, flagged in inventory**. +- **DNS**: iamgmb.com — host/registrar unconfirmed; treat as human handoff until confirmed automatable. + +**Pre-flight**: `ssh core "systemctl is-active diglocate-api.service; grep -ri 'sqlite\|DATABASE_URL' /root/projects/diglocate/backend/*.py"`. Expect `active`; if a DB reference appears, add it to the rsync path list before moving. + +**Verify**: `curl -s http://127.0.0.1:8000/docs -o /dev/null -w '%{http_code}\n'` on app4 → expect 200 (FastAPI auto-docs) or whatever Core returns for the same path (compare both first: `curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8000/docs` on Core). + +### B2. IntelSight API + +- **Current**: Core, `intelsight-api.service`, `python server.py` (not uvicorn-launched, raw script) at `127.0.0.1:8099`, `WorkingDirectory=/root/intelsight-api`. +- **Caddy route**: `my.intelsight.io` → `127.0.0.1:8099` + static; `intelsight.io` and `intelsight.iamgmb.com` are static landing pages only (no backend, can move as plain static rsync whenever convenient, not blocking). +- **Data dir**: `/root/intelsight-api` (64M) incl. `intelsight.db`. +- **DB**: SQLite `intelsight.db`. +- **Cert**: real ACME cert for my.intelsight.io. +- **Backup**: **NONE**. +- **DNS**: intelsight.io — registrar unconfirmed (inventory calls this out as unchecked); human handoff assumed until confirmed otherwise. + +**Verify**: `curl -s http://127.0.0.1:8099/ -o /dev/null -w '%{http_code}\n'` on app4, compare to same on Core. + +### B3. Pipeline API + +- **Current**: Core, `pipeline-api.service`, `python server.py` at `127.0.0.1:8200`, `WorkingDirectory=/root/projects/pipeline`. +- **Caddy route**: `my.itpropartner.com` block includes `127.0.0.1:8200` (multi-target block, also proxies to App3 remotely for a different path — read the live block carefully before recreating on app4, do not copy App3's remote proxy line since that traffic doesn't belong to this migration). +- **Port conflict note**: inventory flags `hermes-control-deck.service` (disabled) also wants port 8200. It is disabled on Core; confirm it stays disabled/is not part of Wave 1's plan for app4 before binding 8200 there, or this will silently fail to start on app4 the day someone enables it. +- **Data dir**: `/root/projects/pipeline` (30M) incl. `pipeline.db`. +- **DB**: SQLite `pipeline.db`. +- **Cert**: real ACME cert for my.itpropartner.com. +- **Backup**: **NONE**. +- **DNS**: my.itpropartner.com — itpropartner.com is SiteGround manual, **human DNS handoff required**. Also note this hostname's Caddy block mixes an App3 remote-proxy path; migrating Pipeline's path alone should not touch that other path's routing on app4 without also confirming with whoever owns the App3 leg of `my.itpropartner.com`. + +**Verify**: `curl -s http://127.0.0.1:8200/ -o /dev/null -w '%{http_code}\n'` on app4 vs Core. + +### B4. PTA registration (TIMAPTA Membership Registration) + +- **Current**: Core, `pta-registration.service`, `uvicorn server:app` at `127.0.0.1:8114`, `WorkingDirectory=/opt/pta-registration`. +- **Caddy route**: `register.timapta.org` (primary) and `pta.iamgmb.com` (duplicate route, same backend) → `127.0.0.1:8114`. +- **Data dir**: `/opt/pta-registration` (53M) incl. `pta.db`. +- **DB**: SQLite `pta.db`. +- **Cert**: real ACME certs for both `register.timapta.org` and `pta.iamgmb.com` (furthest-out expiry in the whole inventory, Dec 9 2026). +- **Backup**: **NONE**. +- **DNS**: `timapta.org` and `iamgmb.com` — registrars unconfirmed for both; treat both as human handoff pending confirmation. `webmail.timapta.org` is a separate self-signed redirect stub, unrelated to this service, do not touch. + +**Verify**: `curl -s http://127.0.0.1:8114/ -o /dev/null -w '%{http_code}\n'` on app4 vs Core; confirm BOTH hostnames' Caddy blocks were recreated (duplicate route is easy to miss). + +### B5. PTA survey (TIMA Location Survey) + +- **Current**: Core, `survey-registration.service`, `uvicorn server:app` at `127.0.0.1:8115`, `WorkingDirectory=/opt/pta-survey`. +- **Caddy route**: `survey.iamgmb.com` → `127.0.0.1:8115`. +- **Data dir**: `/opt/pta-survey` (50M) incl. `survey.db`. +- **DB**: SQLite `survey.db`. +- **Cert**: real ACME cert for survey.iamgmb.com. +- **Backup**: **NONE**. +- **DNS**: iamgmb.com — human handoff pending registrar confirmation. + +**Verify**: `curl -s http://127.0.0.1:8115/ -o /dev/null -w '%{http_code}\n'` on app4 vs Core. + +### B6. Rally (Family Calendar) + +- **Current**: Core, `rally.service`, `python run.py` at `127.0.0.1:8105`, `WorkingDirectory=/opt/rally/backend`. +- **Caddy route**: `rally.iamgmb.com` → `127.0.0.1:8105` + static. +- **Data dir**: `/opt/rally/data/rally.db` (dir 294M — the single largest SQLite-backed dataset in Group B; budget extra rsync time). +- **DB**: SQLite `rally.db`. +- **Cert**: real ACME cert for rally.iamgmb.com. +- **Backup**: none matching this exact DB path found — inventory notes only debug/dump scripts exist, not a scheduled backup job. Treat as **effectively unbacked**. +- **DNS**: iamgmb.com — human handoff pending registrar confirmation. + +**Pre-flight**: given the 294M size, run the bulk rsync well ahead of cutover and confirm free disk on app4 first: `ssh app4 "df -h /opt"`. + +**Verify**: `curl -s http://127.0.0.1:8105/ -o /dev/null -w '%{http_code}\n'` on app4 vs Core. + +### B7. SeeMyTrip (media pipeline) + +- **Current**: Core, `seemytrip.service`, `uvicorn server:app` at `127.0.0.1:8113`, `WorkingDirectory=/opt/seemytrip`. +- **Caddy route**: `seemytrip.iamgmb.com` → `127.0.0.1:8113` + static. +- **Data dir**: `/opt/seemytrip/data` (within a 212M dir — media files, budget rsync time). +- **DB**: SQLite `seemytrip.db`. +- **Cert**: real ACME cert for seemytrip.iamgmb.com. +- **Backup**: **NONE**. +- **DNS**: iamgmb.com — human handoff pending registrar confirmation. + +**Verify**: `curl -s http://127.0.0.1:8113/ -o /dev/null -w '%{http_code}\n'` on app4 vs Core. + +### B8. Shark Attack Fantasy Game + +- **Current**: Core, `shark-game.service`, `python3 server.py` at `0.0.0.0:8083` (note: binds all interfaces, not 127.0.0.1 — unlike every other Group B service; confirm nothing external depends on direct :8083 access bypassing Caddy before assuming a straight rebind is safe). +- **Caddy route**: `shark.iamgmb.com` → `127.0.0.1:8083`. +- **Data dir**: `/root/shark-game/backend/game.db` (dir 127M). +- **DB**: SQLite `game.db`. +- **Cert**: real ACME cert for shark.iamgmb.com. +- **Backup**: **NONE**. +- **Cron dependency**: root crontab on Core runs a daily shark-game scraper and a 15-min shark-draft-reminder job — these are cron entries, not systemd, and are Core-scoped. If Shark Game moves to app4, these two cron jobs must also move (or be re-pointed) or the game data will stop updating even though the web app itself works. Flag for the crontab owner; not fixed by this runbook (read-only constraint, cron untouched by this task). +- **DNS**: iamgmb.com — human handoff pending registrar confirmation. + +**Pre-flight**: `ssh core "crontab -l | grep -i shark"` — capture the exact two cron lines to hand off alongside the service move. + +**Verify**: `curl -s http://127.0.0.1:8083/ -o /dev/null -w '%{http_code}\n'` on app4 vs Core. + +### B9. Shopping Cart Builder + +- **Current**: Core, `shopping-cart.service`, `uvicorn app:app` at `127.0.0.1:8101`, `WorkingDirectory=/opt/shopping-cart`. +- **Caddy route**: `shopping.iamgmb.com` → `127.0.0.1:8101` **and** `127.0.0.1:8210` (Transitpin, see A1) + static — same block serves both services, move them together in the same maintenance window even though they're in different risk groups, to avoid a half-migrated Caddy block. +- **Data dir**: `/opt/shopping-cart` (147M). +- **DB**: none found in shallow scan — verify (`grep -ri 'sqlite\|DATABASE_URL' /opt/shopping-cart/*.py`) before assuming stateless; "Shopping Cart" strongly suggests order/cart state exists somewhere (in-memory? session file? confirm before move). +- **Cert**: real ACME cert for shopping.iamgmb.com. +- **Backup**: **NONE**. +- **DNS**: iamgmb.com — human handoff pending registrar confirmation. + +**Pre-flight**: resolve the DB ambiguity above before treating this as a simple stateless move; if it does process real payment/cart data, escalate this service into Group C instead of moving it here. + +**Verify**: `curl -s http://127.0.0.1:8101/ -o /dev/null -w '%{http_code}\n'` on app4 vs Core. + +### B10. DRE Portal + +- **Current**: Core, `dre-portal.service`, `uvicorn app.main:app` at `127.0.0.1:8093`, `WorkingDirectory=/opt/dre-portal`, `EnvironmentFile=/opt/dre-portal/.env` (contains live secrets per inventory item 7 — copy manually and securely, never via generic backup/rsync of secrets in cleartext logs). +- **Caddy route**: `internal.debtrecoveryexperts.com` (password-protected via Caddy `basic_auth`, confirmed live: `ippadmin` user with a bcrypt hash in the Caddyfile — carry this basic_auth block over verbatim, do not regenerate the hash) and `my.debtrecoveryexperts.com` → both `127.0.0.1:8093` + static. `portal.debtrecoveryexperts.com` is a redirect-only stub to `my.debtrecoveryexperts.com/start`, no backend of its own. +- **Data dir**: `/opt/dre-portal/data/dre.db` (dir 110M). +- **DB**: SQLite `dre.db` — real DRE customer case data. +- **Cert**: real ACME certs for both hostnames. +- **Backup**: **NONE** (flagged in inventory). +- **DNS**: debtrecoveryexperts.com — registrar unconfirmed; human handoff pending confirmation (not itpropartner.com/SiteGround, verify separately — inventory does not state this domain's registrar). + +**Pre-flight**: `ssh core "cat /opt/dre-portal/.env | sed 's/=.*/=REDACTED/'"` to confirm the secret keys needed (do not print values), then manually transfer the real `.env` over `scp` with the ssh key (never through a generic backup path) before starting the service on app4. + +**Verify**: `curl -s -u ippadmin: http://127.0.0.1:8093/ -o /dev/null -w '%{http_code}\n'` on app4 (basic_auth is enforced even on localhost per the Caddy config — confirm whether the app itself also enforces it or only Caddy does) vs same on Core. + +**Note on grouping**: DRE Portal holds real customer case data but not a live-write shared-DB-engine dependency (it's SQLite, single service). Kept in Group B rather than Group C FINAL because it has no payment processing and no cross-service DB engine risk — but it is the highest-sensitivity item in Group B given `internal.debtrecoveryexperts.com`'s basic_auth gate implies internal/sensitive use. If Germaine's risk tolerance differs, this is the first B-item to consider promoting to Group C. + +--- + +## GROUP A — No state / static-adjacent + +(Presented after Group B textually in this file; grouping/risk-order is per the +table in the Grouping-logic section above, not file position. Move order for +execution is: Group A first, then Group B, then Group C FINAL — regardless of +which appears first on the page.) + +### A1. Transitpin (WebSocket relay) + +- **Current**: Core, `transitpin.service` (Node, `/usr/local/bin/node /opt/transitpin/server.js`), binds `127.0.0.1:8210`. +- **Caddy route**: `status.itpropartner.com` block also proxies `/` traffic partly to `127.0.0.1:8210` (shared with Uptime Kuma's 3001 in the same block — already-migrated Uptime Kuma is out of scope here, but this Caddy block is multi-target); `shopping.iamgmb.com` also proxies to `127.0.0.1:8210` for a sub-path. **No dedicated Transitpin-only hostname** — it rides inside two other sites' blocks. Confirm exact path routing by reading the live block before editing (`grep -A15 'status.itpropartner.com {' /etc/caddy/Caddyfile` and same for `shopping.iamgmb.com`). +- **Data dir**: `/opt/transitpin` (496K, no DB — in-memory relay state only, safe to lose in-flight sessions on cutover). +- **DB**: none. +- **Cert**: `status.itpropartner.com` and `shopping.iamgmb.com` both have real ACME certs on Core. +- **Backup status**: not separately backed up; irrelevant, no persistent data. +- **DNS**: `status.itpropartner.com` and `shopping.iamgmb.com` are itpropartner.com / iamgmb.com — confirm iamgmb.com's DNS host (inventory flags this as unchecked — `dig NS iamgmb.com` before cutover). itpropartner.com is SiteGround manual (human handoff required). + +**Pre-flight** +``` +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'systemctl is-active transitpin.service; ss -ltnp | grep 8210; node --version' +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'node --version || echo NEED_NODE_INSTALL' +``` +Expect: `active`, socket on 127.0.0.1:8210, Node version printed. If app4 lacks Node, install the matching major version before moving (`apt-cache policy nodejs` on both, match versions). + +**Move** +``` +rsync -avz -e 'ssh -i /root/.ssh/itpp-infra' root@152.53.192.33:/opt/transitpin/ root@159.195.205.80:/opt/transitpin/ +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'cd /opt/transitpin && npm ci --omit=dev 2>&1 | tail -20' +# Create a systemd unit on app4 mirroring Core's exactly (User=root, WorkingDirectory=/opt/transitpin, ExecStart=/usr/local/bin/node /opt/transitpin/server.js, Restart=always) +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl daemon-reload && systemctl enable --now transitpin.service' +``` +Add the equivalent proxy path to app4's Caddyfile inside the (recreated) `status.itpropartner.com` and `shopping.iamgmb.com` blocks, then: +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile && systemctl reload caddy' +``` + +**Verify** +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl is-active transitpin.service; curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8210/' +curl --resolve status.itpropartner.com:443:159.195.205.80 -s -o /dev/null -w "%{http_code}\n" https://status.itpropartner.com/ +``` +Expect: `active`, a 2xx/101 (websocket upgrade) locally; the `--resolve` curl returns the same response app4 would give once DNS points there (does not require DNS to already be flipped). + +**Rollback**: `systemctl disable --now transitpin.service` on app4, remove the added Caddy stanza, `systemctl reload caddy`. Core was never touched, still serving. + +**DNS**: needs human DNS handoff for both hostnames at cutover (itpropartner.com = SiteGround manual, no API; iamgmb.com host TBD — confirm before assuming automatable). + +--- + +### A2. Outlook upload receiver + +- **Current**: Core, `outlook-upload.service` (`python3 /root/upload-staging/upload_server.py`), `127.0.0.1:8240`, no `EnvironmentFile`. +- **Caddy route**: part of the `core.itpropartner.com` multi-path block (proxies `127.0.0.1:8240` alongside VerdictTank's 8201 and static content) — **not** its own hostname. +- **Data dir**: `/root/upload-staging` (32K: `index.html` + `upload_server.py` + service file copy). Uploaded files land in a staging subpath not captured in the 32K figure at scan time (transient — verify emptiness before move: `ls /root/upload-staging/*/`). +- **DB**: none. +- **Cert**: covered under `core.itpropartner.com`'s existing cert. +- **Backup status**: none (flagged as zero-backup-coverage service in the inventory). +- **DNS**: `core.itpropartner.com` — **this hostname is explicitly called out in the migration plan as staying on Core** ("Core keeps ... core.itpropartner.com"). Moving this service means either (a) leaving it on Core (do nothing — recommended, contradicts nothing in scope) or (b) if genuinely required to move, it needs a NEW path/hostname on app4 since its current route is tied to a host that stays put. **Recommendation: do not move under core.itpropartner.com; if migration is required, cut a new subpath under an app4-hosted domain and update whatever client (Outlook add-in?) posts to it.** Flagging this as a decision point, not silently moving it. + +**Pre-flight** +``` +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'systemctl is-active outlook-upload.service; find /root/upload-staging -mindepth 1 -maxdepth 1' +``` +Expect: `active`; confirm no in-flight large uploads before any move. + +**Move** (only if the domain decision above is resolved in favor of moving): +``` +rsync -avz -e 'ssh -i /root/.ssh/itpp-infra' root@152.53.192.33:/root/upload-staging/ root@159.195.205.80:/root/upload-staging/ +# recreate outlook-upload.service on app4 identically, pick new hostname/path per decision above +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl daemon-reload && systemctl enable --now outlook-upload.service' +``` + +**Verify**: `curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8240/` on app4 → expect 200/404 (whatever Core returns identically — compare both). + +**Rollback**: `systemctl disable --now outlook-upload.service` on app4; nothing else changes since Core's route was never edited. + +**DNS**: no DNS change needed if left on Core (recommended). If moved, needs a **new** hostname decision + human DNS handoff (itpropartner.com is SiteGround manual) or reuse of an existing app4 hostname path. + +--- + +### A3. Twilio MCP, A4. FT360 MCP, A5. DRE MCP (Hermes MCP servers, included per this task's explicit scope) + +These three are structurally Hermes MCP tool servers (stdio, `EnvironmentFile=-/root/.hermes/.env`, no listening TCP port, no Caddy route) but are listed in this task's explicit customer-facing set because their *data* is customer-facing (DRE case data, FleetTracker360 device data, Twilio call/SMS logs). They are grouped here in A because they carry no independent database of their own beyond what upstream APIs (Twilio, FT360, DRE's own DB) hold — moving them changes nothing customer-visible, only which box runs Hermes's tool process. + +| | Twilio MCP | FT360 MCP | DRE MCP | +|---|---|---|---| +| unit | twilio-mcp.service | ft360-mcp.service | dre-mcp.service | +| exec | `/root/docker/twilio-mcp/venv/bin/python3 server.py` | `/opt/ops-portal/venv/bin/python3 /root/docker/ft360-mcp/server.py` (note: **borrows ops-portal's venv**, not its own — verify this venv exists identically on app4 or rebuild ft360-mcp's own venv) | `/root/docker/dre-mcp/venv/bin/python3 server.py` | +| working dir | `/root/docker/twilio-mcp` | `/root/docker/ft360-mcp` | `/root/docker/dre-mcp` | +| env | `/root/.hermes/.env` (shared Hermes secrets file — contains Twilio SIDs, DRE/FT360 keys per inventory item 7) | `/root/.hermes/.env` | `/root/.hermes/.env` | +| data dir size | 171M | 60K | 171M | +| DB | none found | none found | none found (DRE's actual case DB lives in dre-portal, Group B) | +| Caddy route | none — stdio MCP | none | none | +| Backup | none (not customer-DB-bearing) | none | none | + +**IMPORTANT — this is genuinely a Hermes-config change, not just an app move.** +These three run as tools invoked by the Hermes gateway on Core. Moving the +*process* to app4 without also repointing Hermes's MCP server registration would +either break the tool (Hermes on Core can no longer exec a binary on app4) or +require converting them to network-reachable MCP servers (a design change, not a +lift-and-shift). **This runbook does NOT recommend moving these three under +Wave 2** unless the sibling Hermes/Wave-1 team confirms Hermes itself (and its +`.hermes/.env`) is also moving or being made to reach app4 over the network. +Flagging as **needs-decision, coordinate with Wave 1 owner before executing**; +no move commands are given here to avoid prescribing an action that could break +live Hermes tool calls. If Wave 1 confirms these should move, treat identically +to Group A pattern: rsync the `/root/docker/` dir, recreate the systemd +unit with an `EnvironmentFile=-/root/.hermes/.env` that exists identically on +app4 (requires the secrets file to be securely copied — inventory item 7 says do +this manually, never via generic backup), verify with a manual JSON-RPC stdio +ping, rollback = `systemctl disable --now` + re-point Hermes's tool registration +back to Core. + +**DNS**: none of the three have a Caddy hostname — no DNS involvement either way. + +--- + +## GROUP C — FINAL (do not move until a full rehearsed restore has been proven) + +**Gate condition for this entire group**: no service below moves until at least +one full rehearsed restore-from-backup has been executed and verified on app4 +(or a throwaway test box) for that service's specific data (Postgres dump+restore +for HotNow, SQLite file restore for VerdictTank/Auth/DocuSeal). "Rehearsed" means +an actual restore was performed and the restored data was queried/compared, not +that a backup file merely exists. Given the inventory's finding that HotNow, +VerdictTank's users.db, and DocuSeal all currently DO have some backup coverage +in principle (DocuSeal via docuseal-backup cron jobs, Auth API via +auth-api-backup.sh) but HotNow's Postgres DB has **zero** backup coverage today, +**HotNow cannot be rehearsed until a backup mechanism is built for it first** — +this is the single hardest blocker in the whole Wave 2 scope and should be +treated as its own pre-work item, not squeezed into the migration window. + +### C1. Auth API (SSO) + +- **Current**: Core, `auth-api.service`, `python3 -c "import uvicorn; uvicorn.run('server:app', host='127.0.0.1', port=8500, proxy_headers=True)"`, `WorkingDirectory=/root/projects/auth`, `EnvironmentFile=/root/projects/auth/.env`. +- **Caddy route**: `auth.itpropartner.com` → `127.0.0.1:8500` + static. +- **Data dir**: `/root/projects/auth` (32M) incl. `auth.db`. +- **DB**: SQLite `auth.db` — **every other migrated service that relies on SSO breaks if this move goes wrong**, making it the highest blast-radius item in Group C despite being "just SQLite." +- **Cert**: real ACME cert for auth.itpropartner.com. +- **Backup**: **covered** — `auth-api-backup.sh` exists and is confirmed to run (per inventory backup-coverage summary), though it explicitly redacts secret values from `.env` in its backup payload (inventory item 7) — the `.env` itself must be copied manually/securely, not restored from this backup. +- **DNS**: auth.itpropartner.com — SiteGround manual, **human DNS handoff required**. + +**Pre-flight (rehearsal gate)** +``` +# Locate and inspect the existing backup mechanism (read-only) +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'crontab -l | grep -i auth-api; find / -iname "auth-api-backup.sh" 2>/dev/null' +# Perform a rehearsed restore on a scratch path on app4 (NOT into the live auth.db path yet) +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'mkdir -p /root/restore-test/auth && cd /root/restore-test/auth' +# Pull the latest backup archive from Wasabi S3 (same bucket/creds pattern as root-essentials-backup.sh) and restore into the scratch dir, then diff row counts against Core's live auth.db: +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'sqlite3 /root/projects/auth/auth.db "SELECT count(*) FROM sqlite_master; SELECT count(*) FROM users;" 2>&1 || echo "confirm actual table name first"' +``` +Only proceed to the real move once the restored scratch copy's row counts match Core's live counts. + +**Move** (after rehearsal gate passes) +``` +rsync -avz -e 'ssh -i /root/.ssh/itpp-infra' root@152.53.192.33:/root/projects/auth/ root@159.195.205.80:/root/projects/auth/ +# manually scp the real .env (contains live secrets, do not rely on the redacted backup copy) +scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/projects/auth/.env root@159.195.205.80:/root/projects/auth/.env +# recreate auth-api.service identically on app4 +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl daemon-reload && systemctl enable --now auth-api.service' +``` + +**Verify** +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl is-active auth-api.service; curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8500/' +# Functional check: exercise an actual login/token-issue flow end-to-end against app4 before flipping DNS, using --resolve +curl --resolve auth.itpropartner.com:443:159.195.205.80 -s -o /dev/null -w '%{http_code}\n' https://auth.itpropartner.com/ +``` +Expect matching HTTP codes to Core, and a successful token issuance in the functional check — SSO is the one service in this runbook where "the port answers" is not sufficient verification; a real auth round-trip must succeed before this is called done. + +**Rollback**: `systemctl disable --now auth-api.service` on app4. If DNS was flipped, revert `auth.itpropartner.com` to 152.53.192.33 immediately — every dependent app's login breaks for as long as this is down, so this rollback should be treated as sev-1, not routine. + +### C2. VerdictTank API + worker + users.db + +- **Current**: Core, `verdicttank-api.service` (`python3 api.py`) and `verdicttank-worker.service` (`python3 worker.py`), both `WorkingDirectory=/opt/verdicttank`, both `EnvironmentFile=/etc/verdicttank.env` (note: env file lives outside the app dir, in `/etc` — easy to miss in a straight directory rsync). API listens `127.0.0.1:8201`; worker has no listening port (background review-panel processor). +- **Caddy route**: `verdicttank.com` / `www.verdicttank.com` → `127.0.0.1:8201` + static; `ops.verdicttank.com` → same backend; also referenced from `core.itpropartner.com`'s multi-path block (shares that block with Outlook upload, see A2 — confirm core.itpropartner.com's VerdictTank path is intentionally kept on Core or also needs updating). +- **Data dir**: `/opt/verdicttank` (15M) incl. `users.db` (28K live file, confirmed via direct listing) plus multiple `.bak-*` snapshots already sitting next to it from prior manual migrations (`users.db.bak-20260819-124859`, `users.db.bak-pre-migration-20260819-131034`) — evidence the team has manually snapshotted this DB before major changes previously; do the same here (`cp users.db users.db.bak-pre-app4-migration-$(date +%Y%m%d)` on Core, read-only, before the final rsync). +- **DB**: SQLite `users.db` — contains subscription/account data (file naming `api.py.bak-*-subscription-build` strongly implies live subscription logic; treat as payment-adjacent even though no separate payments DB was found). +- **Cert**: real ACME certs for verdicttank.com, www.verdicttank.com, ops.verdicttank.com. +- **Backup**: VerdictTank has `hello-*-collect.py` cron jobs but **no actual DB backup job** for `users.db` — treat as **effectively unbacked** despite the manual `.bak-*` snapshots being evidence of good practice, not a scheduled/automatic job. +- **DNS**: verdicttank.com — registrar unconfirmed; human handoff pending confirmation. + +**Pre-flight (rehearsal gate)** +``` +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'sqlite3 /opt/verdicttank/users.db ".tables"; sqlite3 /opt/verdicttank/users.db "SELECT count(*) FROM users;" 2>&1' +# Take a fresh manual snapshot (read-only cp, not a move) +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'cp /opt/verdicttank/users.db /opt/verdicttank/users.db.bak-pre-app4-migration-$(date +%Y%m%d)' +# Rehearse: copy that snapshot to app4 scratch path, open it, confirm row counts match +rsync -avz -e 'ssh -i /root/.ssh/itpp-infra' root@152.53.192.33:/opt/verdicttank/users.db.bak-pre-app4-migration-* root@159.195.205.80:/root/restore-test/verdicttank/ +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'sqlite3 /root/restore-test/verdicttank/users.db.bak-* "SELECT count(*) FROM users;"' +``` +Only proceed once counts match exactly (SQLite file copies while idle are exact — if Core's live file is being written during the copy, re-run and use `.timeout`/WAL-safe copy, e.g. `sqlite3 users.db ".backup /tmp/users_backup.db"` instead of `cp` for a live DB). + +**Move** +``` +rsync -avz -e 'ssh -i /root/.ssh/itpp-infra' root@152.53.192.33:/opt/verdicttank/ root@159.195.205.80:/opt/verdicttank/ +scp -i /root/.ssh/itpp-infra root@152.53.192.33:/etc/verdicttank.env root@159.195.205.80:/etc/verdicttank.env +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl daemon-reload && systemctl enable --now verdicttank-api.service verdicttank-worker.service' +``` + +**Verify** +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl is-active verdicttank-api.service verdicttank-worker.service; curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8201/' +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'sqlite3 /opt/verdicttank/users.db "SELECT count(*) FROM users;"' +# Compare against Core's live count taken at the same moment +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'sqlite3 /opt/verdicttank/users.db "SELECT count(*) FROM users;"' +``` +Expect both `active`, matching HTTP code, and row counts within the expected drift window (writes between final rsync and cutover) — any large unexplained discrepancy is a stop-ship signal. + +**Rollback**: `systemctl disable --now verdicttank-api.service verdicttank-worker.service` on app4. Revert DNS to 152.53.192.33 if flipped. Core's `users.db` was never modified by this runbook (only read/copied), so it remains authoritative. + +### C3–C5. DocuSeal x3 (docuseal, docuseal-dre, docuseal-modelortho) + +All three are Docker containers on Core, `docuseal/docuseal:latest`, same image +digest `sha256:0defc701c18cf45f9e4212acdabbf5e2dbba11c815e4c2e7565da384accf5947` +across all three (confirmed via `docker inspect`, 2026-09-15) — pin to this exact +digest on app4 per the proven microbin pattern, do not pull `:latest` fresh. + +| | docuseal | docuseal-dre | docuseal-modelortho | +|---|---|---|---| +| Core port | 127.0.0.1:8091 | 127.0.0.1:8094 | 127.0.0.1:8092 | +| Caddy hostname | sign.itpropartner.com | sign.debtrecoveryexperts.com | sign.modelortho.com | +| data dir (Core) | `/root/docker/docuseal/data` (688K) | `/root/docker/docuseal-dre/data` (2.5M) | `/root/docker/docuseal-modelortho/data` (1.4M) | +| compose file | `/root/docker/docuseal/docker-compose.yml` | `/root/docker/docuseal-dre/docker-compose.yml` | `/root/docker/docuseal-modelortho/docker-compose.yml` | +| cert | real ACME | real ACME | real ACME | +| backup | covered (docuseal-backup cron) | covered (docuseal-dre-backup cron) | covered (docuseal-modelortho-backup cron) — **this is the only Group C set with existing backup jobs for all members; still rehearse the restore before moving, do not assume the job has ever actually been test-restored** | +| DNS registrar | itpropartner.com — SiteGround manual | debtrecoveryexperts.com — unconfirmed | modelortho.com — unconfirmed | + +**Why these are signed-legal-document risk, not just "another SQLite app":** +DocuSeal stores signed contracts/agreements in its `/data` volume (SQLite + +uploaded document blobs). Losing or corrupting this during migration is a legal +liability, not just a customer-experience blip — hence FINAL group placement +even though the data volumes themselves are small (under 3MB each). + +**Pre-flight (rehearsal gate, run once per instance — shown for `docuseal`, repeat verbatim for `-dre` and `-modelortho` substituting names/ports)** +``` +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'docker ps --filter name=docuseal --format "{{.Names}}: {{.Status}}"' +# Locate and inspect the existing backup job (read-only) +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'crontab -l | grep -i docuseal; find /root/.hermes/scripts -iname "*docuseal*backup*" 2>/dev/null' +# Perform an actual rehearsed restore into a scratch dir on app4 from the latest S3 backup object, then diff file counts/sizes against Core's live ./data +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'mkdir -p /root/restore-test/docuseal' +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'find /root/docker/docuseal/data -type f | wc -l' +``` +Do not proceed to the live move for ANY of the three until this rehearsal has +actually restored a backup archive and file counts have been diffed successfully +for that instance. + +**Move (per instance, after its rehearsal gate passes)** +``` +rsync -avz -e 'ssh -i /root/.ssh/itpp-infra' root@152.53.192.33:/root/docker/docuseal/ root@159.195.205.80:/root/docker/docuseal/ +``` +On app4, edit the compose file to pin the image by digest (matching the proven +microbin pattern): +``` +image: docuseal/docuseal:latest@sha256:0defc701c18cf45f9e4212acdabbf5e2dbba11c815e4c2e7565da384accf5947 +``` +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'cd /root/docker/docuseal && docker compose up -d' +``` +Repeat identically for `docuseal-dre` (port 8094) and `docuseal-modelortho` (port 8092), each into its own `/root/docker//` dir. + +**Verify (per instance)** +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'docker ps --filter name=docuseal --format "{{.Names}}: {{.Status}}"; curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8091/' +diff <(ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'find /root/docker/docuseal/data -type f | sort') <(ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'find /root/docker/docuseal/data -type f | sort') +``` +Expect `Up`, a 2xx/3xx HTTP code, and an **empty diff** (identical file lists) — +for signed documents, a byte-for-byte file inventory match is the bar, not just +"the app responds." + +**Rollback (per instance)**: `docker compose down` on app4 (does not touch Core's +running container). Revert DNS to 152.53.192.33 if flipped. + +**DNS**: three separate domains, three separate human handoffs — sign.itpropartner.com (SiteGround manual), sign.debtrecoveryexperts.com, sign.modelortho.com (registrars for the latter two unconfirmed, do not assume automatable). + +### C6. HotNow API + its own Postgres DB + Redis DB1 — HIGHEST RISK ITEM IN WAVE 2 + +- **Current**: Core, `hotnow-api.service`, `uvicorn main:app` at `127.0.0.1:8001`, `WorkingDirectory=/root/hotnow-api`, `EnvironmentFile=/root/hotnow-api/.env` (contains live secrets, manual/secure transfer only). +- **Caddy route**: `api.hotnow.io` → `127.0.0.1:8001`; `www.hotnow.io`, `hotnow.io`, `app.hotnow.io`, `admin.hotnow.io` are static/redirect-only, no backend dependency (can move independently, low risk, not gated by this item). +- **Data dir**: `/root/hotnow-api` (66M, app code) — **the real risk is not this directory, it's the two engines below.** +- **DB #1 — Postgres**: database `hotnow` (16MB), owned by role `hotnow_app`, on Core's **host** PostgreSQL 17.10 instance (`postgresql@17-main.service`, port 5432, `127.0.0.1`/`[::1]` only). This is the **only real customer database on all of Core's host Postgres** — moving it means either (a) standing up a fresh Postgres 17 instance on app4 and doing a `pg_dump`/`pg_restore`, or (b) some form of logical replication. Given app4 doesn't appear to run Postgres yet (not seen in the app4 `docker ps -a` baseline check), option (a) is the realistic path. +- **DB #2 — Redis**: HotNow's `.env` points at `redis://localhost:6379/1` (DB1 on Core's host Redis). Redis DB1 was empty (`DBSIZE 0`) at inventory scan time but the `.env` reference confirms it's a real dependency, not dead config — do not assume it's safe to skip just because it was empty during a snapshot. +- **Cert**: `api.hotnow.io` uses a ZeroSSL cert (the only non-Let's-Encrypt cert on Core) — confirm app4's Caddy can complete ZeroSSL issuance the same way, or explicitly force Let's Encrypt on app4 instead (a full reissue is expected regardless, since certs aren't copied per this doc's established pattern). +- **Backup**: **NONE for the Postgres DB, NONE for the app code, NONE for Redis DB1** — this is explicitly the worst-covered service in the entire inventory (a live customer Postgres DB with zero backup jobs). **This is the blocking pre-work item for all of Wave 2's FINAL group**: a backup mechanism (e.g. nightly `pg_dump hotnow | gzip` to the existing Wasabi S3 bucket, mirroring `timetrex-backup.sh`'s `docker exec ... pg_dump` pattern) must be built and run successfully at least once, AND a restore from that backup must be rehearsed, before HotNow moves. +- **DNS**: hotnow.io — registrar unconfirmed; human handoff pending confirmation, separate from itpropartner.com/SiteGround. + +**Pre-flight (rehearsal gate — this is the hard gate for all of Wave 2 FINAL)** +``` +# Confirm today's real state (read-only, safe on production Postgres) +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "sudo -u postgres psql -c '\l+' | grep hotnow" +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "sudo -u postgres psql hotnow -c 'SELECT schemaname, relname, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC LIMIT 10;'" +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "redis-cli -n 1 DBSIZE" +# Confirm app4 has NO Postgres yet (do not silently assume; this determines whether a fresh install is even needed) +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "which psql; systemctl list-units | grep -i postgres; docker ps -a | grep -i postgres" +``` +Build (outside this read-only runbook's scope, coordinate with whoever owns +app4 provisioning) a Postgres 17 instance on app4, then rehearse: +``` +# On Core (read-only, standard pg_dump does not lock/modify the live DB) +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "sudo -u postgres pg_dump hotnow | gzip > /var/tmp/hotnow-rehearsal-$(date +%F).sql.gz" +scp -i /root/.ssh/itpp-infra root@152.53.192.33:/var/tmp/hotnow-rehearsal-*.sql.gz /tmp/ +scp -i /root/.ssh/itpp-infra /tmp/hotnow-rehearsal-*.sql.gz root@159.195.205.80:/var/tmp/ +# On app4, restore into a NEW test database, not the eventual production one, and compare row counts to Core's live query above +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sudo -u postgres createdb hotnow_rehearsal && gunzip -c /var/tmp/hotnow-rehearsal-*.sql.gz | sudo -u postgres psql hotnow_rehearsal" +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sudo -u postgres psql hotnow_rehearsal -c 'SELECT schemaname, relname, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC LIMIT 10;'" +``` +**Do not proceed past this point until the rehearsed restore's row counts match +Core's live counts within an explainable margin.** Clean up the rehearsal DB +afterward (`dropdb hotnow_rehearsal`) — it is not the production database. + +**Move (only after the rehearsal gate above passes AND a real backup job exists and has run at least once)** +``` +# Final dump (as close to cutover as practical) +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "sudo -u postgres pg_dump hotnow | gzip > /var/tmp/hotnow-final-$(date +%F-%H%M).sql.gz" +scp -i /root/.ssh/itpp-infra root@152.53.192.33:/var/tmp/hotnow-final-*.sql.gz /tmp/ && scp -i /root/.ssh/itpp-infra /tmp/hotnow-final-*.sql.gz root@159.195.205.80:/var/tmp/ +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sudo -u postgres createdb hotnow && sudo -u postgres createuser hotnow_app && gunzip -c /var/tmp/hotnow-final-*.sql.gz | sudo -u postgres psql hotnow" +# Redis: DB1 was empty at scan time; if non-empty at cutover time, dump/restore via redis-cli --rdb or DUMP/RESTORE per key +rsync -avz -e 'ssh -i /root/.ssh/itpp-infra' root@152.53.192.33:/root/hotnow-api/ root@159.195.205.80:/root/hotnow-api/ +scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/hotnow-api/.env root@159.195.205.80:/root/hotnow-api/.env +# update .env's DATABASE_URL/REDIS_URL to point at app4's local Postgres/Redis instances before starting +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl daemon-reload && systemctl enable --now hotnow-api.service' +``` + +**Verify** +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl is-active hotnow-api.service; curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8001/' +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sudo -u postgres psql hotnow -c 'SELECT count(*) FROM pg_stat_user_tables;'" +``` +Expect `active`, matching HTTP code vs Core, and table/row counts matching the +final dump's source counts. + +**Rollback**: `systemctl disable --now hotnow-api.service` on app4 (Core's +`hotnow-api.service` and its Postgres/Redis were never touched — Core resumes +serving immediately). If DNS was flipped, revert `api.hotnow.io` to +152.53.192.33. The app4-side Postgres `hotnow` database and any partial data +written post-cutover should be preserved (not dropped) until root-caused, in +case a forward-fix is preferred over a full rollback. + +### C7. Voice Agent + STT + +- **Current**: Core, `voice-agent.service` (`uvicorn agent_server:app` :9101, `EnvironmentFile=/opt/voice-agent/.env`) and `voice-agent-stt.service` (`uvicorn stt_server:app` :9000, no env file — faster-whisper STT, likely stateless/model-only), both `WorkingDirectory=/opt/voice-agent`. +- **Caddy route**: `voice-open.itpropartner.com` → `127.0.0.1:9101`. STT (:9000) has **no direct Caddy route found** — it's likely called internally by the agent process, not exposed publicly; confirm this before assuming it needs its own hostname/cert on app4. +- **Data dir**: `/opt/voice-agent` (470M — second largest in the whole inventory, includes model weights/audio assets budget rsync time). +- **DB**: none found in shallow scan — verify whether call transcripts/session data are persisted anywhere (`grep -ri 'sqlite\|DATABASE_URL' /opt/voice-agent/*.py`) before assuming fully stateless. +- **Cert**: real ACME cert for voice-open.itpropartner.com. (`voice.itpropartner.com`, the SvelteKit-based Hermes Voice frontend at :4331, is a SEPARATE service not in this task's explicit scope — do not conflate the two "voice" hostnames.) +- **Backup**: **NONE for either unit** (flagged in inventory). +- **DNS**: voice-open.itpropartner.com — itpropartner.com is SiteGround manual, **human DNS handoff required**. + +**Why FINAL despite no DB found**: grouped here per the task's explicit instruction to place "anything with a live database or live payment/customer records" in FINAL — Voice Agent's `.env` and the customer-facing nature of live calls (potential in-flight call state, recorded audio, telephony webhook secrets in `.env`) make an uncontrolled mid-call cutover a real customer-impact risk even without a conventional DB file, so it is treated as FINAL-tier caution pending the DB-dependency check above. If that check confirms zero persistence, this can be demoted to Group A/B in a future revision of this document — not changed unilaterally here. + +**Pre-flight** +``` +ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'systemctl is-active voice-agent.service voice-agent-stt.service; grep -ri "sqlite\|DATABASE_URL" /opt/voice-agent/*.py 2>/dev/null' +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'nproc; free -h' # STT (faster-whisper) is CPU/RAM heavy, confirm app4 headroom given it now also hosts everything else in this runbook +``` + +**Move** +``` +rsync -avz -e 'ssh -i /root/.ssh/itpp-infra' root@152.53.192.33:/opt/voice-agent/ root@159.195.205.80:/opt/voice-agent/ +scp -i /root/.ssh/itpp-infra root@152.53.192.33:/opt/voice-agent/.env root@159.195.205.80:/opt/voice-agent/.env +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl daemon-reload && systemctl enable --now voice-agent-stt.service voice-agent.service' +``` + +**Verify** +``` +ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl is-active voice-agent.service voice-agent-stt.service; curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9101/; curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9000/' +curl --resolve voice-open.itpropartner.com:443:159.195.205.80 -s -o /dev/null -w '%{http_code}\n' https://voice-open.itpropartner.com/ +``` +Expect both `active`, matching HTTP codes vs Core, and — given this is a live +telephony-adjacent service — a real end-to-end test call placed against app4 +via `--resolve` (or a Twilio webhook pointed at a temporary test number) before +any DNS flip, not just an HTTP 200 on the root path. + +**Rollback**: `systemctl disable --now voice-agent.service voice-agent-stt.service` on app4. Revert `voice-open.itpropartner.com` DNS to 152.53.192.33 if flipped. Core's units were never stopped. + +--- + +## Summary — DNS dependency at a glance + +| Can move with NO DNS change | Needs human DNS handoff at cutover | +|---|---| +| Twilio MCP, FT360 MCP, DRE MCP (no hostname; needs-decision on the move itself, see Group A) | Transitpin (status.itpropartner.com, shopping.iamgmb.com) | +| Outlook upload (recommended: stays on Core, no move) | DigLocate (dig.iamgmb.com) | +| | IntelSight (my.intelsight.io) | +| | Pipeline API (my.itpropartner.com) | +| | PTA registration (register.timapta.org, pta.iamgmb.com) | +| | PTA survey (survey.iamgmb.com) | +| | Rally (rally.iamgmb.com) | +| | SeeMyTrip (seemytrip.iamgmb.com) | +| | Shark Game (shark.iamgmb.com) | +| | Shopping Cart (shopping.iamgmb.com) | +| | DRE Portal (internal.debtrecoveryexperts.com, my.debtrecoveryexperts.com) | +| | Auth API (auth.itpropartner.com) | +| | VerdictTank (verdicttank.com, www.verdicttank.com, ops.verdicttank.com) | +| | DocuSeal x3 (sign.itpropartner.com, sign.debtrecoveryexperts.com, sign.modelortho.com) | +| | HotNow API (api.hotnow.io) | +| | Voice Agent (voice-open.itpropartner.com) | + +Every single customer-visible cutover in this document requires a DNS change. +`itpropartner.com` changes are SiteGround-manual, no API — Germaine must apply +these himself. All `.iamgmb.com`, `.intelsight.io`, `.debtrecoveryexperts.com`, +`.hotnow.io`, `timapta.org`, and `modelortho.com` records have **unconfirmed** +registrars per the inventory (flagged, not assumed) — verify each with `dig NS +` before assuming any of them are API-automatable; treat all as human +handoff until proven otherwise. + +## Outstanding blockers before Group C can start (recap) + +1. Build and run at least once a Postgres backup job for HotNow's `hotnow` DB (currently zero coverage) — hardest blocker in Wave 2. +2. Rehearse a full restore for HotNow (Postgres), VerdictTank (`users.db`), Auth API (`auth.db`), and all three DocuSeal instances — not just confirm a backup file exists. +3. Close the app4 `root-essentials-backup.sh` gap (`/root/docker` and `/opt/` paths excluded) before Group A even finishes, so newly-landed services aren't immediately unbacked on the new box too. +4. Resolve the Shopping Cart / PTA registration / DigLocate / FT360 MCP "hidden DB?" ambiguities (inventory ambiguity #9) with a real source-code grep before treating them as stateless. +5. Confirm DNS registrar for every non-itpropartner.com domain (`iamgmb.com`, `intelsight.io`, `debtrecoveryexperts.com`, `hotnow.io`, `timapta.org`, `modelortho.com`) — none were confirmed automatable in the source inventory. + + +