itpp-infrastructure/docs/infrastructure/migration-plan-app4-core-bu-2026-09-15.md

migration-plan-app4-core-bu-2026-09-15.md

Owner: IT Pro Partner (Germaine Brown) Created: 2026-09-15 Status: ACTIVE (supersedes the draft app4-migration-plan.md of 2026-08-15; that file is kept for history) Scope: (a) move Core's customer-facing services onto the new app4; (b) stand up core-bu as Core's warm standby with a working failover and failback; (c) update every document, record and reference that names hosts, IPs or service locations.

---

1. What is verified today (2026-09-15)

Everything in this section was measured on the live boxes, not copied from a doc.

1.1 The two new boxes

	app4	core-bu
Role	Core's customer-facing services	Core's warm standby
Hostname	v2202609377162521278.quicksrv.de	v2202609377162521279.megasrv.de
IPv4	159.195.205.80/22	159.195.204.203/22
IPv6	2a0a:4cc0:c2:bcbf:34b3:8cff:fea2:2892	2a0a:4cc0:c2:b3e0:9409:42ff:fe3b:5397
Model	netcup RS 4000 G12, 12 vCPU / 32 GB / 1007 GB	netcup RS 2000 G12, 8 vCPU / 16 GB / 503 GB
Location	Nuremberg (NBG)	Nuremberg (NBG)
RTT from Core	100.5 ms	100.5 ms

core-bu is the exact twin of Core (same 8 vCPU / 16 GB / 503 GB shape), which is what a standby should be.

1.2 Provisioning status: COMPLETE and verified

Both boxes were provisioned to the ITPP standard on 2026-09-15 and each item was verified, not asserted:

• Debian 13, hostname set, timezone America/New_York, 8 GB swap (9 GB on app4).
• ippadmin user with NOPASSWD sudo; itpp-infra key installed for both root and ippadmin.
• ufw active (22/80/443, plus 9100 only from Core and from the tailnet).
• fail2ban active, unattended-upgrades active.
• Docker CE 29.8.0 + Compose v5.5.1 (upstream repo, matching app1's docker-compose-plugin 5.3.1~trixie family).
• node_exporter listening on :9100 - HTTP 200 from Core, closed from a third host.
• awscli + Wasabi credentials in /root/.aws/credentials (mode 600).
• sshd hardened to the fleet convention: PermitRootLogin without-password, PasswordAuthentication no, AllowUsers ippadmin root. Verified four ways per box (root key login, ippadmin key login + sudo -n, password auth refused, non-allowlisted user refused) with the self-reverting lockout guard armed.

1.3 Backups: enrolled and restore-tested

Host	Script	Schedule	Destination	Evidence
app4	root-essentials-backup.sh	04:45 ET	s3://hermes-vps-backups/root-backup/app4/	first run: upload + in-script download/extract verify OK
core-bu	root-essentials-backup.sh	05:15 ET	s3://hermes-vps-backups/root-backup/core-bu/	first run: upload + in-script download/extract verify OK

Note the known gap this replicates: root-essentials-backup.sh excludes *.db by design. Any database these boxes end up hosting needs its own sqlite3 .backup / pg_dump job, exactly as anita-mnz needed hermes-db-backup.sh. This is an explicit Phase 4/5 acceptance item, not an optional nicety.

1.4 Monitoring: registered

A node_exporter job was added to the live Prometheus config (/root/docker/monitoring/prometheus/prometheus.yml) - the job did not previously exist. up=1 verified for core:9100, app4:9100, core-bu:9100.

Finding (pre-existing, not caused by this work): node_exporter is not running on app1, app2, app3 or app1-bu; and the node_exporter target list in /opt/prometheus/prometheus.yml lives in a file Prometheus never loaded (it still names decommissioned wphost02 and 178.156.131.57). Host metrics for the existing fleet were therefore never collected. Tracked in Section 8.

1.5 Credentials

Console/root credentials for both boxes are recorded in /root/.hermes/references/new-servers-2026-09-15.md (mode 600, root only). Password auth is disabled on both boxes, so those passwords are console/rescue only. Vaultwarden was locked at the time of writing, so the Vaultwarden entries are still owed (Section 9, Q3).

---

2. End state

Host	Role after migration
Core (152.53.192.33, RS 2000, Manassas)	Hermes + its direct dependencies (browserless, camofox-browser, SearXNG, Super Search MCP), Prometheus/Telegraf/Grafana, mikrotik-exporter, Caddy for Core-local routes. No customer-facing apps.
app4 (159.195.205.80, RS 4000 G12, Nuremberg)	All customer-facing apps, their databases, and all customer-facing Caddy routes + TLS.
core-bu (159.195.204.203, RS 2000 G12, Nuremberg)	Warm standby for Core (Hermes, its state, its watchdog). Dormant until failover.
app1-bu (5.161.225.131, Hetzner CPX21, Ashburn)	Retirement candidate once core-bu is proven. Only remaining non-netcup box (Section 3.2).
anita-mnz (159.195.16.30, netcup, Manassas)	Unchanged. Anita's dedicated Hermes box.
app1 / app2 / app3 (Manassas)	Unchanged by this plan.

---

3. Deviations and risks you must decide on

3.1 app4 is in Nuremberg, not Manassas (NEW, material)

Measured: 100.5 ms RTT Core -> app4, versus 0.5 ms Core -> app2 (Manassas) and 1.6 ms -> app1-bu (Ashburn). The old draft assumed Manassas. Consequences:

• Customer latency on app4-hosted sites. Typical US East users add roughly 80-110 ms per round trip versus a Manassas host. For static sites this is mostly invisible; for interactive apps (DocuSeal signing flow, Ops Portal, TimeTrex) it is user-visible.
• Core <-> app4 chatter crosses the Atlantic. Any Core->app4 API call, monitoring scrape, backup pull, or Caddy proxy hit pays ~100 ms. This is acceptable if app4 is self-contained, and painful if the two are chatty. Design rule for this migration: app4 must not depend on Core at request time.
• Benefit, and it is real: Core (US) and its standby (EU) now fail independently. A Nuremberg outage does not touch Core, and a Manassas outage does not touch the standby. The old pair (Core + app1-bu Ashburn) were 1.6 ms apart and shared the US East corridor.

Options: (A) accept Nuremberg and design app4 to be self-contained (recommended, zero cost, boxes are paid); (B) re-order app4 as a Manassas RS 4000 and keep the Nuremberg box as the standby. This is a money decision, so it is yours.

3.2 Provider diversity is now unmet

Core, app1, app2, app3, app4, core-bu and anita-mnz are all netcup. app1-bu (Hetzner) is the only other provider, and this plan retires it. Mitigation options: keep app1-bu as the provider-diverse last-resort standby even after core-bu is primary (cheapest option, EUR 31.99/mo), or move the off-site backup/DR device to a non-netcup provider. This must be decided before app1-bu is deleted, and the DR principle that has governed the org so far ("a netcup outage must not kill both live and standby") is currently satisfied by geography but not by provider.

3.3 app4 has no standby of its own

app4 becomes the single host for every customer-facing service. If it dies, customer apps are down until S3 restore. core-bu is shaped for Core, not for the customer tier (16 GB, and it is meant to be dormant). Options: (A) accept S3-restore RTO for app4; (B) let core-bu carry a cold/secondary copy of app4's data; (C) budget a second app-tier box. Recommend a decision now, because it changes what core-bu should replicate.

3.4 DNS authority is split

Confirmed by the old plan's own checklist and this project's history: itpropartner.com is on SiteGround nameservers (manual panel, no API); fleettracker360.com and voipsimplicity.com are on Cloudflare; iamgmb.com, intelsight.io, debtrecoveryexperts.com need per-domain dig NS verification in Phase 0. Every cutover record must be changed in the correct panel or it is a silent no-op. Section 6 lists the records.

---

4. Migration phases

Each phase has a gate: the next phase does not start until the gate's evidence exists.

Phase 0 - Inventory and recon (Core, read-only) - IN PROGRESS

Deliverable: docs/infrastructure/core-service-inventory-2026-09-15.md (+ .json) - every container, unit, port, volume, database, cron job, TLS cert and Caddy route on Core, with sizes and dependencies.

Gate: inventory lists every Caddy site block and its upstream, and explicitly resolves two conflicts that existing docs disagree on:

1. DocuSeal is recorded on Core by the Aug 15 draft but on App1 by backup-plan.md (4:00 AM job).
2. SearXNG likewise. Only the live docker ps / Caddyfile settles it.

Phase 1 - Provision app4 + core-bu, monitoring first - COMPLETE (2026-09-15)

See Section 1. Gate met: both boxes verified; backups restore-tested; both scraped by Prometheus; no customer app touched.

Phase 2 - Access and naming (needs your input)

• Enroll both boxes in Tailscale (needs a reusable auth key or your approval of the login URL - Section 9, Q1).
• Decide DNS names: app4.itpropartner.com and core-bu.itpropartner.com A/AAAA records, added in the correct panel (SiteGround for itpropartner.com). Internal access and monitoring already work by IP, so this is not blocking, but the docs and the recovery manual read better with names.
• Install Caddy on app4 with default_bind 159.195.205.80 (avoids the Tailscale :443 conflict).

Gate: tailscale status shows both nodes; name resolution works from Core.

Phase 3 - Prove the pattern on low-risk apps

• Move microbin (127.0.0.1:8260) first: single container, one volume, no database.
• Move Uptime Kuma second: it is the monitoring tool, so it must be moved carefully and its own downtime window announced.
• For each: stop on Core, rsync the volume, start on app4, verify side-by-side with curl --resolve <domain>:443:159.195.205.80, then flip DNS, then soak 24 h.
• This phase validates the runbook (per-service steps, verification and rollback) before any customer app moves.

Gate: microbin and Uptime Kuma both served from app4 with app4-issued TLS, verified externally, and their S3 backups land from app4 - not from Core - with a restore test on at least one.

Phase 4 - Data foundation + first real app

• Provision Postgres and Redis fresh on app4 (internal-only binds, least privilege, no public 5432/6379).
• Move the Ops Portal backend (:8090), then DocuSeal (SQLite + attachments + its internal Redis), then TimeTrex (Postgres-backed).
• Databases: pg_dump -Fc per database, restore on app4, then compare row counts per major table, not a spot check. SQLite: sqlite3 .backup, never cp.
• New backup jobs on app4 for every database it now hosts (*.db is excluded from the essentials archive).

Gate: row counts match; curl --resolve responses match Core; app4-backup + restore test for each moved DB.

Phase 5 - Customer sites and the voice stack

• rsync every static customer site root (*.iamgmb.com, *.intelsight.io, *.fleettracker360.com, *.debtrecoveryexperts.com) to app4; caddy validate the app4 config; pre-issue TLS.
• Voice stack (voice.*, voice-open.*): enumerate Twilio webhooks and any external endpoints in Phase 0 and update them before the DNS flip, or calls break after cutover.

Gate: every moving domain answers from app4 with a valid cert; voice end-to-end call tested.

Phase 6 - DNS cutover, soak, decommission on Core

• Lower TTL to 60-300 on every moving record 24 h before the flip (correct panel per domain).
• Flip one domain at a time, low traffic first, verifying each (dig +short @1.1.1.1, then curl -sI).
• Keep Core's Caddy blocks as a 301 redirect to app4 during a 24-72 h soak; remove with targeted edits and the caddy-audit hook (never a whole-file rewrite).
• Then: stop/remove the moved containers on Core, retain volumes + images 30 days as rollback, decommission the customer schemas in Core's Postgres/Redis.

Gate: 72 h soak with no rollback; Core runs zero customer-facing apps; rollback path still intact.

Phase 7 - core-bu standby, failover AND failback proven (parallel with 3-6)

• Install the standby package (sync + watchdog with a health-based decision branch, fence-before-takeover, and automatic failback, which the current app1-bu scripts do not have).
• Only one standby may be armed at a time. Disarm app1-bu before arming core-bu, or a Core hiccup makes both answer as the same Telegram bot.
• Prove it with a real, announced test: failover, then failback, then confirm the standby is dormant again.

Gate: failover and failback both demonstrated with evidence, and app1-bu verifiably disarmed.

Phase 8 - Documentation and reference sweep

Deliverable: docs/infrastructure/reference-update-matrix-2026-09-15.md - every artifact that names a host. Includes at minimum: key-inventory.md (done), backup-plan.md (done), CHANGELOG.md (done), app-inventory.csv, server-architecture-plan, server-dr-plans, dr-issue-log, the recovery manual, decommissioned-hosts.json + stale-reference-verify.py (add app1-bu when retired), Prometheus config (live one, and delete/refresh the dead /opt/prometheus/prometheus.yml), Grafana dashboards, Uptime Kuma monitors, health-master-watchdog.py, Hermes cron jobs.json live-config fields, Hudu assets, the ops portal, client-facing runbooks, and any skill that hardcodes a host or IP.

Gate: stale-reference-verify.py and doc-live-verify.py both clean; every doc cites the new IPs.

---

5. Acceptance criteria (whole project)

1. Core hosts no customer-facing app; every moved domain answers from app4 with a valid TLS cert.
2. Every moved service has: a data migration that was verified by counts/sizes, a health check, and a tested rollback.
3. Every database on app4 has its own backup job with a performed restore test (a green cron entry is not evidence).
4. core-bu failover and failback both demonstrated; exactly one standby armed at any time.
5. Documentation matrix closed out: no live surface names a decommissioned host or a stale IP.
6. app1-bu is either retired (with the provider-diversity decision recorded) or explicitly retained as the provider-diverse standby.

---

6. DNS and Caddy change checklist

• dig NS every moving domain; record the authoritative panel in Phase 0.
• Pre-write all moving site blocks into app4's Caddyfile; caddy validate; pre-issue certs.
• default_bind 159.195.205.80 in app4's Caddy global block.
• TTL 60-300 at least 24 h before each flip.
• Flip per domain; verify dig +short @1.1.1.1 and curl -sI https://<domain>.
• Keep Core blocks as 301s for the soak window; then targeted removal + caddy-audit hook.
• Update Http->Https and any CNAME/www records in the same panel as the A record.

---

7. Rollback

• Before each phase: snapshot DNS records, Core Caddyfile, Core docker ps/volume list.
• Phases 3-5: stop on app4, flip DNS back to Core, restart the Core container. Core volumes are untouched.
• Phase 6: with low TTL, the flip back propagates in minutes; Core blocks are retained during soak.
• Data: Core volumes/images retained 30 days. After that, restore from app4's S3 backups (which is why the Phase 1/4 restore tests are mandatory).
• core-bu: failback is part of the design, not an afterthought; the standby stands down on its own.

---

8. Follow-up findings raised by this work (not fixed here)

#	Finding	Impact	Owner
1	node_exporter not running on app1, app2, app3, app1-bu	No host metrics for the fleet	This project (Phase 8)
2	/opt/prometheus/prometheus.yml is a dead file containing decommissioned hosts (wphost02, 178.156.131.57) and was never loaded	Misleading; wasted trust	Phase 8
3	netcup SCP/CCP API auth returns HTTP 500 / 404 (worked in July)	Provisioning automation via API is dead	Separate
4	hermes-standby-sync.sh has a ping-based failback flaw and no failback logic	Standby reliability	Phase 7
5	hermes-snapshot.sh:34 does a live VACUUM INTO	Store churn	Separate

---

9. Open decisions (needed from Germaine)

Q1 - Tailscale: add both boxes to the tailnet. Need either a reusable auth key, or approve the login URL from each box.

Q2 - Nuremberg vs Manassas for app4: accept Nuremberg (design app4 self-contained) or re-order app4 in Manassas and repurpose the Nuremberg box? Section 3.1.

Q3 - Vaultwarden: the CLI is locked. Unlock it (or tell me when) and I will file the two new server items.

Q4 - app1-bu: retire it, or keep it as the provider-diverse standby? Section 3.2 - this is the only remaining non-netcup box.

Q5 - app4 standby scope: accept S3-restore RTO, or should core-bu carry a secondary copy? Section 3.3.

---

Appendix A - Verified evidence log (2026-09-15)

Check	Result
RTT Core -> app4 / core-bu	100.5 ms / 100.5 ms
RTT Core -> app2 (Manassas) / app1-bu (Ashburn)	0.5 ms / 1.6 ms
authorized_keys (root + ippadmin, both boxes)	present, sha256 102c80e5..., 1 line each
Key login from Core	root OK, ippadmin sudo -n -> root, both boxes
Password auth / non-allowlisted user	refused on both
ufw status	active, 22/80/443 + 9100 from Core/tailnet
Docker / Compose	29.8.0 / v5.5.1 both
node_exporter	200 from Core; refused from app2 (third host)
Prometheus up{job="node_exporter"}	core, app4, core-bu = 1
First backup run	upload + download/extract verify OK on both
Credentials file	/root/.hermes/references/new-servers-2026-09-15.md, mode 600