ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
9af0db089bc15666b1c982a0fa7eeac139b32a2f
itpp-infrastructure/docs/infrastructure/app4-migration-plan.md
T

14 KiB
Raw Blame History

app4 Scoping and Migration Plan

Owner: IT Pro Partner (Germaine Brown) Created: 2026-08-15 Status: Draft (for review) Objective: Move every customer-facing app off Core onto a new app4 host. Core becomes the Hermes AI assistant home only, with no customer-facing apps long term.

1. End State

Host Role
Core (RS 2000, 152.53.192.33) Hermes + its direct dependencies + internal monitoring + Caddy for Core-local routes only
app4 (new, netcup) All customer-facing apps, their databases, and all customer-facing Caddy routes

Core keeps: browserless, camofox-browser, Super Search + SearXNG, the Prometheus/Telegraf/Grafana monitoring stack, mikrotik-exporter, Caddy itself, and core.itpropartner.com. Everything else moves.

2. Classification

2.1 STAYS ON CORE (Hermes and its direct dependencies)

Item Type Current Rationale
Caddy reverse proxy systemd (80/443) Core Edge proxy; customer site blocks removed after cutover, default_bind 152.53.192.33 retained
browserless Docker (:3000) Core Hermes headless browser dependency
camofox-browser Docker (:9377) Core Hermes stealth browser dependency
SearXNG Docker (127.0.0.1:8888) Core Super Search search backend
Super Search MCP systemd (:8899) Core Hermes web_search / web_extract MCP
Prometheus Docker Core Internal fleet monitoring (scrapes node_exporter)
Telegraf Docker Core Internal metrics collection
Grafana Docker Core Internal monitoring dashboards
mikrotik-exporter Docker (127.0.0.1:9436) Core MikroTik router metrics for Prometheus
core.itpropartner.com Caddy site Core Hermes / Core admin endpoint

2.2 MOVES TO APP4 (customer-facing apps and routes)

Item Type Current Notes
DocuSeal Docker (127.0.0.1:8091->3000) Core e-sign platform; SQLite (bind-mounted ./data) + internal Redis/Sidekiq
TimeTrex Docker (127.0.0.1:8085) Core Time tracking; Postgres backed
microbin Docker (127.0.0.1:8260) Core Paste bin; lightweight, low blast radius
Uptime Kuma Docker (:3001) Core Public status monitor
Ops Portal backend systemd / uvicorn (:8090) Core FastAPI; SQLite (ops.db)
Postgres host service (:5432) Core Shared; customer schemas move to app4. Core keeps a minimal instance only if a STAYS service still needs it, else decommission
Redis host service (:6379) Core Shared cache; enumerate consumers in Phase 0
sign.itpropartner.com Caddy site Core DocuSeal frontend
ops.itpropartner.com Caddy site Core Ops Portal frontend
uptimekuma.itpropartner.com Caddy site Core Uptime Kuma frontend
my.itpropartner.com Caddy site Core Customer hub
status.itpropartner.com Caddy site Core Public status page
auth.itpropartner.com Caddy site Core Centralized auth
voice.itpropartner.com Caddy site Core Voice agent
voice-open.itpropartner.com Caddy site Core Voice agent (open)
*.iamgmb.com Caddy sites Core Customer sites
*.intelsight.io Caddy sites Core Customer sites
*.fleettracker360.com Caddy sites Core Customer sites
*.debtrecoveryexperts.com Caddy sites Core Customer sites

Postgres / Redis split note: both are shared instances today. They move per-app, not wholesale. Customer databases and caches are provisioned fresh on app4 and populated from dumps. Core keeps a Postgres/Redis instance only if a STAYS service (none currently identified) depends on it, otherwise they are decommissioned on Core after cutover.

Inventory caveat (resolve in Phase 0): the architecture plan (server-architecture-plan skill) records DocuSeal and SearXNG as having moved off Core in 2024/Aug 2026. This plan treats the supplied Core inventory as authoritative and classifies both on Core. Phase 0 must reconcile with docker ps and the live Caddyfile before any move.

3. app4 Sizing Recommendation

Recommendation: netcup RS 4000 G12 (12 vCPU / 32 GB DDR5 ECC / 1 TB NVMe), ~$44/mo, Manassas VA.

Justification:

  • Matches the ITPP standard app tier. app1, app2, and app3 are all RS 4000 G12. Consistency simplifies provisioning, monitoring, DR, and cost accounting.
  • The moved workload is app-tier, not hub-tier. app4 will host a dedicated Postgres + Redis, DocuSeal (Ruby on Rails), TimeTrex (PHP), the Ops Portal backend (FastAPI/uvicorn), the voice stack, and a dozen-plus customer Caddy sites. That is comparable to app1, which already runs an RS 4000.
  • RS 2000 (8 vCPU / 16 GB) is too small. Core today runs everything on an RS 2000 and is being relieved precisely because it is overloaded. Squeezing the entire customer tier back onto a single RS 2000 would recreate the problem.
  • 1 TB NVMe provides headroom for Postgres growth, Docker volumes, voice/audio assets, and backup retention without immediate pressure.
  • Fault isolation: a customer-app outage on app4 no longer competes with Hermes on Core.

Upsize trigger: if the voice stack or customer site count grows materially, or Postgres usage exceeds ~40% of 32 GB, re-evaluate for RS 8000 (16 vCPU / 64 GB / 2 TB). Start at RS 4000.

4. Phased Migration Plan

Phase 0: Inventory and Recon (Core, read-only)

  • Confirm live inventory: docker ps, docker volume ls, ss -tlnp, systemctl list-units --type=service.
  • Extract every moving site block from /etc/caddy/Caddyfile (site name, backend, TLS, redirects).
  • Catalog data locations: Docker named volumes + bind mounts for DocuSeal, TimeTrex, microbin, Uptime Kuma, Ops Portal.
  • Enumerate Postgres databases (psql -l) and map each to its app; enumerate Redis keyspace consumers.
  • Record env files and secret references (Hudu / Vaultwarden) for each moving app.
  • Record cron entries that touch the moving apps or their backups.
  • Verify DNS authority per domain with dig NS <domain> (itpropartner.com is SiteGround manual; fleettracker360.com and voipsimplicity.com are Cloudflare; check iamgmb.com, intelsight.io, debtrecoveryexperts.com individually).
  • Reconcile the DocuSeal / SearXNG inventory caveat from section 2.
  • Produce the runbook: per-app data migration command, per-domain DNS record, and an acceptance checklist.

Phase 1: Provision app4 + Monitoring First

  • Order RS 4000 G12 per server-provisioning-standard: Debian 13, ippadmin user + sudo, itpp-infra SSH key, UFW (open 22, 80, 443), Fail2Ban, unattended-upgrades, Docker + compose plugin, Python, AWS CLI with the cron PATH fix, node_exporter on :9100, Tailscale.
  • Install Caddy on app4 with default_bind <app4-ipv4> to avoid the Tailscale port 443 conflict.
  • Enroll app4 in root-essentials-backup; run one manual backup and verify it lands in S3 (do not rely on the cron entry alone).
  • Add app4:9100 to Core Prometheus targets and Grafana dashboards. Observability exists before any app moves.
  • Do not move any customer app in this phase.

Phase 2: Low-Risk Apps (prove the pattern)

  • Move microbin and Uptime Kuma first. Small, self-contained, low blast radius.
  • microbin: rsync volume, start on app4, verify via curl --resolve.
  • Uptime Kuma: move after microbin; its monitors continue running and its own cutover is the first DNS flip of the whole project.
  • Validate the rsync + healthcheck + rollback playbook on these two before touching customer apps.
  • Confirm app4 S3 backups for these two are working.

Phase 3: Customer Apps + Data

  • Foundation first: provision Postgres and Redis on app4 (least-privilege, internal-only networks).
  • Move Ops Portal backend (:8090), then DocuSeal, then TimeTrex.
  • Bring each up on app4 on internal ports and test side-by-side with Core using curl --resolve <domain>:443:<app4-ip>.
  • Move the voice stack (voice.itpropartner.com, voice-open.itpropartner.com), including any audio assets and external webhook/Twilio endpoint updates.
  • Move static customer sites (*.iamgmb.com, *.intelsight.io, *.fleettracker360.com, *.debtrecoveryexperts.com) by rsyncing web roots.
  • Verify Postgres row counts and Redis state after each app move (see section 5).

Phase 4: DNS Cutover + Decommission on Core

  • Lower TTL on all moving records to 300 (or 60) at least 24h before cutover.
  • Flip DNS per domain, one at a time, low-traffic first, verifying each before the next.
  • Keep critical Core Caddy blocks as a temporary 301 redirect to app4 during a 24 to 72h soak window; remove after verification.
  • After soak: remove customer site blocks from Core Caddyfile (use targeted edits + caddy-audit hook, never rewrite the whole file), stop and remove moved containers on Core, retain volumes and images for 30 days as rollback.
  • Decommission customer schemas in Core Postgres/Redis (or the whole instance if unused by Core).
  • Update Prometheus targets, Uptime Kuma, docs, app-inventory.csv, and the recovery manual.

5. Data Migration Steps

Docker volumes (rsync, app stopped):

# On Core, stop the app, then delta-sync the volume data to app4
docker compose -f /root/docker/<app>/docker-compose.yml stop
rsync -az --delete \
  /var/lib/docker/volumes/<volume>/_data/ \
  ippadmin@app4:/var/lib/docker/volumes/<volume>/_data/
# On app4
docker compose -f /root/docker/<app>/docker-compose.yml up -d

SQLite (online-safe backup, never cp a live DB):

sqlite3 /path/app.db ".backup '/tmp/app-backup.db'"
rsync -az /tmp/app-backup.db ippadmin@app4:/path/app.db

Postgres (per-database custom-format dump):

# On Core
pg_dump -Fc -d <dbname> -f /tmp/<dbname>.dump
rsync -az /tmp/<dbname>.dump ippadmin@app4:/tmp/
# On app4
pg_restore -d <dbname> /tmp/<dbname>.dump

For a full-instance move, use pg_dumpall instead of per-database dumps.

Redis:

# If cache only: rebuild empty on app4. If state matters:
redis-cli BGSAVE   # then rsync dump.rdb with Redis stopped, or configure replication during cutover

Post-migration verification (mandatory, per the Hudu lesson):

  • Compare Postgres row counts for every major table between Core and app4, not just a spot check.
  • Compare Docker volume sizes and file counts after rsync.
  • Hit each domain through app4 with curl --resolve and compare responses against Core side-by-side.
  • Do not declare a migration done on container health alone.

6. Caddy / DNS Change Checklist

  • Verify authoritative nameservers per domain (dig NS). itpropartner.com is SiteGround manual; do not create records via Cloudflare for it.
  • Lower TTL to 300 (or 60) on every moving record at least 24h before cutover.
  • Pre-write the app4 Caddyfile with all moving site blocks; caddy validate it.
  • Pre-issue TLS certs on app4 (on-demand or staging) before DNS flip.
  • Open UFW 80/443 on app4 (netcup blocks them by default) and verify from an external network.
  • Set default_bind <app4-ipv4> in app4 Caddy global block to avoid Tailscale :443 conflict.
  • Flip each A/AAAA record to app4 in the domain's authoritative panel (SiteGround manual or Cloudflare API, per domain).
  • Verify propagation: dig +short @1.1.1.1 <domain>.
  • Verify service and cert on app4: curl -sI https://<domain>.
  • Apply the caddy-audit hook before any Core Caddyfile edit; use targeted edits or patch, never a full rewrite.
  • After soak, remove stale Core blocks and reload Caddy.

7. DR / Backup Implications

  • Repoint backup scripts and cron from Core to app4 for every moved app (docker-volume-sync, any per-app backup jobs, root-essentials-backup).
  • app4 gets its own S3 backup path under the existing Wasabi bucket, keyed by hostname, with least-privilege credentials.
  • The daily Hermes backup on Core stops backing up customer app volumes once they move; confirm the app4 cron owns them before removing Core entries.
  • Add app4 to the DR plan (server-dr-plans.md) and the recovery manual; document what to restore in what order.
  • Decide standby scope: app1-bu is a warm standby for Core, not for customer apps. app4 relies on S3 backups unless a customer-app standby is separately approved.
  • Use /opt/awscli-venv/bin/aws (full path) in every app4 backup script to avoid the silent cron PATH failure.
  • After the first real backup on app4, perform a test restore of one app to prove the backups work, not just the cron entry.

8. Risks and Rollback

Risks

Risk Impact Mitigation
DNS authority confusion (SiteGround vs Cloudflare) Silent no-op record changes, outage Verify dig NS per domain first; route changes through the correct panel
Shared Postgres/Redis partial migration Core app breaks mid-move Move per-app dumps, verify row counts, keep Core DB intact until cutover
Live-file copy corruption (cp on running SQLite/Postgres) Data loss Always stop app or use .backup / pg_dump
Voice stack hidden dependencies (Twilio webhooks, TTS/STT endpoints) Voice breaks after cutover Enumerate external webhooks in Phase 0, update endpoints before DNS flip
TLS issuance failure on app4 Site unreachable Pre-issue certs, confirm UFW 80/443 open, test externally
Caddyfile fragility (whole-file rewrite drops sites) Silent domain loss Targeted edits + caddy-audit hook, never full rewrite
Backup silently failing on app4 (aws not in PATH) No restorable backup Full-path AWS, manual test restore after first backup

Rollback

  • Before each phase, snapshot: current DNS records, Core Caddyfile, and Core Docker state.
  • Phase 2/3 rollback: stop the app on app4, flip DNS back to Core, restart the Core container. Core volumes are untouched and the app returns to its pre-move state.
  • Phase 4 rollback: with low TTL, flipping the A record back to Core propagates in minutes; Core Caddy blocks are retained during the soak window for exactly this purpose.
  • Data rollback: Core volumes and images are retained for 30 days after cutover, so any container can be restarted on Core instantly.
  • After 30 days: restore from app4 S3 backups (this is why a test restore is mandatory in Phase 1).
Reference in New Issue View Git Blame Copy Permalink