# app4 Scoping and Migration Plan **Owner:** IT Pro Partner (Germaine Brown) **Created:** 2026-08-15 **Status:** Draft (for review) **Objective:** Move every customer-facing app off Core onto a new `app4` host. Core becomes the Hermes AI assistant home only, with no customer-facing apps long term. --- ## 1. End State | Host | Role | | --- | --- | | **Core** (RS 2000, 152.53.192.33) | Hermes + its direct dependencies + internal monitoring + Caddy for Core-local routes only | | **app4** (new, netcup) | All customer-facing apps, their databases, and all customer-facing Caddy routes | Core keeps: browserless, camofox-browser, Super Search + SearXNG, the Prometheus/Telegraf/Grafana monitoring stack, mikrotik-exporter, Caddy itself, and core.itpropartner.com. Everything else moves. --- ## 2. Classification ### 2.1 STAYS ON CORE (Hermes and its direct dependencies) | Item | Type | Current | Rationale | | --- | --- | --- | --- | | Caddy reverse proxy | systemd (80/443) | Core | Edge proxy; customer site blocks removed after cutover, `default_bind 152.53.192.33` retained | | browserless | Docker (:3000) | Core | Hermes headless browser dependency | | camofox-browser | Docker (:9377) | Core | Hermes stealth browser dependency | | SearXNG | Docker (127.0.0.1:8888) | Core | Super Search search backend | | Super Search MCP | systemd (:8899) | Core | Hermes `web_search` / `web_extract` MCP | | Prometheus | Docker | Core | Internal fleet monitoring (scrapes node_exporter) | | Telegraf | Docker | Core | Internal metrics collection | | Grafana | Docker | Core | Internal monitoring dashboards | | mikrotik-exporter | Docker (127.0.0.1:9436) | Core | MikroTik router metrics for Prometheus | | core.itpropartner.com | Caddy site | Core | Hermes / Core admin endpoint | ### 2.2 MOVES TO APP4 (customer-facing apps and routes) | Item | Type | Current | Notes | | --- | --- | --- | --- | | DocuSeal | Docker (127.0.0.1:8091->3000) | Core | e-sign platform; SQLite (bind-mounted ./data) + internal Redis/Sidekiq | | TimeTrex | Docker (127.0.0.1:8085) | Core | Time tracking; Postgres backed | | microbin | Docker (127.0.0.1:8260) | Core | Paste bin; lightweight, low blast radius | | Uptime Kuma | Docker (:3001) | Core | Public status monitor | | Ops Portal backend | systemd / uvicorn (:8090) | Core | FastAPI; SQLite (ops.db) | | Postgres | host service (:5432) | Core | Shared; customer schemas move to app4. Core keeps a minimal instance only if a STAYS service still needs it, else decommission | | Redis | host service (:6379) | Core | Shared cache; enumerate consumers in Phase 0 | | sign.itpropartner.com | Caddy site | Core | DocuSeal frontend | | ops.itpropartner.com | Caddy site | Core | Ops Portal frontend | | uptimekuma.itpropartner.com | Caddy site | Core | Uptime Kuma frontend | | my.itpropartner.com | Caddy site | Core | Customer hub | | status.itpropartner.com | Caddy site | Core | Public status page | | auth.itpropartner.com | Caddy site | Core | Centralized auth | | voice.itpropartner.com | Caddy site | Core | Voice agent | | voice-open.itpropartner.com | Caddy site | Core | Voice agent (open) | | *.iamgmb.com | Caddy sites | Core | Customer sites | | *.intelsight.io | Caddy sites | Core | Customer sites | | *.fleettracker360.com | Caddy sites | Core | Customer sites | | *.debtrecoveryexperts.com | Caddy sites | Core | Customer sites | **Postgres / Redis split note:** both are shared instances today. They move per-app, not wholesale. Customer databases and caches are provisioned fresh on app4 and populated from dumps. Core keeps a Postgres/Redis instance only if a STAYS service (none currently identified) depends on it, otherwise they are decommissioned on Core after cutover. **Inventory caveat (resolve in Phase 0):** the architecture plan (`server-architecture-plan` skill) records DocuSeal and SearXNG as having moved off Core in 2024/Aug 2026. This plan treats the supplied Core inventory as authoritative and classifies both on Core. Phase 0 must reconcile with `docker ps` and the live Caddyfile before any move. --- ## 3. app4 Sizing Recommendation **Recommendation: netcup RS 4000 G12 (12 vCPU / 32 GB DDR5 ECC / 1 TB NVMe), ~$44/mo, Manassas VA.** Justification: - Matches the ITPP standard app tier. app1, app2, and app3 are all RS 4000 G12. Consistency simplifies provisioning, monitoring, DR, and cost accounting. - The moved workload is app-tier, not hub-tier. app4 will host a dedicated Postgres + Redis, DocuSeal (Ruby on Rails), TimeTrex (PHP), the Ops Portal backend (FastAPI/uvicorn), the voice stack, and a dozen-plus customer Caddy sites. That is comparable to app1, which already runs an RS 4000. - RS 2000 (8 vCPU / 16 GB) is too small. Core today runs everything on an RS 2000 and is being relieved precisely because it is overloaded. Squeezing the entire customer tier back onto a single RS 2000 would recreate the problem. - 1 TB NVMe provides headroom for Postgres growth, Docker volumes, voice/audio assets, and backup retention without immediate pressure. - Fault isolation: a customer-app outage on app4 no longer competes with Hermes on Core. **Upsize trigger:** if the voice stack or customer site count grows materially, or Postgres usage exceeds ~40% of 32 GB, re-evaluate for RS 8000 (16 vCPU / 64 GB / 2 TB). Start at RS 4000. --- ## 4. Phased Migration Plan ### Phase 0: Inventory and Recon (Core, read-only) - Confirm live inventory: `docker ps`, `docker volume ls`, `ss -tlnp`, `systemctl list-units --type=service`. - Extract every moving site block from `/etc/caddy/Caddyfile` (site name, backend, TLS, redirects). - Catalog data locations: Docker named volumes + bind mounts for DocuSeal, TimeTrex, microbin, Uptime Kuma, Ops Portal. - Enumerate Postgres databases (`psql -l`) and map each to its app; enumerate Redis keyspace consumers. - Record env files and secret references (Hudu / Vaultwarden) for each moving app. - Record cron entries that touch the moving apps or their backups. - Verify DNS authority per domain with `dig NS ` (itpropartner.com is SiteGround manual; fleettracker360.com and voipsimplicity.com are Cloudflare; check iamgmb.com, intelsight.io, debtrecoveryexperts.com individually). - Reconcile the DocuSeal / SearXNG inventory caveat from section 2. - Produce the runbook: per-app data migration command, per-domain DNS record, and an acceptance checklist. ### Phase 1: Provision app4 + Monitoring First - Order RS 4000 G12 per `server-provisioning-standard`: Debian 13, ippadmin user + sudo, itpp-infra SSH key, UFW (open 22, 80, 443), Fail2Ban, unattended-upgrades, Docker + compose plugin, Python, AWS CLI with the cron PATH fix, node_exporter on :9100, Tailscale. - Install Caddy on app4 with `default_bind ` to avoid the Tailscale port 443 conflict. - Enroll app4 in root-essentials-backup; run one manual backup and verify it lands in S3 (do not rely on the cron entry alone). - Add app4:9100 to Core Prometheus targets and Grafana dashboards. Observability exists before any app moves. - Do not move any customer app in this phase. ### Phase 2: Low-Risk Apps (prove the pattern) - Move microbin and Uptime Kuma first. Small, self-contained, low blast radius. - microbin: rsync volume, start on app4, verify via `curl --resolve`. - Uptime Kuma: move after microbin; its monitors continue running and its own cutover is the first DNS flip of the whole project. - Validate the rsync + healthcheck + rollback playbook on these two before touching customer apps. - Confirm app4 S3 backups for these two are working. ### Phase 3: Customer Apps + Data - Foundation first: provision Postgres and Redis on app4 (least-privilege, internal-only networks). - Move Ops Portal backend (:8090), then DocuSeal, then TimeTrex. - Bring each up on app4 on internal ports and test side-by-side with Core using `curl --resolve :443:`. - Move the voice stack (voice.itpropartner.com, voice-open.itpropartner.com), including any audio assets and external webhook/Twilio endpoint updates. - Move static customer sites (*.iamgmb.com, *.intelsight.io, *.fleettracker360.com, *.debtrecoveryexperts.com) by rsyncing web roots. - Verify Postgres row counts and Redis state after each app move (see section 5). ### Phase 4: DNS Cutover + Decommission on Core - Lower TTL on all moving records to 300 (or 60) at least 24h before cutover. - Flip DNS per domain, one at a time, low-traffic first, verifying each before the next. - Keep critical Core Caddy blocks as a temporary 301 redirect to app4 during a 24 to 72h soak window; remove after verification. - After soak: remove customer site blocks from Core Caddyfile (use targeted edits + caddy-audit hook, never rewrite the whole file), stop and remove moved containers on Core, retain volumes and images for 30 days as rollback. - Decommission customer schemas in Core Postgres/Redis (or the whole instance if unused by Core). - Update Prometheus targets, Uptime Kuma, docs, `app-inventory.csv`, and the recovery manual. --- ## 5. Data Migration Steps Docker volumes (rsync, app stopped): ```bash # On Core, stop the app, then delta-sync the volume data to app4 docker compose -f /root/docker//docker-compose.yml stop rsync -az --delete \ /var/lib/docker/volumes//_data/ \ ippadmin@app4:/var/lib/docker/volumes//_data/ # On app4 docker compose -f /root/docker//docker-compose.yml up -d ``` SQLite (online-safe backup, never `cp` a live DB): ```bash sqlite3 /path/app.db ".backup '/tmp/app-backup.db'" rsync -az /tmp/app-backup.db ippadmin@app4:/path/app.db ``` Postgres (per-database custom-format dump): ```bash # On Core pg_dump -Fc -d -f /tmp/.dump rsync -az /tmp/.dump ippadmin@app4:/tmp/ # On app4 pg_restore -d /tmp/.dump ``` For a full-instance move, use `pg_dumpall` instead of per-database dumps. Redis: ```bash # If cache only: rebuild empty on app4. If state matters: redis-cli BGSAVE # then rsync dump.rdb with Redis stopped, or configure replication during cutover ``` Post-migration verification (mandatory, per the Hudu lesson): - Compare Postgres row counts for every major table between Core and app4, not just a spot check. - Compare Docker volume sizes and file counts after rsync. - Hit each domain through app4 with `curl --resolve` and compare responses against Core side-by-side. - Do not declare a migration done on container health alone. --- ## 6. Caddy / DNS Change Checklist - [ ] Verify authoritative nameservers per domain (`dig NS`). itpropartner.com is SiteGround manual; do not create records via Cloudflare for it. - [ ] Lower TTL to 300 (or 60) on every moving record at least 24h before cutover. - [ ] Pre-write the app4 Caddyfile with all moving site blocks; `caddy validate` it. - [ ] Pre-issue TLS certs on app4 (on-demand or staging) before DNS flip. - [ ] Open UFW 80/443 on app4 (netcup blocks them by default) and verify from an external network. - [ ] Set `default_bind ` in app4 Caddy global block to avoid Tailscale :443 conflict. - [ ] Flip each A/AAAA record to app4 in the domain's authoritative panel (SiteGround manual or Cloudflare API, per domain). - [ ] Verify propagation: `dig +short @1.1.1.1 `. - [ ] Verify service and cert on app4: `curl -sI https://`. - [ ] Apply the caddy-audit hook before any Core Caddyfile edit; use targeted edits or `patch`, never a full rewrite. - [ ] After soak, remove stale Core blocks and reload Caddy. --- ## 7. DR / Backup Implications - Repoint backup scripts and cron from Core to app4 for every moved app (docker-volume-sync, any per-app backup jobs, root-essentials-backup). - app4 gets its own S3 backup path under the existing Wasabi bucket, keyed by hostname, with least-privilege credentials. - The daily Hermes backup on Core stops backing up customer app volumes once they move; confirm the app4 cron owns them before removing Core entries. - Add app4 to the DR plan (`server-dr-plans.md`) and the recovery manual; document what to restore in what order. - Decide standby scope: app1-bu is a warm standby for Core, not for customer apps. app4 relies on S3 backups unless a customer-app standby is separately approved. - Use `/opt/awscli-venv/bin/aws` (full path) in every app4 backup script to avoid the silent cron PATH failure. - After the first real backup on app4, perform a test restore of one app to prove the backups work, not just the cron entry. --- ## 8. Risks and Rollback ### Risks | Risk | Impact | Mitigation | | --- | --- | --- | | DNS authority confusion (SiteGround vs Cloudflare) | Silent no-op record changes, outage | Verify `dig NS` per domain first; route changes through the correct panel | | Shared Postgres/Redis partial migration | Core app breaks mid-move | Move per-app dumps, verify row counts, keep Core DB intact until cutover | | Live-file copy corruption (cp on running SQLite/Postgres) | Data loss | Always stop app or use `.backup` / `pg_dump` | | Voice stack hidden dependencies (Twilio webhooks, TTS/STT endpoints) | Voice breaks after cutover | Enumerate external webhooks in Phase 0, update endpoints before DNS flip | | TLS issuance failure on app4 | Site unreachable | Pre-issue certs, confirm UFW 80/443 open, test externally | | Caddyfile fragility (whole-file rewrite drops sites) | Silent domain loss | Targeted edits + caddy-audit hook, never full rewrite | | Backup silently failing on app4 (aws not in PATH) | No restorable backup | Full-path AWS, manual test restore after first backup | ### Rollback - Before each phase, snapshot: current DNS records, Core Caddyfile, and Core Docker state. - Phase 2/3 rollback: stop the app on app4, flip DNS back to Core, restart the Core container. Core volumes are untouched and the app returns to its pre-move state. - Phase 4 rollback: with low TTL, flipping the A record back to Core propagates in minutes; Core Caddy blocks are retained during the soak window for exactly this purpose. - Data rollback: Core volumes and images are retained for 30 days after cutover, so any container can be restarted on Core instantly. - After 30 days: restore from app4 S3 backups (this is why a test restore is mandatory in Phase 1).