Files
itpp-infrastructure/docs/infrastructure/migration-runbook-wave2-customer-2026-09-15.md
T

54 KiB
Raw Blame History

Migration Runbook — WAVE 2: Customer-Facing Services (Core → app4)

Date: 2026-09-15 Scope owner: this document covers the customer-facing service set ONLY. A sibling WAVE 1 document (internal/Hermes-adjacent services — dre-mcp, ft360-mcp, twilio-mcp as Hermes MCP tools, osint-*, pry, super-search, monitoring stack, etc.) is owned by another team and is out of scope here except where a service is genuinely customer-facing and also happens to be Hermes-tool-shaped (see DRE MCP / FT360 MCP / Twilio MCP notes below — included per this task's explicit service list).

Authoritative inventory: core-service-inventory-2026-09-15.md (32KB, read in full before writing this runbook). Treat any conflict between that inventory and older planning docs (app4-migration-plan.md, which undercounted the service list by ~25 services) in the inventory's favor.

Hosts

Host Role IP Spec SSH
Core source, production, DO NOT MODIFY 152.53.192.33 netcup RS 2000, 8vCPU/15GB/503GB, Debian 13 ssh -i /root/.ssh/itpp-infra root@152.53.192.33
app4 destination 159.195.205.80 netcup RS 4000 G12, 12vCPU/32GB/1TB, Nuremberg, Debian 13 ssh -i /root/.ssh/itpp-infra root@159.195.205.80

Verified live on app4 at runbook-authoring time (2026-09-15, read-only checks):

  • Docker 29.8.0, Compose v5.5.1, Caddy v2.11.4 (pinned to Core's Caddy version) — confirmed running.
  • Caddyfile global block: default_bind 159.195.205.80, email info@itpropartner.com.
  • Exactly ONE service already migrated and live: microbin (share.itpropartner.com, container microbin, image pinned by digest danielszabo99/microbin:latest@sha256:6660e5ccad0d764fa3c0032464ffb8f4b4f28c92a2eb9e39202b94cdc5b68909, compose at /root/docker/microbin/docker-compose.yml). This is the proven pattern this runbook replicates for every other service.
  • UFW: 22/80/443 open to Anywhere, 41641/udp (tailscale), plus two narrow Prometheus scrape allows from Core's IP and the tailnet CIDR. No other inbound holes exist yet.
  • docker ps -a on app4 shows only the microbin container — every other service in this runbook is a fresh move, not a resume.

KNOWN BACKUP GAP — read before moving anything

app4:/root/root-essentials-backup.sh backs up .hermes/*, .ssh, .aws, .config/himalaya, shark-game, and projects under /root (verified by reading the script's ARGS loop on app4, 2026-09-15) plus /etc/caddy/Caddyfile if present. It does NOT include /root/docker or /opt. Since the proven pattern (microbin) and most services in this runbook land in /root/docker/<service>/ or keep their existing /opt/<service> tree, every migrated compose stack and every /opt-based systemd service on app4 is currently outside backup coverage the moment it lands, until this gap is closed. This is a pre-existing defect, not introduced by this runbook — flagged here so it is not silently carried forward. Recommendation (not executed by this runbook — read-only constraint): extend root-essentials-backup.sh's ARGS loop on app4 to include docker and the specific /opt/<service> paths, or add a dedicated app4-services-backup.sh mirroring core-services-backup.sh's pattern, before or immediately after Group A completes. Track as a blocking action before Group C (FINAL) is even rehearsed.

Grouping logic

Services are grouped by risk = blast radius of a bad move, which in practice here means: does the service hold a live database with real customer/business data, and/or process payments, and/or is it multi-tenant infrastructure other services depend on (Postgres/Redis)? Group A has zero or trivial state. Group B has a SQLite DB that is important but low-consequence if a short window of writes is lost during a rsync-based cutover. Group C (FINAL) is not moved until a full rehearsed restore has been proven for every service in it — this includes HotNow (live Postgres + Redis + payment-adjacent customer records), VerdictTank (users.db, subscription/payment logic per the .bak-*-subscription-build file found on Core), Auth API (SSO — breaking it breaks login for every other migrated app), and all three DocuSeal instances (signed legal documents).

Group Services Why
A — No state / static-adjacent Transitpin, Outlook upload, Twilio MCP, FT360 MCP, DRE MCP No DB, or trivial re-creatable state; stdio/websocket-only or tiny stateless HTTP
B — SQLite, single-tenant, no payments DigLocate, IntelSight, Pipeline API, PTA registration, PTA survey, Rally, SeeMyTrip, Shark Game, Shopping Cart, DRE Portal Real customer data in SQLite but no live payment flow and no shared DB engine dependency
C — FINAL (rehearsed restore required first) HotNow API + Postgres + Redis DB1, VerdictTank API/worker + users.db, Auth API (SSO), DocuSeal x3 (docuseal, docuseal-dre, docuseal-modelortho), Voice Agent + STT Live DB engine dependency (Postgres/Redis), SSO blast radius, signed legal documents, or subscription/payment-adjacent data

Move order within a group follows the dependency chain noted per-service (e.g. Shopping Cart's Caddy route also serves Transitpin's path, so both move together even though Transitpin is Group A).

Generic pre-flight (run once, before Group A, applies to every service below)

# On app4 — confirm the proven baseline is intact before adding anything
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'docker ps -a; caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile; ufw status'
# Expect: microbin container Up, "Valid configuration", ufw active with 22/80/443 open.

# On Core — snapshot the Caddyfile and systemd unit list so rollback has a known-good diff base
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'md5sum /etc/caddy/Caddyfile; systemctl is-active <service>.service'

Generic rollback primitive used by every service in this doc: Core is never touched (read-only constraint), so rollback is always "stop routing to app4" — either (a) for services with no DNS dependency, delete/disable the new Caddy block on app4 and the old Core route keeps serving unaffected, or (b) for services that required a DNS flip, revert the A/CNAME record at the registrar back to Core's IP (152.53.192.33) — Core's Caddy block and systemd unit were never stopped, so it resumes serving immediately once DNS re-propagates.


GROUP B — SQLite, single-tenant, no live payment flow

(Execution order: after Group A below. Presented first on the page purely because it was authored first; see risk table earlier in this doc for the authoritative A→B→C move order.)

Common pattern for every service below unless noted: FastAPI/uvicorn on 127.0.0.1:<port>, SQLite DB file under its own data dir, systemd unit with User=root, own Python venv. Generic move recipe (repeated per-service with exact paths/ports):

# 1. Pre-flight on Core: confirm service active, note DB file, check for WAL/SHM files (SQLite in-flight writes)
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "systemctl is-active <unit>.service; ls -la <data_dir>/*.db*"
# 2. Quiesce-free rsync (read-only, does not stop Core): copy code+venv+db in one pass, then a final delta pass right before final cutover
rsync -avz --exclude='__pycache__' -e 'ssh -i /root/.ssh/itpp-infra' root@152.53.192.33:<data_dir>/ root@159.195.205.80:<data_dir>/
# 3. Recreate systemd unit on app4 (identical ExecStart/WorkingDirectory/EnvironmentFile/User)
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl daemon-reload && systemctl enable --now <unit>.service"
# 4. Add Caddy site block to app4 Caddyfile mirroring Core's, validate, reload
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile && systemctl reload caddy"
# 5. Verify locally then via --resolve (pre-DNS-flip proof)
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:<port>/"
curl --resolve <hostname>:443:159.195.205.80 -s -o /dev/null -w '%{http_code}\n' https://<hostname>/

Because each service's live SQLite file keeps accepting writes on Core until the DNS flip actually happens, do the rsync twice: once early (bulk copy, service still fully live on Core) and once in the final minutes before the DNS handoff (delta-only, catches writes since the bulk copy) — this keeps the write-loss window to seconds instead of the full rsync duration. All Group B services have this same caveat; not repeated per-service below.

Rollback for all of Group B: systemctl disable --now <unit>.service on app4, remove its Caddy block, systemctl reload caddy. If DNS was already flipped, revert the DNS record to 152.53.192.33 (human handoff for itpropartner.com/iamgmb.com/intelsight.io per registrar — see per-service DNS line). Core's unit and DB file were never touched or stopped, so it is authoritative again the instant traffic returns.

B1. DigLocate

  • Current: Core, diglocate-api.service, uvicorn main:app at 127.0.0.1:8000, WorkingDirectory=/root/projects/diglocate/backend, venv at .venv (note: .venv not venv — different from most others).
  • Caddy route: dig.iamgmb.com127.0.0.1:8000 + static.
  • Data dir: /root/projects/diglocate (118M).
  • DB: none found in shallow scan (inventory ambiguity #9) — verify before move whether a hidden SQLite/Postgres dependency exists (grep -ri 'sqlite\|DATABASE_URL\|psycopg' /root/projects/diglocate/backend/*.py).
  • Cert: real ACME cert for dig.iamgmb.com.
  • Backup: NONE — zero coverage, flagged in inventory.
  • DNS: iamgmb.com — host/registrar unconfirmed; treat as human handoff until confirmed automatable.

Pre-flight: ssh core "systemctl is-active diglocate-api.service; grep -ri 'sqlite\|DATABASE_URL' /root/projects/diglocate/backend/*.py". Expect active; if a DB reference appears, add it to the rsync path list before moving.

Verify: curl -s http://127.0.0.1:8000/docs -o /dev/null -w '%{http_code}\n' on app4 → expect 200 (FastAPI auto-docs) or whatever Core returns for the same path (compare both first: curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8000/docs on Core).

B2. IntelSight API

  • Current: Core, intelsight-api.service, python server.py (not uvicorn-launched, raw script) at 127.0.0.1:8099, WorkingDirectory=/root/intelsight-api.
  • Caddy route: my.intelsight.io127.0.0.1:8099 + static; intelsight.io and intelsight.iamgmb.com are static landing pages only (no backend, can move as plain static rsync whenever convenient, not blocking).
  • Data dir: /root/intelsight-api (64M) incl. intelsight.db.
  • DB: SQLite intelsight.db.
  • Cert: real ACME cert for my.intelsight.io.
  • Backup: NONE.
  • DNS: intelsight.io — registrar unconfirmed (inventory calls this out as unchecked); human handoff assumed until confirmed otherwise.

Verify: curl -s http://127.0.0.1:8099/ -o /dev/null -w '%{http_code}\n' on app4, compare to same on Core.

B3. Pipeline API

  • Current: Core, pipeline-api.service, python server.py at 127.0.0.1:8200, WorkingDirectory=/root/projects/pipeline.
  • Caddy route: my.itpropartner.com block includes 127.0.0.1:8200 (multi-target block, also proxies to App3 remotely for a different path — read the live block carefully before recreating on app4, do not copy App3's remote proxy line since that traffic doesn't belong to this migration).
  • Port conflict note: inventory flags hermes-control-deck.service (disabled) also wants port 8200. It is disabled on Core; confirm it stays disabled/is not part of Wave 1's plan for app4 before binding 8200 there, or this will silently fail to start on app4 the day someone enables it.
  • Data dir: /root/projects/pipeline (30M) incl. pipeline.db.
  • DB: SQLite pipeline.db.
  • Cert: real ACME cert for my.itpropartner.com.
  • Backup: NONE.
  • DNS: my.itpropartner.com — itpropartner.com is SiteGround manual, human DNS handoff required. Also note this hostname's Caddy block mixes an App3 remote-proxy path; migrating Pipeline's path alone should not touch that other path's routing on app4 without also confirming with whoever owns the App3 leg of my.itpropartner.com.

Verify: curl -s http://127.0.0.1:8200/ -o /dev/null -w '%{http_code}\n' on app4 vs Core.

B4. PTA registration (TIMAPTA Membership Registration)

  • Current: Core, pta-registration.service, uvicorn server:app at 127.0.0.1:8114, WorkingDirectory=/opt/pta-registration.
  • Caddy route: register.timapta.org (primary) and pta.iamgmb.com (duplicate route, same backend) → 127.0.0.1:8114.
  • Data dir: /opt/pta-registration (53M) incl. pta.db.
  • DB: SQLite pta.db.
  • Cert: real ACME certs for both register.timapta.org and pta.iamgmb.com (furthest-out expiry in the whole inventory, Dec 9 2026).
  • Backup: NONE.
  • DNS: timapta.org and iamgmb.com — registrars unconfirmed for both; treat both as human handoff pending confirmation. webmail.timapta.org is a separate self-signed redirect stub, unrelated to this service, do not touch.

Verify: curl -s http://127.0.0.1:8114/ -o /dev/null -w '%{http_code}\n' on app4 vs Core; confirm BOTH hostnames' Caddy blocks were recreated (duplicate route is easy to miss).

B5. PTA survey (TIMA Location Survey)

  • Current: Core, survey-registration.service, uvicorn server:app at 127.0.0.1:8115, WorkingDirectory=/opt/pta-survey.
  • Caddy route: survey.iamgmb.com127.0.0.1:8115.
  • Data dir: /opt/pta-survey (50M) incl. survey.db.
  • DB: SQLite survey.db.
  • Cert: real ACME cert for survey.iamgmb.com.
  • Backup: NONE.
  • DNS: iamgmb.com — human handoff pending registrar confirmation.

Verify: curl -s http://127.0.0.1:8115/ -o /dev/null -w '%{http_code}\n' on app4 vs Core.

B6. Rally (Family Calendar)

  • Current: Core, rally.service, python run.py at 127.0.0.1:8105, WorkingDirectory=/opt/rally/backend.
  • Caddy route: rally.iamgmb.com127.0.0.1:8105 + static.
  • Data dir: /opt/rally/data/rally.db (dir 294M — the single largest SQLite-backed dataset in Group B; budget extra rsync time).
  • DB: SQLite rally.db.
  • Cert: real ACME cert for rally.iamgmb.com.
  • Backup: none matching this exact DB path found — inventory notes only debug/dump scripts exist, not a scheduled backup job. Treat as effectively unbacked.
  • DNS: iamgmb.com — human handoff pending registrar confirmation.

Pre-flight: given the 294M size, run the bulk rsync well ahead of cutover and confirm free disk on app4 first: ssh app4 "df -h /opt".

Verify: curl -s http://127.0.0.1:8105/ -o /dev/null -w '%{http_code}\n' on app4 vs Core.

B7. SeeMyTrip (media pipeline)

  • Current: Core, seemytrip.service, uvicorn server:app at 127.0.0.1:8113, WorkingDirectory=/opt/seemytrip.
  • Caddy route: seemytrip.iamgmb.com127.0.0.1:8113 + static.
  • Data dir: /opt/seemytrip/data (within a 212M dir — media files, budget rsync time).
  • DB: SQLite seemytrip.db.
  • Cert: real ACME cert for seemytrip.iamgmb.com.
  • Backup: NONE.
  • DNS: iamgmb.com — human handoff pending registrar confirmation.

Verify: curl -s http://127.0.0.1:8113/ -o /dev/null -w '%{http_code}\n' on app4 vs Core.

B8. Shark Attack Fantasy Game

  • Current: Core, shark-game.service, python3 server.py at 0.0.0.0:8083 (note: binds all interfaces, not 127.0.0.1 — unlike every other Group B service; confirm nothing external depends on direct :8083 access bypassing Caddy before assuming a straight rebind is safe).
  • Caddy route: shark.iamgmb.com127.0.0.1:8083.
  • Data dir: /root/shark-game/backend/game.db (dir 127M).
  • DB: SQLite game.db.
  • Cert: real ACME cert for shark.iamgmb.com.
  • Backup: NONE.
  • Cron dependency: root crontab on Core runs a daily shark-game scraper and a 15-min shark-draft-reminder job — these are cron entries, not systemd, and are Core-scoped. If Shark Game moves to app4, these two cron jobs must also move (or be re-pointed) or the game data will stop updating even though the web app itself works. Flag for the crontab owner; not fixed by this runbook (read-only constraint, cron untouched by this task).
  • DNS: iamgmb.com — human handoff pending registrar confirmation.

Pre-flight: ssh core "crontab -l | grep -i shark" — capture the exact two cron lines to hand off alongside the service move.

Verify: curl -s http://127.0.0.1:8083/ -o /dev/null -w '%{http_code}\n' on app4 vs Core.

B9. Shopping Cart Builder

  • Current: Core, shopping-cart.service, uvicorn app:app at 127.0.0.1:8101, WorkingDirectory=/opt/shopping-cart.
  • Caddy route: shopping.iamgmb.com127.0.0.1:8101 and 127.0.0.1:8210 (Transitpin, see A1) + static — same block serves both services, move them together in the same maintenance window even though they're in different risk groups, to avoid a half-migrated Caddy block.
  • Data dir: /opt/shopping-cart (147M).
  • DB: none found in shallow scan — verify (grep -ri 'sqlite\|DATABASE_URL' /opt/shopping-cart/*.py) before assuming stateless; "Shopping Cart" strongly suggests order/cart state exists somewhere (in-memory? session file? confirm before move).
  • Cert: real ACME cert for shopping.iamgmb.com.
  • Backup: NONE.
  • DNS: iamgmb.com — human handoff pending registrar confirmation.

Pre-flight: resolve the DB ambiguity above before treating this as a simple stateless move; if it does process real payment/cart data, escalate this service into Group C instead of moving it here.

Verify: curl -s http://127.0.0.1:8101/ -o /dev/null -w '%{http_code}\n' on app4 vs Core.

B10. DRE Portal

  • Current: Core, dre-portal.service, uvicorn app.main:app at 127.0.0.1:8093, WorkingDirectory=/opt/dre-portal, EnvironmentFile=/opt/dre-portal/.env (contains live secrets per inventory item 7 — copy manually and securely, never via generic backup/rsync of secrets in cleartext logs).
  • Caddy route: internal.debtrecoveryexperts.com (password-protected via Caddy basic_auth, confirmed live: ippadmin user with a bcrypt hash in the Caddyfile — carry this basic_auth block over verbatim, do not regenerate the hash) and my.debtrecoveryexperts.com → both 127.0.0.1:8093 + static. portal.debtrecoveryexperts.com is a redirect-only stub to my.debtrecoveryexperts.com/start, no backend of its own.
  • Data dir: /opt/dre-portal/data/dre.db (dir 110M).
  • DB: SQLite dre.db — real DRE customer case data.
  • Cert: real ACME certs for both hostnames.
  • Backup: NONE (flagged in inventory).
  • DNS: debtrecoveryexperts.com — registrar unconfirmed; human handoff pending confirmation (not itpropartner.com/SiteGround, verify separately — inventory does not state this domain's registrar).

Pre-flight: ssh core "cat /opt/dre-portal/.env | sed 's/=.*/=REDACTED/'" to confirm the secret keys needed (do not print values), then manually transfer the real .env over scp with the ssh key (never through a generic backup path) before starting the service on app4.

Verify: curl -s -u ippadmin:<known-password> http://127.0.0.1:8093/ -o /dev/null -w '%{http_code}\n' on app4 (basic_auth is enforced even on localhost per the Caddy config — confirm whether the app itself also enforces it or only Caddy does) vs same on Core.

Note on grouping: DRE Portal holds real customer case data but not a live-write shared-DB-engine dependency (it's SQLite, single service). Kept in Group B rather than Group C FINAL because it has no payment processing and no cross-service DB engine risk — but it is the highest-sensitivity item in Group B given internal.debtrecoveryexperts.com's basic_auth gate implies internal/sensitive use. If Germaine's risk tolerance differs, this is the first B-item to consider promoting to Group C.


GROUP A — No state / static-adjacent

(Presented after Group B textually in this file; grouping/risk-order is per the table in the Grouping-logic section above, not file position. Move order for execution is: Group A first, then Group B, then Group C FINAL — regardless of which appears first on the page.)

A1. Transitpin (WebSocket relay)

  • Current: Core, transitpin.service (Node, /usr/local/bin/node /opt/transitpin/server.js), binds 127.0.0.1:8210.
  • Caddy route: status.itpropartner.com block also proxies / traffic partly to 127.0.0.1:8210 (shared with Uptime Kuma's 3001 in the same block — already-migrated Uptime Kuma is out of scope here, but this Caddy block is multi-target); shopping.iamgmb.com also proxies to 127.0.0.1:8210 for a sub-path. No dedicated Transitpin-only hostname — it rides inside two other sites' blocks. Confirm exact path routing by reading the live block before editing (grep -A15 'status.itpropartner.com {' /etc/caddy/Caddyfile and same for shopping.iamgmb.com).
  • Data dir: /opt/transitpin (496K, no DB — in-memory relay state only, safe to lose in-flight sessions on cutover).
  • DB: none.
  • Cert: status.itpropartner.com and shopping.iamgmb.com both have real ACME certs on Core.
  • Backup status: not separately backed up; irrelevant, no persistent data.
  • DNS: status.itpropartner.com and shopping.iamgmb.com are itpropartner.com / iamgmb.com — confirm iamgmb.com's DNS host (inventory flags this as unchecked — dig NS iamgmb.com before cutover). itpropartner.com is SiteGround manual (human handoff required).

Pre-flight

ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'systemctl is-active transitpin.service; ss -ltnp | grep 8210; node --version'
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'node --version || echo NEED_NODE_INSTALL'

Expect: active, socket on 127.0.0.1:8210, Node version printed. If app4 lacks Node, install the matching major version before moving (apt-cache policy nodejs on both, match versions).

Move

rsync -avz -e 'ssh -i /root/.ssh/itpp-infra' root@152.53.192.33:/opt/transitpin/ root@159.195.205.80:/opt/transitpin/
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'cd /opt/transitpin && npm ci --omit=dev 2>&1 | tail -20'
# Create a systemd unit on app4 mirroring Core's exactly (User=root, WorkingDirectory=/opt/transitpin, ExecStart=/usr/local/bin/node /opt/transitpin/server.js, Restart=always)
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl daemon-reload && systemctl enable --now transitpin.service'

Add the equivalent proxy path to app4's Caddyfile inside the (recreated) status.itpropartner.com and shopping.iamgmb.com blocks, then:

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'caddy validate --config /etc/caddy/Caddyfile --adapter caddyfile && systemctl reload caddy'

Verify

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl is-active transitpin.service; curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8210/'
curl --resolve status.itpropartner.com:443:159.195.205.80 -s -o /dev/null -w "%{http_code}\n" https://status.itpropartner.com/

Expect: active, a 2xx/101 (websocket upgrade) locally; the --resolve curl returns the same response app4 would give once DNS points there (does not require DNS to already be flipped).

Rollback: systemctl disable --now transitpin.service on app4, remove the added Caddy stanza, systemctl reload caddy. Core was never touched, still serving.

DNS: needs human DNS handoff for both hostnames at cutover (itpropartner.com = SiteGround manual, no API; iamgmb.com host TBD — confirm before assuming automatable).


A2. Outlook upload receiver

  • Current: Core, outlook-upload.service (python3 /root/upload-staging/upload_server.py), 127.0.0.1:8240, no EnvironmentFile.
  • Caddy route: part of the core.itpropartner.com multi-path block (proxies 127.0.0.1:8240 alongside VerdictTank's 8201 and static content) — not its own hostname.
  • Data dir: /root/upload-staging (32K: index.html + upload_server.py + service file copy). Uploaded files land in a staging subpath not captured in the 32K figure at scan time (transient — verify emptiness before move: ls /root/upload-staging/*/).
  • DB: none.
  • Cert: covered under core.itpropartner.com's existing cert.
  • Backup status: none (flagged as zero-backup-coverage service in the inventory).
  • DNS: core.itpropartner.comthis hostname is explicitly called out in the migration plan as staying on Core ("Core keeps ... core.itpropartner.com"). Moving this service means either (a) leaving it on Core (do nothing — recommended, contradicts nothing in scope) or (b) if genuinely required to move, it needs a NEW path/hostname on app4 since its current route is tied to a host that stays put. Recommendation: do not move under core.itpropartner.com; if migration is required, cut a new subpath under an app4-hosted domain and update whatever client (Outlook add-in?) posts to it. Flagging this as a decision point, not silently moving it.

Pre-flight

ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'systemctl is-active outlook-upload.service; find /root/upload-staging -mindepth 1 -maxdepth 1'

Expect: active; confirm no in-flight large uploads before any move.

Move (only if the domain decision above is resolved in favor of moving):

rsync -avz -e 'ssh -i /root/.ssh/itpp-infra' root@152.53.192.33:/root/upload-staging/ root@159.195.205.80:/root/upload-staging/
# recreate outlook-upload.service on app4 identically, pick new hostname/path per decision above
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl daemon-reload && systemctl enable --now outlook-upload.service'

Verify: curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8240/ on app4 → expect 200/404 (whatever Core returns identically — compare both).

Rollback: systemctl disable --now outlook-upload.service on app4; nothing else changes since Core's route was never edited.

DNS: no DNS change needed if left on Core (recommended). If moved, needs a new hostname decision + human DNS handoff (itpropartner.com is SiteGround manual) or reuse of an existing app4 hostname path.


A3. Twilio MCP, A4. FT360 MCP, A5. DRE MCP (Hermes MCP servers, included per this task's explicit scope)

These three are structurally Hermes MCP tool servers (stdio, EnvironmentFile=-/root/.hermes/.env, no listening TCP port, no Caddy route) but are listed in this task's explicit customer-facing set because their data is customer-facing (DRE case data, FleetTracker360 device data, Twilio call/SMS logs). They are grouped here in A because they carry no independent database of their own beyond what upstream APIs (Twilio, FT360, DRE's own DB) hold — moving them changes nothing customer-visible, only which box runs Hermes's tool process.

Twilio MCP FT360 MCP DRE MCP
unit twilio-mcp.service ft360-mcp.service dre-mcp.service
exec /root/docker/twilio-mcp/venv/bin/python3 server.py /opt/ops-portal/venv/bin/python3 /root/docker/ft360-mcp/server.py (note: borrows ops-portal's venv, not its own — verify this venv exists identically on app4 or rebuild ft360-mcp's own venv) /root/docker/dre-mcp/venv/bin/python3 server.py
working dir /root/docker/twilio-mcp /root/docker/ft360-mcp /root/docker/dre-mcp
env /root/.hermes/.env (shared Hermes secrets file — contains Twilio SIDs, DRE/FT360 keys per inventory item 7) /root/.hermes/.env /root/.hermes/.env
data dir size 171M 60K 171M
DB none found none found none found (DRE's actual case DB lives in dre-portal, Group B)
Caddy route none — stdio MCP none none
Backup none (not customer-DB-bearing) none none

IMPORTANT — this is genuinely a Hermes-config change, not just an app move. These three run as tools invoked by the Hermes gateway on Core. Moving the process to app4 without also repointing Hermes's MCP server registration would either break the tool (Hermes on Core can no longer exec a binary on app4) or require converting them to network-reachable MCP servers (a design change, not a lift-and-shift). This runbook does NOT recommend moving these three under Wave 2 unless the sibling Hermes/Wave-1 team confirms Hermes itself (and its .hermes/.env) is also moving or being made to reach app4 over the network. Flagging as needs-decision, coordinate with Wave 1 owner before executing; no move commands are given here to avoid prescribing an action that could break live Hermes tool calls. If Wave 1 confirms these should move, treat identically to Group A pattern: rsync the /root/docker/<name> dir, recreate the systemd unit with an EnvironmentFile=-/root/.hermes/.env that exists identically on app4 (requires the secrets file to be securely copied — inventory item 7 says do this manually, never via generic backup), verify with a manual JSON-RPC stdio ping, rollback = systemctl disable --now + re-point Hermes's tool registration back to Core.

DNS: none of the three have a Caddy hostname — no DNS involvement either way.


GROUP C — FINAL (do not move until a full rehearsed restore has been proven)

Gate condition for this entire group: no service below moves until at least one full rehearsed restore-from-backup has been executed and verified on app4 (or a throwaway test box) for that service's specific data (Postgres dump+restore for HotNow, SQLite file restore for VerdictTank/Auth/DocuSeal). "Rehearsed" means an actual restore was performed and the restored data was queried/compared, not that a backup file merely exists. Given the inventory's finding that HotNow, VerdictTank's users.db, and DocuSeal all currently DO have some backup coverage in principle (DocuSeal via docuseal-backup cron jobs, Auth API via auth-api-backup.sh) but HotNow's Postgres DB has zero backup coverage today, HotNow cannot be rehearsed until a backup mechanism is built for it first — this is the single hardest blocker in the whole Wave 2 scope and should be treated as its own pre-work item, not squeezed into the migration window.

C1. Auth API (SSO)

  • Current: Core, auth-api.service, python3 -c "import uvicorn; uvicorn.run('server:app', host='127.0.0.1', port=8500, proxy_headers=True)", WorkingDirectory=/root/projects/auth, EnvironmentFile=/root/projects/auth/.env.
  • Caddy route: auth.itpropartner.com127.0.0.1:8500 + static.
  • Data dir: /root/projects/auth (32M) incl. auth.db.
  • DB: SQLite auth.dbevery other migrated service that relies on SSO breaks if this move goes wrong, making it the highest blast-radius item in Group C despite being "just SQLite."
  • Cert: real ACME cert for auth.itpropartner.com.
  • Backup: coveredauth-api-backup.sh exists and is confirmed to run (per inventory backup-coverage summary), though it explicitly redacts secret values from .env in its backup payload (inventory item 7) — the .env itself must be copied manually/securely, not restored from this backup.
  • DNS: auth.itpropartner.com — SiteGround manual, human DNS handoff required.

Pre-flight (rehearsal gate)

# Locate and inspect the existing backup mechanism (read-only)
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'crontab -l | grep -i auth-api; find / -iname "auth-api-backup.sh" 2>/dev/null'
# Perform a rehearsed restore on a scratch path on app4 (NOT into the live auth.db path yet)
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'mkdir -p /root/restore-test/auth && cd /root/restore-test/auth'
# Pull the latest backup archive from Wasabi S3 (same bucket/creds pattern as root-essentials-backup.sh) and restore into the scratch dir, then diff row counts against Core's live auth.db:
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'sqlite3 /root/projects/auth/auth.db "SELECT count(*) FROM sqlite_master; SELECT count(*) FROM users;" 2>&1 || echo "confirm actual table name first"'

Only proceed to the real move once the restored scratch copy's row counts match Core's live counts.

Move (after rehearsal gate passes)

rsync -avz -e 'ssh -i /root/.ssh/itpp-infra' root@152.53.192.33:/root/projects/auth/ root@159.195.205.80:/root/projects/auth/
# manually scp the real .env (contains live secrets, do not rely on the redacted backup copy)
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/projects/auth/.env root@159.195.205.80:/root/projects/auth/.env
# recreate auth-api.service identically on app4
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl daemon-reload && systemctl enable --now auth-api.service'

Verify

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl is-active auth-api.service; curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8500/'
# Functional check: exercise an actual login/token-issue flow end-to-end against app4 before flipping DNS, using --resolve
curl --resolve auth.itpropartner.com:443:159.195.205.80 -s -o /dev/null -w '%{http_code}\n' https://auth.itpropartner.com/

Expect matching HTTP codes to Core, and a successful token issuance in the functional check — SSO is the one service in this runbook where "the port answers" is not sufficient verification; a real auth round-trip must succeed before this is called done.

Rollback: systemctl disable --now auth-api.service on app4. If DNS was flipped, revert auth.itpropartner.com to 152.53.192.33 immediately — every dependent app's login breaks for as long as this is down, so this rollback should be treated as sev-1, not routine.

C2. VerdictTank API + worker + users.db

  • Current: Core, verdicttank-api.service (python3 api.py) and verdicttank-worker.service (python3 worker.py), both WorkingDirectory=/opt/verdicttank, both EnvironmentFile=/etc/verdicttank.env (note: env file lives outside the app dir, in /etc — easy to miss in a straight directory rsync). API listens 127.0.0.1:8201; worker has no listening port (background review-panel processor).
  • Caddy route: verdicttank.com / www.verdicttank.com127.0.0.1:8201 + static; ops.verdicttank.com → same backend; also referenced from core.itpropartner.com's multi-path block (shares that block with Outlook upload, see A2 — confirm core.itpropartner.com's VerdictTank path is intentionally kept on Core or also needs updating).
  • Data dir: /opt/verdicttank (15M) incl. users.db (28K live file, confirmed via direct listing) plus multiple .bak-* snapshots already sitting next to it from prior manual migrations (users.db.bak-20260819-124859, users.db.bak-pre-migration-20260819-131034) — evidence the team has manually snapshotted this DB before major changes previously; do the same here (cp users.db users.db.bak-pre-app4-migration-$(date +%Y%m%d) on Core, read-only, before the final rsync).
  • DB: SQLite users.db — contains subscription/account data (file naming api.py.bak-*-subscription-build strongly implies live subscription logic; treat as payment-adjacent even though no separate payments DB was found).
  • Cert: real ACME certs for verdicttank.com, www.verdicttank.com, ops.verdicttank.com.
  • Backup: VerdictTank has hello-*-collect.py cron jobs but no actual DB backup job for users.db — treat as effectively unbacked despite the manual .bak-* snapshots being evidence of good practice, not a scheduled/automatic job.
  • DNS: verdicttank.com — registrar unconfirmed; human handoff pending confirmation.

Pre-flight (rehearsal gate)

ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'sqlite3 /opt/verdicttank/users.db ".tables"; sqlite3 /opt/verdicttank/users.db "SELECT count(*) FROM users;" 2>&1'
# Take a fresh manual snapshot (read-only cp, not a move)
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'cp /opt/verdicttank/users.db /opt/verdicttank/users.db.bak-pre-app4-migration-$(date +%Y%m%d)'
# Rehearse: copy that snapshot to app4 scratch path, open it, confirm row counts match
rsync -avz -e 'ssh -i /root/.ssh/itpp-infra' root@152.53.192.33:/opt/verdicttank/users.db.bak-pre-app4-migration-* root@159.195.205.80:/root/restore-test/verdicttank/
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'sqlite3 /root/restore-test/verdicttank/users.db.bak-* "SELECT count(*) FROM users;"'

Only proceed once counts match exactly (SQLite file copies while idle are exact — if Core's live file is being written during the copy, re-run and use .timeout/WAL-safe copy, e.g. sqlite3 users.db ".backup /tmp/users_backup.db" instead of cp for a live DB).

Move

rsync -avz -e 'ssh -i /root/.ssh/itpp-infra' root@152.53.192.33:/opt/verdicttank/ root@159.195.205.80:/opt/verdicttank/
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/etc/verdicttank.env root@159.195.205.80:/etc/verdicttank.env
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl daemon-reload && systemctl enable --now verdicttank-api.service verdicttank-worker.service'

Verify

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl is-active verdicttank-api.service verdicttank-worker.service; curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8201/'
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'sqlite3 /opt/verdicttank/users.db "SELECT count(*) FROM users;"'
# Compare against Core's live count taken at the same moment
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'sqlite3 /opt/verdicttank/users.db "SELECT count(*) FROM users;"'

Expect both active, matching HTTP code, and row counts within the expected drift window (writes between final rsync and cutover) — any large unexplained discrepancy is a stop-ship signal.

Rollback: systemctl disable --now verdicttank-api.service verdicttank-worker.service on app4. Revert DNS to 152.53.192.33 if flipped. Core's users.db was never modified by this runbook (only read/copied), so it remains authoritative.

C3C5. DocuSeal x3 (docuseal, docuseal-dre, docuseal-modelortho)

All three are Docker containers on Core, docuseal/docuseal:latest, same image digest sha256:0defc701c18cf45f9e4212acdabbf5e2dbba11c815e4c2e7565da384accf5947 across all three (confirmed via docker inspect, 2026-09-15) — pin to this exact digest on app4 per the proven microbin pattern, do not pull :latest fresh.

docuseal docuseal-dre docuseal-modelortho
Core port 127.0.0.1:8091 127.0.0.1:8094 127.0.0.1:8092
Caddy hostname sign.itpropartner.com sign.debtrecoveryexperts.com sign.modelortho.com
data dir (Core) /root/docker/docuseal/data (688K) /root/docker/docuseal-dre/data (2.5M) /root/docker/docuseal-modelortho/data (1.4M)
compose file /root/docker/docuseal/docker-compose.yml /root/docker/docuseal-dre/docker-compose.yml /root/docker/docuseal-modelortho/docker-compose.yml
cert real ACME real ACME real ACME
backup covered (docuseal-backup cron) covered (docuseal-dre-backup cron) covered (docuseal-modelortho-backup cron) — this is the only Group C set with existing backup jobs for all members; still rehearse the restore before moving, do not assume the job has ever actually been test-restored
DNS registrar itpropartner.com — SiteGround manual debtrecoveryexperts.com — unconfirmed modelortho.com — unconfirmed

Why these are signed-legal-document risk, not just "another SQLite app": DocuSeal stores signed contracts/agreements in its /data volume (SQLite + uploaded document blobs). Losing or corrupting this during migration is a legal liability, not just a customer-experience blip — hence FINAL group placement even though the data volumes themselves are small (under 3MB each).

Pre-flight (rehearsal gate, run once per instance — shown for docuseal, repeat verbatim for -dre and -modelortho substituting names/ports)

ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'docker ps --filter name=docuseal --format "{{.Names}}: {{.Status}}"'
# Locate and inspect the existing backup job (read-only)
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'crontab -l | grep -i docuseal; find /root/.hermes/scripts -iname "*docuseal*backup*" 2>/dev/null'
# Perform an actual rehearsed restore into a scratch dir on app4 from the latest S3 backup object, then diff file counts/sizes against Core's live ./data
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'mkdir -p /root/restore-test/docuseal'
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'find /root/docker/docuseal/data -type f | wc -l'

Do not proceed to the live move for ANY of the three until this rehearsal has actually restored a backup archive and file counts have been diffed successfully for that instance.

Move (per instance, after its rehearsal gate passes)

rsync -avz -e 'ssh -i /root/.ssh/itpp-infra' root@152.53.192.33:/root/docker/docuseal/ root@159.195.205.80:/root/docker/docuseal/

On app4, edit the compose file to pin the image by digest (matching the proven microbin pattern):

image: docuseal/docuseal:latest@sha256:0defc701c18cf45f9e4212acdabbf5e2dbba11c815e4c2e7565da384accf5947
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'cd /root/docker/docuseal && docker compose up -d'

Repeat identically for docuseal-dre (port 8094) and docuseal-modelortho (port 8092), each into its own /root/docker/<name>/ dir.

Verify (per instance)

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'docker ps --filter name=docuseal --format "{{.Names}}: {{.Status}}"; curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8091/'
diff <(ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'find /root/docker/docuseal/data -type f | sort') <(ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'find /root/docker/docuseal/data -type f | sort')

Expect Up, a 2xx/3xx HTTP code, and an empty diff (identical file lists) — for signed documents, a byte-for-byte file inventory match is the bar, not just "the app responds."

Rollback (per instance): docker compose down on app4 (does not touch Core's running container). Revert DNS to 152.53.192.33 if flipped.

DNS: three separate domains, three separate human handoffs — sign.itpropartner.com (SiteGround manual), sign.debtrecoveryexperts.com, sign.modelortho.com (registrars for the latter two unconfirmed, do not assume automatable).

C6. HotNow API + its own Postgres DB + Redis DB1 — HIGHEST RISK ITEM IN WAVE 2

  • Current: Core, hotnow-api.service, uvicorn main:app at 127.0.0.1:8001, WorkingDirectory=/root/hotnow-api, EnvironmentFile=/root/hotnow-api/.env (contains live secrets, manual/secure transfer only).
  • Caddy route: api.hotnow.io127.0.0.1:8001; www.hotnow.io, hotnow.io, app.hotnow.io, admin.hotnow.io are static/redirect-only, no backend dependency (can move independently, low risk, not gated by this item).
  • Data dir: /root/hotnow-api (66M, app code) — the real risk is not this directory, it's the two engines below.
  • DB #1 — Postgres: database hotnow (16MB), owned by role hotnow_app, on Core's host PostgreSQL 17.10 instance (postgresql@17-main.service, port 5432, 127.0.0.1/[::1] only). This is the only real customer database on all of Core's host Postgres — moving it means either (a) standing up a fresh Postgres 17 instance on app4 and doing a pg_dump/pg_restore, or (b) some form of logical replication. Given app4 doesn't appear to run Postgres yet (not seen in the app4 docker ps -a baseline check), option (a) is the realistic path.
  • DB #2 — Redis: HotNow's .env points at redis://localhost:6379/1 (DB1 on Core's host Redis). Redis DB1 was empty (DBSIZE 0) at inventory scan time but the .env reference confirms it's a real dependency, not dead config — do not assume it's safe to skip just because it was empty during a snapshot.
  • Cert: api.hotnow.io uses a ZeroSSL cert (the only non-Let's-Encrypt cert on Core) — confirm app4's Caddy can complete ZeroSSL issuance the same way, or explicitly force Let's Encrypt on app4 instead (a full reissue is expected regardless, since certs aren't copied per this doc's established pattern).
  • Backup: NONE for the Postgres DB, NONE for the app code, NONE for Redis DB1 — this is explicitly the worst-covered service in the entire inventory (a live customer Postgres DB with zero backup jobs). This is the blocking pre-work item for all of Wave 2's FINAL group: a backup mechanism (e.g. nightly pg_dump hotnow | gzip to the existing Wasabi S3 bucket, mirroring timetrex-backup.sh's docker exec ... pg_dump pattern) must be built and run successfully at least once, AND a restore from that backup must be rehearsed, before HotNow moves.
  • DNS: hotnow.io — registrar unconfirmed; human handoff pending confirmation, separate from itpropartner.com/SiteGround.

Pre-flight (rehearsal gate — this is the hard gate for all of Wave 2 FINAL)

# Confirm today's real state (read-only, safe on production Postgres)
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "sudo -u postgres psql -c '\l+' | grep hotnow"
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "sudo -u postgres psql hotnow -c 'SELECT schemaname, relname, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC LIMIT 10;'"
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "redis-cli -n 1 DBSIZE"
# Confirm app4 has NO Postgres yet (do not silently assume; this determines whether a fresh install is even needed)
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "which psql; systemctl list-units | grep -i postgres; docker ps -a | grep -i postgres"

Build (outside this read-only runbook's scope, coordinate with whoever owns app4 provisioning) a Postgres 17 instance on app4, then rehearse:

# On Core (read-only, standard pg_dump does not lock/modify the live DB)
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "sudo -u postgres pg_dump hotnow | gzip > /var/tmp/hotnow-rehearsal-$(date +%F).sql.gz"
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/var/tmp/hotnow-rehearsal-*.sql.gz /tmp/
scp -i /root/.ssh/itpp-infra /tmp/hotnow-rehearsal-*.sql.gz root@159.195.205.80:/var/tmp/
# On app4, restore into a NEW test database, not the eventual production one, and compare row counts to Core's live query above
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sudo -u postgres createdb hotnow_rehearsal && gunzip -c /var/tmp/hotnow-rehearsal-*.sql.gz | sudo -u postgres psql hotnow_rehearsal"
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sudo -u postgres psql hotnow_rehearsal -c 'SELECT schemaname, relname, n_live_tup FROM pg_stat_user_tables ORDER BY n_live_tup DESC LIMIT 10;'"

Do not proceed past this point until the rehearsed restore's row counts match Core's live counts within an explainable margin. Clean up the rehearsal DB afterward (dropdb hotnow_rehearsal) — it is not the production database.

Move (only after the rehearsal gate above passes AND a real backup job exists and has run at least once)

# Final dump (as close to cutover as practical)
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "sudo -u postgres pg_dump hotnow | gzip > /var/tmp/hotnow-final-$(date +%F-%H%M).sql.gz"
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/var/tmp/hotnow-final-*.sql.gz /tmp/ && scp -i /root/.ssh/itpp-infra /tmp/hotnow-final-*.sql.gz root@159.195.205.80:/var/tmp/
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sudo -u postgres createdb hotnow && sudo -u postgres createuser hotnow_app && gunzip -c /var/tmp/hotnow-final-*.sql.gz | sudo -u postgres psql hotnow"
# Redis: DB1 was empty at scan time; if non-empty at cutover time, dump/restore via redis-cli --rdb or DUMP/RESTORE per key
rsync -avz -e 'ssh -i /root/.ssh/itpp-infra' root@152.53.192.33:/root/hotnow-api/ root@159.195.205.80:/root/hotnow-api/
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/hotnow-api/.env root@159.195.205.80:/root/hotnow-api/.env
# update .env's DATABASE_URL/REDIS_URL to point at app4's local Postgres/Redis instances before starting
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl daemon-reload && systemctl enable --now hotnow-api.service'

Verify

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl is-active hotnow-api.service; curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8001/'
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sudo -u postgres psql hotnow -c 'SELECT count(*) FROM pg_stat_user_tables;'"

Expect active, matching HTTP code vs Core, and table/row counts matching the final dump's source counts.

Rollback: systemctl disable --now hotnow-api.service on app4 (Core's hotnow-api.service and its Postgres/Redis were never touched — Core resumes serving immediately). If DNS was flipped, revert api.hotnow.io to 152.53.192.33. The app4-side Postgres hotnow database and any partial data written post-cutover should be preserved (not dropped) until root-caused, in case a forward-fix is preferred over a full rollback.

C7. Voice Agent + STT

  • Current: Core, voice-agent.service (uvicorn agent_server:app :9101, EnvironmentFile=/opt/voice-agent/.env) and voice-agent-stt.service (uvicorn stt_server:app :9000, no env file — faster-whisper STT, likely stateless/model-only), both WorkingDirectory=/opt/voice-agent.
  • Caddy route: voice-open.itpropartner.com127.0.0.1:9101. STT (:9000) has no direct Caddy route found — it's likely called internally by the agent process, not exposed publicly; confirm this before assuming it needs its own hostname/cert on app4.
  • Data dir: /opt/voice-agent (470M — second largest in the whole inventory, includes model weights/audio assets budget rsync time).
  • DB: none found in shallow scan — verify whether call transcripts/session data are persisted anywhere (grep -ri 'sqlite\|DATABASE_URL' /opt/voice-agent/*.py) before assuming fully stateless.
  • Cert: real ACME cert for voice-open.itpropartner.com. (voice.itpropartner.com, the SvelteKit-based Hermes Voice frontend at :4331, is a SEPARATE service not in this task's explicit scope — do not conflate the two "voice" hostnames.)
  • Backup: NONE for either unit (flagged in inventory).
  • DNS: voice-open.itpropartner.com — itpropartner.com is SiteGround manual, human DNS handoff required.

Why FINAL despite no DB found: grouped here per the task's explicit instruction to place "anything with a live database or live payment/customer records" in FINAL — Voice Agent's .env and the customer-facing nature of live calls (potential in-flight call state, recorded audio, telephony webhook secrets in .env) make an uncontrolled mid-call cutover a real customer-impact risk even without a conventional DB file, so it is treated as FINAL-tier caution pending the DB-dependency check above. If that check confirms zero persistence, this can be demoted to Group A/B in a future revision of this document — not changed unilaterally here.

Pre-flight

ssh -i /root/.ssh/itpp-infra root@152.53.192.33 'systemctl is-active voice-agent.service voice-agent-stt.service; grep -ri "sqlite\|DATABASE_URL" /opt/voice-agent/*.py 2>/dev/null'
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'nproc; free -h'  # STT (faster-whisper) is CPU/RAM heavy, confirm app4 headroom given it now also hosts everything else in this runbook

Move

rsync -avz -e 'ssh -i /root/.ssh/itpp-infra' root@152.53.192.33:/opt/voice-agent/ root@159.195.205.80:/opt/voice-agent/
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/opt/voice-agent/.env root@159.195.205.80:/opt/voice-agent/.env
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl daemon-reload && systemctl enable --now voice-agent-stt.service voice-agent.service'

Verify

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 'systemctl is-active voice-agent.service voice-agent-stt.service; curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9101/; curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:9000/'
curl --resolve voice-open.itpropartner.com:443:159.195.205.80 -s -o /dev/null -w '%{http_code}\n' https://voice-open.itpropartner.com/

Expect both active, matching HTTP codes vs Core, and — given this is a live telephony-adjacent service — a real end-to-end test call placed against app4 via --resolve (or a Twilio webhook pointed at a temporary test number) before any DNS flip, not just an HTTP 200 on the root path.

Rollback: systemctl disable --now voice-agent.service voice-agent-stt.service on app4. Revert voice-open.itpropartner.com DNS to 152.53.192.33 if flipped. Core's units were never stopped.


Summary — DNS dependency at a glance

Can move with NO DNS change Needs human DNS handoff at cutover
Twilio MCP, FT360 MCP, DRE MCP (no hostname; needs-decision on the move itself, see Group A) Transitpin (status.itpropartner.com, shopping.iamgmb.com)
Outlook upload (recommended: stays on Core, no move) DigLocate (dig.iamgmb.com)
IntelSight (my.intelsight.io)
Pipeline API (my.itpropartner.com)
PTA registration (register.timapta.org, pta.iamgmb.com)
PTA survey (survey.iamgmb.com)
Rally (rally.iamgmb.com)
SeeMyTrip (seemytrip.iamgmb.com)
Shark Game (shark.iamgmb.com)
Shopping Cart (shopping.iamgmb.com)
DRE Portal (internal.debtrecoveryexperts.com, my.debtrecoveryexperts.com)
Auth API (auth.itpropartner.com)
VerdictTank (verdicttank.com, www.verdicttank.com, ops.verdicttank.com)
DocuSeal x3 (sign.itpropartner.com, sign.debtrecoveryexperts.com, sign.modelortho.com)
HotNow API (api.hotnow.io)
Voice Agent (voice-open.itpropartner.com)

Every single customer-visible cutover in this document requires a DNS change. itpropartner.com changes are SiteGround-manual, no API — Germaine must apply these himself. All .iamgmb.com, .intelsight.io, .debtrecoveryexperts.com, .hotnow.io, timapta.org, and modelortho.com records have unconfirmed registrars per the inventory (flagged, not assumed) — verify each with dig NS <domain> before assuming any of them are API-automatable; treat all as human handoff until proven otherwise.

Outstanding blockers before Group C can start (recap)

  1. Build and run at least once a Postgres backup job for HotNow's hotnow DB (currently zero coverage) — hardest blocker in Wave 2.
  2. Rehearse a full restore for HotNow (Postgres), VerdictTank (users.db), Auth API (auth.db), and all three DocuSeal instances — not just confirm a backup file exists.
  3. Close the app4 root-essentials-backup.sh gap (/root/docker and /opt/<service> paths excluded) before Group A even finishes, so newly-landed services aren't immediately unbacked on the new box too.
  4. Resolve the Shopping Cart / PTA registration / DigLocate / FT360 MCP "hidden DB?" ambiguities (inventory ambiguity #9) with a real source-code grep before treating them as stateless.
  5. Confirm DNS registrar for every non-itpropartner.com domain (iamgmb.com, intelsight.io, debtrecoveryexperts.com, hotnow.io, timapta.org, modelortho.com) — none were confirmed automatable in the source inventory.