232 lines
14 KiB
Markdown
232 lines
14 KiB
Markdown
# app4 Scoping and Migration Plan
|
|
|
|
**Owner:** IT Pro Partner (Germaine Brown)
|
|
**Created:** 2026-08-15
|
|
**Status:** Draft (for review)
|
|
**Objective:** Move every customer-facing app off Core onto a new `app4` host. Core becomes the Hermes AI assistant home only, with no customer-facing apps long term.
|
|
|
|
---
|
|
|
|
## 1. End State
|
|
|
|
| Host | Role |
|
|
| --- | --- |
|
|
| **Core** (RS 2000, 152.53.192.33) | Hermes + its direct dependencies + internal monitoring + Caddy for Core-local routes only |
|
|
| **app4** (new, netcup) | All customer-facing apps, their databases, and all customer-facing Caddy routes |
|
|
|
|
Core keeps: browserless, camofox-browser, Super Search + SearXNG, the Prometheus/Telegraf/Grafana monitoring stack, mikrotik-exporter, Caddy itself, and core.itpropartner.com. Everything else moves.
|
|
|
|
---
|
|
|
|
## 2. Classification
|
|
|
|
### 2.1 STAYS ON CORE (Hermes and its direct dependencies)
|
|
|
|
| Item | Type | Current | Rationale |
|
|
| --- | --- | --- | --- |
|
|
| Caddy reverse proxy | systemd (80/443) | Core | Edge proxy; customer site blocks removed after cutover, `default_bind 152.53.192.33` retained |
|
|
| browserless | Docker (:3000) | Core | Hermes headless browser dependency |
|
|
| camofox-browser | Docker (:9377) | Core | Hermes stealth browser dependency |
|
|
| SearXNG | Docker (127.0.0.1:8888) | Core | Super Search search backend |
|
|
| Super Search MCP | systemd (:8899) | Core | Hermes `web_search` / `web_extract` MCP |
|
|
| Prometheus | Docker | Core | Internal fleet monitoring (scrapes node_exporter) |
|
|
| Telegraf | Docker | Core | Internal metrics collection |
|
|
| Grafana | Docker | Core | Internal monitoring dashboards |
|
|
| mikrotik-exporter | Docker (127.0.0.1:9436) | Core | MikroTik router metrics for Prometheus |
|
|
| core.itpropartner.com | Caddy site | Core | Hermes / Core admin endpoint |
|
|
|
|
### 2.2 MOVES TO APP4 (customer-facing apps and routes)
|
|
|
|
| Item | Type | Current | Notes |
|
|
| --- | --- | --- | --- |
|
|
| DocuSeal | Docker (127.0.0.1:8091->3000) | Core | e-sign platform; SQLite (bind-mounted ./data) + internal Redis/Sidekiq |
|
|
| TimeTrex | Docker (127.0.0.1:8085) | Core | Time tracking; Postgres backed |
|
|
| microbin | Docker (127.0.0.1:8260) | Core | Paste bin; lightweight, low blast radius |
|
|
| Uptime Kuma | Docker (:3001) | Core | Public status monitor |
|
|
| Ops Portal backend | systemd / uvicorn (:8090) | Core | FastAPI; SQLite (ops.db) |
|
|
| Postgres | host service (:5432) | Core | Shared; customer schemas move to app4. Core keeps a minimal instance only if a STAYS service still needs it, else decommission |
|
|
| Redis | host service (:6379) | Core | Shared cache; enumerate consumers in Phase 0 |
|
|
| sign.itpropartner.com | Caddy site | Core | DocuSeal frontend |
|
|
| ops.itpropartner.com | Caddy site | Core | Ops Portal frontend |
|
|
| uptimekuma.itpropartner.com | Caddy site | Core | Uptime Kuma frontend |
|
|
| my.itpropartner.com | Caddy site | Core | Customer hub |
|
|
| status.itpropartner.com | Caddy site | Core | Public status page |
|
|
| auth.itpropartner.com | Caddy site | Core | Centralized auth |
|
|
| voice.itpropartner.com | Caddy site | Core | Voice agent |
|
|
| voice-open.itpropartner.com | Caddy site | Core | Voice agent (open) |
|
|
| *.iamgmb.com | Caddy sites | Core | Customer sites |
|
|
| *.intelsight.io | Caddy sites | Core | Customer sites |
|
|
| *.fleettracker360.com | Caddy sites | Core | Customer sites |
|
|
| *.debtrecoveryexperts.com | Caddy sites | Core | Customer sites |
|
|
|
|
**Postgres / Redis split note:** both are shared instances today. They move per-app, not wholesale. Customer databases and caches are provisioned fresh on app4 and populated from dumps. Core keeps a Postgres/Redis instance only if a STAYS service (none currently identified) depends on it, otherwise they are decommissioned on Core after cutover.
|
|
|
|
**Inventory caveat (resolve in Phase 0):** the architecture plan (`server-architecture-plan` skill) records DocuSeal and SearXNG as having moved off Core in 2024/Aug 2026. This plan treats the supplied Core inventory as authoritative and classifies both on Core. Phase 0 must reconcile with `docker ps` and the live Caddyfile before any move.
|
|
|
|
---
|
|
|
|
## 3. app4 Sizing Recommendation
|
|
|
|
**Recommendation: netcup RS 4000 G12 (12 vCPU / 32 GB DDR5 ECC / 1 TB NVMe), ~$44/mo, Manassas VA.**
|
|
|
|
Justification:
|
|
|
|
- Matches the ITPP standard app tier. app1, app2, and app3 are all RS 4000 G12. Consistency simplifies provisioning, monitoring, DR, and cost accounting.
|
|
- The moved workload is app-tier, not hub-tier. app4 will host a dedicated Postgres + Redis, DocuSeal (Ruby on Rails), TimeTrex (PHP), the Ops Portal backend (FastAPI/uvicorn), the voice stack, and a dozen-plus customer Caddy sites. That is comparable to app1, which already runs an RS 4000.
|
|
- RS 2000 (8 vCPU / 16 GB) is too small. Core today runs everything on an RS 2000 and is being relieved precisely because it is overloaded. Squeezing the entire customer tier back onto a single RS 2000 would recreate the problem.
|
|
- 1 TB NVMe provides headroom for Postgres growth, Docker volumes, voice/audio assets, and backup retention without immediate pressure.
|
|
- Fault isolation: a customer-app outage on app4 no longer competes with Hermes on Core.
|
|
|
|
**Upsize trigger:** if the voice stack or customer site count grows materially, or Postgres usage exceeds ~40% of 32 GB, re-evaluate for RS 8000 (16 vCPU / 64 GB / 2 TB). Start at RS 4000.
|
|
|
|
---
|
|
|
|
## 4. Phased Migration Plan
|
|
|
|
### Phase 0: Inventory and Recon (Core, read-only)
|
|
|
|
- Confirm live inventory: `docker ps`, `docker volume ls`, `ss -tlnp`, `systemctl list-units --type=service`.
|
|
- Extract every moving site block from `/etc/caddy/Caddyfile` (site name, backend, TLS, redirects).
|
|
- Catalog data locations: Docker named volumes + bind mounts for DocuSeal, TimeTrex, microbin, Uptime Kuma, Ops Portal.
|
|
- Enumerate Postgres databases (`psql -l`) and map each to its app; enumerate Redis keyspace consumers.
|
|
- Record env files and secret references (Hudu / Vaultwarden) for each moving app.
|
|
- Record cron entries that touch the moving apps or their backups.
|
|
- Verify DNS authority per domain with `dig NS <domain>` (itpropartner.com is SiteGround manual; fleettracker360.com and voipsimplicity.com are Cloudflare; check iamgmb.com, intelsight.io, debtrecoveryexperts.com individually).
|
|
- Reconcile the DocuSeal / SearXNG inventory caveat from section 2.
|
|
- Produce the runbook: per-app data migration command, per-domain DNS record, and an acceptance checklist.
|
|
|
|
### Phase 1: Provision app4 + Monitoring First
|
|
|
|
- Order RS 4000 G12 per `server-provisioning-standard`: Debian 13, ippadmin user + sudo, itpp-infra SSH key, UFW (open 22, 80, 443), Fail2Ban, unattended-upgrades, Docker + compose plugin, Python, AWS CLI with the cron PATH fix, node_exporter on :9100, Tailscale.
|
|
- Install Caddy on app4 with `default_bind <app4-ipv4>` to avoid the Tailscale port 443 conflict.
|
|
- Enroll app4 in root-essentials-backup; run one manual backup and verify it lands in S3 (do not rely on the cron entry alone).
|
|
- Add app4:9100 to Core Prometheus targets and Grafana dashboards. Observability exists before any app moves.
|
|
- Do not move any customer app in this phase.
|
|
|
|
### Phase 2: Low-Risk Apps (prove the pattern)
|
|
|
|
- Move microbin and Uptime Kuma first. Small, self-contained, low blast radius.
|
|
- microbin: rsync volume, start on app4, verify via `curl --resolve`.
|
|
- Uptime Kuma: move after microbin; its monitors continue running and its own cutover is the first DNS flip of the whole project.
|
|
- Validate the rsync + healthcheck + rollback playbook on these two before touching customer apps.
|
|
- Confirm app4 S3 backups for these two are working.
|
|
|
|
### Phase 3: Customer Apps + Data
|
|
|
|
- Foundation first: provision Postgres and Redis on app4 (least-privilege, internal-only networks).
|
|
- Move Ops Portal backend (:8090), then DocuSeal, then TimeTrex.
|
|
- Bring each up on app4 on internal ports and test side-by-side with Core using `curl --resolve <domain>:443:<app4-ip>`.
|
|
- Move the voice stack (voice.itpropartner.com, voice-open.itpropartner.com), including any audio assets and external webhook/Twilio endpoint updates.
|
|
- Move static customer sites (*.iamgmb.com, *.intelsight.io, *.fleettracker360.com, *.debtrecoveryexperts.com) by rsyncing web roots.
|
|
- Verify Postgres row counts and Redis state after each app move (see section 5).
|
|
|
|
### Phase 4: DNS Cutover + Decommission on Core
|
|
|
|
- Lower TTL on all moving records to 300 (or 60) at least 24h before cutover.
|
|
- Flip DNS per domain, one at a time, low-traffic first, verifying each before the next.
|
|
- Keep critical Core Caddy blocks as a temporary 301 redirect to app4 during a 24 to 72h soak window; remove after verification.
|
|
- After soak: remove customer site blocks from Core Caddyfile (use targeted edits + caddy-audit hook, never rewrite the whole file), stop and remove moved containers on Core, retain volumes and images for 30 days as rollback.
|
|
- Decommission customer schemas in Core Postgres/Redis (or the whole instance if unused by Core).
|
|
- Update Prometheus targets, Uptime Kuma, docs, `app-inventory.csv`, and the recovery manual.
|
|
|
|
---
|
|
|
|
## 5. Data Migration Steps
|
|
|
|
Docker volumes (rsync, app stopped):
|
|
|
|
```bash
|
|
# On Core, stop the app, then delta-sync the volume data to app4
|
|
docker compose -f /root/docker/<app>/docker-compose.yml stop
|
|
rsync -az --delete \
|
|
/var/lib/docker/volumes/<volume>/_data/ \
|
|
ippadmin@app4:/var/lib/docker/volumes/<volume>/_data/
|
|
# On app4
|
|
docker compose -f /root/docker/<app>/docker-compose.yml up -d
|
|
```
|
|
|
|
SQLite (online-safe backup, never `cp` a live DB):
|
|
|
|
```bash
|
|
sqlite3 /path/app.db ".backup '/tmp/app-backup.db'"
|
|
rsync -az /tmp/app-backup.db ippadmin@app4:/path/app.db
|
|
```
|
|
|
|
Postgres (per-database custom-format dump):
|
|
|
|
```bash
|
|
# On Core
|
|
pg_dump -Fc -d <dbname> -f /tmp/<dbname>.dump
|
|
rsync -az /tmp/<dbname>.dump ippadmin@app4:/tmp/
|
|
# On app4
|
|
pg_restore -d <dbname> /tmp/<dbname>.dump
|
|
```
|
|
|
|
For a full-instance move, use `pg_dumpall` instead of per-database dumps.
|
|
|
|
Redis:
|
|
|
|
```bash
|
|
# If cache only: rebuild empty on app4. If state matters:
|
|
redis-cli BGSAVE # then rsync dump.rdb with Redis stopped, or configure replication during cutover
|
|
```
|
|
|
|
Post-migration verification (mandatory, per the Hudu lesson):
|
|
|
|
- Compare Postgres row counts for every major table between Core and app4, not just a spot check.
|
|
- Compare Docker volume sizes and file counts after rsync.
|
|
- Hit each domain through app4 with `curl --resolve` and compare responses against Core side-by-side.
|
|
- Do not declare a migration done on container health alone.
|
|
|
|
---
|
|
|
|
## 6. Caddy / DNS Change Checklist
|
|
|
|
- [ ] Verify authoritative nameservers per domain (`dig NS`). itpropartner.com is SiteGround manual; do not create records via Cloudflare for it.
|
|
- [ ] Lower TTL to 300 (or 60) on every moving record at least 24h before cutover.
|
|
- [ ] Pre-write the app4 Caddyfile with all moving site blocks; `caddy validate` it.
|
|
- [ ] Pre-issue TLS certs on app4 (on-demand or staging) before DNS flip.
|
|
- [ ] Open UFW 80/443 on app4 (netcup blocks them by default) and verify from an external network.
|
|
- [ ] Set `default_bind <app4-ipv4>` in app4 Caddy global block to avoid Tailscale :443 conflict.
|
|
- [ ] Flip each A/AAAA record to app4 in the domain's authoritative panel (SiteGround manual or Cloudflare API, per domain).
|
|
- [ ] Verify propagation: `dig +short @1.1.1.1 <domain>`.
|
|
- [ ] Verify service and cert on app4: `curl -sI https://<domain>`.
|
|
- [ ] Apply the caddy-audit hook before any Core Caddyfile edit; use targeted edits or `patch`, never a full rewrite.
|
|
- [ ] After soak, remove stale Core blocks and reload Caddy.
|
|
|
|
---
|
|
|
|
## 7. DR / Backup Implications
|
|
|
|
- Repoint backup scripts and cron from Core to app4 for every moved app (docker-volume-sync, any per-app backup jobs, root-essentials-backup).
|
|
- app4 gets its own S3 backup path under the existing Wasabi bucket, keyed by hostname, with least-privilege credentials.
|
|
- The daily Hermes backup on Core stops backing up customer app volumes once they move; confirm the app4 cron owns them before removing Core entries.
|
|
- Add app4 to the DR plan (`server-dr-plans.md`) and the recovery manual; document what to restore in what order.
|
|
- Decide standby scope: app1-bu is a warm standby for Core, not for customer apps. app4 relies on S3 backups unless a customer-app standby is separately approved.
|
|
- Use `/opt/awscli-venv/bin/aws` (full path) in every app4 backup script to avoid the silent cron PATH failure.
|
|
- After the first real backup on app4, perform a test restore of one app to prove the backups work, not just the cron entry.
|
|
|
|
---
|
|
|
|
## 8. Risks and Rollback
|
|
|
|
### Risks
|
|
|
|
| Risk | Impact | Mitigation |
|
|
| --- | --- | --- |
|
|
| DNS authority confusion (SiteGround vs Cloudflare) | Silent no-op record changes, outage | Verify `dig NS` per domain first; route changes through the correct panel |
|
|
| Shared Postgres/Redis partial migration | Core app breaks mid-move | Move per-app dumps, verify row counts, keep Core DB intact until cutover |
|
|
| Live-file copy corruption (cp on running SQLite/Postgres) | Data loss | Always stop app or use `.backup` / `pg_dump` |
|
|
| Voice stack hidden dependencies (Twilio webhooks, TTS/STT endpoints) | Voice breaks after cutover | Enumerate external webhooks in Phase 0, update endpoints before DNS flip |
|
|
| TLS issuance failure on app4 | Site unreachable | Pre-issue certs, confirm UFW 80/443 open, test externally |
|
|
| Caddyfile fragility (whole-file rewrite drops sites) | Silent domain loss | Targeted edits + caddy-audit hook, never full rewrite |
|
|
| Backup silently failing on app4 (aws not in PATH) | No restorable backup | Full-path AWS, manual test restore after first backup |
|
|
|
|
### Rollback
|
|
|
|
- Before each phase, snapshot: current DNS records, Core Caddyfile, and Core Docker state.
|
|
- Phase 2/3 rollback: stop the app on app4, flip DNS back to Core, restart the Core container. Core volumes are untouched and the app returns to its pre-move state.
|
|
- Phase 4 rollback: with low TTL, flipping the A record back to Core propagates in minutes; Core Caddy blocks are retained during the soak window for exactly this purpose.
|
|
- Data rollback: Core volumes and images are retained for 30 days after cutover, so any container can be restarted on Core instantly.
|
|
- After 30 days: restore from app4 S3 backups (this is why a test restore is mandatory in Phase 1).
|