# Migration Runbook — WAVE 1 (Internal / Low-Customer-Impact Services) **Date:** 2026-09-15 **Scope:** Core (152.53.192.33) → app4 (159.195.205.80), exactly 6 services: 1. microbin (`:8260`, `share.itpropartner.com`) — **ALREADY MIGRATED, DONE** 2. Uptime Kuma 3. Prometheus + Grafana + node_exporter monitoring stack 4. Grafana specifically (unauthenticated public-bind finding) 5. PRY 6. OSINT API / OSINT Person MCP (`:8902`) + Super Search MCP (`:8899`) **Reference source docs:** `core-service-inventory-2026-09-15.md`, `migration-plan-app4-core-bu-2026-09-15.md`. **Production hosts are READ-ONLY for this runbook's author.** All facts below were captured via read-only SSH (`ss -ltnp`, `docker inspect`, `systemctl cat`, `cat` of configs) on 2026-09-15. No service was started, stopped, or restarted to produce this document. `hermes-maintenance` was never invoked. **SSH access:** `ssh -i /root/.ssh/itpp-infra root@` (key already present on this box). **Proven pattern (established this session on microbin, reference for all 6 services):** 1. Caddy v2.11.4 on app4, pinned to Core's exact version, from the `caddy-stable` apt repo. 2. Service runs via `/root/docker//compose.yaml` (or `docker-compose.yml`), image **pinned by digest**, not `:latest`. 3. app4's global Caddy option block sets `default_bind 159.195.205.80` so a new site block never hijacks another listener (e.g. Tailscale's own :443). 4. `ufw` active on app4: 22/80/443 allowed, default deny incoming otherwise. 5. Cutover = flip DNS at the authoritative panel (SiteGround for `itpropartner.com`, **manual, no API** — Germaine applies these), then Caddy on app4 auto-issues the cert via ACME http-01 on first request. 6. Verify side-by-side with `curl --resolve :443:159.195.205.80 https:///` **before** the DNS flip, so the app layer is proven independent of DNS. **Known defect already fixed (do not re-diagnose):** both app4 and core-bu shipped with an empty Docker data root (`/var/lib/docker` had zero subdirectories → every `docker pull`/`run` failed with `GetImageBlob` / `overlay2: no such file or directory`). Fixed by stopping docker, `rm -rf /var/lib/docker`, restarting docker, and proving with `docker run --rm hello-world`. Both boxes now run containers normally. If a pull/run fails with an overlay2 ENOENT-style error on either box in the steps below, this is the known cause — check `docker info | grep -i 'docker root dir'` and `ls /var/lib/docker` before assuming a new bug. **Known backup gap this runbook must close (Section 8):** app4's `root-essentials-backup.sh` include list covers `.hermes .ssh .aws shark-game projects` (repo-root-relative) plus `/etc/caddy/Caddyfile`, but **not** `/root/docker`. Every compose stack landed on app4 by this runbook (Uptime Kuma, monitoring stack, PRY if containerized, OSINT/Super Search if containerized) is therefore invisible to the nightly backup until Section 8 is applied. --- ## Service 1 of 6 — microbin (`:8260`, `share.itpropartner.com`) — **STATUS: DONE** This is the proven reference pattern; it is documented here as the template the other 5 follow, not as new work. ### Current state (Core) - Container: `microbin` (`danielszabo99/microbin:latest`, image digest `sha256:6660e5ccad0d764fa3c0032464ffb8f4b4f28c92a2eb9e39202b94cdc5b68909`), `unless-stopped`. - Port: `127.0.0.1:8260 -> 8080` (container), not exposed externally on Core. - Compose: `/opt/microbin/docker-compose.yml`. - Data dir: `/opt/microbin/data` (bind mount -> `/app/pasta_data`), **4.0 KB** — no DB engine, flat files only. - Caddy route (Core, `/etc/caddy/Caddyfile` line ~178): `share.itpropartner.com { reverse_proxy 127.0.0.1:8260 }`. - Cert: real ACME (Let's Encrypt) issued to Core. ### What was actually done (already executed, verified in this session) 1. Installed Caddy v2.11.4 on app4 from the `caddy-stable` apt repo (matches Core's `caddy version` output exactly: `v2.11.4 h1:XKxkMTgNSizEvKG6QHue6cAsFOteU2qA61w2tKkCWi0=` on both hosts). 2. Created `/root/docker/microbin/docker-compose.yml` on app4, pinning the image **by digest** (`danielszabo99/microbin@sha256:6660e5ccad0d764fa3c0032464ffb8f4b4f28c92a2eb9e39202b94cdc5b68909`), same env vars as Core's compose, port bound `127.0.0.1:8260:8080`. 3. `docker compose up -d` on app4 — container `microbin` now `Up`, verified via `docker ps`. 4. Wrote app4's `/etc/caddy/Caddyfile`: ``` { default_bind 159.195.205.80 email info@itpropartner.com } # -- MicroBin Share (migrated from Core 2026-09-15, phase 3 of the app4 migration) -- share.itpropartner.com { reverse_proxy 127.0.0.1:8260 encode gzip } ``` 5. Data dir was NOT rsynced from Core because Core's `/opt/microbin/data` was empty (4.0 KB, no user pastes) at scan time — a fresh empty dir on app4 is equivalent. **If Core's data dir is non-empty at your actual cutover time**, rsync it first: `rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/opt/microbin/data/ root@159.195.205.80:/root/docker/microbin/data/` and stop the Core container for the duration of the rsync to avoid a torn copy (microbin uses flat files, not a WAL DB, so this is a low-risk copy). ### Verification (already run, real output) ``` ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker ps --format '{{.Names}}\t{{.Ports}}\t{{.Status}}'" # -> microbin 127.0.0.1:8260->8080/tcp Up ``` App-layer check (curl from app4 itself, since DNS has not flipped yet): ``` ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8260/" # Expected: 200 ``` **Confirmed:** direct curl to app4's microbin returns 200. App layer is fully staged and correct. ### What is NOT done yet (pending, not part of "DONE" claim) - **DNS flip is pending.** `itpropartner.com` is on SiteGround, manual panel, no API — Germaine must change the `share.itpropartner.com` A record from 152.53.192.33 to 159.195.205.80. - Cert on app4 has not been issued yet (Caddy issues on first real HTTPS hit post-DNS-flip via http-01). - No S3 backup job exists yet for app4's `/root/docker/microbin/data` (see Section 8 — same gap applies here). ### Rollback (if flip has NOT happened — current state) Nothing to roll back: Core's container and Caddy route are untouched and still serving `share.itpropartner.com` live. app4's copy is inert until DNS points at it. ### Rollback (if DNS HAS been flipped and a problem appears) 1. Revert the `share.itpropartner.com` A record at SiteGround back to `152.53.192.33`. 2. TTL propagation is the only wait; Core's container was never stopped, so it answers immediately once DNS re-resolves. 3. Do not touch app4's container — leave it running for retry. --- ## Service 2 of 6 — Uptime Kuma ### Current state (Core) - Container: `uptime-kuma` (`louislam/uptime-kuma:latest`, image digest `sha256:3d632903e6af34139a37f18055c4f1bfd9b7205ae1138f1e5e8940ddc1d176f9`, host layer digest `f48d816cb7460cd3b7bb15ed393968b0ae0da4c690443b778b6a5db6b09f527e`), `unless-stopped`. - Port: `0.0.0.0:3001 -> 3001` — **exposed on all interfaces on Core today** (not localhost-bound); reachable directly bypassing Caddy on port 3001 from anywhere that can route to Core's public IP. This is pre-existing Core behavior, not introduced by this migration — replicate `127.0.0.1:3001:3001` on app4 instead and let Caddy be the only door, tightening the surface during the move. - Compose: `/root/docker/uptime-kuma/docker-compose.yml`; volume `./data:/app/data` (relative bind, i.e. `/root/docker/uptime-kuma/data`); has a healthcheck (`curl -f http://localhost:3001`) and resource limits (1 CPU / 512M) already defined — carry these over unchanged. - Data dir: `/root/docker/uptime-kuma/data`, **487 MB**, dominated by `kuma.db` (SQLite, 506,138,624 bytes as of this scan — grows continuously from monitor heartbeat history). - Caddy routes (Core): two site blocks reference it — - `uptimekuma.itpropartner.com { handle /health { respond "OK" 200 } ; reverse_proxy localhost:3001 }` - `status.itpropartner.com { ... handle /api/* { reverse_proxy 127.0.0.1:3001 } ... }` (status page also proxies `/api/msg*` to Transitpin `127.0.0.1:8210` — that portion is out of WAVE 1 scope, do not move it) - Cert: real ACME for `uptimekuma.itpropartner.com`; `status.itpropartner.com` likely shares handling (verify both certs independently at cutover — do not assume one covers the other). - **This is the monitoring tool itself** — its own downtime window must be announced per the migration plan (Phase 3 note), since alerting is blind while it's down. ### Pre-flight checks (read-only, run before touching anything) ``` ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "docker inspect uptime-kuma --format '{{.Image}} {{.State.Status}}'" ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "sqlite3 -readonly /root/docker/uptime-kuma/data/kuma.db 'PRAGMA quick_check;'" # Expected: ok ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "du -sh /root/docker/uptime-kuma/data" # Expected: ~487M (confirm current size before the copy so post-copy size is comparable) ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker info | grep -i 'docker root dir'; ls /var/lib/docker | wc -l" # Confirm app4's Docker data root is populated (known-fixed defect) before pulling any image ``` ### Move commands 1. Recreate the compose file on app4, image pinned by digest, port kept internal-only: ``` ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "mkdir -p /root/docker/uptime-kuma" cat > /tmp/uptime-kuma-compose.yaml <<'EOF' services: uptime-kuma: image: louislam/uptime-kuma@sha256:3d632903e6af34139a37f18055c4f1bfd9b7205ae1138f1e5e8940ddc1d176f9 container_name: uptime-kuma restart: unless-stopped volumes: - ./data:/app/data ports: - "127.0.0.1:3001:3001" healthcheck: test: ["CMD", "curl", "-f", "http://localhost:3001"] interval: 30s timeout: 10s retries: 3 start_period: 10s deploy: resources: limits: cpus: '1' memory: 512M reservations: cpus: '0.25' memory: 128M logging: driver: json-file options: max-size: "10m" max-file: "3" EOF scp -i /root/.ssh/itpp-infra /tmp/uptime-kuma-compose.yaml root@159.195.205.80:/root/docker/uptime-kuma/docker-compose.yml ``` 2. Take a **live-safe SQLite snapshot** on Core (do not `cp` a live WAL DB) and copy it to app4, matching the pattern the existing `core-services-backup.sh` already uses for this exact file: ``` ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "sqlite3 -readonly /root/docker/uptime-kuma/data/kuma.db '.backup /tmp/kuma-migrate.db' && sqlite3 -readonly /tmp/kuma-migrate.db 'PRAGMA quick_check;'" # Expected: ok ``` 3. Announce the monitoring downtime window, then **stop** the Core container (this is the one write-adjacent action required — flagged explicitly since it is the monitoring source of truth; confirm with the requester before executing, this runbook documents the command but does not execute it): ``` ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "docker stop uptime-kuma" ``` 4. Final consistent copy (container stopped, no WAL risk) plus the `docker-tls` subfolder (Kuma's own self-issued monitor certs, if any monitors use them): ``` rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/root/docker/uptime-kuma/data/ root@159.195.205.80:/root/docker/uptime-kuma/data/ ``` 5. Start on app4: ``` ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "cd /root/docker/uptime-kuma && docker compose up -d" ``` 6. Add both Caddy blocks to app4's Caddyfile (mirroring Core, `default_bind` already set globally): ``` uptimekuma.itpropartner.com { handle /health { respond "OK" 200 } reverse_proxy 127.0.0.1:3001 } ``` (`status.itpropartner.com`'s `/api/*` proxy to 3001 is bundled with the Transitpin relay on the same domain — document it here but do not create a standalone `status.itpropartner.com` block until Transitpin's own WAVE migration is scheduled, to avoid a half-built multi-path site block. `caddy validate` app4's file before reload.) ### Verification (exact command + expected output) ``` ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker exec uptime-kuma wget -qO- http://localhost:3001 | head -c 200" # Expected: HTML containing "Uptime Kuma" (or a 200 via curl -o /dev/null -w '%{http_code}') ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sqlite3 -readonly /root/docker/uptime-kuma/data/kuma.db 'SELECT COUNT(*) FROM monitor;'" # Expected: same monitor count as Core's kuma.db at snapshot time — compare with: ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "sqlite3 -readonly /root/docker/uptime-kuma/data/kuma.db 'SELECT COUNT(*) FROM monitor;'" # These two numbers MUST match exactly before cutover. curl --resolve uptimekuma.itpropartner.com:443:159.195.205.80 -sI https://uptimekuma.itpropartner.com/ # Expected (post-DNS-flip, cert issued): HTTP/2 200 ``` ### Rollback 1. `ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker compose -f /root/docker/uptime-kuma/docker-compose.yml down"` (stop app4's copy). 2. `ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "docker start uptime-kuma"` (Core's container and data dir were never deleted — only stopped — so this brings monitoring back exactly where it left off). 3. If DNS was already flipped, revert the `uptimekuma.itpropartner.com` A record to `152.53.192.33`. 4. No data loss: Core's `kuma.db` is untouched by the migration (copy was one-directional, read-only source). --- ## Service 3 of 6 — Prometheus + Grafana + node_exporter monitoring stack Treated as one unit because Prometheus scrapes node_exporter and feeds Grafana's dashboards; moving them together avoids a window where Grafana has no data source. **Grafana's own port/auth finding is broken out separately as Service 4** since it needs an explicit fix, not just a move. ### Current state (Core) - **Prometheus**: container `prometheus` (`prom/prometheus:latest`, `unless-stopped`, `NetworkMode: host`). Config bind-mounted `/root/docker/monitoring/prometheus/prometheus.yml -> /etc/prometheus/prometheus.yml` (ro). Data: named volume `prometheus_data` (`/var/lib/docker/volumes/prometheus_data/_data`, **114.9 MB**). Also bind-mounts `/var/lib/prometheus/textfile` (ro) for host-metrics-exporter's textfile collector. No compose file found — appears to be a manual `docker run` (ambiguity #1 in the inventory doc); reconstruct as a compose file on app4 rather than perpetuating a hand-run container. Listens `*:9090` (host network). `prometheus.yml` (50 lines total) scrape jobs: `prometheus` (self, 127.0.0.1:9090), `mikrotik` (127.0.0.1:9436 via mikrotik-exporter — **stays on Core**, it's scraping Core's local network device), `mikrotik-snmp` (127.0.0.1:9274), `core-telegraf` (127.0.0.1:9273 — **stays on Core**, host-local metrics), `super-search` (172.17.0.1:8899/metrics — Docker bridge IP, will need re-pointing if super-search moves), `node_exporter` (already has 3 targets: `127.0.0.1:9100` labeled `core`, `159.195.205.80:9100` labeled `app4`, `159.195.204.203:9100` labeled `core-bu` — this job already scrapes app4 and core-bu remotely from Core, added during Phase 1 provisioning). - **Grafana**: container `grafana` (`grafana/grafana:11.4.0`, `unless-stopped`, `NetworkMode: host`, no ports block — the image binds directly since it's on host networking). Named volume `grafana_data_final` (`/var/lib/docker/volumes/grafana_data_final/_data`, **~15 MB**) -> `/var/lib/grafana`. Two orphaned volumes exist alongside it on Core (`grafana_data` 50 MB, `grafana_data_v3` 14.6 MB) — **do not migrate these**, they are stale, only `grafana_data_final` is live per the running container's mount. Config: `/root/docker/monitoring/grafana/grafana.ini`, referenced via env vars (`GF_PATHS_CONFIG=/etc/grafana/grafana.ini`) — confirm at move time whether it's a bind mount or baked into the volume; treat the working copy at `/root/docker/monitoring/grafana/grafana.ini` as authoritative source of truth for env/ini settings. - **node_exporter**: NOT a container — a systemd unit (`/etc/systemd/system/node_exporter.service`, `ExecStart=/usr/local/bin/node_exporter`, no flags, `Restart=unless-stopped`). Binds `*:9100`. **app4 already runs its own node_exporter** (systemd, active, listening `*:9100`, confirmed `ss -ltnp | grep 9100` shows `node_exporter` on app4 right now) — this was stood up in Phase 1 provisioning and is already scraped by Core's Prometheus (see the `app4` label above). **node_exporter is therefore ALREADY DONE on app4 — no move action needed for this piece.** - Caddy: **no Caddy site block routes to Prometheus or Grafana today.** Neither `:9090` nor `:3002` appears anywhere in Core's 49 Caddy site blocks. Access today is by direct IP:port only. - Cert: none — neither service is behind TLS today. - Firewall (Core `ufw status numbered`): rule `[7] 8899/tcp ALLOW IN 172.17.0.0/16` (Docker-bridge-only, for super-search's own metrics scrape — not customer-facing) and `[8] 3002/tcp ALLOW IN 100.64.0.0/10` (Tailscale-range-only for Grafana) exist, but **UFW is a second gate behind the actual bind** — see Service 4, because Grafana's `*:3002` listen and this UFW rule together still leave a gap, which is the whole point of the Service 4 finding. ### Pre-flight checks ``` ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "docker inspect prometheus --format '{{.Config.Image}} {{.State.Status}}'; docker inspect grafana --format '{{.Config.Image}} {{.State.Status}}'" ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "du -sh /var/lib/docker/volumes/prometheus_data/_data /var/lib/docker/volumes/grafana_data_final/_data" ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl is-active node_exporter; curl -s http://127.0.0.1:9100/metrics | head -1" # Expected: active / a line starting "# HELP go_gc_duration_seconds" ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "curl -s 'http://127.0.0.1:9090/api/v1/query?query=up' | python3 -m json.tool | head -20" # Baseline: capture current up{} series before the move for a before/after diff ``` ### Move commands 1. Create target dirs on app4: ``` ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "mkdir -p /root/docker/monitoring/prometheus /root/docker/monitoring/grafana" ``` 2. Copy configs (read-only source copy from Core, nothing stopped yet): ``` scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/monitoring/prometheus/prometheus.yml /tmp/prometheus.yml scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/monitoring/grafana/grafana.ini /tmp/grafana.ini ``` 3. **Edit the copy** (never Core's file) before pushing to app4: remove the `mikrotik`, `mikrotik-snmp`, and `core-telegraf` jobs (Core-local, stay on Core) from `/tmp/prometheus.yml`; re-point `super-search`'s target from `172.17.0.1:8899` to wherever Super Search lands (Service 6 — if it also moves to app4 in this wave, target becomes `127.0.0.1:8899`; if it stays put, keep scraping Core's public IP with a dedicated firewall allow, decided at Service 6 time). Push: ``` scp -i /root/.ssh/itpp-infra /tmp/prometheus.yml root@159.195.205.80:/root/docker/monitoring/prometheus/prometheus.yml scp -i /root/.ssh/itpp-infra /tmp/grafana.ini root@159.195.205.80:/root/docker/monitoring/grafana/grafana.ini ``` 4. rsync the live volume data in two passes — a warm first pass while Core's containers still run, then a final short-stop pass for consistency (same pattern as the Uptime Kuma DB): ``` rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/var/lib/docker/volumes/prometheus_data/_data/ /tmp/prom-data-staging/ rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/var/lib/docker/volumes/grafana_data_final/_data/ /tmp/grafana-data-staging/ # announce downtime, then: ssh ... root@152.53.192.33 "docker stop prometheus grafana" rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/var/lib/docker/volumes/prometheus_data/_data/ /tmp/prom-data-staging/ rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/var/lib/docker/volumes/grafana_data_final/_data/ /tmp/grafana-data-staging/ scp -r -i /root/.ssh/itpp-infra /tmp/prom-data-staging root@159.195.205.80:/tmp/ scp -r -i /root/.ssh/itpp-infra /tmp/grafana-data-staging root@159.195.205.80:/tmp/ ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker volume create prometheus_data && docker run --rm -v prometheus_data:/dest -v /tmp/prom-data-staging:/src alpine cp -a /src/. /dest/" ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker volume create grafana_data_final && docker run --rm -v grafana_data_final:/dest -v /tmp/grafana-data-staging:/src alpine cp -a /src/. /dest/" ``` 5. Write `/root/docker/monitoring/docker-compose.yml` on app4, images pinned by digest (resolve fresh: `docker inspect prometheus --format '{{index .RepoDigests 0}}'` on Core for Prometheus; Grafana's tag `11.4.0` is already a pin — resolve its digest the same way), `network_mode: host` preserved for Prometheus to match Core's scrape behavior; Grafana's networking is changed — see Service 4. 6. `docker compose up -d` on app4. ### Verification ``` ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s http://127.0.0.1:9090/-/healthy" # Expected: "Prometheus Server is Healthy." ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s 'http://127.0.0.1:9090/api/v1/query?query=up' | python3 -m json.tool" # Expected: same target set (minus mikrotik/telegraf/core-only jobs) all reporting value "1" ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3002/api/health" # Expected: 200 ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s http://127.0.0.1:3002/api/health" # Expected JSON containing "database":"ok","version":"11.4.0" — confirms grafana.db loaded correctly ``` Series-count parity check (compare Core's pre-stop snapshot vs app4's loaded volume before deleting anything on Core): ``` ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s http://127.0.0.1:9090/api/v1/label/__name__/values | python3 -c 'import json,sys; print(len(json.load(sys.stdin)[\"data\"]))'" # Compare metric-name count to the same query run on Core before stop ``` ### Rollback 1. `ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker compose -f /root/docker/monitoring/docker-compose.yml down"`. 2. `ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "docker start prometheus grafana"` — Core's volumes and config were only read from, never deleted; original containers restart with their original state intact. 3. No DNS is involved (no Caddy route exists either direction today), so rollback is purely stop-app4/ start-Core with no propagation delay. 4. Any staging copy edited on the operator's own box (`/tmp/prometheus.yml`) is disposable; Core's live `prometheus.yml` was never touched, per the read-only constraint on Core. --- ## Service 4 of 6 — Grafana public-bind finding (`*:3002`, unauthenticated, no Caddy/Tailscale gate) This is called out separately from Service 3 because it is a **security finding to fix during the move**, not just a relocation. ### The finding, precisely - Grafana's container runs `NetworkMode: host` and the Grafana process itself binds `*:3002` (confirmed via `ss -ltnp`: `LISTEN 0 4096 *:3002 *:* users:(("grafana",pid=2906,fd=15))`) — i.e. **every** interface on Core, including its public IPv4/IPv6, not just loopback or the Tailscale interface. - It is **not** behind any Caddy site block (grepped all 49 blocks in Core's Caddyfile — zero mention of `3002` or a Grafana upstream). - The only mitigation in place is a UFW rule restricting **inbound port 3002 to the Tailscale CGNAT range** (`100.64.0.0/10`) — `ufw status numbered` line `[8] 3002/tcp ALLOW IN 100.64.0.0/10`. UFW's default policy is `deny (incoming)`, so absent that specific allow rule, 3002 would in fact be closed by default — **but this means the ONLY thing standing between Grafana and the open internet is one UFW line**, with no defense in depth (no app-level auth beyond Grafana's own login, no Caddy layer, no Tailscale-only bind at the app level). - Compounding it: Grafana's `grafana.ini` `[security]` section has `admin_user = admin` / `admin_password = admin` / `disable_initial_admin_creation = false` — **the default admin credentials were never changed**. If the UFW rule is ever misconfigured, misapplied to the wrong interface, or a firewall reload race drops it (a known class of bug on reboot ordering), the admin/admin login is directly exposed to anyone who reaches that interface. ### Fix to apply on app4 (do NOT attempt this on Core — Core is read-only for this task) 1. Do not repeat the `NetworkMode: host` + open-bind pattern on app4. In the Service 3 compose file, add an explicit port mapping and drop host networking for Grafana specifically (Prometheus can keep host networking since it only needs outbound scrape access; Grafana only needs to reach Prometheus on the same host, no host networking required): ```yaml grafana: image: grafana/grafana@sha256: ports: - "127.0.0.1:3002:3000" # loopback only; Caddy or Tailscale serves it externally ``` 2. Add a real Caddy site block on app4 gating it behind HTTP basic auth (mirroring the existing pattern used for `internal.debtrecoveryexperts.com`'s `basic_auth` block in Core's Caddyfile): ``` grafana.itpropartner.com { basic_auth { admin } reverse_proxy 127.0.0.1:3002 } ``` (Generate the hash with `caddy hash-password` — never hand-write a bcrypt hash.) 3. Change the default admin credentials in `grafana.ini` (or via `GF_SECURITY_ADMIN_PASSWORD` env var) on app4's first boot — do not carry `admin/admin` forward. This must happen before app4's Grafana is reachable by anything but loopback. 4. Confirm `ufw status` on app4 does not carry forward the Core Tailscale-3002 rule as the *only* gate — Caddy basic_auth is now gate 1; Tailscale-only access can remain as an additional, not sole, control. ### Verification ``` ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "ss -ltnp | grep 3002" # Expected: 127.0.0.1:3002 only — NOT *:3002 or 0.0.0.0:3002 curl -s -o /dev/null -w '%{http_code}\n' http://159.195.205.80:3002/ # Expected: connection refused / timeout (nothing listening on the public interface) curl -s -o /dev/null -w '%{http_code}\n' https://grafana.itpropartner.com/ # Expected: 401 (basic_auth challenge) before credentials, 200 after ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "grep -E 'admin_password' /root/docker/monitoring/grafana/grafana.ini" # Expected: NOT the literal string "admin" — confirm it was actually changed ``` ### Rollback This fix only touches app4, a not-yet-cutover host; Core is never modified. Rollback is simply: do not flip DNS/Tailscale access to app4's Grafana until the verification above passes. Core's Grafana keeps running with its existing (already-known-weak) posture unchanged — a pre-existing condition to flag to Germaine regardless of migration timing, not something this runbook worsens or is required to fix on Core. --- ## Service 5 of 6 — PRY (unified OSINT search backend) ### Current state (Core) - Not a container — a systemd unit: `pry.service` (`/etc/systemd/system/pry.service`, `Type=simple`, `User=root`, `WorkingDirectory=/root/docker/pry`, `ExecStart=/root/docker/pry/venv/bin/python /root/docker/pry/server.py`, `EnvironmentFile=/root/.hermes/.env`, `Restart=always`, `ProtectSystem=full`, `ProtectHome=read-only`, `ReadWritePaths=/root/docker/pry`, `ReadOnlyPaths=/root/.hermes/.env /root/.config/himalaya/shonuff.pass`). - Port: `127.0.0.1:8905`, confirmed via `ss -ltnp` — loopback only, not directly internet-reachable. - App dir: `/root/docker/pry`, **66 MB total** (mostly its own `venv`). Python 3.13.5 venv, deps from `requirements.txt` (`fastapi>=0.109.0`, `uvicorn[standard]>=0.27.0`, `httpx>=0.26.0`, `python-dotenv>=1.0.0`, `aiosmtplib>=3.0.0` — 5 loose lines, no pinned freeze file, so re-resolve at install time rather than assuming exact matching versions). - Data/DB: **none found** — no `.db`/`.sqlite*` file in `/root/docker/pry`. PRY is stateless app logic; its only persistent asset is its git-tracked source (`.git` present) plus a private `Caddyfile-new` (mode 600, 4842 bytes — appears to be a staged/unused Caddy config someone drafted but never applied; carry it over for reference but do not activate it blindly). - Secrets: `EnvironmentFile=/root/.hermes/.env` — shared secrets file across many services; only PRY-relevant keys need to be extracted/copied to app4's own `.env`, not the whole file (per the inventory doc's finding #7 on live secrets requiring manual, non-rsync transfer). - Caddy route (Core): `http://pry.iamgmb.com { @api path /api/* ; handle @api { reverse_proxy 127.0.0.1:8905 } ; handle { root * /var/www/pry ; try_files ... ; file_server } }` — **HTTP only, no TLS** (`http://` scheme explicit in the Caddyfile block header), and it's a split route: `/api/*` goes to PRY's backend, everything else is a static SPA served from `/var/www/pry` on Core. The static half is out of scope for a backend-only WAVE 1 move — moving PRY's backend without also moving `/var/www/pry`'s static files would break the frontend, so **both must move together** even though the task frames this as one backend service. - Cert: none (HTTP-only site today). ### Pre-flight checks ``` ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "systemctl is-active pry.service; ss -ltnp | grep 8905" ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8905/api/health || curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8905/" # Record baseline response code before moving ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "du -sh /var/www/pry" ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "grep -c '^[A-Z_]*=' /root/.hermes/.env" # Confirms the shared secrets file's line count so a partial extraction can be sanity-checked for completeness ``` ### Move commands 1. Copy the app dir (excluding the venv, which will be rebuilt fresh on app4 to avoid glibc/arch drift): ``` rsync -avz -e "ssh -i /root/.ssh/itpp-infra" --exclude 'venv' --exclude '__pycache__' root@152.53.192.33:/root/docker/pry/ root@159.195.205.80:/root/docker/pry/ ``` 2. Copy the static frontend: ``` rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/var/www/pry/ root@159.195.205.80:/var/www/pry/ ``` 3. Rebuild the venv on app4: ``` ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "cd /root/docker/pry && python3 -m venv venv && ./venv/bin/pip install -r requirements.txt" ``` 4. Manually extract PRY-relevant keys from Core's `/root/.hermes/.env` (grep for the specific vars `server.py` actually reads — do this by inspecting `os.getenv`/`os.environ` calls in `server.py`, not by copying the whole file) into a new, minimal `/root/.hermes/.env` on app4, or a dedicated `/root/docker/pry/.env` + adjust the systemd unit's `EnvironmentFile=` path. This is a manual, secure step — never rsync the shared `.env` wholesale (matches the inventory doc's explicit warning on secret files). 5. Create the systemd unit on app4 (same content as Core's, `WorkingDirectory`/`ExecStart` paths already match since both boxes use `/root/docker/pry`): ``` scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/pry/pry.service /tmp/pry.service # edit EnvironmentFile= path if app4 uses a dedicated .env, then: scp -i /root/.ssh/itpp-infra /tmp/pry.service root@159.195.205.80:/etc/systemd/system/pry.service ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl daemon-reload && systemctl enable --now pry.service" ``` 6. Add the Caddy block to app4, upgraded to real TLS this time (Caddy on app4 will ACME-issue since it's a real domain — no reason to keep it HTTP-only post-move): ``` pry.iamgmb.com { @api path /api/* handle @api { reverse_proxy 127.0.0.1:8905 } handle { root * /var/www/pry try_files {path} {path}.html /index.html file_server } } ``` ### Verification ``` ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl is-active pry.service" # Expected: active ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8905/" # Expected: same status code as the Core baseline captured in pre-flight curl --resolve pry.iamgmb.com:443:159.195.205.80 -sI https://pry.iamgmb.com/ # Expected (post-DNS-flip, cert issued): HTTP/2 200 ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "journalctl -u pry.service -n 20 --no-pager" # Expected: no tracebacks, normal startup log lines matching Core's own startup pattern ``` ### Rollback 1. `ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl stop pry.service"`. 2. Core's `pry.service` was never touched — it is still `active` throughout, so no restart action needed on Core. 3. If DNS was flipped, revert `pry.iamgmb.com` to `152.53.192.33` (note: this domain has no `A` vs `CNAME` redirect complication, it's a direct A record per the plan's DNS checklist). 4. No data to restore (stateless service, no DB). --- ## Service 6 of 6 — OSINT Person MCP (`:8902`) + Super Search MCP (`:8899`) These are grouped because OSINT Person MCP imports Super Search's own module directly (`from super_search import web_search as ss_search, web_extract as ss_extract` in `osint-person-mcp/server.py`) — they have a **code-level dependency**, not just a runtime one, so they must move together or the import breaks. ### Current state (Core) **OSINT Person MCP:** - systemd unit `osint-person.service` (`After=network.target super-search.service` — systemd-level ordering dependency confirming the two are coupled), `WorkingDirectory=/root/docker/osint-person-mcp`, `ExecStart=/root/docker/osint-person-mcp/venv/bin/python3 /root/docker/osint-person-mcp/server.py`, `EnvironmentFile=/root/.hermes/.env`, `Restart=always`. - Port: `127.0.0.1:8902` (confirmed `ss -ltnp`), used as `http://127.0.0.1:8902/mcp` per Hermes' `config.yaml` MCP registration — loopback only. - App dir: `/root/docker/osint-person-mcp`, **314 MB** (Python 3.13.5 venv-heavy). Source files include `server.py` (54,253 bytes, modified same-day as this scan — active development), `reconcile.py`, `super_search.py` (a **local copy** of Super Search's search/extract functions, imported directly — this is the code coupling), `telemetry.py`, plus backup files (`server.py.bak-*`) and a `tests/` dir. No standalone requirements file was found in the directory listing — dependencies live only in the existing `venv`; capture `venv/bin/pip freeze > requirements.txt` on Core before rebuilding on app4, do not guess versions. - Data/DB: none found (no `.db` file); it's a stateless MCP tool server backed by live web calls. **Super Search MCP:** - systemd unit `super-search.service` (`WorkingDirectory=/root/docker/super-search`, `ExecStartPre=/bin/bash /root/docker/super-search/super-search-port-guard.sh` — a pre-start guard script, copy this too, it likely checks for a stale port bind before starting — `ExecStart=.../venv/bin/python3 .../server.py`, `EnvironmentFile=/root/.hermes/.env`, `Restart=always`, `StartLimitBurst=5` over 300s). - Port: `0.0.0.0:8899` — confirmed **bound to all interfaces on Core**, mitigated only by `ufw status numbered` rule `[7] 8899/tcp ALLOW IN 172.17.0.0/16` (Docker-bridge-range only — i.e. only containers on Core's own `docker0`/bridge network can reach it externally-looking-in; this is intentional since Prometheus's `super-search` scrape job targets `172.17.0.1:8899`, the bridge gateway IP, meaning Prometheus itself runs in a container reaching it via the bridge). Registered in Hermes as `http://127.0.0.1:8899/mcp`. - App dir: `/root/docker/super-search`, **1.1 GB** (largest of the six — heavy venv, Python 3.13.5). Has a pinned `requirements.freeze.txt` (167 lines) — **use this file directly on app4**, it's already a proper pin, unlike PRY's loose `requirements.txt` or OSINT Person's missing one. - Data/DB: none found — stateless, backed by live search-provider calls (SearXNG on Core, Exa, Brave, etc. per the skill's provider list); note SearXNG itself (`127.0.0.1:8888`) **stays on Core** per the inventory doc, meaning **if Super Search moves to app4, its SearXNG provider call becomes a cross-Atlantic Core→app4→Core round trip (~100ms each way per the migration plan's RTT finding) unless SearXNG is also exposed to app4**, which is out of WAVE 1 scope. Recommend either: (a) leave Super Search on Core for now despite the task's inclusion of it here, since its own migration-plan classification already says "stays-on-Core (Hermes MCP)" in the inventory doc (contradicting this task's WAVE 1 list — flag this conflict to the requester explicitly before executing), or (b) accept the added latency for one SearXNG-backed provider path while other providers (Exa, Brave, Serper, Tavily) call out directly with no Core round-trip. **This is a real discrepancy between the source inventory and this task's WAVE 1 scope — documented here, not silently resolved.** - Caddy: **neither service has a Caddy route today.** Both are pure loopback/bridge MCP servers consumed only by Hermes' own `config.yaml` `url: http://127.0.0.1:PORT/mcp` entries — there is no public hostname for either in the 49 scanned Caddy blocks. - Cert: none (no public exposure today; loopback-only by design). ### Pre-flight checks ``` ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "systemctl is-active osint-person.service super-search.service" ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8902/mcp; curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8899/mcp" # Record baseline codes (MCP endpoints often 4xx on a bare GET without proper headers — record whatever Core # returns today as the comparison target, don't assume 200) ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "/root/docker/osint-person-mcp/venv/bin/pip freeze > /tmp/osint-person-freeze.txt; wc -l /tmp/osint-person-freeze.txt" ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "du -sh /root/docker/osint-person-mcp /root/docker/super-search" ``` ### Move commands 1. Freeze OSINT Person's deps (not previously pinned) and pull both freeze files: ``` scp -i /root/.ssh/itpp-infra root@152.53.192.33:/tmp/osint-person-freeze.txt /tmp/ scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/super-search/requirements.freeze.txt /tmp/ ``` 2. Copy source (excluding venv/pycache/backup cruft — the `.bak-*` files are large and not needed): ``` rsync -avz -e "ssh -i /root/.ssh/itpp-infra" --exclude 'venv' --exclude '__pycache__' --exclude '*.bak*' root@152.53.192.33:/root/docker/osint-person-mcp/ root@159.195.205.80:/root/docker/osint-person-mcp/ rsync -avz -e "ssh -i /root/.ssh/itpp-infra" --exclude 'venv' --exclude '__pycache__' --exclude '*.bak*' root@152.53.192.33:/root/docker/super-search/ root@159.195.205.80:/root/docker/super-search/ ``` 3. Rebuild both venvs on app4 from the frozen requirements: ``` ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "cd /root/docker/osint-person-mcp && python3 -m venv venv && ./venv/bin/pip install -r /tmp/osint-person-freeze.txt" ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "cd /root/docker/super-search && python3 -m venv venv && ./venv/bin/pip install -r requirements.freeze.txt" ``` 4. Extract only the OSINT/Super-Search-relevant keys from Core's `/root/.hermes/.env` into app4's own `.hermes/.env` (same manual-secret-transfer rule as PRY — do not rsync the whole file). 5. Copy systemd units and the port-guard script, adjusting nothing but confirming paths already match (`/root/docker/...` is identical on both boxes): ``` scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/osint-person-mcp/osint-person.service /tmp/ scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/super-search/super-search.service /tmp/ scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/super-search/super-search-port-guard.sh /tmp/ scp -i /root/.ssh/itpp-infra /tmp/osint-person.service /tmp/super-search.service root@159.195.205.80:/etc/systemd/system/ scp -i /root/.ssh/itpp-infra /tmp/super-search-port-guard.sh root@159.195.205.80:/root/docker/super-search/ ``` 6. **Fix Super Search's bind before enabling** — do not carry `0.0.0.0:8899` forward. If `server.py` hardcodes the bind host, patch app4's copy to `127.0.0.1:8899` (matching OSINT Person's already-safe loopback pattern) unless something on app4 genuinely needs the Docker-bridge-range access Core's UFW rule was protecting (confirm this is even needed on app4 — app4 doesn't run the same Prometheus container-bridge topology as Core unless Service 3 also lands there with the same `172.17.0.1` scrape pattern). 7. Enable both, in dependency order (Super Search first, since OSINT Person's own unit declares `After=... super-search.service`): ``` ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl daemon-reload && systemctl enable --now super-search.service && sleep 3 && systemctl enable --now osint-person.service" ``` 8. Update Hermes' `config.yaml` MCP `url:` entries — but only **after** confirming which box's Hermes actually calls these tools. If it's Core's own Hermes gateway consuming these via `127.0.0.1`, moving the servers to app4 means Core's config must change to `http://159.195.205.80:8902/mcp` / `http://159.195.205.80:8899/mcp`, which reintroduces exactly the cross-Atlantic per-call latency the migration plan's Section 3.1 design rule warns against ("app4 must not depend on Core at request time" — the inverse also applies: Core's live Hermes gateway should not depend on app4 at request time for every-tool-call-latency reasons). **This is a design question for the requester, not a call this runbook makes** — flagged here rather than silently deciding to change `config.yaml`. ### Verification ``` ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl is-active osint-person.service super-search.service" # Expected: active / active ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "ss -ltnp | grep -E ':(8899|8902)\b'" # Expected: 127.0.0.1:8899 and 127.0.0.1:8902 — NOT 0.0.0.0:8899 (confirms the bind fix from step 6 landed) ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8902/mcp; curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8899/mcp" # Expected: same status codes as the Core baseline captured in pre-flight ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "journalctl -u osint-person.service -u super-search.service -n 30 --no-pager" # Expected: no import errors (specifically watch for "ModuleNotFoundError: super_search" — the code-coupling # risk called out above — and no missing-env-var tracebacks) ``` ### Rollback 1. `ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl stop osint-person.service super-search.service"`. 2. Core's two units were never stopped or restarted — they remain `active` throughout, so Hermes' existing `127.0.0.1` MCP URLs keep working with zero interruption regardless of app4's state. 3. If `config.yaml` was changed (step 8), revert the `url:` fields back to `http://127.0.0.1:8902/mcp` and `http://127.0.0.1:8899/mcp` — this requires whichever Hermes profile/gateway consumes them to reload its MCP config (out of scope for this SSH-only runbook; note it as a manual follow-up). 4. No data to restore (both stateless). --- ## Section 8 — Backup coverage gap: `/root/docker` missing from app4's `root-essentials-backup.sh` ### The gap app4's `/root/root-essentials-backup.sh` (v2.0.0, 2026-09-11) builds its include list from this loop: ```bash for p in .hermes/config.yaml .hermes/.env .hermes/skills .hermes/scripts \ .hermes/references .hermes/profiles .hermes/cron .hermes/memories \ .hermes/DR-PLAN.md .hermes/data .hermes/cache/delegation \ .ssh .aws .config/himalaya \ shark-game projects ; do [ -e "$p" ] && ARGS+=("$p") done ``` `/root/docker` (relative: `docker`) is **not in this list**. Every compose stack this runbook lands on app4 — Uptime Kuma's `docker-compose.yml` + its data dir path, the monitoring stack's compose file, PRY's app dir if placed under `/root/docker` (it is), OSINT Person MCP, Super Search — is therefore **entirely invisible** to the nightly 04:45 ET run, even though the script's own `tar` excludes `*.db`/`*.db-shm`/`*.db-wal` regardless (so Uptime Kuma's `kuma.db` and any future SQLite files still need their own `sqlite3 .backup` job even after this fix — this patch closes the "missing compose/config/source" half of the gap, not the "missing DB" half, which is already flagged as a separate Phase 4/5 acceptance item in the migration plan). ### Exact patch ``` ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sed -n '55,60p' /root/root-essentials-backup.sh" # Confirm current line content before editing, since this is a live production script: # .ssh .aws .config/himalaya \ # shark-game projects ; do ``` Change: ```diff - .ssh .aws .config/himalaya \ - shark-game projects ; do + .ssh .aws .config/himalaya \ + shark-game projects docker ; do ``` Apply with `patch`/`sed` on app4 (do NOT run this against Core — Core is read-only for this task, and Core's own backup script coverage is a separate, already-tracked item per the inventory doc's Section on backup gaps): ``` ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sed -i 's/shark-game projects ; do/shark-game projects docker ; do/' /root/root-essentials-backup.sh" ``` ### Verify the patch landed correctly ``` ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "grep -n 'shark-game projects' /root/root-essentials-backup.sh" # Expected: shark-game projects docker ; do ``` ### Restore test (prove the new coverage actually works, not just that the line changed) ``` ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "bash /root/root-essentials-backup.sh 2>&1 | tail -20" # Expected final lines: "[...] Tarball: bytes" (larger than the pre-patch baseline by roughly the size of # /root/docker's non-.db content — compare against a du -sh /root/docker taken beforehand), then # "[...] content check OK - N entries, 3/3 key essentials present", then "Uploading...", then # "Backup verified - archive is valid", then "Done". ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "aws s3 cp s3://hermes-vps-backups/root-backup/\$(hostname -s)/root-essentials-\$(date +%F).tar.gz /tmp/restore-test.tar.gz --endpoint-url https://s3.us-east-1.wasabisys.com --quiet && tar tzf /tmp/restore-test.tar.gz | grep '^docker/' | head -20" # Expected: a non-empty listing of docker//... paths — proves /root/docker is now actually inside the # uploaded archive, not just inside the include-list source, and that the round-trip download+list works. ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "rm -f /tmp/restore-test.tar.gz" ``` ### Rollback (if the patched script misbehaves — e.g. archive size becomes unexpectedly huge from a stray venv) ``` ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sed -i 's/shark-game projects docker ; do/shark-game projects ; do/' /root/root-essentials-backup.sh" ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "grep -n 'shark-game projects' /root/root-essentials-backup.sh" # Expected: shark-game projects ; do (back to original) ``` If venv bloat is the issue (Super Search alone is 1.1 GB, mostly venv), prefer excluding venvs specifically over reverting the whole line: ``` # Add to the tar invocation's --exclude list (near the existing --exclude='*.db' flags): --exclude='docker/*/venv' --exclude='docker/*/*/venv' ``` --- ## Summary table | # | Service | Status | Downtime needed | DNS flip needed | New security work | |---|---|---|---|---|---| | 1 | microbin | **DONE** (app layer), DNS flip pending | None (data dir was empty) | Yes — share.itpropartner.com | None | | 2 | Uptime Kuma | Runbook ready | Yes — brief, for final DB consistency | Yes — uptimekuma.itpropartner.com | Tighten `0.0.0.0:3001` → `127.0.0.1:3001` | | 3 | Prometheus+Grafana+node_exporter | Runbook ready (node_exporter already on app4) | Yes — brief, for volume consistency | None (no Caddy route today) | See Service 4 | | 4 | Grafana public-bind finding | Fix defined | N/A (fix applied pre-cutover) | Optional — grafana.itpropartner.com if exposed | Loopback bind + Caddy basic_auth + change admin/admin | | 5 | PRY | Runbook ready | None (stateless) | Yes — pry.iamgmb.com | Upgrade HTTP-only → real TLS | | 6 | OSINT Person MCP + Super Search MCP | Runbook ready, **scope conflict flagged** | None (stateless) | None (loopback/bridge-only, MCP-internal) | Fix `0.0.0.0:8899` → `127.0.0.1:8899`; resolve Core-Hermes-vs-app4-MCP latency question first | **Open items for the requester before executing Services 3/6:** (a) Service 3's Prometheus scrape config needs a decision on where Super Search ends up before it can be finalized; (b) Service 6 conflicts with the source inventory's own classification of Super Search as "stays-on-Core" — confirm intent before executing.