ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
58de741d94eb57531966e7882e740947fb006afa
itpp-infrastructure/docs/infrastructure/migration-runbook-wave1-internal-2026-09-15.md
T

49 KiB
Raw Blame History

Migration Runbook — WAVE 1 (Internal / Low-Customer-Impact Services)

Date: 2026-09-15 Scope: Core (152.53.192.33) → app4 (159.195.205.80), exactly 6 services:

  1. microbin (:8260, share.itpropartner.com) — ALREADY MIGRATED, DONE
  2. Uptime Kuma
  3. Prometheus + Grafana + node_exporter monitoring stack
  4. Grafana specifically (unauthenticated public-bind finding)
  5. PRY
  6. OSINT API / OSINT Person MCP (:8902) + Super Search MCP (:8899)

Reference source docs: core-service-inventory-2026-09-15.md, migration-plan-app4-core-bu-2026-09-15.md. Production hosts are READ-ONLY for this runbook's author. All facts below were captured via read-only SSH (ss -ltnp, docker inspect, systemctl cat, cat of configs) on 2026-09-15. No service was started, stopped, or restarted to produce this document. hermes-maintenance was never invoked.

SSH access: ssh -i /root/.ssh/itpp-infra root@<ip> (key already present on this box).

Proven pattern (established this session on microbin, reference for all 6 services):

  1. Caddy v2.11.4 on app4, pinned to Core's exact version, from the caddy-stable apt repo.
  2. Service runs via /root/docker/<service>/compose.yaml (or docker-compose.yml), image pinned by digest, not :latest.
  3. app4's global Caddy option block sets default_bind 159.195.205.80 so a new site block never hijacks another listener (e.g. Tailscale's own :443).
  4. ufw active on app4: 22/80/443 allowed, default deny incoming otherwise.
  5. Cutover = flip DNS at the authoritative panel (SiteGround for itpropartner.com, manual, no API — Germaine applies these), then Caddy on app4 auto-issues the cert via ACME http-01 on first request.
  6. Verify side-by-side with curl --resolve <domain>:443:159.195.205.80 https://<domain>/ before the DNS flip, so the app layer is proven independent of DNS.

Known defect already fixed (do not re-diagnose): both app4 and core-bu shipped with an empty Docker data root (/var/lib/docker had zero subdirectories → every docker pull/run failed with GetImageBlob / overlay2: no such file or directory). Fixed by stopping docker, rm -rf /var/lib/docker, restarting docker, and proving with docker run --rm hello-world. Both boxes now run containers normally. If a pull/run fails with an overlay2 ENOENT-style error on either box in the steps below, this is the known cause — check docker info | grep -i 'docker root dir' and ls /var/lib/docker before assuming a new bug.

Known backup gap this runbook must close (Section 8): app4's root-essentials-backup.sh include list covers .hermes .ssh .aws shark-game projects (repo-root-relative) plus /etc/caddy/Caddyfile, but not /root/docker. Every compose stack landed on app4 by this runbook (Uptime Kuma, monitoring stack, PRY if containerized, OSINT/Super Search if containerized) is therefore invisible to the nightly backup until Section 8 is applied.

Service 1 of 6 — microbin (:8260, share.itpropartner.com) — STATUS: DONE

This is the proven reference pattern; it is documented here as the template the other 5 follow, not as new work.

Current state (Core)

  • Container: microbin (danielszabo99/microbin:latest, image digest sha256:6660e5ccad0d764fa3c0032464ffb8f4b4f28c92a2eb9e39202b94cdc5b68909), unless-stopped.
  • Port: 127.0.0.1:8260 -> 8080 (container), not exposed externally on Core.
  • Compose: /opt/microbin/docker-compose.yml.
  • Data dir: /opt/microbin/data (bind mount -> /app/pasta_data), 4.0 KB — no DB engine, flat files only.
  • Caddy route (Core, /etc/caddy/Caddyfile line ~178): share.itpropartner.com { reverse_proxy 127.0.0.1:8260 }.
  • Cert: real ACME (Let's Encrypt) issued to Core.

What was actually done (already executed, verified in this session)

  1. Installed Caddy v2.11.4 on app4 from the caddy-stable apt repo (matches Core's caddy version output exactly: v2.11.4 h1:XKxkMTgNSizEvKG6QHue6cAsFOteU2qA61w2tKkCWi0= on both hosts).
  2. Created /root/docker/microbin/docker-compose.yml on app4, pinning the image by digest (danielszabo99/microbin@sha256:6660e5ccad0d764fa3c0032464ffb8f4b4f28c92a2eb9e39202b94cdc5b68909), same env vars as Core's compose, port bound 127.0.0.1:8260:8080.
  3. docker compose up -d on app4 — container microbin now Up, verified via docker ps.
  4. Wrote app4's /etc/caddy/Caddyfile: 
    {
    default_bind 159.195.205.80
    email info@itpropartner.com
}
# -- MicroBin Share (migrated from Core 2026-09-15, phase 3 of the app4 migration) --
share.itpropartner.com {
    reverse_proxy 127.0.0.1:8260
    encode gzip
}
  5. Data dir was NOT rsynced from Core because Core's /opt/microbin/data was empty (4.0 KB, no user pastes) at scan time — a fresh empty dir on app4 is equivalent. If Core's data dir is non-empty at your actual cutover time, rsync it first: rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/opt/microbin/data/ root@159.195.205.80:/root/docker/microbin/data/ and stop the Core container for the duration of the rsync to avoid a torn copy (microbin uses flat files, not a WAL DB, so this is a low-risk copy).

Verification (already run, real output)

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker ps --format '{{.Names}}\t{{.Ports}}\t{{.Status}}'"
# -> microbin   127.0.0.1:8260->8080/tcp   Up

App-layer check (curl from app4 itself, since DNS has not flipped yet):

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8260/"
# Expected: 200

Confirmed: direct curl to app4's microbin returns 200. App layer is fully staged and correct.

What is NOT done yet (pending, not part of "DONE" claim)

  • DNS flip is pending. itpropartner.com is on SiteGround, manual panel, no API — Germaine must change the share.itpropartner.com A record from 152.53.192.33 to 159.195.205.80.
  • Cert on app4 has not been issued yet (Caddy issues on first real HTTPS hit post-DNS-flip via http-01).
  • No S3 backup job exists yet for app4's /root/docker/microbin/data (see Section 8 — same gap applies here).

Rollback (if flip has NOT happened — current state)

Nothing to roll back: Core's container and Caddy route are untouched and still serving share.itpropartner.com live. app4's copy is inert until DNS points at it.

Rollback (if DNS HAS been flipped and a problem appears)

  1. Revert the share.itpropartner.com A record at SiteGround back to 152.53.192.33.
  2. TTL propagation is the only wait; Core's container was never stopped, so it answers immediately once DNS re-resolves.
  3. Do not touch app4's container — leave it running for retry.

Service 2 of 6 — Uptime Kuma

Current state (Core)

  • Container: uptime-kuma (louislam/uptime-kuma:latest, image digest sha256:3d632903e6af34139a37f18055c4f1bfd9b7205ae1138f1e5e8940ddc1d176f9, host layer digest f48d816cb7460cd3b7bb15ed393968b0ae0da4c690443b778b6a5db6b09f527e), unless-stopped.
  • Port: 0.0.0.0:3001 -> 3001 — exposed on all interfaces on Core today (not localhost-bound); reachable directly bypassing Caddy on port 3001 from anywhere that can route to Core's public IP. This is pre-existing Core behavior, not introduced by this migration — replicate 127.0.0.1:3001:3001 on app4 instead and let Caddy be the only door, tightening the surface during the move.
  • Compose: /root/docker/uptime-kuma/docker-compose.yml; volume ./data:/app/data (relative bind, i.e. /root/docker/uptime-kuma/data); has a healthcheck (curl -f http://localhost:3001) and resource limits (1 CPU / 512M) already defined — carry these over unchanged.
  • Data dir: /root/docker/uptime-kuma/data, 487 MB, dominated by kuma.db (SQLite, 506,138,624 bytes as of this scan — grows continuously from monitor heartbeat history).
  • Caddy routes (Core): two site blocks reference it —
    • uptimekuma.itpropartner.com { handle /health { respond "OK" 200 } ; reverse_proxy localhost:3001 }
    • status.itpropartner.com { ... handle /api/* { reverse_proxy 127.0.0.1:3001 } ... } (status page also proxies /api/msg* to Transitpin 127.0.0.1:8210 — that portion is out of WAVE 1 scope, do not move it)
  • Cert: real ACME for uptimekuma.itpropartner.com; status.itpropartner.com likely shares handling (verify both certs independently at cutover — do not assume one covers the other).
  • This is the monitoring tool itself — its own downtime window must be announced per the migration plan (Phase 3 note), since alerting is blind while it's down.

Pre-flight checks (read-only, run before touching anything)

ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "docker inspect uptime-kuma --format '{{.Image}} {{.State.Status}}'"
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "sqlite3 -readonly /root/docker/uptime-kuma/data/kuma.db 'PRAGMA quick_check;'"
# Expected: ok
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "du -sh /root/docker/uptime-kuma/data"
# Expected: ~487M (confirm current size before the copy so post-copy size is comparable)
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker info | grep -i 'docker root dir'; ls /var/lib/docker | wc -l"
# Confirm app4's Docker data root is populated (known-fixed defect) before pulling any image

Move commands

  1. Recreate the compose file on app4, image pinned by digest, port kept internal-only:
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "mkdir -p /root/docker/uptime-kuma"
cat > /tmp/uptime-kuma-compose.yaml <<'EOF'
services:
  uptime-kuma:
    image: louislam/uptime-kuma@sha256:3d632903e6af34139a37f18055c4f1bfd9b7205ae1138f1e5e8940ddc1d176f9
    container_name: uptime-kuma
    restart: unless-stopped
    volumes:
      - ./data:/app/data
    ports:
      - "127.0.0.1:3001:3001"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3001"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 10s
    deploy:
      resources:
        limits:
          cpus: '1'
          memory: 512M
        reservations:
          cpus: '0.25'
          memory: 128M
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "3"
EOF
scp -i /root/.ssh/itpp-infra /tmp/uptime-kuma-compose.yaml root@159.195.205.80:/root/docker/uptime-kuma/docker-compose.yml
  1. Take a live-safe SQLite snapshot on Core (do not cp a live WAL DB) and copy it to app4, matching the pattern the existing core-services-backup.sh already uses for this exact file:
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "sqlite3 -readonly /root/docker/uptime-kuma/data/kuma.db '.backup /tmp/kuma-migrate.db' && sqlite3 -readonly /tmp/kuma-migrate.db 'PRAGMA quick_check;'"
# Expected: ok
  1. Announce the monitoring downtime window, then stop the Core container (this is the one write-adjacent action required — flagged explicitly since it is the monitoring source of truth; confirm with the requester before executing, this runbook documents the command but does not execute it):
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "docker stop uptime-kuma"
  1. Final consistent copy (container stopped, no WAL risk) plus the docker-tls subfolder (Kuma's own self-issued monitor certs, if any monitors use them):
rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/root/docker/uptime-kuma/data/ root@159.195.205.80:/root/docker/uptime-kuma/data/
  1. Start on app4:
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "cd /root/docker/uptime-kuma && docker compose up -d"
  1. Add both Caddy blocks to app4's Caddyfile (mirroring Core, default_bind already set globally):
uptimekuma.itpropartner.com {
    handle /health {
        respond "OK" 200
    }
    reverse_proxy 127.0.0.1:3001
}

(status.itpropartner.com's /api/* proxy to 3001 is bundled with the Transitpin relay on the same domain — document it here but do not create a standalone status.itpropartner.com block until Transitpin's own WAVE migration is scheduled, to avoid a half-built multi-path site block. caddy validate app4's file before reload.)

Verification (exact command + expected output)

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker exec uptime-kuma wget -qO- http://localhost:3001 | head -c 200"
# Expected: HTML containing "<title>Uptime Kuma</title>" (or a 200 via curl -o /dev/null -w '%{http_code}')

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sqlite3 -readonly /root/docker/uptime-kuma/data/kuma.db 'SELECT COUNT(*) FROM monitor;'"
# Expected: same monitor count as Core's kuma.db at snapshot time — compare with:
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "sqlite3 -readonly /root/docker/uptime-kuma/data/kuma.db 'SELECT COUNT(*) FROM monitor;'"
# These two numbers MUST match exactly before cutover.

curl --resolve uptimekuma.itpropartner.com:443:159.195.205.80 -sI https://uptimekuma.itpropartner.com/
# Expected (post-DNS-flip, cert issued): HTTP/2 200

Rollback

  1. ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker compose -f /root/docker/uptime-kuma/docker-compose.yml down" (stop app4's copy).
  2. ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "docker start uptime-kuma" (Core's container and data dir were never deleted — only stopped — so this brings monitoring back exactly where it left off).
  3. If DNS was already flipped, revert the uptimekuma.itpropartner.com A record to 152.53.192.33.
  4. No data loss: Core's kuma.db is untouched by the migration (copy was one-directional, read-only source).

Service 3 of 6 — Prometheus + Grafana + node_exporter monitoring stack

Treated as one unit because Prometheus scrapes node_exporter and feeds Grafana's dashboards; moving them together avoids a window where Grafana has no data source. Grafana's own port/auth finding is broken out separately as Service 4 since it needs an explicit fix, not just a move.

Current state (Core)

  • Prometheus: container prometheus (prom/prometheus:latest, unless-stopped, NetworkMode: host). Config bind-mounted /root/docker/monitoring/prometheus/prometheus.yml -> /etc/prometheus/prometheus.yml (ro). Data: named volume prometheus_data (/var/lib/docker/volumes/prometheus_data/_data, 114.9 MB). Also bind-mounts /var/lib/prometheus/textfile (ro) for host-metrics-exporter's textfile collector. No compose file found — appears to be a manual docker run (ambiguity #1 in the inventory doc); reconstruct as a compose file on app4 rather than perpetuating a hand-run container. Listens *:9090 (host network). prometheus.yml (50 lines total) scrape jobs: prometheus (self, 127.0.0.1:9090), mikrotik (127.0.0.1:9436 via mikrotik-exporter — stays on Core, it's scraping Core's local network device), mikrotik-snmp (127.0.0.1:9274), core-telegraf (127.0.0.1:9273 — stays on Core, host-local metrics), super-search (172.17.0.1:8899/metrics — Docker bridge IP, will need re-pointing if super-search moves), node_exporter (already has 3 targets: 127.0.0.1:9100 labeled core, 159.195.205.80:9100 labeled app4, 159.195.204.203:9100 labeled core-bu — this job already scrapes app4 and core-bu remotely from Core, added during Phase 1 provisioning).
  • Grafana: container grafana (grafana/grafana:11.4.0, unless-stopped, NetworkMode: host, no ports block — the image binds directly since it's on host networking). Named volume grafana_data_final (/var/lib/docker/volumes/grafana_data_final/_data, ~15 MB) -> /var/lib/grafana. Two orphaned volumes exist alongside it on Core (grafana_data 50 MB, grafana_data_v3 14.6 MB) — do not migrate these, they are stale, only grafana_data_final is live per the running container's mount. Config: /root/docker/monitoring/grafana/grafana.ini, referenced via env vars (GF_PATHS_CONFIG=/etc/grafana/grafana.ini) — confirm at move time whether it's a bind mount or baked into the volume; treat the working copy at /root/docker/monitoring/grafana/grafana.ini as authoritative source of truth for env/ini settings.
  • node_exporter: NOT a container — a systemd unit (/etc/systemd/system/node_exporter.service, ExecStart=/usr/local/bin/node_exporter, no flags, Restart=unless-stopped). Binds *:9100. app4 already runs its own node_exporter (systemd, active, listening *:9100, confirmed ss -ltnp | grep 9100 shows node_exporter on app4 right now) — this was stood up in Phase 1 provisioning and is already scraped by Core's Prometheus (see the app4 label above). node_exporter is therefore ALREADY DONE on app4 — no move action needed for this piece.
  • Caddy: no Caddy site block routes to Prometheus or Grafana today. Neither :9090 nor :3002 appears anywhere in Core's 49 Caddy site blocks. Access today is by direct IP:port only.
  • Cert: none — neither service is behind TLS today.
  • Firewall (Core ufw status numbered): rule [7] 8899/tcp ALLOW IN 172.17.0.0/16 (Docker-bridge-only, for super-search's own metrics scrape — not customer-facing) and [8] 3002/tcp ALLOW IN 100.64.0.0/10 (Tailscale-range-only for Grafana) exist, but UFW is a second gate behind the actual bind — see Service 4, because Grafana's *:3002 listen and this UFW rule together still leave a gap, which is the whole point of the Service 4 finding.

Pre-flight checks

ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "docker inspect prometheus --format '{{.Config.Image}} {{.State.Status}}'; docker inspect grafana --format '{{.Config.Image}} {{.State.Status}}'"
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "du -sh /var/lib/docker/volumes/prometheus_data/_data /var/lib/docker/volumes/grafana_data_final/_data"
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl is-active node_exporter; curl -s http://127.0.0.1:9100/metrics | head -1"
# Expected: active / a line starting "# HELP go_gc_duration_seconds"
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "curl -s 'http://127.0.0.1:9090/api/v1/query?query=up' | python3 -m json.tool | head -20"
# Baseline: capture current up{} series before the move for a before/after diff

Move commands

  1. Create target dirs on app4:
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "mkdir -p /root/docker/monitoring/prometheus /root/docker/monitoring/grafana"
  1. Copy configs (read-only source copy from Core, nothing stopped yet):
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/monitoring/prometheus/prometheus.yml /tmp/prometheus.yml
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/monitoring/grafana/grafana.ini /tmp/grafana.ini
  1. Edit the copy (never Core's file) before pushing to app4: remove the mikrotik, mikrotik-snmp, and core-telegraf jobs (Core-local, stay on Core) from /tmp/prometheus.yml; re-point super-search's target from 172.17.0.1:8899 to wherever Super Search lands (Service 6 — if it also moves to app4 in this wave, target becomes 127.0.0.1:8899; if it stays put, keep scraping Core's public IP with a dedicated firewall allow, decided at Service 6 time). Push:
scp -i /root/.ssh/itpp-infra /tmp/prometheus.yml root@159.195.205.80:/root/docker/monitoring/prometheus/prometheus.yml
scp -i /root/.ssh/itpp-infra /tmp/grafana.ini root@159.195.205.80:/root/docker/monitoring/grafana/grafana.ini
  1. rsync the live volume data in two passes — a warm first pass while Core's containers still run, then a final short-stop pass for consistency (same pattern as the Uptime Kuma DB):
rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/var/lib/docker/volumes/prometheus_data/_data/ /tmp/prom-data-staging/
rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/var/lib/docker/volumes/grafana_data_final/_data/ /tmp/grafana-data-staging/
# announce downtime, then: ssh ... root@152.53.192.33 "docker stop prometheus grafana"
rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/var/lib/docker/volumes/prometheus_data/_data/ /tmp/prom-data-staging/
rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/var/lib/docker/volumes/grafana_data_final/_data/ /tmp/grafana-data-staging/
scp -r -i /root/.ssh/itpp-infra /tmp/prom-data-staging root@159.195.205.80:/tmp/
scp -r -i /root/.ssh/itpp-infra /tmp/grafana-data-staging root@159.195.205.80:/tmp/
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker volume create prometheus_data && docker run --rm -v prometheus_data:/dest -v /tmp/prom-data-staging:/src alpine cp -a /src/. /dest/"
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker volume create grafana_data_final && docker run --rm -v grafana_data_final:/dest -v /tmp/grafana-data-staging:/src alpine cp -a /src/. /dest/"
  1. Write /root/docker/monitoring/docker-compose.yml on app4, images pinned by digest (resolve fresh: docker inspect prometheus --format '{{index .RepoDigests 0}}' on Core for Prometheus; Grafana's tag 11.4.0 is already a pin — resolve its digest the same way), network_mode: host preserved for Prometheus to match Core's scrape behavior; Grafana's networking is changed — see Service 4.
  2. docker compose up -d on app4.

Verification

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s http://127.0.0.1:9090/-/healthy"
# Expected: "Prometheus Server is Healthy."
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s 'http://127.0.0.1:9090/api/v1/query?query=up' | python3 -m json.tool"
# Expected: same target set (minus mikrotik/telegraf/core-only jobs) all reporting value "1"
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3002/api/health"
# Expected: 200
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s http://127.0.0.1:3002/api/health"
# Expected JSON containing "database":"ok","version":"11.4.0" — confirms grafana.db loaded correctly

Series-count parity check (compare Core's pre-stop snapshot vs app4's loaded volume before deleting anything on Core):

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s http://127.0.0.1:9090/api/v1/label/__name__/values | python3 -c 'import json,sys; print(len(json.load(sys.stdin)[\"data\"]))'"
# Compare metric-name count to the same query run on Core before stop

Rollback

  1. ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker compose -f /root/docker/monitoring/docker-compose.yml down".
  2. ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "docker start prometheus grafana" — Core's volumes and config were only read from, never deleted; original containers restart with their original state intact.
  3. No DNS is involved (no Caddy route exists either direction today), so rollback is purely stop-app4/ start-Core with no propagation delay.
  4. Any staging copy edited on the operator's own box (/tmp/prometheus.yml) is disposable; Core's live prometheus.yml was never touched, per the read-only constraint on Core.

Service 4 of 6 — Grafana public-bind finding (*:3002, unauthenticated, no Caddy/Tailscale gate)

This is called out separately from Service 3 because it is a security finding to fix during the move, not just a relocation.

The finding, precisely

  • Grafana's container runs NetworkMode: host and the Grafana process itself binds *:3002 (confirmed via ss -ltnp: LISTEN 0 4096 *:3002 *:* users:(("grafana",pid=2906,fd=15))) — i.e. every interface on Core, including its public IPv4/IPv6, not just loopback or the Tailscale interface.
  • It is not behind any Caddy site block (grepped all 49 blocks in Core's Caddyfile — zero mention of 3002 or a Grafana upstream).
  • The only mitigation in place is a UFW rule restricting inbound port 3002 to the Tailscale CGNAT range (100.64.0.0/10) — ufw status numbered line [8] 3002/tcp ALLOW IN 100.64.0.0/10. UFW's default policy is deny (incoming), so absent that specific allow rule, 3002 would in fact be closed by default — but this means the ONLY thing standing between Grafana and the open internet is one UFW line, with no defense in depth (no app-level auth beyond Grafana's own login, no Caddy layer, no Tailscale-only bind at the app level).
  • Compounding it: Grafana's grafana.ini [security] section has admin_user = admin / admin_password = admin / disable_initial_admin_creation = false — the default admin credentials were never changed. If the UFW rule is ever misconfigured, misapplied to the wrong interface, or a firewall reload race drops it (a known class of bug on reboot ordering), the admin/admin login is directly exposed to anyone who reaches that interface.

Fix to apply on app4 (do NOT attempt this on Core — Core is read-only for this task)

  1. Do not repeat the NetworkMode: host + open-bind pattern on app4. In the Service 3 compose file, add an explicit port mapping and drop host networking for Grafana specifically (Prometheus can keep host networking since it only needs outbound scrape access; Grafana only needs to reach Prometheus on the same host, no host networking required): 
    grafana:
  image: grafana/grafana@sha256:<resolve-at-execution-time>
  ports:
    - "127.0.0.1:3002:3000"   # loopback only; Caddy or Tailscale serves it externally
  2. Add a real Caddy site block on app4 gating it behind HTTP basic auth (mirroring the existing pattern used for internal.debtrecoveryexperts.com's basic_auth block in Core's Caddyfile): 
    grafana.itpropartner.com {
    basic_auth {
        admin <bcrypt-hash-generated-with-caddy-hash-password>
    }
    reverse_proxy 127.0.0.1:3002
}
    (Generate the hash with caddy hash-password — never hand-write a bcrypt hash.)
  3. Change the default admin credentials in grafana.ini (or via GF_SECURITY_ADMIN_PASSWORD env var) on app4's first boot — do not carry admin/admin forward. This must happen before app4's Grafana is reachable by anything but loopback.
  4. Confirm ufw status on app4 does not carry forward the Core Tailscale-3002 rule as the only gate — Caddy basic_auth is now gate 1; Tailscale-only access can remain as an additional, not sole, control.

Verification

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "ss -ltnp | grep 3002"
# Expected: 127.0.0.1:3002 only — NOT *:3002 or 0.0.0.0:3002
curl -s -o /dev/null -w '%{http_code}\n' http://159.195.205.80:3002/
# Expected: connection refused / timeout (nothing listening on the public interface)
curl -s -o /dev/null -w '%{http_code}\n' https://grafana.itpropartner.com/
# Expected: 401 (basic_auth challenge) before credentials, 200 after
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "grep -E 'admin_password' /root/docker/monitoring/grafana/grafana.ini"
# Expected: NOT the literal string "admin" — confirm it was actually changed

Rollback

This fix only touches app4, a not-yet-cutover host; Core is never modified. Rollback is simply: do not flip DNS/Tailscale access to app4's Grafana until the verification above passes. Core's Grafana keeps running with its existing (already-known-weak) posture unchanged — a pre-existing condition to flag to Germaine regardless of migration timing, not something this runbook worsens or is required to fix on Core.

Service 5 of 6 — PRY (unified OSINT search backend)

Current state (Core)

  • Not a container — a systemd unit: pry.service (/etc/systemd/system/pry.service, Type=simple, User=root, WorkingDirectory=/root/docker/pry, ExecStart=/root/docker/pry/venv/bin/python /root/docker/pry/server.py, EnvironmentFile=/root/.hermes/.env, Restart=always, ProtectSystem=full, ProtectHome=read-only, ReadWritePaths=/root/docker/pry, ReadOnlyPaths=/root/.hermes/.env /root/.config/himalaya/shonuff.pass).
  • Port: 127.0.0.1:8905, confirmed via ss -ltnp — loopback only, not directly internet-reachable.
  • App dir: /root/docker/pry, 66 MB total (mostly its own venv). Python 3.13.5 venv, deps from requirements.txt (fastapi>=0.109.0, uvicorn[standard]>=0.27.0, httpx>=0.26.0, python-dotenv>=1.0.0, aiosmtplib>=3.0.0 — 5 loose lines, no pinned freeze file, so re-resolve at install time rather than assuming exact matching versions).
  • Data/DB: none found — no .db/.sqlite* file in /root/docker/pry. PRY is stateless app logic; its only persistent asset is its git-tracked source (.git present) plus a private Caddyfile-new (mode 600, 4842 bytes — appears to be a staged/unused Caddy config someone drafted but never applied; carry it over for reference but do not activate it blindly).
  • Secrets: EnvironmentFile=/root/.hermes/.env — shared secrets file across many services; only PRY-relevant keys need to be extracted/copied to app4's own .env, not the whole file (per the inventory doc's finding #7 on live secrets requiring manual, non-rsync transfer).
  • Caddy route (Core): http://pry.iamgmb.com { @api path /api/* ; handle @api { reverse_proxy 127.0.0.1:8905 } ; handle { root * /var/www/pry ; try_files ... ; file_server } } — HTTP only, no TLS (http:// scheme explicit in the Caddyfile block header), and it's a split route: /api/* goes to PRY's backend, everything else is a static SPA served from /var/www/pry on Core. The static half is out of scope for a backend-only WAVE 1 move — moving PRY's backend without also moving /var/www/pry's static files would break the frontend, so both must move together even though the task frames this as one backend service.
  • Cert: none (HTTP-only site today).

Pre-flight checks

ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "systemctl is-active pry.service; ss -ltnp | grep 8905"
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8905/api/health || curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8905/"
# Record baseline response code before moving
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "du -sh /var/www/pry"
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "grep -c '^[A-Z_]*=' /root/.hermes/.env"
# Confirms the shared secrets file's line count so a partial extraction can be sanity-checked for completeness

Move commands

  1. Copy the app dir (excluding the venv, which will be rebuilt fresh on app4 to avoid glibc/arch drift):
rsync -avz -e "ssh -i /root/.ssh/itpp-infra" --exclude 'venv' --exclude '__pycache__' root@152.53.192.33:/root/docker/pry/ root@159.195.205.80:/root/docker/pry/
  1. Copy the static frontend:
rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/var/www/pry/ root@159.195.205.80:/var/www/pry/
  1. Rebuild the venv on app4:
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "cd /root/docker/pry && python3 -m venv venv && ./venv/bin/pip install -r requirements.txt"
  1. Manually extract PRY-relevant keys from Core's /root/.hermes/.env (grep for the specific vars server.py actually reads — do this by inspecting os.getenv/os.environ calls in server.py, not by copying the whole file) into a new, minimal /root/.hermes/.env on app4, or a dedicated /root/docker/pry/.env + adjust the systemd unit's EnvironmentFile= path. This is a manual, secure step — never rsync the shared .env wholesale (matches the inventory doc's explicit warning on secret files).
  2. Create the systemd unit on app4 (same content as Core's, WorkingDirectory/ExecStart paths already match since both boxes use /root/docker/pry):
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/pry/pry.service /tmp/pry.service
# edit EnvironmentFile= path if app4 uses a dedicated .env, then:
scp -i /root/.ssh/itpp-infra /tmp/pry.service root@159.195.205.80:/etc/systemd/system/pry.service
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl daemon-reload && systemctl enable --now pry.service"
  1. Add the Caddy block to app4, upgraded to real TLS this time (Caddy on app4 will ACME-issue since it's a real domain — no reason to keep it HTTP-only post-move):
pry.iamgmb.com {
    @api path /api/*
    handle @api {
        reverse_proxy 127.0.0.1:8905
    }
    handle {
        root * /var/www/pry
        try_files {path} {path}.html /index.html
        file_server
    }
}

Verification

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl is-active pry.service"
# Expected: active
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8905/"
# Expected: same status code as the Core baseline captured in pre-flight
curl --resolve pry.iamgmb.com:443:159.195.205.80 -sI https://pry.iamgmb.com/
# Expected (post-DNS-flip, cert issued): HTTP/2 200
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "journalctl -u pry.service -n 20 --no-pager"
# Expected: no tracebacks, normal startup log lines matching Core's own startup pattern

Rollback

  1. ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl stop pry.service".
  2. Core's pry.service was never touched — it is still active throughout, so no restart action needed on Core.
  3. If DNS was flipped, revert pry.iamgmb.com to 152.53.192.33 (note: this domain has no A vs CNAME redirect complication, it's a direct A record per the plan's DNS checklist).
  4. No data to restore (stateless service, no DB).

Service 6 of 6 — OSINT Person MCP (:8902) + Super Search MCP (:8899)

These are grouped because OSINT Person MCP imports Super Search's own module directly (from super_search import web_search as ss_search, web_extract as ss_extract in osint-person-mcp/server.py) — they have a code-level dependency, not just a runtime one, so they must move together or the import breaks.

Current state (Core)

OSINT Person MCP:

  • systemd unit osint-person.service (After=network.target super-search.service — systemd-level ordering dependency confirming the two are coupled), WorkingDirectory=/root/docker/osint-person-mcp, ExecStart=/root/docker/osint-person-mcp/venv/bin/python3 /root/docker/osint-person-mcp/server.py, EnvironmentFile=/root/.hermes/.env, Restart=always.
  • Port: 127.0.0.1:8902 (confirmed ss -ltnp), used as http://127.0.0.1:8902/mcp per Hermes' config.yaml MCP registration — loopback only.
  • App dir: /root/docker/osint-person-mcp, 314 MB (Python 3.13.5 venv-heavy). Source files include server.py (54,253 bytes, modified same-day as this scan — active development), reconcile.py, super_search.py (a local copy of Super Search's search/extract functions, imported directly — this is the code coupling), telemetry.py, plus backup files (server.py.bak-*) and a tests/ dir. No standalone requirements file was found in the directory listing — dependencies live only in the existing venv; capture venv/bin/pip freeze > requirements.txt on Core before rebuilding on app4, do not guess versions.
  • Data/DB: none found (no .db file); it's a stateless MCP tool server backed by live web calls.

Super Search MCP:

  • systemd unit super-search.service (WorkingDirectory=/root/docker/super-search, ExecStartPre=/bin/bash /root/docker/super-search/super-search-port-guard.sh — a pre-start guard script, copy this too, it likely checks for a stale port bind before starting — ExecStart=.../venv/bin/python3 .../server.py, EnvironmentFile=/root/.hermes/.env, Restart=always, StartLimitBurst=5 over 300s).
  • Port: 0.0.0.0:8899 — confirmed bound to all interfaces on Core, mitigated only by ufw status numbered rule [7] 8899/tcp ALLOW IN 172.17.0.0/16 (Docker-bridge-range only — i.e. only containers on Core's own docker0/bridge network can reach it externally-looking-in; this is intentional since Prometheus's super-search scrape job targets 172.17.0.1:8899, the bridge gateway IP, meaning Prometheus itself runs in a container reaching it via the bridge). Registered in Hermes as http://127.0.0.1:8899/mcp.
  • App dir: /root/docker/super-search, 1.1 GB (largest of the six — heavy venv, Python 3.13.5). Has a pinned requirements.freeze.txt (167 lines) — use this file directly on app4, it's already a proper pin, unlike PRY's loose requirements.txt or OSINT Person's missing one.
  • Data/DB: none found — stateless, backed by live search-provider calls (SearXNG on Core, Exa, Brave, etc. per the skill's provider list); note SearXNG itself (127.0.0.1:8888) stays on Core per the inventory doc, meaning if Super Search moves to app4, its SearXNG provider call becomes a cross-Atlantic Core→app4→Core round trip (~100ms each way per the migration plan's RTT finding) unless SearXNG is also exposed to app4, which is out of WAVE 1 scope. Recommend either: (a) leave Super Search on Core for now despite the task's inclusion of it here, since its own migration-plan classification already says "stays-on-Core (Hermes MCP)" in the inventory doc (contradicting this task's WAVE 1 list — flag this conflict to the requester explicitly before executing), or (b) accept the added latency for one SearXNG-backed provider path while other providers (Exa, Brave, Serper, Tavily) call out directly with no Core round-trip. This is a real discrepancy between the source inventory and this task's WAVE 1 scope — documented here, not silently resolved.
  • Caddy: neither service has a Caddy route today. Both are pure loopback/bridge MCP servers consumed only by Hermes' own config.yaml url: http://127.0.0.1:PORT/mcp entries — there is no public hostname for either in the 49 scanned Caddy blocks.
  • Cert: none (no public exposure today; loopback-only by design).

Pre-flight checks

ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "systemctl is-active osint-person.service super-search.service"
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8902/mcp; curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8899/mcp"
# Record baseline codes (MCP endpoints often 4xx on a bare GET without proper headers — record whatever Core
# returns today as the comparison target, don't assume 200)
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "/root/docker/osint-person-mcp/venv/bin/pip freeze > /tmp/osint-person-freeze.txt; wc -l /tmp/osint-person-freeze.txt"
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "du -sh /root/docker/osint-person-mcp /root/docker/super-search"

Move commands

  1. Freeze OSINT Person's deps (not previously pinned) and pull both freeze files:
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/tmp/osint-person-freeze.txt /tmp/
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/super-search/requirements.freeze.txt /tmp/
  1. Copy source (excluding venv/pycache/backup cruft — the .bak-* files are large and not needed):
rsync -avz -e "ssh -i /root/.ssh/itpp-infra" --exclude 'venv' --exclude '__pycache__' --exclude '*.bak*' root@152.53.192.33:/root/docker/osint-person-mcp/ root@159.195.205.80:/root/docker/osint-person-mcp/
rsync -avz -e "ssh -i /root/.ssh/itpp-infra" --exclude 'venv' --exclude '__pycache__' --exclude '*.bak*' root@152.53.192.33:/root/docker/super-search/ root@159.195.205.80:/root/docker/super-search/
  1. Rebuild both venvs on app4 from the frozen requirements:
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "cd /root/docker/osint-person-mcp && python3 -m venv venv && ./venv/bin/pip install -r /tmp/osint-person-freeze.txt"
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "cd /root/docker/super-search && python3 -m venv venv && ./venv/bin/pip install -r requirements.freeze.txt"
  1. Extract only the OSINT/Super-Search-relevant keys from Core's /root/.hermes/.env into app4's own .hermes/.env (same manual-secret-transfer rule as PRY — do not rsync the whole file).
  2. Copy systemd units and the port-guard script, adjusting nothing but confirming paths already match (/root/docker/... is identical on both boxes):
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/osint-person-mcp/osint-person.service /tmp/
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/super-search/super-search.service /tmp/
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/super-search/super-search-port-guard.sh /tmp/
scp -i /root/.ssh/itpp-infra /tmp/osint-person.service /tmp/super-search.service root@159.195.205.80:/etc/systemd/system/
scp -i /root/.ssh/itpp-infra /tmp/super-search-port-guard.sh root@159.195.205.80:/root/docker/super-search/
  1. Fix Super Search's bind before enabling — do not carry 0.0.0.0:8899 forward. If server.py hardcodes the bind host, patch app4's copy to 127.0.0.1:8899 (matching OSINT Person's already-safe loopback pattern) unless something on app4 genuinely needs the Docker-bridge-range access Core's UFW rule was protecting (confirm this is even needed on app4 — app4 doesn't run the same Prometheus container-bridge topology as Core unless Service 3 also lands there with the same 172.17.0.1 scrape pattern).
  2. Enable both, in dependency order (Super Search first, since OSINT Person's own unit declares After=... super-search.service):
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl daemon-reload && systemctl enable --now super-search.service && sleep 3 && systemctl enable --now osint-person.service"
  1. Update Hermes' config.yaml MCP url: entries — but only after confirming which box's Hermes actually calls these tools. If it's Core's own Hermes gateway consuming these via 127.0.0.1, moving the servers to app4 means Core's config must change to http://159.195.205.80:8902/mcp / http://159.195.205.80:8899/mcp, which reintroduces exactly the cross-Atlantic per-call latency the migration plan's Section 3.1 design rule warns against ("app4 must not depend on Core at request time" — the inverse also applies: Core's live Hermes gateway should not depend on app4 at request time for every-tool-call-latency reasons). This is a design question for the requester, not a call this runbook makes — flagged here rather than silently deciding to change config.yaml.

Verification

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl is-active osint-person.service super-search.service"
# Expected: active / active
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "ss -ltnp | grep -E ':(8899|8902)\b'"
# Expected: 127.0.0.1:8899 and 127.0.0.1:8902 — NOT 0.0.0.0:8899 (confirms the bind fix from step 6 landed)
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8902/mcp; curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8899/mcp"
# Expected: same status codes as the Core baseline captured in pre-flight
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "journalctl -u osint-person.service -u super-search.service -n 30 --no-pager"
# Expected: no import errors (specifically watch for "ModuleNotFoundError: super_search" — the code-coupling
# risk called out above — and no missing-env-var tracebacks)

Rollback

  1. ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl stop osint-person.service super-search.service".
  2. Core's two units were never stopped or restarted — they remain active throughout, so Hermes' existing 127.0.0.1 MCP URLs keep working with zero interruption regardless of app4's state.
  3. If config.yaml was changed (step 8), revert the url: fields back to http://127.0.0.1:8902/mcp and http://127.0.0.1:8899/mcp — this requires whichever Hermes profile/gateway consumes them to reload its MCP config (out of scope for this SSH-only runbook; note it as a manual follow-up).
  4. No data to restore (both stateless).

Section 8 — Backup coverage gap: /root/docker missing from app4's root-essentials-backup.sh

The gap

app4's /root/root-essentials-backup.sh (v2.0.0, 2026-09-11) builds its include list from this loop:

for p in .hermes/config.yaml .hermes/.env .hermes/skills .hermes/scripts \
         .hermes/references .hermes/profiles .hermes/cron .hermes/memories \
         .hermes/DR-PLAN.md .hermes/data .hermes/cache/delegation \
         .ssh .aws .config/himalaya \
         shark-game projects ; do
    [ -e "$p" ] && ARGS+=("$p")
done

/root/docker (relative: docker) is not in this list. Every compose stack this runbook lands on app4 — Uptime Kuma's docker-compose.yml + its data dir path, the monitoring stack's compose file, PRY's app dir if placed under /root/docker (it is), OSINT Person MCP, Super Search — is therefore entirely invisible to the nightly 04:45 ET run, even though the script's own tar excludes *.db/*.db-shm/*.db-wal regardless (so Uptime Kuma's kuma.db and any future SQLite files still need their own sqlite3 .backup job even after this fix — this patch closes the "missing compose/config/source" half of the gap, not the "missing DB" half, which is already flagged as a separate Phase 4/5 acceptance item in the migration plan).

Exact patch

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sed -n '55,60p' /root/root-essentials-backup.sh"
# Confirm current line content before editing, since this is a live production script:
#   .ssh .aws .config/himalaya \
#          shark-game projects ; do

Change:

-         .ssh .aws .config/himalaya \
-         shark-game projects ; do
+         .ssh .aws .config/himalaya \
+         shark-game projects docker ; do

Apply with patch/sed on app4 (do NOT run this against Core — Core is read-only for this task, and Core's own backup script coverage is a separate, already-tracked item per the inventory doc's Section on backup gaps):

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sed -i 's/shark-game projects ; do/shark-game projects docker ; do/' /root/root-essentials-backup.sh"

Verify the patch landed correctly

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "grep -n 'shark-game projects' /root/root-essentials-backup.sh"
# Expected: shark-game projects docker ; do

Restore test (prove the new coverage actually works, not just that the line changed)

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "bash /root/root-essentials-backup.sh 2>&1 | tail -20"
# Expected final lines: "[...] Tarball: <N> bytes" (larger than the pre-patch baseline by roughly the size of
# /root/docker's non-.db content — compare against a du -sh /root/docker taken beforehand), then
# "[...] content check OK - N entries, 3/3 key essentials present", then "Uploading...", then
# "Backup verified - archive is valid", then "Done".
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "aws s3 cp s3://hermes-vps-backups/root-backup/\$(hostname -s)/root-essentials-\$(date +%F).tar.gz /tmp/restore-test.tar.gz --endpoint-url https://s3.us-east-1.wasabisys.com --quiet && tar tzf /tmp/restore-test.tar.gz | grep '^docker/' | head -20"
# Expected: a non-empty listing of docker/<service>/... paths — proves /root/docker is now actually inside the
# uploaded archive, not just inside the include-list source, and that the round-trip download+list works.
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "rm -f /tmp/restore-test.tar.gz"

Rollback (if the patched script misbehaves — e.g. archive size becomes unexpectedly huge from a stray venv)

ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sed -i 's/shark-game projects docker ; do/shark-game projects ; do/' /root/root-essentials-backup.sh"
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "grep -n 'shark-game projects' /root/root-essentials-backup.sh"
# Expected: shark-game projects ; do  (back to original)

If venv bloat is the issue (Super Search alone is 1.1 GB, mostly venv), prefer excluding venvs specifically over reverting the whole line:

# Add to the tar invocation's --exclude list (near the existing --exclude='*.db' flags):
--exclude='docker/*/venv' --exclude='docker/*/*/venv'

Summary table

# Service Status Downtime needed DNS flip needed New security work
1 microbin DONE (app layer), DNS flip pending None (data dir was empty) Yes — share.itpropartner.com None
2 Uptime Kuma Runbook ready Yes — brief, for final DB consistency Yes — uptimekuma.itpropartner.com Tighten 0.0.0.0:3001 → 127.0.0.1:3001
3 Prometheus+Grafana+node_exporter Runbook ready (node_exporter already on app4) Yes — brief, for volume consistency None (no Caddy route today) See Service 4
4 Grafana public-bind finding Fix defined N/A (fix applied pre-cutover) Optional — grafana.itpropartner.com if exposed Loopback bind + Caddy basic_auth + change admin/admin
5 PRY Runbook ready None (stateless) Yes — pry.iamgmb.com Upgrade HTTP-only → real TLS
6 OSINT Person MCP + Super Search MCP Runbook ready, scope conflict flagged None (stateless) None (loopback/bridge-only, MCP-internal) Fix 0.0.0.0:8899 → 127.0.0.1:8899; resolve Core-Hermes-vs-app4-MCP latency question first

Open items for the requester before executing Services 3/6: (a) Service 3's Prometheus scrape config needs a decision on where Super Search ends up before it can be finalized; (b) Service 6 conflicts with the source inventory's own classification of Super Search as "stays-on-Core" — confirm intent before executing.

Reference in New Issue View Git Blame Copy Permalink