migration plan: corrected scope + waves, backup gap reconciliation, Wave1/Wave2 runbooks, core-bu standby package; backup-plan: docker-stack-defs + app-data jobs recorded (2026-09-15)
This commit is contained in:
@@ -0,0 +1,756 @@
|
||||
# Migration Runbook — WAVE 1 (Internal / Low-Customer-Impact Services)
|
||||
|
||||
**Date:** 2026-09-15
|
||||
**Scope:** Core (152.53.192.33) → app4 (159.195.205.80), exactly 6 services:
|
||||
1. microbin (`:8260`, `share.itpropartner.com`) — **ALREADY MIGRATED, DONE**
|
||||
2. Uptime Kuma
|
||||
3. Prometheus + Grafana + node_exporter monitoring stack
|
||||
4. Grafana specifically (unauthenticated public-bind finding)
|
||||
5. PRY
|
||||
6. OSINT API / OSINT Person MCP (`:8902`) + Super Search MCP (`:8899`)
|
||||
|
||||
**Reference source docs:** `core-service-inventory-2026-09-15.md`, `migration-plan-app4-core-bu-2026-09-15.md`.
|
||||
**Production hosts are READ-ONLY for this runbook's author.** All facts below were captured via read-only SSH
|
||||
(`ss -ltnp`, `docker inspect`, `systemctl cat`, `cat` of configs) on 2026-09-15. No service was started, stopped,
|
||||
or restarted to produce this document. `hermes-maintenance` was never invoked.
|
||||
|
||||
**SSH access:** `ssh -i /root/.ssh/itpp-infra root@<ip>` (key already present on this box).
|
||||
|
||||
**Proven pattern (established this session on microbin, reference for all 6 services):**
|
||||
1. Caddy v2.11.4 on app4, pinned to Core's exact version, from the `caddy-stable` apt repo.
|
||||
2. Service runs via `/root/docker/<service>/compose.yaml` (or `docker-compose.yml`), image **pinned by digest**,
|
||||
not `:latest`.
|
||||
3. app4's global Caddy option block sets `default_bind 159.195.205.80` so a new site block never hijacks another
|
||||
listener (e.g. Tailscale's own :443).
|
||||
4. `ufw` active on app4: 22/80/443 allowed, default deny incoming otherwise.
|
||||
5. Cutover = flip DNS at the authoritative panel (SiteGround for `itpropartner.com`, **manual, no API** —
|
||||
Germaine applies these), then Caddy on app4 auto-issues the cert via ACME http-01 on first request.
|
||||
6. Verify side-by-side with `curl --resolve <domain>:443:159.195.205.80 https://<domain>/` **before** the DNS
|
||||
flip, so the app layer is proven independent of DNS.
|
||||
|
||||
**Known defect already fixed (do not re-diagnose):** both app4 and core-bu shipped with an empty Docker data
|
||||
root (`/var/lib/docker` had zero subdirectories → every `docker pull`/`run` failed with `GetImageBlob` /
|
||||
`overlay2: no such file or directory`). Fixed by stopping docker, `rm -rf /var/lib/docker`, restarting docker,
|
||||
and proving with `docker run --rm hello-world`. Both boxes now run containers normally. If a pull/run fails with
|
||||
an overlay2 ENOENT-style error on either box in the steps below, this is the known cause — check
|
||||
`docker info | grep -i 'docker root dir'` and `ls /var/lib/docker` before assuming a new bug.
|
||||
|
||||
**Known backup gap this runbook must close (Section 8):** app4's `root-essentials-backup.sh` include list
|
||||
covers `.hermes .ssh .aws shark-game projects` (repo-root-relative) plus `/etc/caddy/Caddyfile`, but **not**
|
||||
`/root/docker`. Every compose stack landed on app4 by this runbook (Uptime Kuma, monitoring stack, PRY if
|
||||
containerized, OSINT/Super Search if containerized) is therefore invisible to the nightly backup until Section 8
|
||||
is applied.
|
||||
|
||||
---
|
||||
|
||||
## Service 1 of 6 — microbin (`:8260`, `share.itpropartner.com`) — **STATUS: DONE**
|
||||
|
||||
This is the proven reference pattern; it is documented here as the template the other 5 follow, not as new work.
|
||||
|
||||
### Current state (Core)
|
||||
- Container: `microbin` (`danielszabo99/microbin:latest`, image digest
|
||||
`sha256:6660e5ccad0d764fa3c0032464ffb8f4b4f28c92a2eb9e39202b94cdc5b68909`), `unless-stopped`.
|
||||
- Port: `127.0.0.1:8260 -> 8080` (container), not exposed externally on Core.
|
||||
- Compose: `/opt/microbin/docker-compose.yml`.
|
||||
- Data dir: `/opt/microbin/data` (bind mount -> `/app/pasta_data`), **4.0 KB** — no DB engine, flat files only.
|
||||
- Caddy route (Core, `/etc/caddy/Caddyfile` line ~178): `share.itpropartner.com { reverse_proxy 127.0.0.1:8260 }`.
|
||||
- Cert: real ACME (Let's Encrypt) issued to Core.
|
||||
|
||||
### What was actually done (already executed, verified in this session)
|
||||
1. Installed Caddy v2.11.4 on app4 from the `caddy-stable` apt repo (matches Core's `caddy version` output
|
||||
exactly: `v2.11.4 h1:XKxkMTgNSizEvKG6QHue6cAsFOteU2qA61w2tKkCWi0=` on both hosts).
|
||||
2. Created `/root/docker/microbin/docker-compose.yml` on app4, pinning the image **by digest**
|
||||
(`danielszabo99/microbin@sha256:6660e5ccad0d764fa3c0032464ffb8f4b4f28c92a2eb9e39202b94cdc5b68909`), same
|
||||
env vars as Core's compose, port bound `127.0.0.1:8260:8080`.
|
||||
3. `docker compose up -d` on app4 — container `microbin` now `Up`, verified via `docker ps`.
|
||||
4. Wrote app4's `/etc/caddy/Caddyfile`:
|
||||
```
|
||||
{
|
||||
default_bind 159.195.205.80
|
||||
email info@itpropartner.com
|
||||
}
|
||||
# -- MicroBin Share (migrated from Core 2026-09-15, phase 3 of the app4 migration) --
|
||||
share.itpropartner.com {
|
||||
reverse_proxy 127.0.0.1:8260
|
||||
encode gzip
|
||||
}
|
||||
```
|
||||
5. Data dir was NOT rsynced from Core because Core's `/opt/microbin/data` was empty (4.0 KB, no user pastes) at
|
||||
scan time — a fresh empty dir on app4 is equivalent. **If Core's data dir is non-empty at your actual cutover
|
||||
time**, rsync it first: `rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/opt/microbin/data/ root@159.195.205.80:/root/docker/microbin/data/` and stop the Core container for the duration of the rsync to avoid a torn copy (microbin uses flat files, not a WAL DB, so this is a low-risk copy).
|
||||
|
||||
### Verification (already run, real output)
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker ps --format '{{.Names}}\t{{.Ports}}\t{{.Status}}'"
|
||||
# -> microbin 127.0.0.1:8260->8080/tcp Up
|
||||
```
|
||||
App-layer check (curl from app4 itself, since DNS has not flipped yet):
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8260/"
|
||||
# Expected: 200
|
||||
```
|
||||
**Confirmed:** direct curl to app4's microbin returns 200. App layer is fully staged and correct.
|
||||
|
||||
### What is NOT done yet (pending, not part of "DONE" claim)
|
||||
- **DNS flip is pending.** `itpropartner.com` is on SiteGround, manual panel, no API — Germaine must change the
|
||||
`share.itpropartner.com` A record from 152.53.192.33 to 159.195.205.80.
|
||||
- Cert on app4 has not been issued yet (Caddy issues on first real HTTPS hit post-DNS-flip via http-01).
|
||||
- No S3 backup job exists yet for app4's `/root/docker/microbin/data` (see Section 8 — same gap applies here).
|
||||
|
||||
### Rollback (if flip has NOT happened — current state)
|
||||
Nothing to roll back: Core's container and Caddy route are untouched and still serving
|
||||
`share.itpropartner.com` live. app4's copy is inert until DNS points at it.
|
||||
|
||||
### Rollback (if DNS HAS been flipped and a problem appears)
|
||||
1. Revert the `share.itpropartner.com` A record at SiteGround back to `152.53.192.33`.
|
||||
2. TTL propagation is the only wait; Core's container was never stopped, so it answers immediately once DNS
|
||||
re-resolves.
|
||||
3. Do not touch app4's container — leave it running for retry.
|
||||
|
||||
---
|
||||
|
||||
## Service 2 of 6 — Uptime Kuma
|
||||
|
||||
### Current state (Core)
|
||||
- Container: `uptime-kuma` (`louislam/uptime-kuma:latest`, image digest
|
||||
`sha256:3d632903e6af34139a37f18055c4f1bfd9b7205ae1138f1e5e8940ddc1d176f9`, host layer digest
|
||||
`f48d816cb7460cd3b7bb15ed393968b0ae0da4c690443b778b6a5db6b09f527e`), `unless-stopped`.
|
||||
- Port: `0.0.0.0:3001 -> 3001` — **exposed on all interfaces on Core today** (not localhost-bound); reachable
|
||||
directly bypassing Caddy on port 3001 from anywhere that can route to Core's public IP. This is pre-existing
|
||||
Core behavior, not introduced by this migration — replicate `127.0.0.1:3001:3001` on app4 instead and let
|
||||
Caddy be the only door, tightening the surface during the move.
|
||||
- Compose: `/root/docker/uptime-kuma/docker-compose.yml`; volume `./data:/app/data` (relative bind, i.e.
|
||||
`/root/docker/uptime-kuma/data`); has a healthcheck (`curl -f http://localhost:3001`) and resource limits
|
||||
(1 CPU / 512M) already defined — carry these over unchanged.
|
||||
- Data dir: `/root/docker/uptime-kuma/data`, **487 MB**, dominated by `kuma.db` (SQLite, 506,138,624 bytes as of
|
||||
this scan — grows continuously from monitor heartbeat history).
|
||||
- Caddy routes (Core): two site blocks reference it —
|
||||
- `uptimekuma.itpropartner.com { handle /health { respond "OK" 200 } ; reverse_proxy localhost:3001 }`
|
||||
- `status.itpropartner.com { ... handle /api/* { reverse_proxy 127.0.0.1:3001 } ... }` (status page also
|
||||
proxies `/api/msg*` to Transitpin `127.0.0.1:8210` — that portion is out of WAVE 1 scope, do not move it)
|
||||
- Cert: real ACME for `uptimekuma.itpropartner.com`; `status.itpropartner.com` likely shares handling (verify
|
||||
both certs independently at cutover — do not assume one covers the other).
|
||||
- **This is the monitoring tool itself** — its own downtime window must be announced per the migration plan
|
||||
(Phase 3 note), since alerting is blind while it's down.
|
||||
|
||||
### Pre-flight checks (read-only, run before touching anything)
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "docker inspect uptime-kuma --format '{{.Image}} {{.State.Status}}'"
|
||||
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "sqlite3 -readonly /root/docker/uptime-kuma/data/kuma.db 'PRAGMA quick_check;'"
|
||||
# Expected: ok
|
||||
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "du -sh /root/docker/uptime-kuma/data"
|
||||
# Expected: ~487M (confirm current size before the copy so post-copy size is comparable)
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker info | grep -i 'docker root dir'; ls /var/lib/docker | wc -l"
|
||||
# Confirm app4's Docker data root is populated (known-fixed defect) before pulling any image
|
||||
```
|
||||
|
||||
### Move commands
|
||||
1. Recreate the compose file on app4, image pinned by digest, port kept internal-only:
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "mkdir -p /root/docker/uptime-kuma"
|
||||
cat > /tmp/uptime-kuma-compose.yaml <<'EOF'
|
||||
services:
|
||||
uptime-kuma:
|
||||
image: louislam/uptime-kuma@sha256:3d632903e6af34139a37f18055c4f1bfd9b7205ae1138f1e5e8940ddc1d176f9
|
||||
container_name: uptime-kuma
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
- ./data:/app/data
|
||||
ports:
|
||||
- "127.0.0.1:3001:3001"
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-f", "http://localhost:3001"]
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: '1'
|
||||
memory: 512M
|
||||
reservations:
|
||||
cpus: '0.25'
|
||||
memory: 128M
|
||||
logging:
|
||||
driver: json-file
|
||||
options:
|
||||
max-size: "10m"
|
||||
max-file: "3"
|
||||
EOF
|
||||
scp -i /root/.ssh/itpp-infra /tmp/uptime-kuma-compose.yaml root@159.195.205.80:/root/docker/uptime-kuma/docker-compose.yml
|
||||
```
|
||||
2. Take a **live-safe SQLite snapshot** on Core (do not `cp` a live WAL DB) and copy it to app4, matching the
|
||||
pattern the existing `core-services-backup.sh` already uses for this exact file:
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "sqlite3 -readonly /root/docker/uptime-kuma/data/kuma.db '.backup /tmp/kuma-migrate.db' && sqlite3 -readonly /tmp/kuma-migrate.db 'PRAGMA quick_check;'"
|
||||
# Expected: ok
|
||||
```
|
||||
3. Announce the monitoring downtime window, then **stop** the Core container (this is the one write-adjacent
|
||||
action required — flagged explicitly since it is the monitoring source of truth; confirm with the requester
|
||||
before executing, this runbook documents the command but does not execute it):
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "docker stop uptime-kuma"
|
||||
```
|
||||
4. Final consistent copy (container stopped, no WAL risk) plus the `docker-tls` subfolder (Kuma's own
|
||||
self-issued monitor certs, if any monitors use them):
|
||||
```
|
||||
rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/root/docker/uptime-kuma/data/ root@159.195.205.80:/root/docker/uptime-kuma/data/
|
||||
```
|
||||
5. Start on app4:
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "cd /root/docker/uptime-kuma && docker compose up -d"
|
||||
```
|
||||
6. Add both Caddy blocks to app4's Caddyfile (mirroring Core, `default_bind` already set globally):
|
||||
```
|
||||
uptimekuma.itpropartner.com {
|
||||
handle /health {
|
||||
respond "OK" 200
|
||||
}
|
||||
reverse_proxy 127.0.0.1:3001
|
||||
}
|
||||
```
|
||||
(`status.itpropartner.com`'s `/api/*` proxy to 3001 is bundled with the Transitpin relay on the same domain —
|
||||
document it here but do not create a standalone `status.itpropartner.com` block until Transitpin's own WAVE
|
||||
migration is scheduled, to avoid a half-built multi-path site block. `caddy validate` app4's file before reload.)
|
||||
|
||||
### Verification (exact command + expected output)
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker exec uptime-kuma wget -qO- http://localhost:3001 | head -c 200"
|
||||
# Expected: HTML containing "<title>Uptime Kuma</title>" (or a 200 via curl -o /dev/null -w '%{http_code}')
|
||||
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sqlite3 -readonly /root/docker/uptime-kuma/data/kuma.db 'SELECT COUNT(*) FROM monitor;'"
|
||||
# Expected: same monitor count as Core's kuma.db at snapshot time — compare with:
|
||||
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "sqlite3 -readonly /root/docker/uptime-kuma/data/kuma.db 'SELECT COUNT(*) FROM monitor;'"
|
||||
# These two numbers MUST match exactly before cutover.
|
||||
|
||||
curl --resolve uptimekuma.itpropartner.com:443:159.195.205.80 -sI https://uptimekuma.itpropartner.com/
|
||||
# Expected (post-DNS-flip, cert issued): HTTP/2 200
|
||||
```
|
||||
|
||||
### Rollback
|
||||
1. `ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker compose -f /root/docker/uptime-kuma/docker-compose.yml down"` (stop app4's copy).
|
||||
2. `ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "docker start uptime-kuma"` (Core's container and data dir
|
||||
were never deleted — only stopped — so this brings monitoring back exactly where it left off).
|
||||
3. If DNS was already flipped, revert the `uptimekuma.itpropartner.com` A record to `152.53.192.33`.
|
||||
4. No data loss: Core's `kuma.db` is untouched by the migration (copy was one-directional, read-only source).
|
||||
|
||||
---
|
||||
|
||||
## Service 3 of 6 — Prometheus + Grafana + node_exporter monitoring stack
|
||||
|
||||
Treated as one unit because Prometheus scrapes node_exporter and feeds Grafana's dashboards; moving them
|
||||
together avoids a window where Grafana has no data source. **Grafana's own port/auth finding is broken out
|
||||
separately as Service 4** since it needs an explicit fix, not just a move.
|
||||
|
||||
### Current state (Core)
|
||||
- **Prometheus**: container `prometheus` (`prom/prometheus:latest`, `unless-stopped`, `NetworkMode: host`).
|
||||
Config bind-mounted `/root/docker/monitoring/prometheus/prometheus.yml -> /etc/prometheus/prometheus.yml`
|
||||
(ro). Data: named volume `prometheus_data` (`/var/lib/docker/volumes/prometheus_data/_data`, **114.9 MB**).
|
||||
Also bind-mounts `/var/lib/prometheus/textfile` (ro) for host-metrics-exporter's textfile collector. No
|
||||
compose file found — appears to be a manual `docker run` (ambiguity #1 in the inventory doc); reconstruct as
|
||||
a compose file on app4 rather than perpetuating a hand-run container.
|
||||
Listens `*:9090` (host network).
|
||||
`prometheus.yml` (50 lines total) scrape jobs: `prometheus` (self, 127.0.0.1:9090), `mikrotik` (127.0.0.1:9436
|
||||
via mikrotik-exporter — **stays on Core**, it's scraping Core's local network device), `mikrotik-snmp`
|
||||
(127.0.0.1:9274), `core-telegraf` (127.0.0.1:9273 — **stays on Core**, host-local metrics), `super-search`
|
||||
(172.17.0.1:8899/metrics — Docker bridge IP, will need re-pointing if super-search moves), `node_exporter`
|
||||
(already has 3 targets: `127.0.0.1:9100` labeled `core`, `159.195.205.80:9100` labeled `app4`,
|
||||
`159.195.204.203:9100` labeled `core-bu` — this job already scrapes app4 and core-bu remotely from Core, added
|
||||
during Phase 1 provisioning).
|
||||
- **Grafana**: container `grafana` (`grafana/grafana:11.4.0`, `unless-stopped`, `NetworkMode: host`, no ports
|
||||
block — the image binds directly since it's on host networking). Named volume `grafana_data_final`
|
||||
(`/var/lib/docker/volumes/grafana_data_final/_data`, **~15 MB**) -> `/var/lib/grafana`. Two orphaned volumes
|
||||
exist alongside it on Core (`grafana_data` 50 MB, `grafana_data_v3` 14.6 MB) — **do not migrate these**, they
|
||||
are stale, only `grafana_data_final` is live per the running container's mount.
|
||||
Config: `/root/docker/monitoring/grafana/grafana.ini`, referenced via env vars
|
||||
(`GF_PATHS_CONFIG=/etc/grafana/grafana.ini`) — confirm at move time whether it's a bind mount or baked into
|
||||
the volume; treat the working copy at `/root/docker/monitoring/grafana/grafana.ini` as authoritative source
|
||||
of truth for env/ini settings.
|
||||
- **node_exporter**: NOT a container — a systemd unit (`/etc/systemd/system/node_exporter.service`,
|
||||
`ExecStart=/usr/local/bin/node_exporter`, no flags, `Restart=unless-stopped`). Binds `*:9100`.
|
||||
**app4 already runs its own node_exporter** (systemd, active, listening `*:9100`, confirmed
|
||||
`ss -ltnp | grep 9100` shows `node_exporter` on app4 right now) — this was stood up in Phase 1 provisioning
|
||||
and is already scraped by Core's Prometheus (see the `app4` label above). **node_exporter is therefore
|
||||
ALREADY DONE on app4 — no move action needed for this piece.**
|
||||
- Caddy: **no Caddy site block routes to Prometheus or Grafana today.** Neither `:9090` nor `:3002` appears
|
||||
anywhere in Core's 49 Caddy site blocks. Access today is by direct IP:port only.
|
||||
- Cert: none — neither service is behind TLS today.
|
||||
- Firewall (Core `ufw status numbered`): rule `[7] 8899/tcp ALLOW IN 172.17.0.0/16` (Docker-bridge-only, for
|
||||
super-search's own metrics scrape — not customer-facing) and `[8] 3002/tcp ALLOW IN 100.64.0.0/10`
|
||||
(Tailscale-range-only for Grafana) exist, but **UFW is a second gate behind the actual bind** — see Service 4,
|
||||
because Grafana's `*:3002` listen and this UFW rule together still leave a gap, which is the whole point of
|
||||
the Service 4 finding.
|
||||
|
||||
### Pre-flight checks
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "docker inspect prometheus --format '{{.Config.Image}} {{.State.Status}}'; docker inspect grafana --format '{{.Config.Image}} {{.State.Status}}'"
|
||||
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "du -sh /var/lib/docker/volumes/prometheus_data/_data /var/lib/docker/volumes/grafana_data_final/_data"
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl is-active node_exporter; curl -s http://127.0.0.1:9100/metrics | head -1"
|
||||
# Expected: active / a line starting "# HELP go_gc_duration_seconds"
|
||||
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "curl -s 'http://127.0.0.1:9090/api/v1/query?query=up' | python3 -m json.tool | head -20"
|
||||
# Baseline: capture current up{} series before the move for a before/after diff
|
||||
```
|
||||
|
||||
### Move commands
|
||||
1. Create target dirs on app4:
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "mkdir -p /root/docker/monitoring/prometheus /root/docker/monitoring/grafana"
|
||||
```
|
||||
2. Copy configs (read-only source copy from Core, nothing stopped yet):
|
||||
```
|
||||
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/monitoring/prometheus/prometheus.yml /tmp/prometheus.yml
|
||||
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/monitoring/grafana/grafana.ini /tmp/grafana.ini
|
||||
```
|
||||
3. **Edit the copy** (never Core's file) before pushing to app4: remove the `mikrotik`, `mikrotik-snmp`, and
|
||||
`core-telegraf` jobs (Core-local, stay on Core) from `/tmp/prometheus.yml`; re-point `super-search`'s target
|
||||
from `172.17.0.1:8899` to wherever Super Search lands (Service 6 — if it also moves to app4 in this wave,
|
||||
target becomes `127.0.0.1:8899`; if it stays put, keep scraping Core's public IP with a dedicated firewall
|
||||
allow, decided at Service 6 time). Push:
|
||||
```
|
||||
scp -i /root/.ssh/itpp-infra /tmp/prometheus.yml root@159.195.205.80:/root/docker/monitoring/prometheus/prometheus.yml
|
||||
scp -i /root/.ssh/itpp-infra /tmp/grafana.ini root@159.195.205.80:/root/docker/monitoring/grafana/grafana.ini
|
||||
```
|
||||
4. rsync the live volume data in two passes — a warm first pass while Core's containers still run, then a
|
||||
final short-stop pass for consistency (same pattern as the Uptime Kuma DB):
|
||||
```
|
||||
rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/var/lib/docker/volumes/prometheus_data/_data/ /tmp/prom-data-staging/
|
||||
rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/var/lib/docker/volumes/grafana_data_final/_data/ /tmp/grafana-data-staging/
|
||||
# announce downtime, then: ssh ... root@152.53.192.33 "docker stop prometheus grafana"
|
||||
rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/var/lib/docker/volumes/prometheus_data/_data/ /tmp/prom-data-staging/
|
||||
rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/var/lib/docker/volumes/grafana_data_final/_data/ /tmp/grafana-data-staging/
|
||||
scp -r -i /root/.ssh/itpp-infra /tmp/prom-data-staging root@159.195.205.80:/tmp/
|
||||
scp -r -i /root/.ssh/itpp-infra /tmp/grafana-data-staging root@159.195.205.80:/tmp/
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker volume create prometheus_data && docker run --rm -v prometheus_data:/dest -v /tmp/prom-data-staging:/src alpine cp -a /src/. /dest/"
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker volume create grafana_data_final && docker run --rm -v grafana_data_final:/dest -v /tmp/grafana-data-staging:/src alpine cp -a /src/. /dest/"
|
||||
```
|
||||
5. Write `/root/docker/monitoring/docker-compose.yml` on app4, images pinned by digest (resolve fresh:
|
||||
`docker inspect prometheus --format '{{index .RepoDigests 0}}'` on Core for Prometheus; Grafana's tag
|
||||
`11.4.0` is already a pin — resolve its digest the same way), `network_mode: host` preserved for Prometheus
|
||||
to match Core's scrape behavior; Grafana's networking is changed — see Service 4.
|
||||
6. `docker compose up -d` on app4.
|
||||
|
||||
### Verification
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s http://127.0.0.1:9090/-/healthy"
|
||||
# Expected: "Prometheus Server is Healthy."
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s 'http://127.0.0.1:9090/api/v1/query?query=up' | python3 -m json.tool"
|
||||
# Expected: same target set (minus mikrotik/telegraf/core-only jobs) all reporting value "1"
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:3002/api/health"
|
||||
# Expected: 200
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s http://127.0.0.1:3002/api/health"
|
||||
# Expected JSON containing "database":"ok","version":"11.4.0" — confirms grafana.db loaded correctly
|
||||
```
|
||||
Series-count parity check (compare Core's pre-stop snapshot vs app4's loaded volume before deleting anything
|
||||
on Core):
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s http://127.0.0.1:9090/api/v1/label/__name__/values | python3 -c 'import json,sys; print(len(json.load(sys.stdin)[\"data\"]))'"
|
||||
# Compare metric-name count to the same query run on Core before stop
|
||||
```
|
||||
|
||||
### Rollback
|
||||
1. `ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "docker compose -f /root/docker/monitoring/docker-compose.yml down"`.
|
||||
2. `ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "docker start prometheus grafana"` — Core's volumes and
|
||||
config were only read from, never deleted; original containers restart with their original state intact.
|
||||
3. No DNS is involved (no Caddy route exists either direction today), so rollback is purely stop-app4/
|
||||
start-Core with no propagation delay.
|
||||
4. Any staging copy edited on the operator's own box (`/tmp/prometheus.yml`) is disposable; Core's live
|
||||
`prometheus.yml` was never touched, per the read-only constraint on Core.
|
||||
|
||||
---
|
||||
|
||||
## Service 4 of 6 — Grafana public-bind finding (`*:3002`, unauthenticated, no Caddy/Tailscale gate)
|
||||
|
||||
This is called out separately from Service 3 because it is a **security finding to fix during the move**, not
|
||||
just a relocation.
|
||||
|
||||
### The finding, precisely
|
||||
- Grafana's container runs `NetworkMode: host` and the Grafana process itself binds `*:3002` (confirmed via
|
||||
`ss -ltnp`: `LISTEN 0 4096 *:3002 *:* users:(("grafana",pid=2906,fd=15))`) — i.e. **every** interface on Core,
|
||||
including its public IPv4/IPv6, not just loopback or the Tailscale interface.
|
||||
- It is **not** behind any Caddy site block (grepped all 49 blocks in Core's Caddyfile — zero mention of `3002`
|
||||
or a Grafana upstream).
|
||||
- The only mitigation in place is a UFW rule restricting **inbound port 3002 to the Tailscale CGNAT range**
|
||||
(`100.64.0.0/10`) — `ufw status numbered` line `[8] 3002/tcp ALLOW IN 100.64.0.0/10`. UFW's default policy is
|
||||
`deny (incoming)`, so absent that specific allow rule, 3002 would in fact be closed by default — **but this
|
||||
means the ONLY thing standing between Grafana and the open internet is one UFW line**, with no defense in
|
||||
depth (no app-level auth beyond Grafana's own login, no Caddy layer, no Tailscale-only bind at the app level).
|
||||
- Compounding it: Grafana's `grafana.ini` `[security]` section has
|
||||
`admin_user = admin` / `admin_password = admin` / `disable_initial_admin_creation = false` — **the default
|
||||
admin credentials were never changed**. If the UFW rule is ever misconfigured, misapplied to the wrong
|
||||
interface, or a firewall reload race drops it (a known class of bug on reboot ordering), the admin/admin
|
||||
login is directly exposed to anyone who reaches that interface.
|
||||
|
||||
### Fix to apply on app4 (do NOT attempt this on Core — Core is read-only for this task)
|
||||
1. Do not repeat the `NetworkMode: host` + open-bind pattern on app4. In the Service 3 compose file, add an
|
||||
explicit port mapping and drop host networking for Grafana specifically (Prometheus can keep host
|
||||
networking since it only needs outbound scrape access; Grafana only needs to reach Prometheus on the same
|
||||
host, no host networking required):
|
||||
```yaml
|
||||
grafana:
|
||||
image: grafana/grafana@sha256:<resolve-at-execution-time>
|
||||
ports:
|
||||
- "127.0.0.1:3002:3000" # loopback only; Caddy or Tailscale serves it externally
|
||||
```
|
||||
2. Add a real Caddy site block on app4 gating it behind HTTP basic auth (mirroring the existing pattern used
|
||||
for `internal.debtrecoveryexperts.com`'s `basic_auth` block in Core's Caddyfile):
|
||||
```
|
||||
grafana.itpropartner.com {
|
||||
basic_auth {
|
||||
admin <bcrypt-hash-generated-with-caddy-hash-password>
|
||||
}
|
||||
reverse_proxy 127.0.0.1:3002
|
||||
}
|
||||
```
|
||||
(Generate the hash with `caddy hash-password` — never hand-write a bcrypt hash.)
|
||||
3. Change the default admin credentials in `grafana.ini` (or via `GF_SECURITY_ADMIN_PASSWORD` env var) on
|
||||
app4's first boot — do not carry `admin/admin` forward. This must happen before app4's Grafana is reachable
|
||||
by anything but loopback.
|
||||
4. Confirm `ufw status` on app4 does not carry forward the Core Tailscale-3002 rule as the *only* gate — Caddy
|
||||
basic_auth is now gate 1; Tailscale-only access can remain as an additional, not sole, control.
|
||||
|
||||
### Verification
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "ss -ltnp | grep 3002"
|
||||
# Expected: 127.0.0.1:3002 only — NOT *:3002 or 0.0.0.0:3002
|
||||
curl -s -o /dev/null -w '%{http_code}\n' http://159.195.205.80:3002/
|
||||
# Expected: connection refused / timeout (nothing listening on the public interface)
|
||||
curl -s -o /dev/null -w '%{http_code}\n' https://grafana.itpropartner.com/
|
||||
# Expected: 401 (basic_auth challenge) before credentials, 200 after
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "grep -E 'admin_password' /root/docker/monitoring/grafana/grafana.ini"
|
||||
# Expected: NOT the literal string "admin" — confirm it was actually changed
|
||||
```
|
||||
|
||||
### Rollback
|
||||
This fix only touches app4, a not-yet-cutover host; Core is never modified. Rollback is simply: do not flip
|
||||
DNS/Tailscale access to app4's Grafana until the verification above passes. Core's Grafana keeps running with
|
||||
its existing (already-known-weak) posture unchanged — a pre-existing condition to flag to Germaine regardless
|
||||
of migration timing, not something this runbook worsens or is required to fix on Core.
|
||||
|
||||
---
|
||||
|
||||
## Service 5 of 6 — PRY (unified OSINT search backend)
|
||||
|
||||
### Current state (Core)
|
||||
- Not a container — a systemd unit: `pry.service`
|
||||
(`/etc/systemd/system/pry.service`, `Type=simple`, `User=root`,
|
||||
`WorkingDirectory=/root/docker/pry`, `ExecStart=/root/docker/pry/venv/bin/python /root/docker/pry/server.py`,
|
||||
`EnvironmentFile=/root/.hermes/.env`, `Restart=always`, `ProtectSystem=full`, `ProtectHome=read-only`,
|
||||
`ReadWritePaths=/root/docker/pry`, `ReadOnlyPaths=/root/.hermes/.env /root/.config/himalaya/shonuff.pass`).
|
||||
- Port: `127.0.0.1:8905`, confirmed via `ss -ltnp` — loopback only, not directly internet-reachable.
|
||||
- App dir: `/root/docker/pry`, **66 MB total** (mostly its own `venv`). Python 3.13.5 venv, deps from
|
||||
`requirements.txt` (`fastapi>=0.109.0`, `uvicorn[standard]>=0.27.0`, `httpx>=0.26.0`, `python-dotenv>=1.0.0`,
|
||||
`aiosmtplib>=3.0.0` — 5 loose lines, no pinned freeze file, so re-resolve at install time rather than
|
||||
assuming exact matching versions).
|
||||
- Data/DB: **none found** — no `.db`/`.sqlite*` file in `/root/docker/pry`. PRY is stateless app logic; its
|
||||
only persistent asset is its git-tracked source (`.git` present) plus a private `Caddyfile-new` (mode 600,
|
||||
4842 bytes — appears to be a staged/unused Caddy config someone drafted but never applied; carry it over for
|
||||
reference but do not activate it blindly).
|
||||
- Secrets: `EnvironmentFile=/root/.hermes/.env` — shared secrets file across many services; only PRY-relevant
|
||||
keys need to be extracted/copied to app4's own `.env`, not the whole file (per the inventory doc's finding #7
|
||||
on live secrets requiring manual, non-rsync transfer).
|
||||
- Caddy route (Core): `http://pry.iamgmb.com { @api path /api/* ; handle @api { reverse_proxy 127.0.0.1:8905 } ; handle { root * /var/www/pry ; try_files ... ; file_server } }` — **HTTP only, no TLS** (`http://` scheme
|
||||
explicit in the Caddyfile block header), and it's a split route: `/api/*` goes to PRY's backend, everything
|
||||
else is a static SPA served from `/var/www/pry` on Core. The static half is out of scope for a backend-only
|
||||
WAVE 1 move — moving PRY's backend without also moving `/var/www/pry`'s static files would break the
|
||||
frontend, so **both must move together** even though the task frames this as one backend service.
|
||||
- Cert: none (HTTP-only site today).
|
||||
|
||||
### Pre-flight checks
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "systemctl is-active pry.service; ss -ltnp | grep 8905"
|
||||
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8905/api/health || curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8905/"
|
||||
# Record baseline response code before moving
|
||||
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "du -sh /var/www/pry"
|
||||
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "grep -c '^[A-Z_]*=' /root/.hermes/.env"
|
||||
# Confirms the shared secrets file's line count so a partial extraction can be sanity-checked for completeness
|
||||
```
|
||||
|
||||
### Move commands
|
||||
1. Copy the app dir (excluding the venv, which will be rebuilt fresh on app4 to avoid glibc/arch drift):
|
||||
```
|
||||
rsync -avz -e "ssh -i /root/.ssh/itpp-infra" --exclude 'venv' --exclude '__pycache__' root@152.53.192.33:/root/docker/pry/ root@159.195.205.80:/root/docker/pry/
|
||||
```
|
||||
2. Copy the static frontend:
|
||||
```
|
||||
rsync -avz -e "ssh -i /root/.ssh/itpp-infra" root@152.53.192.33:/var/www/pry/ root@159.195.205.80:/var/www/pry/
|
||||
```
|
||||
3. Rebuild the venv on app4:
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "cd /root/docker/pry && python3 -m venv venv && ./venv/bin/pip install -r requirements.txt"
|
||||
```
|
||||
4. Manually extract PRY-relevant keys from Core's `/root/.hermes/.env` (grep for the specific vars `server.py`
|
||||
actually reads — do this by inspecting `os.getenv`/`os.environ` calls in `server.py`, not by copying the
|
||||
whole file) into a new, minimal `/root/.hermes/.env` on app4, or a dedicated `/root/docker/pry/.env` +
|
||||
adjust the systemd unit's `EnvironmentFile=` path. This is a manual, secure step — never rsync the shared
|
||||
`.env` wholesale (matches the inventory doc's explicit warning on secret files).
|
||||
5. Create the systemd unit on app4 (same content as Core's, `WorkingDirectory`/`ExecStart` paths already match
|
||||
since both boxes use `/root/docker/pry`):
|
||||
```
|
||||
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/pry/pry.service /tmp/pry.service
|
||||
# edit EnvironmentFile= path if app4 uses a dedicated .env, then:
|
||||
scp -i /root/.ssh/itpp-infra /tmp/pry.service root@159.195.205.80:/etc/systemd/system/pry.service
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl daemon-reload && systemctl enable --now pry.service"
|
||||
```
|
||||
6. Add the Caddy block to app4, upgraded to real TLS this time (Caddy on app4 will ACME-issue since it's a
|
||||
real domain — no reason to keep it HTTP-only post-move):
|
||||
```
|
||||
pry.iamgmb.com {
|
||||
@api path /api/*
|
||||
handle @api {
|
||||
reverse_proxy 127.0.0.1:8905
|
||||
}
|
||||
handle {
|
||||
root * /var/www/pry
|
||||
try_files {path} {path}.html /index.html
|
||||
file_server
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Verification
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl is-active pry.service"
|
||||
# Expected: active
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8905/"
|
||||
# Expected: same status code as the Core baseline captured in pre-flight
|
||||
curl --resolve pry.iamgmb.com:443:159.195.205.80 -sI https://pry.iamgmb.com/
|
||||
# Expected (post-DNS-flip, cert issued): HTTP/2 200
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "journalctl -u pry.service -n 20 --no-pager"
|
||||
# Expected: no tracebacks, normal startup log lines matching Core's own startup pattern
|
||||
```
|
||||
|
||||
### Rollback
|
||||
1. `ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl stop pry.service"`.
|
||||
2. Core's `pry.service` was never touched — it is still `active` throughout, so no restart action needed on
|
||||
Core.
|
||||
3. If DNS was flipped, revert `pry.iamgmb.com` to `152.53.192.33` (note: this domain has no `A` vs `CNAME`
|
||||
redirect complication, it's a direct A record per the plan's DNS checklist).
|
||||
4. No data to restore (stateless service, no DB).
|
||||
|
||||
---
|
||||
|
||||
## Service 6 of 6 — OSINT Person MCP (`:8902`) + Super Search MCP (`:8899`)
|
||||
|
||||
These are grouped because OSINT Person MCP imports Super Search's own module directly
|
||||
(`from super_search import web_search as ss_search, web_extract as ss_extract` in
|
||||
`osint-person-mcp/server.py`) — they have a **code-level dependency**, not just a runtime one, so they must
|
||||
move together or the import breaks.
|
||||
|
||||
### Current state (Core)
|
||||
|
||||
**OSINT Person MCP:**
|
||||
- systemd unit `osint-person.service` (`After=network.target super-search.service` — systemd-level ordering
|
||||
dependency confirming the two are coupled), `WorkingDirectory=/root/docker/osint-person-mcp`,
|
||||
`ExecStart=/root/docker/osint-person-mcp/venv/bin/python3 /root/docker/osint-person-mcp/server.py`,
|
||||
`EnvironmentFile=/root/.hermes/.env`, `Restart=always`.
|
||||
- Port: `127.0.0.1:8902` (confirmed `ss -ltnp`), used as `http://127.0.0.1:8902/mcp` per Hermes'
|
||||
`config.yaml` MCP registration — loopback only.
|
||||
- App dir: `/root/docker/osint-person-mcp`, **314 MB** (Python 3.13.5 venv-heavy). Source files include
|
||||
`server.py` (54,253 bytes, modified same-day as this scan — active development), `reconcile.py`,
|
||||
`super_search.py` (a **local copy** of Super Search's search/extract functions, imported directly — this is
|
||||
the code coupling), `telemetry.py`, plus backup files (`server.py.bak-*`) and a `tests/` dir. No standalone
|
||||
requirements file was found in the directory listing — dependencies live only in the existing `venv`;
|
||||
capture `venv/bin/pip freeze > requirements.txt` on Core before rebuilding on app4, do not guess versions.
|
||||
- Data/DB: none found (no `.db` file); it's a stateless MCP tool server backed by live web calls.
|
||||
|
||||
**Super Search MCP:**
|
||||
- systemd unit `super-search.service` (`WorkingDirectory=/root/docker/super-search`,
|
||||
`ExecStartPre=/bin/bash /root/docker/super-search/super-search-port-guard.sh` — a pre-start guard script,
|
||||
copy this too, it likely checks for a stale port bind before starting — `ExecStart=.../venv/bin/python3
|
||||
.../server.py`, `EnvironmentFile=/root/.hermes/.env`, `Restart=always`, `StartLimitBurst=5` over 300s).
|
||||
- Port: `0.0.0.0:8899` — confirmed **bound to all interfaces on Core**, mitigated only by
|
||||
`ufw status numbered` rule `[7] 8899/tcp ALLOW IN 172.17.0.0/16` (Docker-bridge-range only — i.e. only
|
||||
containers on Core's own `docker0`/bridge network can reach it externally-looking-in; this is intentional
|
||||
since Prometheus's `super-search` scrape job targets `172.17.0.1:8899`, the bridge gateway IP, meaning
|
||||
Prometheus itself runs in a container reaching it via the bridge). Registered in Hermes as
|
||||
`http://127.0.0.1:8899/mcp`.
|
||||
- App dir: `/root/docker/super-search`, **1.1 GB** (largest of the six — heavy venv, Python 3.13.5). Has a
|
||||
pinned `requirements.freeze.txt` (167 lines) — **use this file directly on app4**, it's already a proper pin,
|
||||
unlike PRY's loose `requirements.txt` or OSINT Person's missing one.
|
||||
- Data/DB: none found — stateless, backed by live search-provider calls (SearXNG on Core, Exa, Brave, etc. per
|
||||
the skill's provider list); note SearXNG itself (`127.0.0.1:8888`) **stays on Core** per the inventory doc,
|
||||
meaning **if Super Search moves to app4, its SearXNG provider call becomes a cross-Atlantic Core→app4→Core
|
||||
round trip (~100ms each way per the migration plan's RTT finding) unless SearXNG is also exposed to app4**,
|
||||
which is out of WAVE 1 scope. Recommend either: (a) leave Super Search on Core for now despite the task's
|
||||
inclusion of it here, since its own migration-plan classification already says "stays-on-Core (Hermes MCP)"
|
||||
in the inventory doc (contradicting this task's WAVE 1 list — flag this conflict to the requester explicitly
|
||||
before executing), or (b) accept the added latency for one SearXNG-backed provider path while other
|
||||
providers (Exa, Brave, Serper, Tavily) call out directly with no Core round-trip. **This is a real
|
||||
discrepancy between the source inventory and this task's WAVE 1 scope — documented here, not silently
|
||||
resolved.**
|
||||
- Caddy: **neither service has a Caddy route today.** Both are pure loopback/bridge MCP servers consumed only
|
||||
by Hermes' own `config.yaml` `url: http://127.0.0.1:PORT/mcp` entries — there is no public hostname for
|
||||
either in the 49 scanned Caddy blocks.
|
||||
- Cert: none (no public exposure today; loopback-only by design).
|
||||
|
||||
### Pre-flight checks
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "systemctl is-active osint-person.service super-search.service"
|
||||
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8902/mcp; curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8899/mcp"
|
||||
# Record baseline codes (MCP endpoints often 4xx on a bare GET without proper headers — record whatever Core
|
||||
# returns today as the comparison target, don't assume 200)
|
||||
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "/root/docker/osint-person-mcp/venv/bin/pip freeze > /tmp/osint-person-freeze.txt; wc -l /tmp/osint-person-freeze.txt"
|
||||
ssh -i /root/.ssh/itpp-infra root@152.53.192.33 "du -sh /root/docker/osint-person-mcp /root/docker/super-search"
|
||||
```
|
||||
|
||||
### Move commands
|
||||
1. Freeze OSINT Person's deps (not previously pinned) and pull both freeze files:
|
||||
```
|
||||
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/tmp/osint-person-freeze.txt /tmp/
|
||||
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/super-search/requirements.freeze.txt /tmp/
|
||||
```
|
||||
2. Copy source (excluding venv/pycache/backup cruft — the `.bak-*` files are large and not needed):
|
||||
```
|
||||
rsync -avz -e "ssh -i /root/.ssh/itpp-infra" --exclude 'venv' --exclude '__pycache__' --exclude '*.bak*' root@152.53.192.33:/root/docker/osint-person-mcp/ root@159.195.205.80:/root/docker/osint-person-mcp/
|
||||
rsync -avz -e "ssh -i /root/.ssh/itpp-infra" --exclude 'venv' --exclude '__pycache__' --exclude '*.bak*' root@152.53.192.33:/root/docker/super-search/ root@159.195.205.80:/root/docker/super-search/
|
||||
```
|
||||
3. Rebuild both venvs on app4 from the frozen requirements:
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "cd /root/docker/osint-person-mcp && python3 -m venv venv && ./venv/bin/pip install -r /tmp/osint-person-freeze.txt"
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "cd /root/docker/super-search && python3 -m venv venv && ./venv/bin/pip install -r requirements.freeze.txt"
|
||||
```
|
||||
4. Extract only the OSINT/Super-Search-relevant keys from Core's `/root/.hermes/.env` into app4's own
|
||||
`.hermes/.env` (same manual-secret-transfer rule as PRY — do not rsync the whole file).
|
||||
5. Copy systemd units and the port-guard script, adjusting nothing but confirming paths already match
|
||||
(`/root/docker/...` is identical on both boxes):
|
||||
```
|
||||
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/osint-person-mcp/osint-person.service /tmp/
|
||||
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/super-search/super-search.service /tmp/
|
||||
scp -i /root/.ssh/itpp-infra root@152.53.192.33:/root/docker/super-search/super-search-port-guard.sh /tmp/
|
||||
scp -i /root/.ssh/itpp-infra /tmp/osint-person.service /tmp/super-search.service root@159.195.205.80:/etc/systemd/system/
|
||||
scp -i /root/.ssh/itpp-infra /tmp/super-search-port-guard.sh root@159.195.205.80:/root/docker/super-search/
|
||||
```
|
||||
6. **Fix Super Search's bind before enabling** — do not carry `0.0.0.0:8899` forward. If `server.py` hardcodes
|
||||
the bind host, patch app4's copy to `127.0.0.1:8899` (matching OSINT Person's already-safe loopback pattern)
|
||||
unless something on app4 genuinely needs the Docker-bridge-range access Core's UFW rule was protecting
|
||||
(confirm this is even needed on app4 — app4 doesn't run the same Prometheus container-bridge topology as
|
||||
Core unless Service 3 also lands there with the same `172.17.0.1` scrape pattern).
|
||||
7. Enable both, in dependency order (Super Search first, since OSINT Person's own unit declares
|
||||
`After=... super-search.service`):
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl daemon-reload && systemctl enable --now super-search.service && sleep 3 && systemctl enable --now osint-person.service"
|
||||
```
|
||||
8. Update Hermes' `config.yaml` MCP `url:` entries — but only **after** confirming which box's Hermes actually
|
||||
calls these tools. If it's Core's own Hermes gateway consuming these via `127.0.0.1`, moving the servers to
|
||||
app4 means Core's config must change to `http://159.195.205.80:8902/mcp` /
|
||||
`http://159.195.205.80:8899/mcp`, which reintroduces exactly the cross-Atlantic per-call latency the
|
||||
migration plan's Section 3.1 design rule warns against ("app4 must not depend on Core at request time" — the
|
||||
inverse also applies: Core's live Hermes gateway should not depend on app4 at request time for
|
||||
every-tool-call-latency reasons). **This is a design question for the requester, not a call this runbook
|
||||
makes** — flagged here rather than silently deciding to change `config.yaml`.
|
||||
|
||||
### Verification
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl is-active osint-person.service super-search.service"
|
||||
# Expected: active / active
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "ss -ltnp | grep -E ':(8899|8902)\b'"
|
||||
# Expected: 127.0.0.1:8899 and 127.0.0.1:8902 — NOT 0.0.0.0:8899 (confirms the bind fix from step 6 landed)
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8902/mcp; curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8899/mcp"
|
||||
# Expected: same status codes as the Core baseline captured in pre-flight
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "journalctl -u osint-person.service -u super-search.service -n 30 --no-pager"
|
||||
# Expected: no import errors (specifically watch for "ModuleNotFoundError: super_search" — the code-coupling
|
||||
# risk called out above — and no missing-env-var tracebacks)
|
||||
```
|
||||
|
||||
### Rollback
|
||||
1. `ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "systemctl stop osint-person.service super-search.service"`.
|
||||
2. Core's two units were never stopped or restarted — they remain `active` throughout, so Hermes' existing
|
||||
`127.0.0.1` MCP URLs keep working with zero interruption regardless of app4's state.
|
||||
3. If `config.yaml` was changed (step 8), revert the `url:` fields back to `http://127.0.0.1:8902/mcp` and
|
||||
`http://127.0.0.1:8899/mcp` — this requires whichever Hermes profile/gateway consumes them to reload its MCP
|
||||
config (out of scope for this SSH-only runbook; note it as a manual follow-up).
|
||||
4. No data to restore (both stateless).
|
||||
|
||||
---
|
||||
|
||||
## Section 8 — Backup coverage gap: `/root/docker` missing from app4's `root-essentials-backup.sh`
|
||||
|
||||
### The gap
|
||||
app4's `/root/root-essentials-backup.sh` (v2.0.0, 2026-09-11) builds its include list from this loop:
|
||||
```bash
|
||||
for p in .hermes/config.yaml .hermes/.env .hermes/skills .hermes/scripts \
|
||||
.hermes/references .hermes/profiles .hermes/cron .hermes/memories \
|
||||
.hermes/DR-PLAN.md .hermes/data .hermes/cache/delegation \
|
||||
.ssh .aws .config/himalaya \
|
||||
shark-game projects ; do
|
||||
[ -e "$p" ] && ARGS+=("$p")
|
||||
done
|
||||
```
|
||||
`/root/docker` (relative: `docker`) is **not in this list**. Every compose stack this runbook lands on app4 —
|
||||
Uptime Kuma's `docker-compose.yml` + its data dir path, the monitoring stack's compose file, PRY's app dir if
|
||||
placed under `/root/docker` (it is), OSINT Person MCP, Super Search — is therefore **entirely invisible** to
|
||||
the nightly 04:45 ET run, even though the script's own `tar` excludes `*.db`/`*.db-shm`/`*.db-wal` regardless
|
||||
(so Uptime Kuma's `kuma.db` and any future SQLite files still need their own `sqlite3 .backup` job even after
|
||||
this fix — this patch closes the "missing compose/config/source" half of the gap, not the "missing DB" half,
|
||||
which is already flagged as a separate Phase 4/5 acceptance item in the migration plan).
|
||||
|
||||
### Exact patch
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sed -n '55,60p' /root/root-essentials-backup.sh"
|
||||
# Confirm current line content before editing, since this is a live production script:
|
||||
# .ssh .aws .config/himalaya \
|
||||
# shark-game projects ; do
|
||||
```
|
||||
Change:
|
||||
```diff
|
||||
- .ssh .aws .config/himalaya \
|
||||
- shark-game projects ; do
|
||||
+ .ssh .aws .config/himalaya \
|
||||
+ shark-game projects docker ; do
|
||||
```
|
||||
Apply with `patch`/`sed` on app4 (do NOT run this against Core — Core is read-only for this task, and Core's
|
||||
own backup script coverage is a separate, already-tracked item per the inventory doc's Section on backup gaps):
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sed -i 's/shark-game projects ; do/shark-game projects docker ; do/' /root/root-essentials-backup.sh"
|
||||
```
|
||||
|
||||
### Verify the patch landed correctly
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "grep -n 'shark-game projects' /root/root-essentials-backup.sh"
|
||||
# Expected: shark-game projects docker ; do
|
||||
```
|
||||
|
||||
### Restore test (prove the new coverage actually works, not just that the line changed)
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "bash /root/root-essentials-backup.sh 2>&1 | tail -20"
|
||||
# Expected final lines: "[...] Tarball: <N> bytes" (larger than the pre-patch baseline by roughly the size of
|
||||
# /root/docker's non-.db content — compare against a du -sh /root/docker taken beforehand), then
|
||||
# "[...] content check OK - N entries, 3/3 key essentials present", then "Uploading...", then
|
||||
# "Backup verified - archive is valid", then "Done".
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "aws s3 cp s3://hermes-vps-backups/root-backup/\$(hostname -s)/root-essentials-\$(date +%F).tar.gz /tmp/restore-test.tar.gz --endpoint-url https://s3.us-east-1.wasabisys.com --quiet && tar tzf /tmp/restore-test.tar.gz | grep '^docker/' | head -20"
|
||||
# Expected: a non-empty listing of docker/<service>/... paths — proves /root/docker is now actually inside the
|
||||
# uploaded archive, not just inside the include-list source, and that the round-trip download+list works.
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "rm -f /tmp/restore-test.tar.gz"
|
||||
```
|
||||
|
||||
### Rollback (if the patched script misbehaves — e.g. archive size becomes unexpectedly huge from a stray venv)
|
||||
```
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "sed -i 's/shark-game projects docker ; do/shark-game projects ; do/' /root/root-essentials-backup.sh"
|
||||
ssh -i /root/.ssh/itpp-infra root@159.195.205.80 "grep -n 'shark-game projects' /root/root-essentials-backup.sh"
|
||||
# Expected: shark-game projects ; do (back to original)
|
||||
```
|
||||
If venv bloat is the issue (Super Search alone is 1.1 GB, mostly venv), prefer excluding venvs specifically
|
||||
over reverting the whole line:
|
||||
```
|
||||
# Add to the tar invocation's --exclude list (near the existing --exclude='*.db' flags):
|
||||
--exclude='docker/*/venv' --exclude='docker/*/*/venv'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Summary table
|
||||
|
||||
| # | Service | Status | Downtime needed | DNS flip needed | New security work |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | microbin | **DONE** (app layer), DNS flip pending | None (data dir was empty) | Yes — share.itpropartner.com | None |
|
||||
| 2 | Uptime Kuma | Runbook ready | Yes — brief, for final DB consistency | Yes — uptimekuma.itpropartner.com | Tighten `0.0.0.0:3001` → `127.0.0.1:3001` |
|
||||
| 3 | Prometheus+Grafana+node_exporter | Runbook ready (node_exporter already on app4) | Yes — brief, for volume consistency | None (no Caddy route today) | See Service 4 |
|
||||
| 4 | Grafana public-bind finding | Fix defined | N/A (fix applied pre-cutover) | Optional — grafana.itpropartner.com if exposed | Loopback bind + Caddy basic_auth + change admin/admin |
|
||||
| 5 | PRY | Runbook ready | None (stateless) | Yes — pry.iamgmb.com | Upgrade HTTP-only → real TLS |
|
||||
| 6 | OSINT Person MCP + Super Search MCP | Runbook ready, **scope conflict flagged** | None (stateless) | None (loopback/bridge-only, MCP-internal) | Fix `0.0.0.0:8899` → `127.0.0.1:8899`; resolve Core-Hermes-vs-app4-MCP latency question first |
|
||||
|
||||
**Open items for the requester before executing Services 3/6:** (a) Service 3's Prometheus scrape config needs
|
||||
a decision on where Super Search ends up before it can be finalized; (b) Service 6 conflicts with the source
|
||||
inventory's own classification of Super Search as "stays-on-Core" — confirm intent before executing.
|
||||
|
||||
Reference in New Issue
Block a user