itpp-infrastructure/CHANGELOG.md

# itpp-infrastructure — CHANGELOG

## 2026-09-11 — Anita's Hermes Profile Moved to a Dedicated Box (anita-mnz)

- **Infra move:** Anita's assistant profile moved off shared Core (`152.53.241.111`) to a dedicated box **anita-mnz `159.195.16.30`** (netcup, Manassas VA, 8 vCPU / 15 GB / 503 GB). Cutover 15:53 EDT, ~2 minutes dark, **zero messages lost**. She keeps the same Telegram bot and chat.
- **Why:** repeated `state.db` corruption on Core under co-tenant memory pressure (4th recurrence, `system_prompts` then `sessions` pages). A dedicated box removes the co-tenancy.
- **Method:** clean staged base + row-tolerant live-tail graft, so the corrupt live file never transferred. Signatures matched exactly: `99283/99283/166244414/157` (messages/maxid/contentbytes/sessions), `integrity_check ok`.
- **Post-move state:** Core `hermes-gateway-anita.service` stopped and **disabled** (no double-poller risk; standby app1-bu carries no anita unit); target gateway `active`+`enabled`, telegram `connected`, 0 `getUpdates` conflicts.
- **Backup:** her own box now runs the 3 AM root-essentials backup → `s3://hermes-vps-backups/root-backup/anita-mnz/`. Full run tested end-to-end 2026-09-11 15:56 (163 MB, upload + download/extract verify OK).
- **Core change:** `hermes-live-sync` no longer snapshots the frozen `profiles/anita` copy (it would advertise a stale db as "live").
- **MCP servers:** stripped from her profile (decision 2026-09-11: "She doesn't need access to those"). The `mcp_servers` block (`dre`, `osint-person`, `super-search` → `127.0.0.1:8900/8902/8899`) was removed from `anita/config.yaml`, which is why her log had been retry-parking those three every ~5 minutes since the move. Backed up first as `config.yaml.bak-mcpstrip-*`; 13 top-level keys verified intact. Takes effect on her next gateway restart.
- **Retired / order cancelled:** Nuremberg `89.58.44.96` (`v2202609377162518632.nicesrv.de`). **Correction to how this was first written here:** it does not "hold nothing". It holds a bare default-profile Hermes install: no `state.db`, empty `sessions/` and `cron/`, no systemd user units, no gateway process (only node_exporter, containerd, sshd). No unique data, so nothing needs preserving before cancellation. Recorded in `/root/.hermes/references/decommissioned-hosts.json`.
- **Monitoring fix:** `health-master-watchdog.py` watch-listed `hermes-gateway-anita.service` as a LOCAL user unit on Core, so it would have alerted forever once that unit was disabled. Local check removed; `anita-mnz` (`159.195.16.30`) added to `REMOTE_SERVERS` and a new `REMOTE_USER_UNITS` remote user-unit check added (SSH + `XDG_RUNTIME_DIR`). Verified live: no false alert for her gateway, and her box answers as active.
- **Docs:** full incident + pitfalls in `/root/.hermes/references/dr-issue-log.md`; transferable procedure in the `hermes-migration` skill.

## 2026-08-17 - Scirium v2 Proposal Deployed to /scirium/

- **v2 proposal deployed** to `proposals.itpropartner.com/scirium/` (index.html + 04-business-proposal-v2.md + critical-review.html). Assembled from 4 parallel remediation teams (marketing, technical, financial, legal), SOM reconciled with Financial as authority, build cost corrected to ~$167K (was $68K), verdict: GO with conditions (churn gate, acquisition-maturation gate, trademark clearance, Phase 0 DLP spike). Status: DRAFT FOR REVIEW pending Germaine review before VerdictTank resubmission.
- **URL rename completed:** v1 (codename Wall-O) frozen at `proposals.itpropartner.com/wall-o/` with a SUPERSEDED banner pointing to `/scirium/`. v2 is live at `/scirium/`. This closes the pending item from the 2026-08-16 changelog entry.
- **Sources:** v1 at `projects/scirium/04-business-proposal.md`; v2 at `projects/scirium/04-business-proposal-v2.md`; team remediation sections under `/tmp/scirium-v2/output/` (not repo-bound).

## 2026-08-16 — Wall-O Renamed to Scirium

- **Product renamed Wall-O → Scirium** (coined from Latin "scire" = to know). Applies going forward; "Wall-O" retired to internal codename history only.
- **Domain:** `scirium.com` selected. `.com`/`.io`/`.ai`/`.co`/`.app` all available (RDAP 404 + empty NS cross-check). Trademarkia: 0 results for "scirium".
- **Cloudflare at-cost pricing (verified 2026-08-16):** `.com` $10.44/yr, `.io` $50/yr (renewal ~$51.75), `.ai` $70/yr (min 2-year term = $140; rising to $80/yr on 2026-03-05), `.co` $15 first yr / $30 renewal, `.app` $14.20/yr, `.dev` $10.18/yr.
- **Source folder moved** `projects/wall-o/` → `projects/scirium/`. Legacy 4 docs still carry "Wall-O" internally; rebranded by the docs team as part of the v1/v2 documentation package.
- **Deployed proposal URL** (`proposals.itpropartner.com/wall-o/`) unchanged pending redeploy under `/scirium/`.

## 2026-08-12 — app3 Web Docroot Migration to Per-Site Users

- **Change:** every app3 nginx vhost moved off the shared `/home/ippadmin/htdocs/` root to a per-site dedicated Linux user with docroot `/home/<site-user>/htdocs/<domain>` (security hardening — no more single-owner web tree).
- **Verified mappings (live nginx configs, 2026-08-14):** mockups → `/home/mockups`, proposals → `/home/proposals`, docs → `/home/docs`, support → `/home/support`, my.verdicttank.com → `/home/myverdicttank`, verdicttank.com → `/home/gmb`, my.transitpin.com → `/home/transitpin-dash`.
- **Consequence:** 10+ skills and their reference/script files still referenced the old `/home/ippadmin/htdocs/` paths, causing a wrong-tree deploy on 2026-08-14. Remediated across SKILL.md, references/, and scripts/ (28 files, incl. singular `mockup`/`proposal` domain typos).
- **Rule:** always read `/etc/nginx/sites-enabled/<domain>.conf` to confirm the real docroot before deploying. Never assume `ippadmin` owns a site's files.

## 2026-08-08 — Hexclave Renamed → Stack Auth

- **Hexclave** renamed to **Stack Auth**. Now running at `auth2.itpropartner.com` on app3.
- This is the same service (customer-facing authentication), same server, same Docker stack — only the name changed.
- Old references to "Hexclave" in scripts, docs, and backups should be updated to "Stack Auth" / `stack-auth`.
- **Rule going forward:** any rename of critical infrastructure gets a changelog entry at the time of the rename, not discovered later.

## 2026-08-06 — Fallback Chain Overhaul & Two-Key Strategy

- **Root cause:** Aug 5 admin-ai budget cap + 4 dead fallback legs = $45 Anthropic burn in 10 hours
- Rotated all 5 fallback provider keys (new keys for deepseek, google, xai, anthropic, openai)
- Added F5: gpt-4.1-nano via OpenAI direct (independent infrastructure)
- Fixed F3: grok-4.6 → grok-4.5 (grok-4.6 never existed — LiteLLM catalog ghost)
- Documented two-key strategy: operational keys (admin-ai only) vs fallback keys (direct, daily-capped)
- Added to operational chain: claude-haiku-4-5 (lightweight), grok-4.5 (auditor 2), deepseek-v4-flash (batch)
- Synced Anita profile with identical fallback chain + provider keys
- Admin-ai budget raised: $20 → $30/day
- Updated: model-chain.md, operational-models.md

## 2026-07-16 — Audit Remediation

- Created CHANGELOG.md (missing per project documentation standard)
- Project directory: `/root/projects/itpp-infrastructure`