itpp-infrastructure/CHANGELOG.md

itpp-infrastructure — CHANGELOG

2026-09-11 — Anita's Hermes Profile Moved to a Dedicated Box (anita-mnz)

• Infra move: Anita's assistant profile moved off shared Core (152.53.241.111) to a dedicated box anita-mnz 159.195.16.30 (netcup, Manassas VA, 8 vCPU / 15 GB / 503 GB). Cutover 15:53 EDT, ~2 minutes dark, zero messages lost. She keeps the same Telegram bot and chat.
• Why: repeated state.db corruption on Core under co-tenant memory pressure (4th recurrence, system_prompts then sessions pages). A dedicated box removes the co-tenancy.
• Method: clean staged base + row-tolerant live-tail graft, so the corrupt live file never transferred. Signatures matched exactly: 99283/99283/166244414/157 (messages/maxid/contentbytes/sessions), integrity_check ok.
• Post-move state: Core hermes-gateway-anita.service stopped and disabled (no double-poller risk; standby app1-bu carries no anita unit); target gateway active+enabled, telegram connected, 0 getUpdates conflicts.
• Backup: her own box now runs the 3 AM root-essentials backup → s3://hermes-vps-backups/root-backup/anita-mnz/. Full run tested end-to-end 2026-09-11 15:56 (163 MB, upload + download/extract verify OK).
• Core change: hermes-live-sync no longer snapshots the frozen profiles/anita copy (it would advertise a stale db as "live").
• MCP servers: stripped from her profile (decision 2026-09-11: "She doesn't need access to those"). The mcp_servers block (dre, osint-person, super-search → 127.0.0.1:8900/8902/8899) was removed from anita/config.yaml, which is why her log had been retry-parking those three every ~5 minutes since the move. Backed up first as config.yaml.bak-mcpstrip-*; 13 top-level keys verified intact. Takes effect on her next gateway restart.
• Retired / order cancelled: Nuremberg 89.58.44.96 (v2202609377162518632.nicesrv.de). Correction to how this was first written here: it does not "hold nothing". It holds a bare default-profile Hermes install: no state.db, empty sessions/ and cron/, no systemd user units, no gateway process (only node_exporter, containerd, sshd). No unique data, so nothing needs preserving before cancellation. Recorded in /root/.hermes/references/decommissioned-hosts.json.
• Monitoring fix: health-master-watchdog.py watch-listed hermes-gateway-anita.service as a LOCAL user unit on Core, so it would have alerted forever once that unit was disabled. Local check removed; anita-mnz (159.195.16.30) added to REMOTE_SERVERS and a new REMOTE_USER_UNITS remote user-unit check added (SSH + XDG_RUNTIME_DIR). Verified live: no false alert for her gateway, and her box answers as active.
• Docs: full incident + pitfalls in /root/.hermes/references/dr-issue-log.md; transferable procedure in the hermes-migration skill.

2026-08-17 - Scirium v2 Proposal Deployed to /scirium/

• v2 proposal deployed to proposals.itpropartner.com/scirium/ (index.html + 04-business-proposal-v2.md + critical-review.html). Assembled from 4 parallel remediation teams (marketing, technical, financial, legal), SOM reconciled with Financial as authority, build cost corrected to ~$167K (was $68K), verdict: GO with conditions (churn gate, acquisition-maturation gate, trademark clearance, Phase 0 DLP spike). Status: DRAFT FOR REVIEW pending Germaine review before VerdictTank resubmission.
• URL rename completed: v1 (codename Wall-O) frozen at proposals.itpropartner.com/wall-o/ with a SUPERSEDED banner pointing to /scirium/. v2 is live at /scirium/. This closes the pending item from the 2026-08-16 changelog entry.
• Sources: v1 at projects/scirium/04-business-proposal.md; v2 at projects/scirium/04-business-proposal-v2.md; team remediation sections under /tmp/scirium-v2/output/ (not repo-bound).

2026-08-16 — Wall-O Renamed to Scirium

• Product renamed Wall-O → Scirium (coined from Latin "scire" = to know). Applies going forward; "Wall-O" retired to internal codename history only.
• Domain: scirium.com selected. .com/.io/.ai/.co/.app all available (RDAP 404 + empty NS cross-check). Trademarkia: 0 results for "scirium".
• Cloudflare at-cost pricing (verified 2026-08-16): .com $10.44/yr, .io $50/yr (renewal ~$51.75), .ai $70/yr (min 2-year term = $140; rising to $80/yr on 2026-03-05), .co $15 first yr / $30 renewal, .app $14.20/yr, .dev $10.18/yr.
• Source folder moved projects/wall-o/ → projects/scirium/. Legacy 4 docs still carry "Wall-O" internally; rebranded by the docs team as part of the v1/v2 documentation package.
• Deployed proposal URL (proposals.itpropartner.com/wall-o/) unchanged pending redeploy under /scirium/.

2026-08-12 — app3 Web Docroot Migration to Per-Site Users

• Change: every app3 nginx vhost moved off the shared /home/ippadmin/htdocs/ root to a per-site dedicated Linux user with docroot /home/<site-user>/htdocs/<domain> (security hardening — no more single-owner web tree).
• Verified mappings (live nginx configs, 2026-08-14): mockups → /home/mockups, proposals → /home/proposals, docs → /home/docs, support → /home/support, my.verdicttank.com → /home/myverdicttank, verdicttank.com → /home/gmb, my.transitpin.com → /home/transitpin-dash.
• Consequence: 10+ skills and their reference/script files still referenced the old /home/ippadmin/htdocs/ paths, causing a wrong-tree deploy on 2026-08-14. Remediated across SKILL.md, references/, and scripts/ (28 files, incl. singular mockup/proposal domain typos).
• Rule: always read /etc/nginx/sites-enabled/<domain>.conf to confirm the real docroot before deploying. Never assume ippadmin owns a site's files.

2026-08-08 — Hexclave Renamed → Stack Auth

• Hexclave renamed to Stack Auth. Now running at auth2.itpropartner.com on app3.
• This is the same service (customer-facing authentication), same server, same Docker stack — only the name changed.
• Old references to "Hexclave" in scripts, docs, and backups should be updated to "Stack Auth" / stack-auth.
• Rule going forward: any rename of critical infrastructure gets a changelog entry at the time of the rename, not discovered later.

2026-08-06 — Fallback Chain Overhaul & Two-Key Strategy

• Root cause: Aug 5 admin-ai budget cap + 4 dead fallback legs = $45 Anthropic burn in 10 hours
• Rotated all 5 fallback provider keys (new keys for deepseek, google, xai, anthropic, openai)
• Added F5: gpt-4.1-nano via OpenAI direct (independent infrastructure)
• Fixed F3: grok-4.6 → grok-4.5 (grok-4.6 never existed — LiteLLM catalog ghost)
• Documented two-key strategy: operational keys (admin-ai only) vs fallback keys (direct, daily-capped)
• Added to operational chain: claude-haiku-4-5 (lightweight), grok-4.5 (auditor 2), deepseek-v4-flash (batch)
• Synced Anita profile with identical fallback chain + provider keys
• Admin-ai budget raised: $20 → $30/day
• Updated: model-chain.md, operational-models.md

2026-07-16 — Audit Remediation

• Created CHANGELOG.md (missing per project documentation standard)
• Project directory: /root/projects/itpp-infrastructure