Sync docs, audit artifacts, project notes, and VerdictTank proposal docs
- audit/phase-one + phase-two: security audit briefs, findings, credential-rotation plan, Docker-USER hardening scripts, rollback refs - disaster-recovery/restore-test-log.md + backup-dr-audit-2026-08-10.md - clients/ (modelortho SEO audit, ai-biz-dev competitive landscape), notes/ (tiktok strategy) - projects/: front-desk-voice-agent, seo-visibility-checker product plan, hotnow-savannah HTML, resend-transactional-email, backup-dashboard-enhancements, code-review-graph, seo-ci-architecture - proposals/verdicttank/: architecture v4.0, methodology, judge-pool review, consolidation reasoning, cross-check review - docs/super-search/firecrawl-provider-strategy.md - updates: CHANGELOG, model-chain, projects-master-readme, intelsight.io - .gitignore: exclude nested standalone repos (seo-tool, venturebuilt)
This commit is contained in:
root
@@ -0,0 +1,45 @@
|
||||
# ITPP Phase Two Change Log
|
||||
|
||||
Engagement: ITPP Phase Two (remediation). Supersedes the placeholder framework.
|
||||
Started: 2026-08-13
|
||||
Conductor: Sho'Nuff
|
||||
Authority: Germaine Brown (sole authorizer per P&P section 2 and section 8)
|
||||
|
||||
This log records every live change made during Phase Two, per Policy and Procedure section 2:
|
||||
(a) what changed, (b) who authorized it, (c) when, and (d) how to roll back.
|
||||
|
||||
Every entry is written at the moment the change is made, not discovered later.
|
||||
Passwords and secrets are never recorded in plaintext here. The entry points to the
|
||||
Vaultwarden item that holds the new value.
|
||||
|
||||
---
|
||||
|
||||
## Schema
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| ID | Sequential entry number (P2-###) |
|
||||
| Date | When the change was applied (ET) |
|
||||
| Change | What changed (old -> new where applicable) |
|
||||
| Authorized by | Germaine, or a referenced approval (e.g. "Phase Two fast-track C7") |
|
||||
| Rollback | How to undo it |
|
||||
| Status | done / in-progress / rolled-back |
|
||||
| Notes | Dependencies verified, LiteLLM-adjacent flag, exemption reference if any |
|
||||
|
||||
---
|
||||
|
||||
## Entries
|
||||
|
||||
| ID | Date | Change | Authorized by | Rollback | Status | Notes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| P2-C3-001 | 2026-08-13 | Enrolled Wazuh agents 4.9.2 on 5 hosts: core, app2, app3, app1-bu, wphost02 (manager app1 152.53.36.131; agent IDs 001-005, all Active) | Phase Two fast-track C3 (Germaine) | Disable/remove agent service on each host, then remove agent record from manager via agent_control -r `<id>` | done | LiteLLM-adjacent=no. Verified via agent_control -l on manager: all 5 new agents + agent 000 (manager) show Active. No password/secret recorded (authd enrollment used use_password=no, no credential involved). |
|
||||
| P2-C7-001 | 2026-08-13 | Rotated Grafana admin password (default 'admin' removed; 24-char random value stored in Vaultwarden item 'Grafana - Core Admin'); restricted port 3002 from Anywhere to Tailscale 100.64.0.0/10 | Phase Two fast-track C7 (Germaine) | Recreate container with GF_SECURITY_ADMIN_PASSWORD=admin (see rollback/grafana-inspect.json); ufw allow 3002/tcp to restore Anywhere | done | LiteLLM-adjacent=no. Verified: admin/admin login -> 401, new password -> 200, UFW shows only 100.64.0.0/10 rule. Password persisted in Vaultwarden item d3785ae9-085d-4e40-ba1b-69b66fef9e34. |
|
||||
| P2-C8-001 | 2026-08-13 | wphost02 full DB backup coverage; closed the "6 of 8 unprotected" gap. RECON: found the offsite S3 script (`/root/backup.sh` on wphost02, using `mysqldump --all-databases`) was already being triggered daily via a Core-side crontab entry (`0 5 * * * ssh ... root@5.161.62.38 '/root/backup.sh'`, running since ≥2026-07-19); this was NOT visible from wphost02's own crontab, which is why the earlier finding only checked local `db-dump.sh` (2 of 8 DBs, 7-day local retention) and missed it. Verified via journalctl (`wphost02-backup` tag on Core) and a fresh manual run of backup.sh: all 8 WordPress DBs (apextrackexperience_1781549652, boxpilotlogistics_1770339547, debtrecoveryexperts_1778934554, iAmGMB_1764020288, katiewattsdesign_1735425014, mainWP_1717713767, vigilanttac_1728911691, voipsimplicity_1732250845) are present in the all-databases.sql dump and in per-site tarballs, landing daily in s3://hermes-vps-backups/wphost02-backup/<date>/ (Wasabi). Also extended wphost02's local `/root/db-dump.sh` (still only covered 2/8 DBs on local disk, 7-day retention) to dump all 8 DBs individually for local defense-in-depth; ran it successfully (22 files, all 8 today's dumps present). No new crontab entry was needed; the offsite S3 job was already wired in on Core and already covers all 8 DBs; only the local-disk script was extended. | Phase Two Workstream 1 C8 (Germaine) | restore db-dump.sh from /root/db-dump.sh.bak-20260813 on wphost02 (scp back or `cp db-dump.sh.bak-20260813 db-dump.sh`); no crontab line was added so there is nothing to remove there; the pre-existing Core crontab line `0 5 * * * ssh -i /root/.ssh/itpp-infra ... root@5.161.62.38 '/root/backup.sh'` is unchanged from before this task. | done | LiteLLM-adjacent=no. No decommission action taken; no live WordPress site or config modified; no new S3 credentials created (existing Wasabi creds in wphost02:/root/.aws/credentials, already valid, reused as-is). Verified via S3 object listing (10 objects, fresh timestamp) and journalctl, not by trusting script exit code alone. |
|
||||
| P2-C1-001 | 2026-08-13 | LiteLLM fallback switch (pre-C1 safety gate): Hermes primary model provider admin-ai (LiteLLM on app1) -> deepseek (direct to api.deepseek.com). Also switched delegation.provider and delegation.model to direct deepseek, and delegation.fallback provider admin-ai -> deepseek. Model stays deepseek-v4-pro (no downgrade; direct endpoint serves both deepseek-v4-pro and deepseek-v4-flash). | Phase Two safety rule (fallback switch required before any LiteLLM-adjacent change); C1 app1 portion is LiteLLM-adjacent | Restore config from /root/.hermes/config.yaml.bak-20260813-C1, or revert via: hermes config set model.provider admin-ai; hermes config set delegation.provider admin-ai; hermes config set delegation.model claude-sonnet-5; and set delegation.fallback provider back to admin-ai. | done | LiteLLM-adjacent=yes (this IS the fallback switch). Verified direct deepseek /v1/models returns both deepseek-v4-pro and deepseek-v4-flash, and /v1/chat/completions returns a valid choices array with no error. DEEPSEEK_API_KEY present in .env and valid. Note: the running gateway session continues on admin-ai until next restart but is protected by the already-active direct-first fallback chain (deepseek -> google -> xai -> anthropic -> openai). Residual admin-ai references remain only in vision paths (auxiliary.vision and top-level vision), non-blocking for C1 terminal/SSH work. |
|
||||
| P2-C9-001 | 2026-08-13 | Warm-standby DB snapshot consistency: hermes-live-sync.sh now exports state.db and memory_store.db via SQLite online backup (.backup) into a consistent snapshot uploaded as the authoritative S3 object, and excludes the raw live WAL/SHM files from the sync. Deleted obsolete transient state.db-wal/shm and memory_store.db-wal/shm objects from s3://hermes-vps-backups/live/. | Phase Two Workstream 1 C9 (Germaine) | cp ~/.hermes/scripts/hermes-live-sync.sh.bak-20260813-C9 ~/.hermes/scripts/hermes-live-sync.sh (raw file copy behavior restored) | done | LiteLLM-adjacent=no. Verified: .backup of the 2.98 GB live state.db completed in 9s and passed PRAGMA integrity_check (ok); fixed script ran exit 0 and uploaded the 2980638720-byte state.db and 1871872-byte memory_store.db snapshots at 15:30 ET. Root cause: aws s3 sync of a live WAL-mode SQLite DB can capture a torn checkpoint (evidence: state.db.corrupted plus 3 malformed-backup-* objects in S3 dated 2026-07-09). Residual follow-up: profiles/anita/state.db* still synced raw because top-level excludes do not match profile paths; and ~7.5 GB of stale corrupted/malformed S3 objects left in place pending a cleanup decision. |
|
||||
| P2-C4-001 | 2026-08-13 | Restored least-privilege file permissions on app1-bu standby: chmod 600 on 11 world-readable secret-bearing files (.hermes/.env, .hermes/config.yaml, profiles/anita/.env, profiles/anita/google_client_secret.json, docker/vaultwarden/.env, docker/twenty/.env, references/glc-credentials.md, references/liberty-credentials.md, .aws/credentials.bak, and two state-snapshot .env files). Fixed root cause: added umask 077 to hermes-standby-sync.sh, hermes-standby-watchdog.sh, and hermes-standby-restore.sh (standby) plus hermes-live-sync.sh (Core) so aws s3 sync no longer re-creates synced files as 644 and the Core .snapshots output is no longer transiently world-readable. | Phase Two Workstream 1 C4 (Germaine) | Restore the three .bak-20260813-C4 script backups (cp each .bak-20260813-C4 back over its script); re-chmod 644 the files (not recommended) | done | LiteLLM-adjacent=no. Root cause: aws s3 sync does not preserve POSIX permissions (S3 objects carry no mode bits), so the 10-min standby sync re-created Core 600 files as 644 under the default umask 022. Verified: functional test of aws s3 sync under umask 077 lands a downloaded object as 600; bash -n passes on all three edited scripts. Core already holds the same files at 600, so no Core change was needed. |
|
||||
| P2-C1-002 | 2026-08-13 | DOCKER-USER firewall hardening made persistent on Core, app1, app2, app3. Live IPv4 rules (applied earlier 2026-08-13) now survive reboot and docker restart via /usr/local/sbin/itpp-docker-user.sh (idempotent flush+rebuild of DOCKER-USER, IPv4 + IPv6 where present) wired as an ExecStartPost drop-in at /etc/systemd/system/docker.service.d/itpp-docker-user.conf on each host. Closed to Tailscale-only (100.64.0.0/10 + ESTABLISHED,RELATED only): Core browserless 3000, camofox 9377, uptime-kuma 3001; app1 komodo 9120, twenty 3003, wazuh dashboard 5601, wazuh indexer 9200; app2 bookstack 6875 (direct), minio 9001, gitea ssh 3022, ragflow 9380-9384/9392/9393, infinity 23817/23820, unifi admin 8443/8843/8880, unms 81/8444; app3 buzz relay 3000. Preserved public: app1 wazuh manager 1514/1515/55000 tcp + 514 udp; app2 dns 53 tcp+udp, support-api 6880, unifi inform 8080, traccar 8082 + 5000-5150 tcp+udp, unms-nginx 8089, bookstack 6875 via ctorigdstport proxy path, unms-netflow 2055 udp, unifi 3478/10001 udp. | Phase Two Workstream 1 C1 (Germaine) | On each host: rm /etc/systemd/system/docker.service.d/itpp-docker-user.conf && systemctl daemon-reload; then iptables -F DOCKER-USER (and ip6tables -F DOCKER-USER where present); rm /usr/local/sbin/itpp-docker-user.sh | done | LiteLLM-adjacent=no. Rules scoped -i eth0 so docker-bridge traffic is untouched. Verified live: iptables -L DOCKER-USER -n -v on all 4 hosts shows correct chains with -i eth0 and active packet counters. Persistence proven end-to-end: ufw reload on Core preserved FORWARD integration and DOCKER-USER rules; systemctl restart docker on app3 flushed DOCKER-USER (counters reset) then ExecStartPost re-applied rules immediately, all 8 app3 containers returned healthy (hexclave-server health:starting then healthy). Pre-existing issues noted, not C1: crm.debtrecoveryexperts.com and crm.intelsight.io 502 because Core Caddy proxies to Core-local localhost:3003 where nothing listens (Twenty runs app1:3003); admin-ai /v1/models 401 despite /health 200 (auth, not firewall); infinity 23817 returns 000 from minio while 23820 returns 404 (port semantics). |
|
||||
| P2-C4-002 | 2026-08-13 | C4 credential rotation: D1 approved (rotate A and B, revoke C). Phase 0 inventory complete. Rollback snapshots taken: /root/.hermes/config.yaml.bak-20260813-C4-pre and /root/.hermes/.env.bak-20260813-C4-pre, both chmod 600. | Germaine (A rotate, B rotate, C revoke, 2026-08-13) | Restore both C4-pre backups | done | LiteLLM-adjacent=yes (this entry is the go/no-go gate for LiteLLM key changes). No keys changed yet; decision record only. Phase 0 verdict: A=7 rotate, B=7 rotate, C=4 revoke. |
|
||||
| P2-C4-003 | 2026-08-13 | Revoked 4 dead AI keys from plaintext: removed AI21_API_KEY, ALIBABA_API_KEY (duplicate, x2), ZAI_API_KEY, NVIDIA_API_KEY from Core /root/.hermes/.env and app1-bu /root/.hermes/.env (5 lines to 0), plus the stale NVIDIA NIM comment line. PARALLEL_API_KEY reclassified: it is a live Super Search provider (Parallel.ai Search, fallback #11 in server.py), not an orphan, moved to Phase 3 service tokens (Keep). | Germaine (C revoke) | Restore /root/.hermes/.env.bak-20260813-C4-pre on Core and app1-bu .env from its prior state | done | LiteLLM-adjacent=no (plaintext .env removal only; no LiteLLM credential touched). Verified before/after line counts and mode 600 on both hosts. Residual: NVIDIA LiteLLM credential (0 working models) still in app1 Postgres, deletion deferred to the LiteLLM step. |
|
||||
| P2-C4-004 | 2026-08-14 | Rotated 5 assistant-owned portal API keys (b1 set): Cohere, Fireworks, Mistral, Perplexity, FAL. Applied across three layers: (1) LiteLLM credentials PATCHed via /credentials/{name} on app1 (Cohere, Fireworks, Mistral, Perplexity, Fal) — 5/5 HTTP 200; (2) Core /root/.hermes/.env swapped (COHERE_API_KEY, FIREWORKS_API_KEY, MISTRAL_API_KEY, PERPLEXITY_API_KEY, FAL_KEY) with backup /root/.hermes/.env.bak-20260814-b1; (3) Vaultwarden — created 3 new 'X - API Key' items (Cohere, Fireworks, Mistral; username info@itpropartner.com) and updated Perplexity ('console.perplexity.ai') and FAL ('fal.ai') item passwords in place. Mistral+Cohere keys generated in-console as 'LiteLLM-20260813'; Fireworks/FAL/Perplexity keys supplied by Germaine. | Germaine (C4 D1 GO + b1 direction 2026-08-14) | Core .env: cp /root/.hermes/.env.bak-20260814-b1 /root/.hermes/.env. LiteLLM: re-PATCH the 5 credentials with pre-rotation values (old keys remain active provider-side; recoverable from .env.bak-20260813-C4-pre and Vaultwarden password history). Vaultwarden: restore Perplexity/FAL item passwords via bw item password history; delete the 3 new 'X - API Key' items. | done | LiteLLM-adjacent=yes. Verified: all 5 keys live against providers (Mistral/Cohere/Fireworks/Perplexity HTTP 200; FAL past auth gateway); LiteLLM PATCH 5/5 HTTP 200; .env values correct with pre-existing dual NETCUP_API_KEY quirk preserved; Vaultwarden re-read 5/5 PASS. Active consumers: Perplexity -> Super Search (super-search.service restarted; health_check reports perplexity:ok, server v2.4.0); FAL -> image_generate (picks up new key on next Hermes restart; running process holds prior key in os.environ). Cohere/Fireworks/Mistral have no active Core consumer (LiteLLM 'use if set' only). Old keys NOT revoked provider-side. |
|
||||
| P2-C4-005 | 2026-08-14 | Rotated the admin-ai LiteLLM master key: old (len 66, sk-litel...) -> new (len 67, sk-x2top..., sk- + 64 mixed-case alphanumeric, no special chars). Applied to: (1) app1 /root/docker/litellm/.env LITELLM_MASTER_KEY and UI_PASSWORD (backup .env.bak-20260814-adminai); (2) Core /root/.hermes/scripts/cost-alert-watchdog.py MASTER_KEY (backup .bak-20260814-adminai). Recreated litellm container via `docker compose up -d --force-recreate litellm` (a bare `docker restart` does NOT re-read env vars — caught during verification). Vaultwarden consolidated: new key in item 'LiteLLM (admin-ai) Master Key' 0772c50c; updated 'admin-ai.itpropartner.com' bb01ef49 to the new key; deleted stale 'LiteLLM Master Key' 446da6ca and duplicate 'admin-ai.itpropartner.com' 8c8a5eb1. | Germaine (added new key to Vaultwarden as 'LiteLLM (admin-ai) Master Key', 2026-08-14) | Restore app1 .env from .env.bak-20260814-adminai and watchdog from .bak-20260814-adminai, then `docker compose up -d --force-recreate litellm` (old key is now revoked 401, so rollback requires restoring the old value to .env first); restore deleted Vaultwarden items from trash. | done | LiteLLM-adjacent=yes. Verified on app1 localhost:4000 — new key HTTP 200 on /spend/keys and /global/spend, old key HTTP 401 (revoked); cost-alert-watchdog ran clean against the new key (surfaced real alert: hermes-agent-v5 at 88% budget); Core 25-char virtual key 'Hermes (Sho'Nuff) LiteLLM Virtual API Key' (0aed1ca4) unaffected, /v1/chat/completions HTTP 200; LITELLM_SALT_KEY unchanged. Core admin-ai provider uses the 25-char virtual key (not the master key), so no Core config.yaml change was required. |
|
||||
Reference in New Issue
Block a user