Sync docs, audit artifacts, project notes, and VerdictTank proposal docs
- audit/phase-one + phase-two: security audit briefs, findings, credential-rotation plan, Docker-USER hardening scripts, rollback refs - disaster-recovery/restore-test-log.md + backup-dr-audit-2026-08-10.md - clients/ (modelortho SEO audit, ai-biz-dev competitive landscape), notes/ (tiktok strategy) - projects/: front-desk-voice-agent, seo-visibility-checker product plan, hotnow-savannah HTML, resend-transactional-email, backup-dashboard-enhancements, code-review-graph, seo-ci-architecture - proposals/verdicttank/: architecture v4.0, methodology, judge-pool review, consolidation reasoning, cross-check review - docs/super-search/firecrawl-provider-strategy.md - updates: CHANGELOG, model-chain, projects-master-readme, intelsight.io - .gitignore: exclude nested standalone repos (seo-tool, venturebuilt)
This commit is contained in:
root
@@ -0,0 +1,127 @@
|
||||
# C4 Credential Rotation Plan (Phase Two, WS1)
|
||||
|
||||
Status: APPROVED (D1). Phase 0 complete, Phase 1 in progress.
|
||||
Date: 2026-08-13
|
||||
Owner: Sho'Nuff
|
||||
|
||||
## 0. What this plan covers (and what it does not)
|
||||
|
||||
- The C4 **permission subset** is DONE. Logged as `P2-C4-001` (app1-bu chmod 600 on 11 secret files + umask 077 root-cause fix). This plan does not re-open it.
|
||||
- This plan covers the **second half of C4**: inventory every plaintext credential, rotate them, and move them to Vaultwarden.
|
||||
- Source-hygiene follow-ups are included as Phase 6 (they are rotation-adjacent, not permission work).
|
||||
|
||||
## 1. The two decisions this plan needs from you (up front)
|
||||
|
||||
| # | Decision | What it unlocks |
|
||||
|---|---|---|
|
||||
| D1 | **APPROVED 2026-08-13** ("A - Rotate, B - Rotate, C - Revoke"): rotate live AI keys (A+B), revoke dead (C) | Phase 2 (in progress) |
|
||||
| D2 | **Go/no-go: rotate the MSP-backdoor shared credential, plus approve the per-portal order** | Phase 3 |
|
||||
|
||||
Everything else in this plan runs under those two gates. Phases 4-6 are lower-risk hygiene and I will flag them individually before executing.
|
||||
|
||||
## 2. Evidence base (phase-one report, section 3.1 C4)
|
||||
|
||||
1. Two unsanitized copies of `key-inventory.md` on Core (one inside Hermes's reference dir).
|
||||
2. Private `scripts` repo hardcodes the MSP-backdoor admin password reused across client onboardings (Git-A Finding 1). Single most consequential live credential.
|
||||
3. app1-bu `/root/.hermes/.env` (mode 644) holds ~20 plaintext secrets: root passwords for app1/app2/app3, Telegram bot token, Cloudflare/Netcup/SyncroMSP tokens, and eight AI-provider keys.
|
||||
4. `rally.service`, `seemytrip.service`, `giftaroast.service` hardcode `JWT_SECRET`, `DEEPSEEK_API_KEY`, `ADMIN_AI_KEY`, and Twilio credentials in world-readable unit files.
|
||||
5. app3 MySQL root password in `/opt/backup-restore/snapshot.sh` (mode 775) and `/root/backup.sh`.
|
||||
6. Public repo `itpp-infrastructure` re-leaks live admin passwords in a prior audit report (section 7, deferred per Germaine).
|
||||
|
||||
Recommended order from the report: app1-bu .env first, then key-inventory.md, then systemd units, then app3 snapshot.sh.
|
||||
|
||||
## 3. Safety rules (non-negotiable, apply to every phase)
|
||||
|
||||
1. **Fallback-first.** Before any LiteLLM-adjacent or credential-affecting change, Hermes stays pinned to direct DeepSeek. This is already in force since C1 (`P2-C1-001`); re-confirm at start of each phase.
|
||||
2. **Vaultwarden-before-changes.** Retrieve a credential from Vaultwarden before touching the live system, never from a plaintext file, shell history, or a prior report.
|
||||
3. **No autonomous key rotation.** Every rotation happens only after the go/no-go above, in the order listed.
|
||||
4. **Verify before done.** Every rotation is followed by a live check (API call, login, health endpoint). No "should work" statements.
|
||||
5. **Log + rollback.** Each change is written to `change-log.md` at the moment it is made, with the rollback path stated before the change.
|
||||
6. **Zero artifact style.** No em dashes or en dashes in any plan or change-log entry. All credential values appear as `[REDACTED]`.
|
||||
|
||||
## 4. Phase 0: Inventory (read-only, no changes)
|
||||
|
||||
Goal: a single authoritative list of every live plaintext credential, cross-referenced against Vaultwarden.
|
||||
|
||||
Steps:
|
||||
1. Re-run the secret-sprawl scan on the three hosts with the raw hit counts (6,296 Core / 5,504 app1-bu / 1,275 app2) and triage down to live, consequential secrets only.
|
||||
2. Categorize every secret:
|
||||
- (A) AI provider keys
|
||||
- (B) MSP-backdoor shared credential
|
||||
- (C) server root passwords (app1/app2/app3)
|
||||
- (D) service tokens (Telegram, Cloudflare, Netcup, SyncroMSP, Twilio)
|
||||
- (E) JWT secrets and DB passwords (systemd units, app3 snapshot/backup scripts)
|
||||
- (F) source-repo leaks (key-inventory.md copies, scripts repo, public repo section 7)
|
||||
3. Cross-reference each item against Vaultwarden: mark `already vaulted`, `missing`, or `stale`.
|
||||
4. Deliverable: an inventory table with a rotation priority, a current location, and a target Vaultwarden item for each credential.
|
||||
|
||||
Effort note: this is where the L-XL sizing sits. The raw scan counts are noise; the consequential list is far smaller.
|
||||
|
||||
## 4a. Phase 0 result (2026-08-13)
|
||||
|
||||
- Inventory complete. The report's "eight keys" was an undercount: the live AI surface is ~19 upstream keys across three layers (config.yaml literals, .env plaintext, LiteLLM encrypted credentials), many duplicated.
|
||||
- GEMINI_API_KEY and GOOGLE_AI_STUDIO_KEY hold the same value under two names (one rotation, two lines to update).
|
||||
- Verdict: A (fallback chain, 7) = Rotate. B (operational LLM, 7) = Rotate. C (dead, 4) = Revoke.
|
||||
- Correction: PARALLEL_API_KEY is not dead. It is a live Super Search provider (Parallel.ai Search, fallback #11 in server.py), reclassified as a Phase 3 service token (Keep).
|
||||
- Vault gaps: DeepSeek, OpenAI, Google/Gemini, Groq, MiniMax have no Vaultwarden item; Cohere/Fireworks/Perplexity/Mistral have portal logins only.
|
||||
|
||||
## 5. Phase 1: AI provider keys (needs D1)
|
||||
|
||||
Sequence (fallback-first, rotate non-active providers before the active one):
|
||||
|
||||
1. Confirm Hermes is pinned to direct DeepSeek and the direct endpoint is healthy (`/v1/models` + a test completion).
|
||||
2. From Phase 0 inventory, confirm the exact list of eight AI providers and which LiteLLM / env locations hold each key.
|
||||
3. Rotate the keys for all providers EXCEPT the one Hermes is actively pinned to (DeepSeek), one at a time:
|
||||
- Generate a new key at the provider portal.
|
||||
- Store the new key in Vaultwarden (target item from Phase 0).
|
||||
- Update LiteLLM config and/or env to reference the vault-backed value.
|
||||
- Verify the provider still resolves through the fallback chain (test call).
|
||||
- Write the change to `change-log.md` with rollback.
|
||||
4. Rotate the pinned provider (DeepSeek) LAST, immediately updating Core `.env` and LiteLLM before any subsequent API call can fail.
|
||||
5. Re-verify the full fallback chain end to end and switch Hermes back to its normal provider only after all eight keys are confirmed live.
|
||||
|
||||
Rollback: each step keeps the prior key in the change-log rollback note until the next step verifies green.
|
||||
|
||||
## 6. Phase 2: MSP-backdoor shared credential (needs D2)
|
||||
|
||||
Sequence (this is the highest-consequence single credential):
|
||||
|
||||
1. Identify the exact shared admin credential and every client system / vendor portal currently using it (inventory from `scripts` repo + onboarding records).
|
||||
2. Produce a per-portal rotation order, ordered by blast radius (most-impacted or most-exposed first), and get your sign-off on the order.
|
||||
3. Vaultwarden-before-changes: confirm the replacement credential is generated and vaulted before touching any live system.
|
||||
4. Rotate one portal at a time: update the portal, verify login with the new credential, update any dependent automation/env, then move to the next.
|
||||
5. Remove the hardcoded password from the private `scripts` repo and replace it with a Vaultwarden lookup or a `[REDACTED]` placeholder.
|
||||
6. Log every portal change in `change-log.md` with rollback.
|
||||
|
||||
## 7. Phase 3: server root passwords and service tokens (lower risk)
|
||||
|
||||
- Rotate app1/app2/app3 root passwords; store in Vaultwarden; update the one place that currently references them in plaintext (app1-bu `.env`).
|
||||
- Rotate service tokens: Telegram bot token, Cloudflare, Netcup, SyncroMSP, Twilio. Each token rotation re-issues at the vendor console, then updates the consuming service and any env/unit file.
|
||||
- Verify each service still functions after its token rotates (send a test, poll an endpoint, etc.).
|
||||
|
||||
## 8. Phase 4: JWT secrets and DB passwords
|
||||
|
||||
- Move `JWT_SECRET`, `DEEPSEEK_API_KEY`, `ADMIN_AI_KEY`, and Twilio creds out of `rally.service`, `seemytrip.service`, `giftaroast.service` into vault-backed env or a `chmod 600` env file, then reload the units.
|
||||
- Rotate the app3 MySQL root password in `/opt/backup-restore/snapshot.sh` and `/root/backup.sh`; store in Vaultwarden; update the scripts to source it from a 600-mode file.
|
||||
- Verify each service/backup still runs after the change.
|
||||
|
||||
## 9. Phase 5: source hygiene (flagged separately)
|
||||
|
||||
- Redact or delete the two `key-inventory.md` copies on Core.
|
||||
- Confirm the `scripts` repo MSP password removal from Phase 2 is committed and pushed.
|
||||
- Public repo `itpp-infrastructure` re-leak (report section 7): deferred per your earlier instruction; restating it here so it is not silently dropped. Confirm whether to keep it deferred or fold it into this plan.
|
||||
|
||||
## 10. Verification gates (plan-wide)
|
||||
|
||||
- [ ] Phase 0 inventory complete and cross-referenced against Vaultwarden
|
||||
- [ ] Fallback chain verified end to end after all AI key rotations
|
||||
- [ ] Every portal login verified after MSP-backdoor rotation
|
||||
- [ ] Every service/unit verified functional after its token/JWT/DB rotation
|
||||
- [ ] `change-log.md` has a dated entry with rollback for every single change
|
||||
- [ ] No plaintext credential remains in any world-readable location (re-run the scan)
|
||||
|
||||
## 11. Approval needed
|
||||
|
||||
- D1: AI provider key rotation go/no-go (unlocks Phase 1)
|
||||
- D2: MSP-backdoor rotation go/no-go + per-portal order (unlocks Phase 2)
|
||||
- Phases 3-5: flag individually before execution (no blanket approval implied)
|
||||
@@ -0,0 +1,45 @@
|
||||
# ITPP Phase Two Change Log
|
||||
|
||||
Engagement: ITPP Phase Two (remediation). Supersedes the placeholder framework.
|
||||
Started: 2026-08-13
|
||||
Conductor: Sho'Nuff
|
||||
Authority: Germaine Brown (sole authorizer per P&P section 2 and section 8)
|
||||
|
||||
This log records every live change made during Phase Two, per Policy and Procedure section 2:
|
||||
(a) what changed, (b) who authorized it, (c) when, and (d) how to roll back.
|
||||
|
||||
Every entry is written at the moment the change is made, not discovered later.
|
||||
Passwords and secrets are never recorded in plaintext here. The entry points to the
|
||||
Vaultwarden item that holds the new value.
|
||||
|
||||
---
|
||||
|
||||
## Schema
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| ID | Sequential entry number (P2-###) |
|
||||
| Date | When the change was applied (ET) |
|
||||
| Change | What changed (old -> new where applicable) |
|
||||
| Authorized by | Germaine, or a referenced approval (e.g. "Phase Two fast-track C7") |
|
||||
| Rollback | How to undo it |
|
||||
| Status | done / in-progress / rolled-back |
|
||||
| Notes | Dependencies verified, LiteLLM-adjacent flag, exemption reference if any |
|
||||
|
||||
---
|
||||
|
||||
## Entries
|
||||
|
||||
| ID | Date | Change | Authorized by | Rollback | Status | Notes |
|
||||
|---|---|---|---|---|---|---|
|
||||
| P2-C3-001 | 2026-08-13 | Enrolled Wazuh agents 4.9.2 on 5 hosts: core, app2, app3, app1-bu, wphost02 (manager app1 152.53.36.131; agent IDs 001-005, all Active) | Phase Two fast-track C3 (Germaine) | Disable/remove agent service on each host, then remove agent record from manager via agent_control -r `<id>` | done | LiteLLM-adjacent=no. Verified via agent_control -l on manager: all 5 new agents + agent 000 (manager) show Active. No password/secret recorded (authd enrollment used use_password=no, no credential involved). |
|
||||
| P2-C7-001 | 2026-08-13 | Rotated Grafana admin password (default 'admin' removed; 24-char random value stored in Vaultwarden item 'Grafana - Core Admin'); restricted port 3002 from Anywhere to Tailscale 100.64.0.0/10 | Phase Two fast-track C7 (Germaine) | Recreate container with GF_SECURITY_ADMIN_PASSWORD=admin (see rollback/grafana-inspect.json); ufw allow 3002/tcp to restore Anywhere | done | LiteLLM-adjacent=no. Verified: admin/admin login -> 401, new password -> 200, UFW shows only 100.64.0.0/10 rule. Password persisted in Vaultwarden item d3785ae9-085d-4e40-ba1b-69b66fef9e34. |
|
||||
| P2-C8-001 | 2026-08-13 | wphost02 full DB backup coverage; closed the "6 of 8 unprotected" gap. RECON: found the offsite S3 script (`/root/backup.sh` on wphost02, using `mysqldump --all-databases`) was already being triggered daily via a Core-side crontab entry (`0 5 * * * ssh ... root@5.161.62.38 '/root/backup.sh'`, running since ≥2026-07-19); this was NOT visible from wphost02's own crontab, which is why the earlier finding only checked local `db-dump.sh` (2 of 8 DBs, 7-day local retention) and missed it. Verified via journalctl (`wphost02-backup` tag on Core) and a fresh manual run of backup.sh: all 8 WordPress DBs (apextrackexperience_1781549652, boxpilotlogistics_1770339547, debtrecoveryexperts_1778934554, iAmGMB_1764020288, katiewattsdesign_1735425014, mainWP_1717713767, vigilanttac_1728911691, voipsimplicity_1732250845) are present in the all-databases.sql dump and in per-site tarballs, landing daily in s3://hermes-vps-backups/wphost02-backup/<date>/ (Wasabi). Also extended wphost02's local `/root/db-dump.sh` (still only covered 2/8 DBs on local disk, 7-day retention) to dump all 8 DBs individually for local defense-in-depth; ran it successfully (22 files, all 8 today's dumps present). No new crontab entry was needed; the offsite S3 job was already wired in on Core and already covers all 8 DBs; only the local-disk script was extended. | Phase Two Workstream 1 C8 (Germaine) | restore db-dump.sh from /root/db-dump.sh.bak-20260813 on wphost02 (scp back or `cp db-dump.sh.bak-20260813 db-dump.sh`); no crontab line was added so there is nothing to remove there; the pre-existing Core crontab line `0 5 * * * ssh -i /root/.ssh/itpp-infra ... root@5.161.62.38 '/root/backup.sh'` is unchanged from before this task. | done | LiteLLM-adjacent=no. No decommission action taken; no live WordPress site or config modified; no new S3 credentials created (existing Wasabi creds in wphost02:/root/.aws/credentials, already valid, reused as-is). Verified via S3 object listing (10 objects, fresh timestamp) and journalctl, not by trusting script exit code alone. |
|
||||
| P2-C1-001 | 2026-08-13 | LiteLLM fallback switch (pre-C1 safety gate): Hermes primary model provider admin-ai (LiteLLM on app1) -> deepseek (direct to api.deepseek.com). Also switched delegation.provider and delegation.model to direct deepseek, and delegation.fallback provider admin-ai -> deepseek. Model stays deepseek-v4-pro (no downgrade; direct endpoint serves both deepseek-v4-pro and deepseek-v4-flash). | Phase Two safety rule (fallback switch required before any LiteLLM-adjacent change); C1 app1 portion is LiteLLM-adjacent | Restore config from /root/.hermes/config.yaml.bak-20260813-C1, or revert via: hermes config set model.provider admin-ai; hermes config set delegation.provider admin-ai; hermes config set delegation.model claude-sonnet-5; and set delegation.fallback provider back to admin-ai. | done | LiteLLM-adjacent=yes (this IS the fallback switch). Verified direct deepseek /v1/models returns both deepseek-v4-pro and deepseek-v4-flash, and /v1/chat/completions returns a valid choices array with no error. DEEPSEEK_API_KEY present in .env and valid. Note: the running gateway session continues on admin-ai until next restart but is protected by the already-active direct-first fallback chain (deepseek -> google -> xai -> anthropic -> openai). Residual admin-ai references remain only in vision paths (auxiliary.vision and top-level vision), non-blocking for C1 terminal/SSH work. |
|
||||
| P2-C9-001 | 2026-08-13 | Warm-standby DB snapshot consistency: hermes-live-sync.sh now exports state.db and memory_store.db via SQLite online backup (.backup) into a consistent snapshot uploaded as the authoritative S3 object, and excludes the raw live WAL/SHM files from the sync. Deleted obsolete transient state.db-wal/shm and memory_store.db-wal/shm objects from s3://hermes-vps-backups/live/. | Phase Two Workstream 1 C9 (Germaine) | cp ~/.hermes/scripts/hermes-live-sync.sh.bak-20260813-C9 ~/.hermes/scripts/hermes-live-sync.sh (raw file copy behavior restored) | done | LiteLLM-adjacent=no. Verified: .backup of the 2.98 GB live state.db completed in 9s and passed PRAGMA integrity_check (ok); fixed script ran exit 0 and uploaded the 2980638720-byte state.db and 1871872-byte memory_store.db snapshots at 15:30 ET. Root cause: aws s3 sync of a live WAL-mode SQLite DB can capture a torn checkpoint (evidence: state.db.corrupted plus 3 malformed-backup-* objects in S3 dated 2026-07-09). Residual follow-up: profiles/anita/state.db* still synced raw because top-level excludes do not match profile paths; and ~7.5 GB of stale corrupted/malformed S3 objects left in place pending a cleanup decision. |
|
||||
| P2-C4-001 | 2026-08-13 | Restored least-privilege file permissions on app1-bu standby: chmod 600 on 11 world-readable secret-bearing files (.hermes/.env, .hermes/config.yaml, profiles/anita/.env, profiles/anita/google_client_secret.json, docker/vaultwarden/.env, docker/twenty/.env, references/glc-credentials.md, references/liberty-credentials.md, .aws/credentials.bak, and two state-snapshot .env files). Fixed root cause: added umask 077 to hermes-standby-sync.sh, hermes-standby-watchdog.sh, and hermes-standby-restore.sh (standby) plus hermes-live-sync.sh (Core) so aws s3 sync no longer re-creates synced files as 644 and the Core .snapshots output is no longer transiently world-readable. | Phase Two Workstream 1 C4 (Germaine) | Restore the three .bak-20260813-C4 script backups (cp each .bak-20260813-C4 back over its script); re-chmod 644 the files (not recommended) | done | LiteLLM-adjacent=no. Root cause: aws s3 sync does not preserve POSIX permissions (S3 objects carry no mode bits), so the 10-min standby sync re-created Core 600 files as 644 under the default umask 022. Verified: functional test of aws s3 sync under umask 077 lands a downloaded object as 600; bash -n passes on all three edited scripts. Core already holds the same files at 600, so no Core change was needed. |
|
||||
| P2-C1-002 | 2026-08-13 | DOCKER-USER firewall hardening made persistent on Core, app1, app2, app3. Live IPv4 rules (applied earlier 2026-08-13) now survive reboot and docker restart via /usr/local/sbin/itpp-docker-user.sh (idempotent flush+rebuild of DOCKER-USER, IPv4 + IPv6 where present) wired as an ExecStartPost drop-in at /etc/systemd/system/docker.service.d/itpp-docker-user.conf on each host. Closed to Tailscale-only (100.64.0.0/10 + ESTABLISHED,RELATED only): Core browserless 3000, camofox 9377, uptime-kuma 3001; app1 komodo 9120, twenty 3003, wazuh dashboard 5601, wazuh indexer 9200; app2 bookstack 6875 (direct), minio 9001, gitea ssh 3022, ragflow 9380-9384/9392/9393, infinity 23817/23820, unifi admin 8443/8843/8880, unms 81/8444; app3 buzz relay 3000. Preserved public: app1 wazuh manager 1514/1515/55000 tcp + 514 udp; app2 dns 53 tcp+udp, support-api 6880, unifi inform 8080, traccar 8082 + 5000-5150 tcp+udp, unms-nginx 8089, bookstack 6875 via ctorigdstport proxy path, unms-netflow 2055 udp, unifi 3478/10001 udp. | Phase Two Workstream 1 C1 (Germaine) | On each host: rm /etc/systemd/system/docker.service.d/itpp-docker-user.conf && systemctl daemon-reload; then iptables -F DOCKER-USER (and ip6tables -F DOCKER-USER where present); rm /usr/local/sbin/itpp-docker-user.sh | done | LiteLLM-adjacent=no. Rules scoped -i eth0 so docker-bridge traffic is untouched. Verified live: iptables -L DOCKER-USER -n -v on all 4 hosts shows correct chains with -i eth0 and active packet counters. Persistence proven end-to-end: ufw reload on Core preserved FORWARD integration and DOCKER-USER rules; systemctl restart docker on app3 flushed DOCKER-USER (counters reset) then ExecStartPost re-applied rules immediately, all 8 app3 containers returned healthy (hexclave-server health:starting then healthy). Pre-existing issues noted, not C1: crm.debtrecoveryexperts.com and crm.intelsight.io 502 because Core Caddy proxies to Core-local localhost:3003 where nothing listens (Twenty runs app1:3003); admin-ai /v1/models 401 despite /health 200 (auth, not firewall); infinity 23817 returns 000 from minio while 23820 returns 404 (port semantics). |
|
||||
| P2-C4-002 | 2026-08-13 | C4 credential rotation: D1 approved (rotate A and B, revoke C). Phase 0 inventory complete. Rollback snapshots taken: /root/.hermes/config.yaml.bak-20260813-C4-pre and /root/.hermes/.env.bak-20260813-C4-pre, both chmod 600. | Germaine (A rotate, B rotate, C revoke, 2026-08-13) | Restore both C4-pre backups | done | LiteLLM-adjacent=yes (this entry is the go/no-go gate for LiteLLM key changes). No keys changed yet; decision record only. Phase 0 verdict: A=7 rotate, B=7 rotate, C=4 revoke. |
|
||||
| P2-C4-003 | 2026-08-13 | Revoked 4 dead AI keys from plaintext: removed AI21_API_KEY, ALIBABA_API_KEY (duplicate, x2), ZAI_API_KEY, NVIDIA_API_KEY from Core /root/.hermes/.env and app1-bu /root/.hermes/.env (5 lines to 0), plus the stale NVIDIA NIM comment line. PARALLEL_API_KEY reclassified: it is a live Super Search provider (Parallel.ai Search, fallback #11 in server.py), not an orphan, moved to Phase 3 service tokens (Keep). | Germaine (C revoke) | Restore /root/.hermes/.env.bak-20260813-C4-pre on Core and app1-bu .env from its prior state | done | LiteLLM-adjacent=no (plaintext .env removal only; no LiteLLM credential touched). Verified before/after line counts and mode 600 on both hosts. Residual: NVIDIA LiteLLM credential (0 working models) still in app1 Postgres, deletion deferred to the LiteLLM step. |
|
||||
| P2-C4-004 | 2026-08-14 | Rotated 5 assistant-owned portal API keys (b1 set): Cohere, Fireworks, Mistral, Perplexity, FAL. Applied across three layers: (1) LiteLLM credentials PATCHed via /credentials/{name} on app1 (Cohere, Fireworks, Mistral, Perplexity, Fal) — 5/5 HTTP 200; (2) Core /root/.hermes/.env swapped (COHERE_API_KEY, FIREWORKS_API_KEY, MISTRAL_API_KEY, PERPLEXITY_API_KEY, FAL_KEY) with backup /root/.hermes/.env.bak-20260814-b1; (3) Vaultwarden — created 3 new 'X - API Key' items (Cohere, Fireworks, Mistral; username info@itpropartner.com) and updated Perplexity ('console.perplexity.ai') and FAL ('fal.ai') item passwords in place. Mistral+Cohere keys generated in-console as 'LiteLLM-20260813'; Fireworks/FAL/Perplexity keys supplied by Germaine. | Germaine (C4 D1 GO + b1 direction 2026-08-14) | Core .env: cp /root/.hermes/.env.bak-20260814-b1 /root/.hermes/.env. LiteLLM: re-PATCH the 5 credentials with pre-rotation values (old keys remain active provider-side; recoverable from .env.bak-20260813-C4-pre and Vaultwarden password history). Vaultwarden: restore Perplexity/FAL item passwords via bw item password history; delete the 3 new 'X - API Key' items. | done | LiteLLM-adjacent=yes. Verified: all 5 keys live against providers (Mistral/Cohere/Fireworks/Perplexity HTTP 200; FAL past auth gateway); LiteLLM PATCH 5/5 HTTP 200; .env values correct with pre-existing dual NETCUP_API_KEY quirk preserved; Vaultwarden re-read 5/5 PASS. Active consumers: Perplexity -> Super Search (super-search.service restarted; health_check reports perplexity:ok, server v2.4.0); FAL -> image_generate (picks up new key on next Hermes restart; running process holds prior key in os.environ). Cohere/Fireworks/Mistral have no active Core consumer (LiteLLM 'use if set' only). Old keys NOT revoked provider-side. |
|
||||
| P2-C4-005 | 2026-08-14 | Rotated the admin-ai LiteLLM master key: old (len 66, sk-litel...) -> new (len 67, sk-x2top..., sk- + 64 mixed-case alphanumeric, no special chars). Applied to: (1) app1 /root/docker/litellm/.env LITELLM_MASTER_KEY and UI_PASSWORD (backup .env.bak-20260814-adminai); (2) Core /root/.hermes/scripts/cost-alert-watchdog.py MASTER_KEY (backup .bak-20260814-adminai). Recreated litellm container via `docker compose up -d --force-recreate litellm` (a bare `docker restart` does NOT re-read env vars — caught during verification). Vaultwarden consolidated: new key in item 'LiteLLM (admin-ai) Master Key' 0772c50c; updated 'admin-ai.itpropartner.com' bb01ef49 to the new key; deleted stale 'LiteLLM Master Key' 446da6ca and duplicate 'admin-ai.itpropartner.com' 8c8a5eb1. | Germaine (added new key to Vaultwarden as 'LiteLLM (admin-ai) Master Key', 2026-08-14) | Restore app1 .env from .env.bak-20260814-adminai and watchdog from .bak-20260814-adminai, then `docker compose up -d --force-recreate litellm` (old key is now revoked 401, so rollback requires restoring the old value to .env first); restore deleted Vaultwarden items from trash. | done | LiteLLM-adjacent=yes. Verified on app1 localhost:4000 — new key HTTP 200 on /spend/keys and /global/spend, old key HTTP 401 (revoked); cost-alert-watchdog ran clean against the new key (surfaced real alert: hermes-agent-v5 at 88% budget); Core 25-char virtual key 'Hermes (Sho'Nuff) LiteLLM Virtual API Key' (0aed1ca4) unaffected, /v1/chat/completions HTTP 200; LITELLM_SALT_KEY unchanged. Core admin-ai provider uses the 25-char virtual key (not the master key), so no Core config.yaml change was required. |
|
||||
@@ -0,0 +1,83 @@
|
||||
# ITPP Phase Two - Remediation Cost Estimate
|
||||
|
||||
**Date:** 2026-08-13
|
||||
**Status:** AWAITING GERMAINE APPROVAL (C7 + C3 pre-approved fast-track, already executing)
|
||||
**Conductor:** Sho'Nuff
|
||||
|
||||
---
|
||||
|
||||
## Anchor (Phase One actuals, for calibration)
|
||||
|
||||
Phase One was a read-only audit of 6 hosts plus the repo estate. From `brief.md` and the final report:
|
||||
|
||||
- Approved estimate: subtotal ~$5.40
|
||||
- Realistic range: $8-10
|
||||
- Ceiling: ~$13
|
||||
- **Actual: exceeded the ceiling.** Driven by claude-sonnet-5 subagent usage and re-runs from 429 rate-limit failures.
|
||||
- 7-day estate-wide DeepSeek total: $61.81 (all usage, not audit-only).
|
||||
|
||||
Phase Two is **live-change remediation** (investigate, change, verify, rollback-test, log), roughly 2-4x the per-unit cost of a read-only finding. The estimate below carries margin because Phase One actuals exceeded estimate. Under the $20/day cap, this spreads across ~8-12 execution days.
|
||||
|
||||
Effort labels are taken verbatim from the Phase One report recommendations table (S = under 1h, M = half day, L = 1-2 days, XL = multi-day).
|
||||
|
||||
---
|
||||
|
||||
## Workstream 1 - Critical findings (itemized, per requirement)
|
||||
|
||||
| Item | Finding (report ref) | Effort | Estimate |
|
||||
|---|---|---|---|
|
||||
| C1 | Docker published-port UFW bypass, ~20 public consoles | M-L | $5-14 |
|
||||
| C4 | Plaintext credentials, inventory + rotate + vault | L-XL | $12-28 |
|
||||
| C5 | LiteLLM Postgres backup gap (pg_dump wrong DB name) | S-M | $2-5 |
|
||||
| C6 | app3 shared MySQL, no tenant boundary (~24 sites) | L-XL | $12-28 |
|
||||
| C8 | wphost02 backup gap + decommission (split-brain) | M | $3-8 |
|
||||
| C9 | Warm standby state-DB sync (RPO ~28d stale) | M | $3-8 |
|
||||
| C10 | Single shared SSH key, passwordless root (Indep D4) | M | $3-8 |
|
||||
| D3 | Public repo re-leaks admin creds (EX-001, deferred) | M | $3-8 |
|
||||
| C3 | Wazuh agents on 5 hosts (FAST-TRACK, in progress) | M | $3-8 |
|
||||
| C7 | Grafana rotate + Tailscale (FAST-TRACK, in progress) | S | $1-3 |
|
||||
|
||||
**WS1 subtotal: ~$47-118**
|
||||
|
||||
## Workstreams 2-6 (workstream-level)
|
||||
|
||||
Scope drawn from Phase One report sections noted.
|
||||
|
||||
| Workstream | Scope (report ref) | Estimate |
|
||||
|---|---|---|
|
||||
| WS2 - High findings | MFA enforcement (Gitea/CloudPanel/Vaultwarden/Grafana, report #11), fail2ban + unattended-upgrades + auditd (#12), backup gaps + restore-tests (#13), UNMS to UISP + image pinning (#14), port 8200 + runaway processes + gateway systemd (#15), retire NOPASSWD:ALL for named sudo (#10) | $20-50 |
|
||||
| WS3 - Segmentation | C2 Tailscale ACL tags + three-tier model (internal/client/product) from report section 6 | $12-28 |
|
||||
| WS4 - Git reorg | Repo estate cleanup (report 2.3) + private-repo credential exposures (Git-A) | $5-14 |
|
||||
| WS5 - Documentation | Say-do gap fixes across 12 doc sections (report section 5) | $3-10 |
|
||||
| WS6 - Operationalize P&P | Stand up Running Exemptions doc + change-management cadence (bootstrapped) | $3-10 |
|
||||
|
||||
**WS2-6 subtotal: ~$43-112**
|
||||
|
||||
## Independence / verification buffer
|
||||
|
||||
Conductor independence checks (Sonnet 5) on consequential live-change "is this safe to do live" calls, per the Phase One mitigation pattern: $6-12
|
||||
|
||||
---
|
||||
|
||||
## Totals
|
||||
|
||||
| Line | Range |
|
||||
|---|---|
|
||||
| WS1 (Critical) | $47-118 |
|
||||
| WS2-6 | $43-112 |
|
||||
| Independence buffer | $6-12 |
|
||||
| **TOTAL** | **~$96-242 (midpoint ~$170)** |
|
||||
|
||||
Pacing: ~8-12 days at the $20/day cap. No opus-tier models; primary DeepSeek V4 Pro, Sonnet 5 only for independence checks.
|
||||
|
||||
## Scope-variance flags (flag before absorbing)
|
||||
|
||||
- **C4** inventory may be larger than the report captured (6,296 Core / 5,504 app1-bu / 1,275 app2 secret-sprawl hits need per-file triage).
|
||||
- **C6** app3 DB segmentation is structural and may surface new per-tenant requirements mid-flight.
|
||||
- **D3** depends on Germaine's repo decision (currently deferred as EX-001).
|
||||
- **C1** on app1 is LiteLLM-adjacent (Hermes's own inference path); fallback chain will be used and logged.
|
||||
|
||||
## Pre-flight confirmation
|
||||
|
||||
- Fallback chain verified intact: primary = admin-ai (LiteLLM on app1); fallbacks = deepseek (direct api.deepseek.com), google, xai, anthropic, openai, all configured direct.
|
||||
- C7 (Core) and C3 (agent enrollment, manager ports already published) are NOT LiteLLM-adjacent; no fallback switch required.
|
||||
@@ -0,0 +1,223 @@
|
||||
[
|
||||
{
|
||||
"Id": "66114deb715b3c90199bff9be1b4bbce3522e3152d6b4c0ff15256cbe2dfccf7",
|
||||
"Created": "2026-07-16T06:40:35.087889714Z",
|
||||
"Path": "/run.sh",
|
||||
"Args": [],
|
||||
"State": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 670022,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-07-16T06:40:35.232657942Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z"
|
||||
},
|
||||
"Image": "sha256:c0b69935a2469f0add84fa71a1632a2594c78aeeb016ba9f5d8728a4b4c1b976",
|
||||
"ResolvConfPath": "/var/lib/docker/containers/66114deb715b3c90199bff9be1b4bbce3522e3152d6b4c0ff15256cbe2dfccf7/resolv.conf",
|
||||
"HostnamePath": "/var/lib/docker/containers/66114deb715b3c90199bff9be1b4bbce3522e3152d6b4c0ff15256cbe2dfccf7/hostname",
|
||||
"HostsPath": "/var/lib/docker/containers/66114deb715b3c90199bff9be1b4bbce3522e3152d6b4c0ff15256cbe2dfccf7/hosts",
|
||||
"LogPath": "/var/lib/docker/containers/66114deb715b3c90199bff9be1b4bbce3522e3152d6b4c0ff15256cbe2dfccf7/66114deb715b3c90199bff9be1b4bbce3522e3152d6b4c0ff15256cbe2dfccf7-json.log",
|
||||
"Name": "/grafana",
|
||||
"RestartCount": 0,
|
||||
"Driver": "overlay2",
|
||||
"Platform": "linux",
|
||||
"MountLabel": "",
|
||||
"ProcessLabel": "",
|
||||
"AppArmorProfile": "docker-default",
|
||||
"ExecIDs": null,
|
||||
"HostConfig": {
|
||||
"Binds": [
|
||||
"grafana_data_final:/var/lib/grafana"
|
||||
],
|
||||
"ContainerIDFile": "",
|
||||
"LogConfig": {
|
||||
"Type": "json-file",
|
||||
"Config": {}
|
||||
},
|
||||
"NetworkMode": "host",
|
||||
"PortBindings": {},
|
||||
"RestartPolicy": {
|
||||
"Name": "unless-stopped",
|
||||
"MaximumRetryCount": 0
|
||||
},
|
||||
"AutoRemove": false,
|
||||
"VolumeDriver": "",
|
||||
"VolumesFrom": null,
|
||||
"ConsoleSize": [
|
||||
0,
|
||||
0
|
||||
],
|
||||
"CapAdd": null,
|
||||
"CapDrop": null,
|
||||
"CgroupnsMode": "private",
|
||||
"Dns": [],
|
||||
"DnsOptions": [],
|
||||
"DnsSearch": [],
|
||||
"ExtraHosts": null,
|
||||
"GroupAdd": null,
|
||||
"IpcMode": "private",
|
||||
"Cgroup": "",
|
||||
"Links": null,
|
||||
"OomScoreAdj": 0,
|
||||
"PidMode": "",
|
||||
"Privileged": false,
|
||||
"PublishAllPorts": false,
|
||||
"ReadonlyRootfs": false,
|
||||
"SecurityOpt": null,
|
||||
"UTSMode": "",
|
||||
"UsernsMode": "",
|
||||
"ShmSize": 67108864,
|
||||
"Runtime": "runc",
|
||||
"Isolation": "",
|
||||
"CpuShares": 0,
|
||||
"Memory": 0,
|
||||
"NanoCpus": 0,
|
||||
"CgroupParent": "",
|
||||
"BlkioWeight": 0,
|
||||
"BlkioWeightDevice": [],
|
||||
"BlkioDeviceReadBps": [],
|
||||
"BlkioDeviceWriteBps": [],
|
||||
"BlkioDeviceReadIOps": [],
|
||||
"BlkioDeviceWriteIOps": [],
|
||||
"CpuPeriod": 0,
|
||||
"CpuQuota": 0,
|
||||
"CpuRealtimePeriod": 0,
|
||||
"CpuRealtimeRuntime": 0,
|
||||
"CpusetCpus": "",
|
||||
"CpusetMems": "",
|
||||
"Devices": [],
|
||||
"DeviceCgroupRules": null,
|
||||
"DeviceRequests": null,
|
||||
"MemoryReservation": 0,
|
||||
"MemorySwap": 0,
|
||||
"MemorySwappiness": null,
|
||||
"OomKillDisable": null,
|
||||
"PidsLimit": null,
|
||||
"Ulimits": [],
|
||||
"CpuCount": 0,
|
||||
"CpuPercent": 0,
|
||||
"IOMaximumIOps": 0,
|
||||
"IOMaximumBandwidth": 0,
|
||||
"MaskedPaths": [
|
||||
"/proc/asound",
|
||||
"/proc/acpi",
|
||||
"/proc/kcore",
|
||||
"/proc/keys",
|
||||
"/proc/latency_stats",
|
||||
"/proc/timer_list",
|
||||
"/proc/timer_stats",
|
||||
"/proc/sched_debug",
|
||||
"/proc/scsi",
|
||||
"/sys/firmware",
|
||||
"/sys/devices/virtual/powercap"
|
||||
],
|
||||
"ReadonlyPaths": [
|
||||
"/proc/bus",
|
||||
"/proc/fs",
|
||||
"/proc/irq",
|
||||
"/proc/sys",
|
||||
"/proc/sysrq-trigger"
|
||||
]
|
||||
},
|
||||
"GraphDriver": {
|
||||
"Data": {
|
||||
"LowerDir": "/var/lib/docker/overlay2/3ab0a39f1573bc37ee3fc9203d8f876bb9ed9df325b39ca776e3ef86da2bc9ff-init/diff:/var/lib/docker/overlay2/96d3092d12c0e0b0c1bd01a5818f003bb7fab32246a893224296a67799841669/diff:/var/lib/docker/overlay2/4da7e8838b31487c835f93853d4a81e5cd06434fa34d04fdbbf2141cb95abc13/diff:/var/lib/docker/overlay2/8e1e7eaf9bac203b5e4a54e1c80c317f6483d40610b1776e9f6911aa18c41a72/diff:/var/lib/docker/overlay2/02a0a88da9ab58ca40a03ab418e1248317e5917fa318654d4cf60bf394b21b07/diff:/var/lib/docker/overlay2/9cc7ea00867e008c469d86f0bf2244d3e60737a665648609a303c93c8e3805cf/diff:/var/lib/docker/overlay2/8b1f24758f00124ab15b7b5b9957594184426ad6aa1911d76c5add384d1613ed/diff:/var/lib/docker/overlay2/bd4250cfbd3624493c084c81954c59a9744c13a863da7cd3ef348f0accf19719/diff:/var/lib/docker/overlay2/041749d1c02cf48d8a3257d604c5a7cfae504fc36f492854a4e056c0562cf599/diff:/var/lib/docker/overlay2/21d92f72aced24d754ce4a6daf302e9dd122273b48fdd107e8ede5c1018d4a4d/diff:/var/lib/docker/overlay2/2311d9ffcf0c02b1565678784a4f5bb120660ae0c021094f152cbd9747a5b916/diff",
|
||||
"MergedDir": "/var/lib/docker/overlay2/3ab0a39f1573bc37ee3fc9203d8f876bb9ed9df325b39ca776e3ef86da2bc9ff/merged",
|
||||
"UpperDir": "/var/lib/docker/overlay2/3ab0a39f1573bc37ee3fc9203d8f876bb9ed9df325b39ca776e3ef86da2bc9ff/diff",
|
||||
"WorkDir": "/var/lib/docker/overlay2/3ab0a39f1573bc37ee3fc9203d8f876bb9ed9df325b39ca776e3ef86da2bc9ff/work"
|
||||
},
|
||||
"Name": "overlay2"
|
||||
},
|
||||
"Mounts": [
|
||||
{
|
||||
"Type": "volume",
|
||||
"Name": "grafana_data_final",
|
||||
"Source": "/var/lib/docker/volumes/grafana_data_final/_data",
|
||||
"Destination": "/var/lib/grafana",
|
||||
"Driver": "local",
|
||||
"Mode": "z",
|
||||
"RW": true,
|
||||
"Propagation": ""
|
||||
}
|
||||
],
|
||||
"Config": {
|
||||
"Hostname": "core",
|
||||
"Domainname": "",
|
||||
"User": "472",
|
||||
"AttachStdin": false,
|
||||
"AttachStdout": false,
|
||||
"AttachStderr": false,
|
||||
"ExposedPorts": {
|
||||
"3000/tcp": {}
|
||||
},
|
||||
"Tty": false,
|
||||
"OpenStdin": false,
|
||||
"StdinOnce": false,
|
||||
"Env": [
|
||||
"GF_SECURITY_ADMIN_USER=admin",
|
||||
"GF_SECURITY_ADMIN_PASSWORD=admin",
|
||||
"GF_SERVER_HTTP_PORT=3002",
|
||||
"PATH=/usr/share/grafana/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
|
||||
"GF_PATHS_CONFIG=/etc/grafana/grafana.ini",
|
||||
"GF_PATHS_DATA=/var/lib/grafana",
|
||||
"GF_PATHS_HOME=/usr/share/grafana",
|
||||
"GF_PATHS_LOGS=/var/log/grafana",
|
||||
"GF_PATHS_PLUGINS=/var/lib/grafana/plugins",
|
||||
"GF_PATHS_PROVISIONING=/etc/grafana/provisioning"
|
||||
],
|
||||
"Cmd": null,
|
||||
"Image": "grafana/grafana:11.4.0",
|
||||
"Volumes": null,
|
||||
"WorkingDir": "/usr/share/grafana",
|
||||
"Entrypoint": [
|
||||
"/run.sh"
|
||||
],
|
||||
"OnBuild": null,
|
||||
"Labels": {
|
||||
"maintainer": "Grafana Labs <hello@grafana.com>"
|
||||
}
|
||||
},
|
||||
"NetworkSettings": {
|
||||
"Bridge": "",
|
||||
"SandboxID": "fb05509631e71c8372e97ea4f43331331476ce53ecef4e52494b20e96c9debc0",
|
||||
"SandboxKey": "/var/run/docker/netns/default",
|
||||
"Ports": {},
|
||||
"HairpinMode": false,
|
||||
"LinkLocalIPv6Address": "",
|
||||
"LinkLocalIPv6PrefixLen": 0,
|
||||
"SecondaryIPAddresses": null,
|
||||
"SecondaryIPv6Addresses": null,
|
||||
"EndpointID": "",
|
||||
"Gateway": "",
|
||||
"GlobalIPv6Address": "",
|
||||
"GlobalIPv6PrefixLen": 0,
|
||||
"IPAddress": "",
|
||||
"IPPrefixLen": 0,
|
||||
"IPv6Gateway": "",
|
||||
"MacAddress": "",
|
||||
"Networks": {
|
||||
"host": {
|
||||
"IPAMConfig": null,
|
||||
"Links": null,
|
||||
"Aliases": null,
|
||||
"MacAddress": "",
|
||||
"NetworkID": "f7ebc0a30ba605067a1c4f5b22c519f3c3cb4f46e8a4b4359787cf0fe8d09a37",
|
||||
"EndpointID": "458bbfe47f84e5aab9820274ff509a215b8b24bc99e8cbb388081a0d75f028b0",
|
||||
"Gateway": "",
|
||||
"IPAddress": "",
|
||||
"IPPrefixLen": 0,
|
||||
"IPv6Gateway": "",
|
||||
"GlobalIPv6Address": "",
|
||||
"GlobalIPv6PrefixLen": 0,
|
||||
"DriverOpts": null,
|
||||
"DNSNames": null
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,37 @@
|
||||
# Running Exemptions Document
|
||||
|
||||
Authority: Germaine Brown is the sole authorizer of any departure from the
|
||||
Policy & Procedure document (P&P section 8). No exemption is assumed; it is requested,
|
||||
justified, approved, and recorded here.
|
||||
|
||||
## Schema
|
||||
|
||||
| Field | Meaning |
|
||||
|---|---|
|
||||
| ID | EX-### |
|
||||
| Date | When requested/approved (ET) |
|
||||
| Requester | Who raised it |
|
||||
| Provision departed from | Which P&P section/policy |
|
||||
| Request + justification | The business reason |
|
||||
| Authorization | Germaine's decision |
|
||||
| One-time / ongoing | Duration |
|
||||
| Follow-up date | For ongoing exemptions |
|
||||
| Status | open / closed |
|
||||
|
||||
---
|
||||
|
||||
## Entries
|
||||
|
||||
### EX-001: Public-repo credential exposure (D3) deferred
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| ID | EX-001 |
|
||||
| Date | 2026-08-13 (raised in Phase One, re-surfaced at Phase Two start) |
|
||||
| Requester | Germaine Brown |
|
||||
| Provision departed from | P&P section 1 (no live credentials in any repo): the public repo `itpp-infrastructure` re-leaks live admin credentials |
|
||||
| Request + justification | Defer the repo lockdown ("leave the repo alone for now") while Phase One was read-only |
|
||||
| Authorization | Deferred by Germaine in Phase One |
|
||||
| One-time / ongoing | One-time, with a Phase Two follow-up |
|
||||
| Follow-up date | Phase Two start (now): this is Workstream 1 item 10 (D3/EX-001) |
|
||||
| Status | OPEN: pending Germaine's updated decision now that Phase Two is starting |
|
||||
@@ -0,0 +1,30 @@
|
||||
#!/bin/bash
|
||||
# ITPP Phase Two C1 - DOCKER-USER hardening (management consoles Tailscale-only)
|
||||
# Host: app1 (152.53.36.131) | Applied 2026-08-13 | Authorized: Phase Two Workstream 1 C1 (Germaine)
|
||||
# Idempotent: flushes and rebuilds DOCKER-USER (IPv4 + IPv6). Safe to re-run manually.
|
||||
# Closes (Tailscale-only): komodo 9120, twenty-server 3003, wazuh dashboard 5601, wazuh indexer 9200.
|
||||
# Preserves (public): wazuh manager 1514/1515/55000 tcp + 514 udp (5 active agents).
|
||||
# NOTE: litellm is 127.0.0.1:4000 behind Caddy (admin-ai.itpropartner.com) - untouched here.
|
||||
set -eu
|
||||
|
||||
if ! iptables -L DOCKER-USER -n >/dev/null 2>&1; then
|
||||
iptables -N DOCKER-USER
|
||||
fi
|
||||
iptables -F DOCKER-USER
|
||||
iptables -A DOCKER-USER -s 100.64.0.0/10 -j RETURN
|
||||
iptables -A DOCKER-USER -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
|
||||
iptables -A DOCKER-USER -i eth0 -p tcp -m multiport --dports 1514,1515,55000 -j RETURN
|
||||
iptables -A DOCKER-USER -i eth0 -p udp -m multiport --dports 514 -j RETURN
|
||||
iptables -A DOCKER-USER -i eth0 -j DROP
|
||||
iptables -A DOCKER-USER -j RETURN
|
||||
|
||||
if ip6tables -L DOCKER-USER -n >/dev/null 2>&1; then
|
||||
ip6tables -F DOCKER-USER
|
||||
ip6tables -A DOCKER-USER -s fd7a:115c:a1e0::/48 -j RETURN
|
||||
ip6tables -A DOCKER-USER -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
|
||||
ip6tables -A DOCKER-USER -i eth0 -p tcp -m multiport --dports 1514,1515,55000 -j RETURN
|
||||
ip6tables -A DOCKER-USER -i eth0 -p udp -m multiport --dports 514 -j RETURN
|
||||
ip6tables -A DOCKER-USER -i eth0 -j DROP
|
||||
ip6tables -A DOCKER-USER -j RETURN
|
||||
fi
|
||||
exit 0
|
||||
@@ -0,0 +1,42 @@
|
||||
#!/bin/bash
|
||||
# ITPP Phase Two C1 - DOCKER-USER hardening (management consoles Tailscale-only)
|
||||
# Host: app2 (152.53.39.202) | Applied 2026-08-13 | Authorized: Phase Two Workstream 1 C1 (Germaine)
|
||||
# Idempotent: flushes and rebuilds DOCKER-USER (IPv4 + IPv6). Safe to re-run manually.
|
||||
#
|
||||
# Closes (Tailscale-only, dropped by the final eth0 DROP; admin UIs still reachable via Caddy :443 -> localhost):
|
||||
# unifi-controller admin UI 8443 + guest portal 8843/8880 (unifi.itpropartner.com -> Caddy -> localhost:8443);
|
||||
# unms-nginx admin UI 8444 + http 81 (unms.forefrontwireless.com -> Caddy -> localhost:8444);
|
||||
# minio console 9001, infinity 23817+23820, ragflow 9380-9384+9392+9393 (ragflow.itpropartner.com -> Caddy -> localhost:9392);
|
||||
# gitea ssh 3022 (web is 127.0.0.1:3001 -> Caddy git.itpropartner.com).
|
||||
# Preserves (public):
|
||||
# technitium DNS 53 tcp+udp;
|
||||
# bookstack 6875 (published 6875 -> container 80, matched via ctorigdstport; app3 support site proxies here);
|
||||
# support-api 6880 (app3 support.itpropartner.com -> 152.53.39.202:6880);
|
||||
# traccar 8082 + 5000-5150 tcp+udp (GPS devices + Core Caddy cross-proxy);
|
||||
# unifi-controller 8080 tcp (23 remote devices inform here) + 3478/10001 udp (STUN/discovery);
|
||||
# unms-nginx 8089 tcp + unms-netflow 2055 udp (UISP device-facing; conservative - see change log notes).
|
||||
set -eu
|
||||
|
||||
if ! iptables -L DOCKER-USER -n >/dev/null 2>&1; then
|
||||
iptables -N DOCKER-USER
|
||||
fi
|
||||
iptables -F DOCKER-USER
|
||||
iptables -A DOCKER-USER -s 100.64.0.0/10 -j RETURN
|
||||
iptables -A DOCKER-USER -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
|
||||
iptables -A DOCKER-USER -i eth0 -p tcp -m multiport --dports 53,6880,8080,8082,8089,5000:5150 -j RETURN
|
||||
iptables -A DOCKER-USER -i eth0 -p tcp -m conntrack --ctorigdstport 6875 -j RETURN
|
||||
iptables -A DOCKER-USER -i eth0 -p udp -m multiport --dports 53,2055,3478,10001,5000:5150 -j RETURN
|
||||
iptables -A DOCKER-USER -i eth0 -j DROP
|
||||
iptables -A DOCKER-USER -j RETURN
|
||||
|
||||
if ip6tables -L DOCKER-USER -n >/dev/null 2>&1; then
|
||||
ip6tables -F DOCKER-USER
|
||||
ip6tables -A DOCKER-USER -s fd7a:115c:a1e0::/48 -j RETURN
|
||||
ip6tables -A DOCKER-USER -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
|
||||
ip6tables -A DOCKER-USER -i eth0 -p tcp -m multiport --dports 53,6880,8080,8082,8089,5000:5150 -j RETURN
|
||||
ip6tables -A DOCKER-USER -i eth0 -p tcp -m conntrack --ctorigdstport 6875 -j RETURN
|
||||
ip6tables -A DOCKER-USER -i eth0 -p udp -m multiport --dports 53,2055,3478,10001,5000:5150 -j RETURN
|
||||
ip6tables -A DOCKER-USER -i eth0 -j DROP
|
||||
ip6tables -A DOCKER-USER -j RETURN
|
||||
fi
|
||||
exit 0
|
||||
@@ -0,0 +1,25 @@
|
||||
#!/bin/bash
|
||||
# ITPP Phase Two C1 - DOCKER-USER hardening (management consoles Tailscale-only)
|
||||
# Host: app3 (152.53.241.111) | Applied 2026-08-13 | Authorized: Phase Two Workstream 1 C1 (Germaine)
|
||||
# Idempotent: flushes and rebuilds DOCKER-USER (IPv4 + IPv6). Safe to re-run manually.
|
||||
# Closes (Tailscale-only): buzz-prod-relay 3000/tcp (publicly served via nginx loopback buzz.iamgmb.com).
|
||||
# Preserves: none (hexclave 8101-8102 is loopback; nginx 80/443 is host INPUT chain).
|
||||
set -eu
|
||||
|
||||
if ! iptables -L DOCKER-USER -n >/dev/null 2>&1; then
|
||||
iptables -N DOCKER-USER
|
||||
fi
|
||||
iptables -F DOCKER-USER
|
||||
iptables -A DOCKER-USER -s 100.64.0.0/10 -j RETURN
|
||||
iptables -A DOCKER-USER -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
|
||||
iptables -A DOCKER-USER -i eth0 -j DROP
|
||||
iptables -A DOCKER-USER -j RETURN
|
||||
|
||||
if ip6tables -L DOCKER-USER -n >/dev/null 2>&1; then
|
||||
ip6tables -F DOCKER-USER
|
||||
ip6tables -A DOCKER-USER -s fd7a:115c:a1e0::/48 -j RETURN
|
||||
ip6tables -A DOCKER-USER -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
|
||||
ip6tables -A DOCKER-USER -i eth0 -j DROP
|
||||
ip6tables -A DOCKER-USER -j RETURN
|
||||
fi
|
||||
exit 0
|
||||
@@ -0,0 +1,21 @@
|
||||
#!/bin/bash
|
||||
# ITPP Phase Two C1 - DOCKER-USER hardening (management consoles Tailscale-only)
|
||||
# Host: core (152.53.192.33) | Applied 2026-08-13 | Authorized: Phase Two Workstream 1 C1 (Germaine)
|
||||
# Idempotent: flushes and rebuilds the DOCKER-USER chain. Safe to re-run manually.
|
||||
# Closes (Tailscale-only): browserless 3000/tcp, camofox-browser 9377/tcp, uptime-kuma 3001/tcp.
|
||||
# Preserves: none (no public Docker services on core; Caddy 80/443 is host INPUT chain).
|
||||
set -eu
|
||||
|
||||
if ! iptables -L DOCKER-USER -n >/dev/null 2>&1; then
|
||||
iptables -N DOCKER-USER
|
||||
fi
|
||||
iptables -F DOCKER-USER
|
||||
iptables -A DOCKER-USER -s 100.64.0.0/10 -j RETURN
|
||||
iptables -A DOCKER-USER -m conntrack --ctstate ESTABLISHED,RELATED -j RETURN
|
||||
iptables -A DOCKER-USER -i eth0 -j DROP
|
||||
iptables -A DOCKER-USER -j RETURN
|
||||
|
||||
# IPv6: Docker ip6tables integration is off on core (no ip6tables DOCKER-USER chain,
|
||||
# no IPv6 DNAT). [::] publishes are userland docker-proxy (INPUT path) and are already
|
||||
# blocked by UFW IPv6 default-deny (only 22/80/443/8080-tailscale/51821/8890 allowed).
|
||||
exit 0
|
||||
@@ -0,0 +1,12 @@
|
||||
[Unit]
|
||||
Description=ITPP P2-C1 DOCKER-USER firewall hardening (Tailscale-only management consoles)
|
||||
After=docker.service
|
||||
Wants=docker.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/itpp-docker-user.sh
|
||||
RemainAfterExit=yes
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
Reference in New Issue
Block a user