root
f5175f1ce0
- audit/phase-one + phase-two: security audit briefs, findings, credential-rotation plan, Docker-USER hardening scripts, rollback refs - disaster-recovery/restore-test-log.md + backup-dr-audit-2026-08-10.md - clients/ (modelortho SEO audit, ai-biz-dev competitive landscape), notes/ (tiktok strategy) - projects/: front-desk-voice-agent, seo-visibility-checker product plan, hotnow-savannah HTML, resend-transactional-email, backup-dashboard-enhancements, code-review-graph, seo-ci-architecture - proposals/verdicttank/: architecture v4.0, methodology, judge-pool review, consolidation reasoning, cross-check review - docs/super-search/firecrawl-provider-strategy.md - updates: CHANGELOG, model-chain, projects-master-readme, intelsight.io - .gitignore: exclude nested standalone repos (seo-tool, venturebuilt)
128 lines
8.9 KiB
Markdown
128 lines
8.9 KiB
Markdown
# C4 Credential Rotation Plan (Phase Two, WS1)
|
|
|
|
Status: APPROVED (D1). Phase 0 complete, Phase 1 in progress.
|
|
Date: 2026-08-13
|
|
Owner: Sho'Nuff
|
|
|
|
## 0. What this plan covers (and what it does not)
|
|
|
|
- The C4 **permission subset** is DONE. Logged as `P2-C4-001` (app1-bu chmod 600 on 11 secret files + umask 077 root-cause fix). This plan does not re-open it.
|
|
- This plan covers the **second half of C4**: inventory every plaintext credential, rotate them, and move them to Vaultwarden.
|
|
- Source-hygiene follow-ups are included as Phase 6 (they are rotation-adjacent, not permission work).
|
|
|
|
## 1. The two decisions this plan needs from you (up front)
|
|
|
|
| # | Decision | What it unlocks |
|
|
|---|---|---|
|
|
| D1 | **APPROVED 2026-08-13** ("A - Rotate, B - Rotate, C - Revoke"): rotate live AI keys (A+B), revoke dead (C) | Phase 2 (in progress) |
|
|
| D2 | **Go/no-go: rotate the MSP-backdoor shared credential, plus approve the per-portal order** | Phase 3 |
|
|
|
|
Everything else in this plan runs under those two gates. Phases 4-6 are lower-risk hygiene and I will flag them individually before executing.
|
|
|
|
## 2. Evidence base (phase-one report, section 3.1 C4)
|
|
|
|
1. Two unsanitized copies of `key-inventory.md` on Core (one inside Hermes's reference dir).
|
|
2. Private `scripts` repo hardcodes the MSP-backdoor admin password reused across client onboardings (Git-A Finding 1). Single most consequential live credential.
|
|
3. app1-bu `/root/.hermes/.env` (mode 644) holds ~20 plaintext secrets: root passwords for app1/app2/app3, Telegram bot token, Cloudflare/Netcup/SyncroMSP tokens, and eight AI-provider keys.
|
|
4. `rally.service`, `seemytrip.service`, `giftaroast.service` hardcode `JWT_SECRET`, `DEEPSEEK_API_KEY`, `ADMIN_AI_KEY`, and Twilio credentials in world-readable unit files.
|
|
5. app3 MySQL root password in `/opt/backup-restore/snapshot.sh` (mode 775) and `/root/backup.sh`.
|
|
6. Public repo `itpp-infrastructure` re-leaks live admin passwords in a prior audit report (section 7, deferred per Germaine).
|
|
|
|
Recommended order from the report: app1-bu .env first, then key-inventory.md, then systemd units, then app3 snapshot.sh.
|
|
|
|
## 3. Safety rules (non-negotiable, apply to every phase)
|
|
|
|
1. **Fallback-first.** Before any LiteLLM-adjacent or credential-affecting change, Hermes stays pinned to direct DeepSeek. This is already in force since C1 (`P2-C1-001`); re-confirm at start of each phase.
|
|
2. **Vaultwarden-before-changes.** Retrieve a credential from Vaultwarden before touching the live system, never from a plaintext file, shell history, or a prior report.
|
|
3. **No autonomous key rotation.** Every rotation happens only after the go/no-go above, in the order listed.
|
|
4. **Verify before done.** Every rotation is followed by a live check (API call, login, health endpoint). No "should work" statements.
|
|
5. **Log + rollback.** Each change is written to `change-log.md` at the moment it is made, with the rollback path stated before the change.
|
|
6. **Zero artifact style.** No em dashes or en dashes in any plan or change-log entry. All credential values appear as `[REDACTED]`.
|
|
|
|
## 4. Phase 0: Inventory (read-only, no changes)
|
|
|
|
Goal: a single authoritative list of every live plaintext credential, cross-referenced against Vaultwarden.
|
|
|
|
Steps:
|
|
1. Re-run the secret-sprawl scan on the three hosts with the raw hit counts (6,296 Core / 5,504 app1-bu / 1,275 app2) and triage down to live, consequential secrets only.
|
|
2. Categorize every secret:
|
|
- (A) AI provider keys
|
|
- (B) MSP-backdoor shared credential
|
|
- (C) server root passwords (app1/app2/app3)
|
|
- (D) service tokens (Telegram, Cloudflare, Netcup, SyncroMSP, Twilio)
|
|
- (E) JWT secrets and DB passwords (systemd units, app3 snapshot/backup scripts)
|
|
- (F) source-repo leaks (key-inventory.md copies, scripts repo, public repo section 7)
|
|
3. Cross-reference each item against Vaultwarden: mark `already vaulted`, `missing`, or `stale`.
|
|
4. Deliverable: an inventory table with a rotation priority, a current location, and a target Vaultwarden item for each credential.
|
|
|
|
Effort note: this is where the L-XL sizing sits. The raw scan counts are noise; the consequential list is far smaller.
|
|
|
|
## 4a. Phase 0 result (2026-08-13)
|
|
|
|
- Inventory complete. The report's "eight keys" was an undercount: the live AI surface is ~19 upstream keys across three layers (config.yaml literals, .env plaintext, LiteLLM encrypted credentials), many duplicated.
|
|
- GEMINI_API_KEY and GOOGLE_AI_STUDIO_KEY hold the same value under two names (one rotation, two lines to update).
|
|
- Verdict: A (fallback chain, 7) = Rotate. B (operational LLM, 7) = Rotate. C (dead, 4) = Revoke.
|
|
- Correction: PARALLEL_API_KEY is not dead. It is a live Super Search provider (Parallel.ai Search, fallback #11 in server.py), reclassified as a Phase 3 service token (Keep).
|
|
- Vault gaps: DeepSeek, OpenAI, Google/Gemini, Groq, MiniMax have no Vaultwarden item; Cohere/Fireworks/Perplexity/Mistral have portal logins only.
|
|
|
|
## 5. Phase 1: AI provider keys (needs D1)
|
|
|
|
Sequence (fallback-first, rotate non-active providers before the active one):
|
|
|
|
1. Confirm Hermes is pinned to direct DeepSeek and the direct endpoint is healthy (`/v1/models` + a test completion).
|
|
2. From Phase 0 inventory, confirm the exact list of eight AI providers and which LiteLLM / env locations hold each key.
|
|
3. Rotate the keys for all providers EXCEPT the one Hermes is actively pinned to (DeepSeek), one at a time:
|
|
- Generate a new key at the provider portal.
|
|
- Store the new key in Vaultwarden (target item from Phase 0).
|
|
- Update LiteLLM config and/or env to reference the vault-backed value.
|
|
- Verify the provider still resolves through the fallback chain (test call).
|
|
- Write the change to `change-log.md` with rollback.
|
|
4. Rotate the pinned provider (DeepSeek) LAST, immediately updating Core `.env` and LiteLLM before any subsequent API call can fail.
|
|
5. Re-verify the full fallback chain end to end and switch Hermes back to its normal provider only after all eight keys are confirmed live.
|
|
|
|
Rollback: each step keeps the prior key in the change-log rollback note until the next step verifies green.
|
|
|
|
## 6. Phase 2: MSP-backdoor shared credential (needs D2)
|
|
|
|
Sequence (this is the highest-consequence single credential):
|
|
|
|
1. Identify the exact shared admin credential and every client system / vendor portal currently using it (inventory from `scripts` repo + onboarding records).
|
|
2. Produce a per-portal rotation order, ordered by blast radius (most-impacted or most-exposed first), and get your sign-off on the order.
|
|
3. Vaultwarden-before-changes: confirm the replacement credential is generated and vaulted before touching any live system.
|
|
4. Rotate one portal at a time: update the portal, verify login with the new credential, update any dependent automation/env, then move to the next.
|
|
5. Remove the hardcoded password from the private `scripts` repo and replace it with a Vaultwarden lookup or a `[REDACTED]` placeholder.
|
|
6. Log every portal change in `change-log.md` with rollback.
|
|
|
|
## 7. Phase 3: server root passwords and service tokens (lower risk)
|
|
|
|
- Rotate app1/app2/app3 root passwords; store in Vaultwarden; update the one place that currently references them in plaintext (app1-bu `.env`).
|
|
- Rotate service tokens: Telegram bot token, Cloudflare, Netcup, SyncroMSP, Twilio. Each token rotation re-issues at the vendor console, then updates the consuming service and any env/unit file.
|
|
- Verify each service still functions after its token rotates (send a test, poll an endpoint, etc.).
|
|
|
|
## 8. Phase 4: JWT secrets and DB passwords
|
|
|
|
- Move `JWT_SECRET`, `DEEPSEEK_API_KEY`, `ADMIN_AI_KEY`, and Twilio creds out of `rally.service`, `seemytrip.service`, `giftaroast.service` into vault-backed env or a `chmod 600` env file, then reload the units.
|
|
- Rotate the app3 MySQL root password in `/opt/backup-restore/snapshot.sh` and `/root/backup.sh`; store in Vaultwarden; update the scripts to source it from a 600-mode file.
|
|
- Verify each service/backup still runs after the change.
|
|
|
|
## 9. Phase 5: source hygiene (flagged separately)
|
|
|
|
- Redact or delete the two `key-inventory.md` copies on Core.
|
|
- Confirm the `scripts` repo MSP password removal from Phase 2 is committed and pushed.
|
|
- Public repo `itpp-infrastructure` re-leak (report section 7): deferred per your earlier instruction; restating it here so it is not silently dropped. Confirm whether to keep it deferred or fold it into this plan.
|
|
|
|
## 10. Verification gates (plan-wide)
|
|
|
|
- [ ] Phase 0 inventory complete and cross-referenced against Vaultwarden
|
|
- [ ] Fallback chain verified end to end after all AI key rotations
|
|
- [ ] Every portal login verified after MSP-backdoor rotation
|
|
- [ ] Every service/unit verified functional after its token/JWT/DB rotation
|
|
- [ ] `change-log.md` has a dated entry with rollback for every single change
|
|
- [ ] No plaintext credential remains in any world-readable location (re-run the scan)
|
|
|
|
## 11. Approval needed
|
|
|
|
- D1: AI provider key rotation go/no-go (unlocks Phase 1)
|
|
- D2: MSP-backdoor rotation go/no-go + per-portal order (unlocks Phase 2)
|
|
- Phases 3-5: flag individually before execution (no blanket approval implied)
|