itpp-infrastructure/audit/phase-two/c4-credential-rotation-plan.md

C4 Credential Rotation Plan (Phase Two, WS1)

Status: APPROVED (D1). Phase 0 complete, Phase 1 in progress. Date: 2026-08-13 Owner: Sho'Nuff

0. What this plan covers (and what it does not)

• The C4 permission subset is DONE. Logged as P2-C4-001 (app1-bu chmod 600 on 11 secret files + umask 077 root-cause fix). This plan does not re-open it.
• This plan covers the second half of C4: inventory every plaintext credential, rotate them, and move them to Vaultwarden.
• Source-hygiene follow-ups are included as Phase 6 (they are rotation-adjacent, not permission work).

1. The two decisions this plan needs from you (up front)

#	Decision	What it unlocks
D1	APPROVED 2026-08-13 ("A - Rotate, B - Rotate, C - Revoke"): rotate live AI keys (A+B), revoke dead (C)	Phase 2 (in progress)
D2	Go/no-go: rotate the MSP-backdoor shared credential, plus approve the per-portal order	Phase 3

Everything else in this plan runs under those two gates. Phases 4-6 are lower-risk hygiene and I will flag them individually before executing.

2. Evidence base (phase-one report, section 3.1 C4)

1. Two unsanitized copies of key-inventory.md on Core (one inside Hermes's reference dir).
2. Private scripts repo hardcodes the MSP-backdoor admin password reused across client onboardings (Git-A Finding 1). Single most consequential live credential.
3. app1-bu /root/.hermes/.env (mode 644) holds ~20 plaintext secrets: root passwords for app1/app2/app3, Telegram bot token, Cloudflare/Netcup/SyncroMSP tokens, and eight AI-provider keys.
4. rally.service, seemytrip.service, giftaroast.service hardcode JWT_SECRET, DEEPSEEK_API_KEY, ADMIN_AI_KEY, and Twilio credentials in world-readable unit files.
5. app3 MySQL root password in /opt/backup-restore/snapshot.sh (mode 775) and /root/backup.sh.
6. Public repo itpp-infrastructure re-leaks live admin passwords in a prior audit report (section 7, deferred per Germaine).

Recommended order from the report: app1-bu .env first, then key-inventory.md, then systemd units, then app3 snapshot.sh.

3. Safety rules (non-negotiable, apply to every phase)

1. Fallback-first. Before any LiteLLM-adjacent or credential-affecting change, Hermes stays pinned to direct DeepSeek. This is already in force since C1 (P2-C1-001); re-confirm at start of each phase.
2. Vaultwarden-before-changes. Retrieve a credential from Vaultwarden before touching the live system, never from a plaintext file, shell history, or a prior report.
3. No autonomous key rotation. Every rotation happens only after the go/no-go above, in the order listed.
4. Verify before done. Every rotation is followed by a live check (API call, login, health endpoint). No "should work" statements.
5. Log + rollback. Each change is written to change-log.md at the moment it is made, with the rollback path stated before the change.
6. Zero artifact style. No em dashes or en dashes in any plan or change-log entry. All credential values appear as [REDACTED].

4. Phase 0: Inventory (read-only, no changes)

Goal: a single authoritative list of every live plaintext credential, cross-referenced against Vaultwarden.

Steps:

1. Re-run the secret-sprawl scan on the three hosts with the raw hit counts (6,296 Core / 5,504 app1-bu / 1,275 app2) and triage down to live, consequential secrets only.
2. Categorize every secret:
   • (A) AI provider keys
   • (B) MSP-backdoor shared credential
   • (C) server root passwords (app1/app2/app3)
   • (D) service tokens (Telegram, Cloudflare, Netcup, SyncroMSP, Twilio)
   • (E) JWT secrets and DB passwords (systemd units, app3 snapshot/backup scripts)
   • (F) source-repo leaks (key-inventory.md copies, scripts repo, public repo section 7)
3. Cross-reference each item against Vaultwarden: mark already vaulted, missing, or stale.
4. Deliverable: an inventory table with a rotation priority, a current location, and a target Vaultwarden item for each credential.

Effort note: this is where the L-XL sizing sits. The raw scan counts are noise; the consequential list is far smaller.

4a. Phase 0 result (2026-08-13)

• Inventory complete. The report's "eight keys" was an undercount: the live AI surface is ~19 upstream keys across three layers (config.yaml literals, .env plaintext, LiteLLM encrypted credentials), many duplicated.
• GEMINI_API_KEY and GOOGLE_AI_STUDIO_KEY hold the same value under two names (one rotation, two lines to update).
• Verdict: A (fallback chain, 7) = Rotate. B (operational LLM, 7) = Rotate. C (dead, 4) = Revoke.
• Correction: PARALLEL_API_KEY is not dead. It is a live Super Search provider (Parallel.ai Search, fallback #11 in server.py), reclassified as a Phase 3 service token (Keep).
• Vault gaps: DeepSeek, OpenAI, Google/Gemini, Groq, MiniMax have no Vaultwarden item; Cohere/Fireworks/Perplexity/Mistral have portal logins only.

5. Phase 1: AI provider keys (needs D1)

Sequence (fallback-first, rotate non-active providers before the active one):

1. Confirm Hermes is pinned to direct DeepSeek and the direct endpoint is healthy (/v1/models + a test completion).
2. From Phase 0 inventory, confirm the exact list of eight AI providers and which LiteLLM / env locations hold each key.
3. Rotate the keys for all providers EXCEPT the one Hermes is actively pinned to (DeepSeek), one at a time:
   • Generate a new key at the provider portal.
   • Store the new key in Vaultwarden (target item from Phase 0).
   • Update LiteLLM config and/or env to reference the vault-backed value.
   • Verify the provider still resolves through the fallback chain (test call).
   • Write the change to change-log.md with rollback.
4. Rotate the pinned provider (DeepSeek) LAST, immediately updating Core .env and LiteLLM before any subsequent API call can fail.
5. Re-verify the full fallback chain end to end and switch Hermes back to its normal provider only after all eight keys are confirmed live.

Rollback: each step keeps the prior key in the change-log rollback note until the next step verifies green.

6. Phase 2: MSP-backdoor shared credential (needs D2)

Sequence (this is the highest-consequence single credential):

1. Identify the exact shared admin credential and every client system / vendor portal currently using it (inventory from scripts repo + onboarding records).
2. Produce a per-portal rotation order, ordered by blast radius (most-impacted or most-exposed first), and get your sign-off on the order.
3. Vaultwarden-before-changes: confirm the replacement credential is generated and vaulted before touching any live system.
4. Rotate one portal at a time: update the portal, verify login with the new credential, update any dependent automation/env, then move to the next.
5. Remove the hardcoded password from the private scripts repo and replace it with a Vaultwarden lookup or a [REDACTED] placeholder.
6. Log every portal change in change-log.md with rollback.

7. Phase 3: server root passwords and service tokens (lower risk)

• Rotate app1/app2/app3 root passwords; store in Vaultwarden; update the one place that currently references them in plaintext (app1-bu .env).
• Rotate service tokens: Telegram bot token, Cloudflare, Netcup, SyncroMSP, Twilio. Each token rotation re-issues at the vendor console, then updates the consuming service and any env/unit file.
• Verify each service still functions after its token rotates (send a test, poll an endpoint, etc.).

8. Phase 4: JWT secrets and DB passwords

• Move JWT_SECRET, DEEPSEEK_API_KEY, ADMIN_AI_KEY, and Twilio creds out of rally.service, seemytrip.service, giftaroast.service into vault-backed env or a chmod 600 env file, then reload the units.
• Rotate the app3 MySQL root password in /opt/backup-restore/snapshot.sh and /root/backup.sh; store in Vaultwarden; update the scripts to source it from a 600-mode file.
• Verify each service/backup still runs after the change.

9. Phase 5: source hygiene (flagged separately)

• Redact or delete the two key-inventory.md copies on Core.
• Confirm the scripts repo MSP password removal from Phase 2 is committed and pushed.
• Public repo itpp-infrastructure re-leak (report section 7): deferred per your earlier instruction; restating it here so it is not silently dropped. Confirm whether to keep it deferred or fold it into this plan.

10. Verification gates (plan-wide)

• Phase 0 inventory complete and cross-referenced against Vaultwarden
• Fallback chain verified end to end after all AI key rotations
• Every portal login verified after MSP-backdoor rotation
• Every service/unit verified functional after its token/JWT/DB rotation
• change-log.md has a dated entry with rollback for every single change
• No plaintext credential remains in any world-readable location (re-run the scan)

11. Approval needed

• D1: AI provider key rotation go/no-go (unlocks Phase 1)
• D2: MSP-backdoor rotation go/no-go + per-portal order (unlocks Phase 2)
• Phases 3-5: flag individually before execution (no blanket approval implied)