# C4 Credential Rotation Plan (Phase Two, WS1)

Status: APPROVED (D1). Phase 0 complete, Phase 1 in progress.
Date: 2026-08-13
Owner: Sho'Nuff

## 0. What this plan covers (and what it does not)

- The C4 **permission subset** is DONE. Logged as `P2-C4-001` (app1-bu chmod 600 on 11 secret files + umask 077 root-cause fix). This plan does not re-open it.
- This plan covers the **second half of C4**: inventory every plaintext credential, rotate them, and move them to Vaultwarden.
- Source-hygiene follow-ups are included as Phase 6 (they are rotation-adjacent, not permission work).

## 1. The two decisions this plan needs from you (up front)

| # | Decision | What it unlocks |
|---|---|---|
| D1 | **APPROVED 2026-08-13** ("A - Rotate, B - Rotate, C - Revoke"): rotate live AI keys (A+B), revoke dead (C) | Phase 2 (in progress) |
| D2 | **Go/no-go: rotate the MSP-backdoor shared credential, plus approve the per-portal order** | Phase 3 |

Everything else in this plan runs under those two gates. Phases 4-6 are lower-risk hygiene and I will flag them individually before executing.

## 2. Evidence base (phase-one report, section 3.1 C4)

1. Two unsanitized copies of `key-inventory.md` on Core (one inside Hermes's reference dir).
2. Private `scripts` repo hardcodes the MSP-backdoor admin password reused across client onboardings (Git-A Finding 1). Single most consequential live credential.
3. app1-bu `/root/.hermes/.env` (mode 644) holds ~20 plaintext secrets: root passwords for app1/app2/app3, Telegram bot token, Cloudflare/Netcup/SyncroMSP tokens, and eight AI-provider keys.
4. `rally.service`, `seemytrip.service`, `giftaroast.service` hardcode `JWT_SECRET`, `DEEPSEEK_API_KEY`, `ADMIN_AI_KEY`, and Twilio credentials in world-readable unit files.
5. app3 MySQL root password in `/opt/backup-restore/snapshot.sh` (mode 775) and `/root/backup.sh`.
6. Public repo `itpp-infrastructure` re-leaks live admin passwords in a prior audit report (section 7, deferred per Germaine).

Recommended order from the report: app1-bu .env first, then key-inventory.md, then systemd units, then app3 snapshot.sh.

## 3. Safety rules (non-negotiable, apply to every phase)

1. **Fallback-first.** Before any LiteLLM-adjacent or credential-affecting change, Hermes stays pinned to direct DeepSeek. This is already in force since C1 (`P2-C1-001`); re-confirm at start of each phase.
2. **Vaultwarden-before-changes.** Retrieve a credential from Vaultwarden before touching the live system, never from a plaintext file, shell history, or a prior report.
3. **No autonomous key rotation.** Every rotation happens only after the go/no-go above, in the order listed.
4. **Verify before done.** Every rotation is followed by a live check (API call, login, health endpoint). No "should work" statements.
5. **Log + rollback.** Each change is written to `change-log.md` at the moment it is made, with the rollback path stated before the change.
6. **Zero artifact style.** No em dashes or en dashes in any plan or change-log entry. All credential values appear as `[REDACTED]`.

## 4. Phase 0: Inventory (read-only, no changes)

Goal: a single authoritative list of every live plaintext credential, cross-referenced against Vaultwarden.

Steps:
1. Re-run the secret-sprawl scan on the three hosts with the raw hit counts (6,296 Core / 5,504 app1-bu / 1,275 app2) and triage down to live, consequential secrets only.
2. Categorize every secret:
   - (A) AI provider keys
   - (B) MSP-backdoor shared credential
   - (C) server root passwords (app1/app2/app3)
   - (D) service tokens (Telegram, Cloudflare, Netcup, SyncroMSP, Twilio)
   - (E) JWT secrets and DB passwords (systemd units, app3 snapshot/backup scripts)
   - (F) source-repo leaks (key-inventory.md copies, scripts repo, public repo section 7)
3. Cross-reference each item against Vaultwarden: mark `already vaulted`, `missing`, or `stale`.
4. Deliverable: an inventory table with a rotation priority, a current location, and a target Vaultwarden item for each credential.

Effort note: this is where the L-XL sizing sits. The raw scan counts are noise; the consequential list is far smaller.

## 4a. Phase 0 result (2026-08-13)

- Inventory complete. The report's "eight keys" was an undercount: the live AI surface is ~19 upstream keys across three layers (config.yaml literals, .env plaintext, LiteLLM encrypted credentials), many duplicated.
- GEMINI_API_KEY and GOOGLE_AI_STUDIO_KEY hold the same value under two names (one rotation, two lines to update).
- Verdict: A (fallback chain, 7) = Rotate. B (operational LLM, 7) = Rotate. C (dead, 4) = Revoke.
- Correction: PARALLEL_API_KEY is not dead. It is a live Super Search provider (Parallel.ai Search, fallback #11 in server.py), reclassified as a Phase 3 service token (Keep).
- Vault gaps: DeepSeek, OpenAI, Google/Gemini, Groq, MiniMax have no Vaultwarden item; Cohere/Fireworks/Perplexity/Mistral have portal logins only.

## 5. Phase 1: AI provider keys (needs D1)

Sequence (fallback-first, rotate non-active providers before the active one):

1. Confirm Hermes is pinned to direct DeepSeek and the direct endpoint is healthy (`/v1/models` + a test completion).
2. From Phase 0 inventory, confirm the exact list of eight AI providers and which LiteLLM / env locations hold each key.
3. Rotate the keys for all providers EXCEPT the one Hermes is actively pinned to (DeepSeek), one at a time:
   - Generate a new key at the provider portal.
   - Store the new key in Vaultwarden (target item from Phase 0).
   - Update LiteLLM config and/or env to reference the vault-backed value.
   - Verify the provider still resolves through the fallback chain (test call).
   - Write the change to `change-log.md` with rollback.
4. Rotate the pinned provider (DeepSeek) LAST, immediately updating Core `.env` and LiteLLM before any subsequent API call can fail.
5. Re-verify the full fallback chain end to end and switch Hermes back to its normal provider only after all eight keys are confirmed live.

Rollback: each step keeps the prior key in the change-log rollback note until the next step verifies green.

## 6. Phase 2: MSP-backdoor shared credential (needs D2)

Sequence (this is the highest-consequence single credential):

1. Identify the exact shared admin credential and every client system / vendor portal currently using it (inventory from `scripts` repo + onboarding records).
2. Produce a per-portal rotation order, ordered by blast radius (most-impacted or most-exposed first), and get your sign-off on the order.
3. Vaultwarden-before-changes: confirm the replacement credential is generated and vaulted before touching any live system.
4. Rotate one portal at a time: update the portal, verify login with the new credential, update any dependent automation/env, then move to the next.
5. Remove the hardcoded password from the private `scripts` repo and replace it with a Vaultwarden lookup or a `[REDACTED]` placeholder.
6. Log every portal change in `change-log.md` with rollback.

## 7. Phase 3: server root passwords and service tokens (lower risk)

- Rotate app1/app2/app3 root passwords; store in Vaultwarden; update the one place that currently references them in plaintext (app1-bu `.env`).
- Rotate service tokens: Telegram bot token, Cloudflare, Netcup, SyncroMSP, Twilio. Each token rotation re-issues at the vendor console, then updates the consuming service and any env/unit file.
- Verify each service still functions after its token rotates (send a test, poll an endpoint, etc.).

## 8. Phase 4: JWT secrets and DB passwords

- Move `JWT_SECRET`, `DEEPSEEK_API_KEY`, `ADMIN_AI_KEY`, and Twilio creds out of `rally.service`, `seemytrip.service`, `giftaroast.service` into vault-backed env or a `chmod 600` env file, then reload the units.
- Rotate the app3 MySQL root password in `/opt/backup-restore/snapshot.sh` and `/root/backup.sh`; store in Vaultwarden; update the scripts to source it from a 600-mode file.
- Verify each service/backup still runs after the change.

## 9. Phase 5: source hygiene (flagged separately)

- Redact or delete the two `key-inventory.md` copies on Core.
- Confirm the `scripts` repo MSP password removal from Phase 2 is committed and pushed.
- Public repo `itpp-infrastructure` re-leak (report section 7): deferred per your earlier instruction; restating it here so it is not silently dropped. Confirm whether to keep it deferred or fold it into this plan.

## 10. Verification gates (plan-wide)

- [ ] Phase 0 inventory complete and cross-referenced against Vaultwarden
- [ ] Fallback chain verified end to end after all AI key rotations
- [ ] Every portal login verified after MSP-backdoor rotation
- [ ] Every service/unit verified functional after its token/JWT/DB rotation
- [ ] `change-log.md` has a dated entry with rollback for every single change
- [ ] No plaintext credential remains in any world-readable location (re-run the scan)

## 11. Approval needed

- D1: AI provider key rotation go/no-go (unlocks Phase 1)
- D2: MSP-backdoor rotation go/no-go + per-portal order (unlocks Phase 2)
- Phases 3-5: flag individually before execution (no blanket approval implied)