itpp-infrastructure/audit/phase-one/policy-and-procedure.md

ITPP Infrastructure - Policy & Procedure Document

Version: 1.0 (Phase One deliverable) Date: 2026-08-13 Owner: Germaine Brown (final authority on all exemptions) Enforcement: Sho'Nuff (Hermes) as policy-adherence gate; see Skill Spec for the enforcement mechanism.

---

1. Purpose

This document defines the operating rules for IT Pro Partner (ITPP) infrastructure. It exists to close the gap between what the documentation claims and what the estate actually does - the single most consistent theme of the Phase One audit. Every rule below maps to a finding that was either observed in the wild or missing in practice.

---

2. Change Management

Policy: No live configuration change to production infrastructure without a traceable record of (a) what changed, (b) who authorized it, (c) when, and (d) how to roll back.

Procedure:

1. Before any change to Core/app1/app2/app3/app1-bu/wphost02, the credential is retrieved from Vaultwarden (never from a local plaintext file, shell history, or a prior report).
2. Change is recorded in the changelog at the moment it is made - old name → new name, date, reason - not discovered later. (Applies to renames, IP changes, credential rotations, and config moves.)
3. Any change that affects a dependency must verify the dependent records (A, CNAME, env files, backup targets, watchdog targets) before being declared complete.
4. Rollback path is stated in the change record before the change is applied.

Mapping: Audit C1/C4/C9 - the Docker/UFW bypass, plaintext-credential sprawl, and the stale standby all trace to undocumented or unverified changes.

---

3. Documentation Synchronization (Docs-Sync)

Policy: Documentation describes what is actually running, verified against live systems, not memory. No document may claim a control that does not exist.

Procedure:

1. Any infra change must include a corresponding update to docs.itpropartner.com (or the canonical docs repo) in the same work session.
2. Say-do verification: claims like "credentials sanitized", "validated routing", or "whole-site auth gate" must be backed by a check against the live system at the time the claim is written.
3. Stale sections are archived, not silently retained: any section with no update in 30 days is moved to an archive/ tree or explicitly marked stale.
4. The docs build must have a single authoritative source. Duplicate builds (e.g. the app3 split-brain /home/ippadmin/htdocs/ copy) are deleted, and the served copy is wired to a publish CI job.
5. Credential-sensitive material is never published to an unauthenticated path, even in changelogs or historical reports.

Mapping: Docs-W say-do gaps (unsanitized key inventory, unverified auth gate, public credential in changelog), Sys-C backup-plan undercount (27 vs 34 targets), split-brain docs build.

---

4. Backup & Disaster Recovery Standards

Policy: "Backed up" means restore-tested. A backup that has never been restored is an unproven assumption, not a backup. Every service is in the backup matrix.

Procedure:

1. Coverage: every live service has a scheduled backup with a documented destination, retention, and RTO/RPO. The master backup-plan.md must match live discovery (currently 27 documented vs 34 live - reconcile to 34).
2. Restore testing: any new backup target is restore-tested within 7 days of being added. The full set is re-tested on a rolling cadence (at minimum, one restore test per critical service per quarter). Restore tests are logged in restore-test-log.md with evidence (object count, table count, sample-record verification).
3. Database correctness: pg_dump/mysqldump targets are verified against the live database name from config at schedule time, and a failed dump is treated as a failure (alerts, does not fail silently).
4. 3-2-1: critical data exists in 3 copies, on 2 media, 1 off-site. Wasabi S3 is the off-site target; host-local-only backups (e.g. wphost02) are a violation, not an acceptable state.
5. Provider diversity: at least one backup device remains off-premise and on a different provider than the live estate (netcup vs Hetzner). A netcup outage must not take down both the live estate and its standby.
6. Standby readiness: the standby must be data-ready to the documented RPO. If state databases are intentionally excluded from sync, the RPO is documented as "config-only" and the DR plan reflects it - the contradiction between a "warm standby" claim and a 4-week-stale state DB must not persist.

Mapping: C5 (LiteLLM DB never backed up), C8 (wphost02 75% unprotected), C9 (standby not data-ready), Sys-C (2/34 restore-tested), D2 (untested Hudu/UNMS/UniFi).

---

5. Access & Offboarding

Policy: Least privilege. No shared omnipotent credentials. Every person (or agent) has a named, revocable identity.

Procedure:

1. No shared root keys. The single itpp-infra key is split into per-host or per-role keys; each has a named owner and is revocable independently.
2. No blanket NOPASSWD:ALL. Privileged escalation is via named sudoers entries scoped to the commands required; the ippadmin NOPASSWD:ALL and clpctlWrapper ALL rules are retired.
3. MFA is mandatory on every admin console that supports it (Gitea, CloudPanel, Vaultwarden, Grafana, Wazuh, Hudu, LiteLLM). Open registration is closed; captcha is enabled where guest signup exists.
4. Credentials live in Vaultwarden. Plaintext copies in .env, systemd units, scripts, or the filesystem are rotated and removed. Vaultwarden is the single source of truth.
5. Offboarding is immediate on separation: revoke keys, deactivate accounts, rotate any secret the person had access to, and verify no active sessions remain. The shared-key era's inability to do this cleanly is the reason for rules 1-2.
6. Audit trail: every admin action on a production host is attributable. Onboarding/offboarding events are logged with a timestamp and actor.

Mapping: Sec-A (single key, shared admin accounts, NOPASSWD:ALL), C7 (Grafana default creds), Sec-B (no MFA anywhere, open Gitea registration).

---

6. Segmentation for New Entities

Policy: New hosts, sites, or products are placed in a named trust tier at creation time, never added to the flat "everything everywhere" group.

Procedure:

1. Three tiers: internal (ops tooling), client (client sites), product (micro-SaaS). A new entity is assigned to exactly one tier on day one.
2. Tailscale ACLs enforce the tier boundaries - tags are applied before the host is reachable, and the default allow-all is removed.
3. Network exposure: any Docker service binds to 127.0.0.1 and is reached via the reverse proxy; no new service publishes directly to 0.0.0.0. A DOCKER-USER UFW chain is the standing gate for any exception.
4. Database isolation: new sites/products get their own database user and schema, never shared credentials on a shared engine. Products get their own DB server where isolation is a product requirement.
5. DNS: every new host has A/CNAME/SPF/DMARC records verified at creation; no decommissioned IP is left pointed at in DNS.

Mapping: C1 (Docker bypass), C2 (no segmentation), C6 (shared Percona), NetEng-A DNS hygiene findings.

---

7. Recurring Audit Cadence

Policy: The Phase One audit is a baseline, not a one-off. It repeats on a fixed cadence with a fixed scope.

Procedure:

1. Quarterly - the full read-only audit re-runs (the Phase One subagent roster and methodology are reused; see Skill Spec / audit brief).
2. Monthly - a lighter sweep: secrets-in-plaintext grep, open-port diff, backup freshness check, restore-test log review, patch-lag check.
3. Ad hoc - on any security advisory affecting a deployed component (old-image findings like rabbitmq/browserless/grafana), an immediate targeted audit of that component runs.
4. Findings lifecycle: every finding is tracked in the DR issue log (/root/.hermes/references/dr-issue-log.md) with root cause, fix, and verification date. Findings do not silently age out - they are resolved or explicitly accepted by Germaine.
5. Independence: severity ratings are re-checked by an independent model instance before a report is finalized (the Batch 3 "Indep" step), to catch false positives and overstated severities.

Mapping: Sec-B patch-lag findings, old-image findings, D1/D2 (which the independence check is designed to catch).

---

8. Exemptions

Policy: Only Germaine authorizes a departure from this document. No exemption is assumed - it is requested, justified, approved, and recorded.

Procedure:

1. Any request to depart from a policy is raised as an exemption request with a business justification.
2. Germaine approves or denies. Approval is recorded in the Running Exemptions Document with: date, requester, the provision being departed from, the request + business justification, Germaine's authorization, and whether it is one-time or ongoing.
3. Ongoing exemptions carry a follow-up review date. One-time exemptions are closed when the exception ends.
4. The Phase One example: the public-repo itpp-infrastructure credential exposure was surfaced as a Critical finding; Germaine deferred action ("leave the repo alone for now"). That deferral is recorded as an open one-time exemption with a Phase Two follow-up, not silently dropped.

Mapping: D3 (public-repo deferral) - this is the template entry for the Running Exemptions Document.

---

End of Policy & Procedure Document v1.0.