root
f5175f1ce0
- audit/phase-one + phase-two: security audit briefs, findings, credential-rotation plan, Docker-USER hardening scripts, rollback refs - disaster-recovery/restore-test-log.md + backup-dr-audit-2026-08-10.md - clients/ (modelortho SEO audit, ai-biz-dev competitive landscape), notes/ (tiktok strategy) - projects/: front-desk-voice-agent, seo-visibility-checker product plan, hotnow-savannah HTML, resend-transactional-email, backup-dashboard-enhancements, code-review-graph, seo-ci-architecture - proposals/verdicttank/: architecture v4.0, methodology, judge-pool review, consolidation reasoning, cross-check review - docs/super-search/firecrawl-provider-strategy.md - updates: CHANGELOG, model-chain, projects-master-readme, intelsight.io - .gitignore: exclude nested standalone repos (seo-tool, venturebuilt)
122 lines
9.4 KiB
Markdown
122 lines
9.4 KiB
Markdown
# ITPP Infrastructure - Policy & Procedure Document
|
|
|
|
**Version:** 1.0 (Phase One deliverable)
|
|
**Date:** 2026-08-13
|
|
**Owner:** Germaine Brown (final authority on all exemptions)
|
|
**Enforcement:** Sho'Nuff (Hermes) as policy-adherence gate; see Skill Spec for the enforcement mechanism.
|
|
|
|
---
|
|
|
|
## 1. Purpose
|
|
|
|
This document defines the operating rules for IT Pro Partner (ITPP) infrastructure. It exists to close the gap between what the documentation claims and what the estate actually does - the single most consistent theme of the Phase One audit. Every rule below maps to a finding that was either observed in the wild or missing in practice.
|
|
|
|
---
|
|
|
|
## 2. Change Management
|
|
|
|
**Policy:** No live configuration change to production infrastructure without a traceable record of (a) what changed, (b) who authorized it, (c) when, and (d) how to roll back.
|
|
|
|
**Procedure:**
|
|
1. Before any change to Core/app1/app2/app3/app1-bu/wphost02, the credential is retrieved from Vaultwarden (never from a local plaintext file, shell history, or a prior report).
|
|
2. Change is recorded in the changelog at the moment it is made - **old name → new name, date, reason** - not discovered later. (Applies to renames, IP changes, credential rotations, and config moves.)
|
|
3. Any change that affects a dependency must verify the dependent records (A, CNAME, env files, backup targets, watchdog targets) before being declared complete.
|
|
4. Rollback path is stated in the change record before the change is applied.
|
|
|
|
**Mapping:** Audit C1/C4/C9 - the Docker/UFW bypass, plaintext-credential sprawl, and the stale standby all trace to undocumented or unverified changes.
|
|
|
|
---
|
|
|
|
## 3. Documentation Synchronization (Docs-Sync)
|
|
|
|
**Policy:** Documentation describes what is *actually running*, verified against live systems, not memory. No document may claim a control that does not exist.
|
|
|
|
**Procedure:**
|
|
1. Any infra change must include a corresponding update to `docs.itpropartner.com` (or the canonical docs repo) in the same work session.
|
|
2. Say-do verification: claims like "credentials sanitized", "validated routing", or "whole-site auth gate" must be backed by a check against the live system at the time the claim is written.
|
|
3. Stale sections are archived, not silently retained: any section with no update in 30 days is moved to an `archive/` tree or explicitly marked stale.
|
|
4. The docs build must have a single authoritative source. Duplicate builds (e.g. the app3 split-brain `/home/ippadmin/htdocs/` copy) are deleted, and the served copy is wired to a publish CI job.
|
|
5. Credential-sensitive material is never published to an unauthenticated path, even in changelogs or historical reports.
|
|
|
|
**Mapping:** Docs-W say-do gaps (unsanitized key inventory, unverified auth gate, public credential in changelog), Sys-C backup-plan undercount (27 vs 34 targets), split-brain docs build.
|
|
|
|
---
|
|
|
|
## 4. Backup & Disaster Recovery Standards
|
|
|
|
**Policy:** "Backed up" means **restore-tested**. A backup that has never been restored is an unproven assumption, not a backup. Every service is in the backup matrix.
|
|
|
|
**Procedure:**
|
|
1. **Coverage:** every live service has a scheduled backup with a documented destination, retention, and RTO/RPO. The master `backup-plan.md` must match live discovery (currently 27 documented vs 34 live - reconcile to 34).
|
|
2. **Restore testing:** any new backup target is restore-tested within 7 days of being added. The full set is re-tested on a rolling cadence (at minimum, one restore test per critical service per quarter). Restore tests are logged in `restore-test-log.md` with evidence (object count, table count, sample-record verification).
|
|
3. **Database correctness:** `pg_dump`/`mysqldump` targets are verified against the live database name from config at schedule time, and a failed dump is treated as a failure (alerts, does not fail silently).
|
|
4. **3-2-1:** critical data exists in 3 copies, on 2 media, 1 off-site. Wasabi S3 is the off-site target; host-local-only backups (e.g. wphost02) are a violation, not an acceptable state.
|
|
5. **Provider diversity:** at least one backup device remains off-premise and on a different provider than the live estate (netcup vs Hetzner). A netcup outage must not take down both the live estate and its standby.
|
|
6. **Standby readiness:** the standby must be data-ready to the documented RPO. If state databases are intentionally excluded from sync, the RPO is documented as "config-only" and the DR plan reflects it - the contradiction between a "warm standby" claim and a 4-week-stale state DB must not persist.
|
|
|
|
**Mapping:** C5 (LiteLLM DB never backed up), C8 (wphost02 75% unprotected), C9 (standby not data-ready), Sys-C (2/34 restore-tested), D2 (untested Hudu/UNMS/UniFi).
|
|
|
|
---
|
|
|
|
## 5. Access & Offboarding
|
|
|
|
**Policy:** Least privilege. No shared omnipotent credentials. Every person (or agent) has a named, revocable identity.
|
|
|
|
**Procedure:**
|
|
1. **No shared root keys.** The single `itpp-infra` key is split into per-host or per-role keys; each has a named owner and is revocable independently.
|
|
2. **No blanket `NOPASSWD:ALL`.** Privileged escalation is via named sudoers entries scoped to the commands required; the `ippadmin NOPASSWD:ALL` and `clpctlWrapper ALL` rules are retired.
|
|
3. **MFA is mandatory** on every admin console that supports it (Gitea, CloudPanel, Vaultwarden, Grafana, Wazuh, Hudu, LiteLLM). Open registration is closed; captcha is enabled where guest signup exists.
|
|
4. **Credentials live in Vaultwarden.** Plaintext copies in `.env`, systemd units, scripts, or the filesystem are rotated and removed. Vaultwarden is the single source of truth.
|
|
5. **Offboarding** is immediate on separation: revoke keys, deactivate accounts, rotate any secret the person had access to, and verify no active sessions remain. The shared-key era's inability to do this cleanly is the reason for rules 1-2.
|
|
6. **Audit trail:** every admin action on a production host is attributable. Onboarding/offboarding events are logged with a timestamp and actor.
|
|
|
|
**Mapping:** Sec-A (single key, shared admin accounts, NOPASSWD:ALL), C7 (Grafana default creds), Sec-B (no MFA anywhere, open Gitea registration).
|
|
|
|
---
|
|
|
|
## 6. Segmentation for New Entities
|
|
|
|
**Policy:** New hosts, sites, or products are placed in a named trust tier at creation time, never added to the flat "everything everywhere" group.
|
|
|
|
**Procedure:**
|
|
1. **Three tiers:** `internal` (ops tooling), `client` (client sites), `product` (micro-SaaS). A new entity is assigned to exactly one tier on day one.
|
|
2. **Tailscale ACLs** enforce the tier boundaries - tags are applied before the host is reachable, and the default allow-all is removed.
|
|
3. **Network exposure:** any Docker service binds to `127.0.0.1` and is reached via the reverse proxy; no new service publishes directly to 0.0.0.0. A `DOCKER-USER` UFW chain is the standing gate for any exception.
|
|
4. **Database isolation:** new sites/products get their own database user and schema, never shared credentials on a shared engine. Products get their own DB server where isolation is a product requirement.
|
|
5. **DNS:** every new host has A/CNAME/SPF/DMARC records verified at creation; no decommissioned IP is left pointed at in DNS.
|
|
|
|
**Mapping:** C1 (Docker bypass), C2 (no segmentation), C6 (shared Percona), NetEng-A DNS hygiene findings.
|
|
|
|
---
|
|
|
|
## 7. Recurring Audit Cadence
|
|
|
|
**Policy:** The Phase One audit is a baseline, not a one-off. It repeats on a fixed cadence with a fixed scope.
|
|
|
|
**Procedure:**
|
|
1. **Quarterly** - the full read-only audit re-runs (the Phase One subagent roster and methodology are reused; see Skill Spec / audit brief).
|
|
2. **Monthly** - a lighter sweep: secrets-in-plaintext grep, open-port diff, backup freshness check, restore-test log review, patch-lag check.
|
|
3. **Ad hoc** - on any security advisory affecting a deployed component (old-image findings like rabbitmq/browserless/grafana), an immediate targeted audit of that component runs.
|
|
4. **Findings lifecycle:** every finding is tracked in the DR issue log (`/root/.hermes/references/dr-issue-log.md`) with root cause, fix, and verification date. Findings do not silently age out - they are resolved or explicitly accepted by Germaine.
|
|
5. **Independence:** severity ratings are re-checked by an independent model instance before a report is finalized (the Batch 3 "Indep" step), to catch false positives and overstated severities.
|
|
|
|
**Mapping:** Sec-B patch-lag findings, old-image findings, D1/D2 (which the independence check is designed to catch).
|
|
|
|
---
|
|
|
|
## 8. Exemptions
|
|
|
|
**Policy:** Only Germaine authorizes a departure from this document. No exemption is assumed - it is requested, justified, approved, and recorded.
|
|
|
|
**Procedure:**
|
|
1. Any request to depart from a policy is raised as an exemption request with a business justification.
|
|
2. Germaine approves or denies. Approval is recorded in the **Running Exemptions Document** with: date, requester, the provision being departed from, the request + business justification, Germaine's authorization, and whether it is one-time or ongoing.
|
|
3. Ongoing exemptions carry a follow-up review date. One-time exemptions are closed when the exception ends.
|
|
4. The Phase One example: the public-repo `itpp-infrastructure` credential exposure was surfaced as a Critical finding; Germaine deferred action ("leave the repo alone for now"). That deferral is recorded as an open one-time exemption with a Phase Two follow-up, not silently dropped.
|
|
|
|
**Mapping:** D3 (public-repo deferral) - this is the template entry for the Running Exemptions Document.
|
|
|
|
---
|
|
|
|
*End of Policy & Procedure Document v1.0.*
|