Initial resurrection kit — 81 scripts, 62 references, configs, systemd units, crons, Docker compose files, Caddy config, master README

references/2-Backup-Security-Standard.md

@@ -0,0 +1,137 @@
+IT PRO PARTNER — DOCUMENT 2 OF 4
+Backup & Security Hardening Standard
+Immutability, credential separation, and a second backup destination
+Field
+Value
+Document status
+ACTIVE
+Date
+July 9, 2026
+Applies to
+All servers listed in the Infrastructure Inventory
+Owner
+Network Services Team
+
+1. Why This Exists
+The original backup standard put system config, Docker volumes, database dumps, and Hermes state all in one Wasabi S3-compatible bucket, written using the same credentials the production servers use for everything else. That means a compromised or misbehaving production server could delete or overwrite its own backups — and it means a single provider/account outage takes down backup storage, restore source, and warm-standby sync simultaneously. This document closes both gaps.
+2. Backup Immutability — Required, Not Planned
+2.1 Object Lock
+Enable Object Lock (compliance mode) on the primary Wasabi bucket for all daily and 15-minute snapshot objects. Once written, an object cannot be deleted or overwritten until its retention period expires — including by an account holder with full credentials. This is the single highest-value control against ransomware and accidental deletion.
+Data Class
+Object Lock Retention
+Hermes state (15-min snapshots)
+72 hours
+Daily system/config/DB/app backups
+90 days
+Weekly rollups (new — see Section 5)
+6 months
+Monthly rollups (new — see Section 5)
+24 months
+
+2.2 Bucket versioning
+Enable versioning in addition to Object Lock, so an overwrite creates a new version rather than destroying the prior one, even for objects outside the Object Lock retention window.
+2.3 Credential separation — write-only vs. restore/admin
+Production servers must only ever hold write-only credentials. They can create new backup objects; they cannot list, read, delete, or modify existing ones. A separate, tightly held restore/admin credential — used only by a human or the DR automation during an actual restore — is the only credential with read/delete rights.
+{
+  "Effect": "Allow",
+  "Action": ["s3:PutObject"],
+  "Resource": "arn:aws:s3:::hermes-vps-backups/<server>/*"
+}
+// Explicitly no s3:GetObject, s3:DeleteObject, s3:ListBucket on the production-server credential
+Production server IAM/API keys: PutObject only, scoped to that server's own prefix.
+Restore/admin credential: full read + write + delete, held outside production servers (password manager / vault), MFA-gated where the provider supports it.
+No server should hold a credential capable of deleting its own historical backups.
+2.4 Access logging and alerting
+Enable bucket access logging on the Wasabi bucket.
+Alert on any DeleteObject, mass overwrite, or use of the restore/admin credential outside a declared incident window.
+Alert if a production server's write-only credential is used for anything other than PutObject.
+3. Second Backup Destination — Required
+Wasabi remains the primary backup and restore path because it is already integrated. A second, independently-owned destination is added purely as insurance against a Wasabi account, bucket, or provider-level failure — it is not a replacement for the primary path.
+Layer
+Provider
+Purpose
+Sync Method
+Primary
+Wasabi (existing bucket)
+Normal daily backup + restore path
+Direct write from each server (current process)
+Secondary
+Independent provider/account (e.g. Backblaze B2, or a second Wasabi account in a different region)
+Protects against primary provider/account failure or compromise
+One-way replication job, primary → secondary, nightly
+Local standby cache
+app1-bu disk
+Fast failover without depending on any S3 endpoint at the moment of failover
+Continuous sync of Core/Hermes state only
+
+Why a separate account, not just a separate bucket: If the Wasabi account itself is compromised or suspended, a second bucket in the same account offers no protection. The secondary destination must sit under different credentials and ideally a different provider entirely.
+Replication approach: a scheduled job (e.g., rclone) reads from the primary bucket using the restore/admin credential and writes to the secondary destination. Production servers never write directly to the secondary — this keeps the write-only credential model intact and avoids doubling the attack surface.
+4. Local Standby Cache (app1-bu)
+The warm standby must not depend on reaching S3 at the exact moment of failover. app1-bu keeps a locally cached copy of the most recent Hermes state, refreshed continuously (targeting the same 15-minute RPO as the S3 snapshots), so it can start service even if the S3 endpoint is temporarily unreachable during the outage that triggered failover in the first place.
+5. Retention Model
+Flat 90-day daily / 48-hour snapshot retention is fine for routine recovery but offers no protection against slow-developing corruption discovered after 90 days. Add tiered rollups:
+Backup Type
+Retention
+15-minute Hermes snapshots
+72 hours
+Daily backups
+90 days
+Weekly rollups
+6–12 months
+Monthly rollups
+12–36 months
+Yearly archive
+Optional — compliance-dependent
+6. Per-Application Backup Inventory — Verification Required
+The backup standard states Docker volumes and database dumps are covered daily. Confirm this is true for every host, not assumed. Track it in a living table:
+Host
+Docker Volumes Enumerated?
+Included in Backup Job?
+DB Dump Scheduled?
+Last Verified
+Core
+☐
+☐
+☐
+
+app1
+☐
+☐
+☐
+
+app2
+☐
+☐
+☐
+
+app3
+☐
+☐
+☐
+
+app1-bu
+☐
+☐
+N/A
+
+wphost02
+☐
+☐
+☐ (MariaDB)
+
+fleettracker360
+☐
+☐
+☐ (MariaDB)
+
+
+Known gap to close first: wphost02 and fleettracker360 both run MariaDB — confirm scheduled dumps exist, upload to S3, are non-empty, and are restorable. Add an alert: no successful DB backup for a service in more than 26 hours = critical.
+7. Monitoring & Alerting Requirements
+Backup job success/failure — alert same day on failure.
+Backup age / RPO compliance — alert if the newest backup for any service exceeds its RPO target.
+Standby sync freshness — alert if app1-bu's local cache falls more than 15 minutes stale.
+Standby cannot reach S3 — alert immediately, since this affects both backup and failover readiness.
+Disk usage on every host — warning at 80%, critical at 90%, emergency at 95%.
+Mass delete / unusual access on the backup bucket.
+
+Next: see Per-Server Runbooks for the exact restore commands per host, and the DR Testing & Validation Schedule to verify these controls are actually working.