IT PRO PARTNER — DOCUMENT 2 OF 4 Backup & Security Hardening Standard Immutability, credential separation, and a second backup destination Field Value Document status ACTIVE Date July 9, 2026 Applies to All servers listed in the Infrastructure Inventory Owner Network Services Team 1. Why This Exists The original backup standard put system config, Docker volumes, database dumps, and Hermes state all in one Wasabi S3-compatible bucket, written using the same credentials the production servers use for everything else. That means a compromised or misbehaving production server could delete or overwrite its own backups — and it means a single provider/account outage takes down backup storage, restore source, and warm-standby sync simultaneously. This document closes both gaps. 2. Backup Immutability — Required, Not Planned 2.1 Object Lock Enable Object Lock (compliance mode) on the primary Wasabi bucket for all daily and 15-minute snapshot objects. Once written, an object cannot be deleted or overwritten until its retention period expires — including by an account holder with full credentials. This is the single highest-value control against ransomware and accidental deletion. Data Class Object Lock Retention Hermes state (15-min snapshots) 72 hours Daily system/config/DB/app backups 90 days Weekly rollups (new — see Section 5) 6 months Monthly rollups (new — see Section 5) 24 months 2.2 Bucket versioning Enable versioning in addition to Object Lock, so an overwrite creates a new version rather than destroying the prior one, even for objects outside the Object Lock retention window. 2.3 Credential separation — write-only vs. restore/admin Production servers must only ever hold write-only credentials. They can create new backup objects; they cannot list, read, delete, or modify existing ones. A separate, tightly held restore/admin credential — used only by a human or the DR automation during an actual restore — is the only credential with read/delete rights. { "Effect": "Allow", "Action": ["s3:PutObject"], "Resource": "arn:aws:s3:::hermes-vps-backups//*" } // Explicitly no s3:GetObject, s3:DeleteObject, s3:ListBucket on the production-server credential Production server IAM/API keys: PutObject only, scoped to that server's own prefix. Restore/admin credential: full read + write + delete, held outside production servers (password manager / vault), MFA-gated where the provider supports it. No server should hold a credential capable of deleting its own historical backups. 2.4 Access logging and alerting Enable bucket access logging on the Wasabi bucket. Alert on any DeleteObject, mass overwrite, or use of the restore/admin credential outside a declared incident window. Alert if a production server's write-only credential is used for anything other than PutObject. 3. Second Backup Destination — Required Wasabi remains the primary backup and restore path because it is already integrated. A second, independently-owned destination is added purely as insurance against a Wasabi account, bucket, or provider-level failure — it is not a replacement for the primary path. Layer Provider Purpose Sync Method Primary Wasabi (existing bucket) Normal daily backup + restore path Direct write from each server (current process) Secondary Independent provider/account (e.g. Backblaze B2, or a second Wasabi account in a different region) Protects against primary provider/account failure or compromise One-way replication job, primary → secondary, nightly Local standby cache app1-bu disk Fast failover without depending on any S3 endpoint at the moment of failover Continuous sync of Core/Hermes state only Why a separate account, not just a separate bucket: If the Wasabi account itself is compromised or suspended, a second bucket in the same account offers no protection. The secondary destination must sit under different credentials and ideally a different provider entirely. Replication approach: a scheduled job (e.g., rclone) reads from the primary bucket using the restore/admin credential and writes to the secondary destination. Production servers never write directly to the secondary — this keeps the write-only credential model intact and avoids doubling the attack surface. 4. Local Standby Cache (app1-bu) The warm standby must not depend on reaching S3 at the exact moment of failover. app1-bu keeps a locally cached copy of the most recent Hermes state, refreshed continuously (targeting the same 15-minute RPO as the S3 snapshots), so it can start service even if the S3 endpoint is temporarily unreachable during the outage that triggered failover in the first place. 5. Retention Model Flat 90-day daily / 48-hour snapshot retention is fine for routine recovery but offers no protection against slow-developing corruption discovered after 90 days. Add tiered rollups: Backup Type Retention 15-minute Hermes snapshots 72 hours Daily backups 90 days Weekly rollups 6–12 months Monthly rollups 12–36 months Yearly archive Optional — compliance-dependent 6. Per-Application Backup Inventory — Verification Required The backup standard states Docker volumes and database dumps are covered daily. Confirm this is true for every host, not assumed. Track it in a living table: Host Docker Volumes Enumerated? Included in Backup Job? DB Dump Scheduled? Last Verified Core ☐ ☐ ☐ app1 ☐ ☐ ☐ app2 ☐ ☐ ☐ app3 ☐ ☐ ☐ app1-bu ☐ ☐ N/A wphost02 ☐ ☐ ☐ (MariaDB) fleettracker360 ☐ ☐ ☐ (MariaDB) Known gap to close first: wphost02 and fleettracker360 both run MariaDB — confirm scheduled dumps exist, upload to S3, are non-empty, and are restorable. Add an alert: no successful DB backup for a service in more than 26 hours = critical. 7. Monitoring & Alerting Requirements Backup job success/failure — alert same day on failure. Backup age / RPO compliance — alert if the newest backup for any service exceeds its RPO target. Standby sync freshness — alert if app1-bu's local cache falls more than 15 minutes stale. Standby cannot reach S3 — alert immediately, since this affects both backup and failover readiness. Disk usage on every host — warning at 80%, critical at 90%, emergency at 95%. Mass delete / unusual access on the backup bucket. Next: see Per-Server Runbooks for the exact restore commands per host, and the DR Testing & Validation Schedule to verify these controls are actually working.