137 lines
6.3 KiB
Markdown
137 lines
6.3 KiB
Markdown
IT PRO PARTNER — DOCUMENT 2 OF 4
|
||
Backup & Security Hardening Standard
|
||
Immutability, credential separation, and a second backup destination
|
||
Field
|
||
Value
|
||
Document status
|
||
ACTIVE
|
||
Date
|
||
July 9, 2026
|
||
Applies to
|
||
All servers listed in the Infrastructure Inventory
|
||
Owner
|
||
Network Services Team
|
||
|
||
1. Why This Exists
|
||
The original backup standard put system config, Docker volumes, database dumps, and Hermes state all in one Wasabi S3-compatible bucket, written using the same credentials the production servers use for everything else. That means a compromised or misbehaving production server could delete or overwrite its own backups — and it means a single provider/account outage takes down backup storage, restore source, and warm-standby sync simultaneously. This document closes both gaps.
|
||
2. Backup Immutability — Required, Not Planned
|
||
2.1 Object Lock
|
||
Enable Object Lock (compliance mode) on the primary Wasabi bucket for all daily and 15-minute snapshot objects. Once written, an object cannot be deleted or overwritten until its retention period expires — including by an account holder with full credentials. This is the single highest-value control against ransomware and accidental deletion.
|
||
Data Class
|
||
Object Lock Retention
|
||
Hermes state (15-min snapshots)
|
||
72 hours
|
||
Daily system/config/DB/app backups
|
||
90 days
|
||
Weekly rollups (new — see Section 5)
|
||
6 months
|
||
Monthly rollups (new — see Section 5)
|
||
24 months
|
||
|
||
2.2 Bucket versioning
|
||
Enable versioning in addition to Object Lock, so an overwrite creates a new version rather than destroying the prior one, even for objects outside the Object Lock retention window.
|
||
2.3 Credential separation — write-only vs. restore/admin
|
||
Production servers must only ever hold write-only credentials. They can create new backup objects; they cannot list, read, delete, or modify existing ones. A separate, tightly held restore/admin credential — used only by a human or the DR automation during an actual restore — is the only credential with read/delete rights.
|
||
{
|
||
"Effect": "Allow",
|
||
"Action": ["s3:PutObject"],
|
||
"Resource": "arn:aws:s3:::hermes-vps-backups/<server>/*"
|
||
}
|
||
// Explicitly no s3:GetObject, s3:DeleteObject, s3:ListBucket on the production-server credential
|
||
Production server IAM/API keys: PutObject only, scoped to that server's own prefix.
|
||
Restore/admin credential: full read + write + delete, held outside production servers (password manager / vault), MFA-gated where the provider supports it.
|
||
No server should hold a credential capable of deleting its own historical backups.
|
||
2.4 Access logging and alerting
|
||
Enable bucket access logging on the Wasabi bucket.
|
||
Alert on any DeleteObject, mass overwrite, or use of the restore/admin credential outside a declared incident window.
|
||
Alert if a production server's write-only credential is used for anything other than PutObject.
|
||
3. Second Backup Destination — Required
|
||
Wasabi remains the primary backup and restore path because it is already integrated. A second, independently-owned destination is added purely as insurance against a Wasabi account, bucket, or provider-level failure — it is not a replacement for the primary path.
|
||
Layer
|
||
Provider
|
||
Purpose
|
||
Sync Method
|
||
Primary
|
||
Wasabi (existing bucket)
|
||
Normal daily backup + restore path
|
||
Direct write from each server (current process)
|
||
Secondary
|
||
Independent provider/account (e.g. Backblaze B2, or a second Wasabi account in a different region)
|
||
Protects against primary provider/account failure or compromise
|
||
One-way replication job, primary → secondary, nightly
|
||
Local standby cache
|
||
app1-bu disk
|
||
Fast failover without depending on any S3 endpoint at the moment of failover
|
||
Continuous sync of Core/Hermes state only
|
||
|
||
Why a separate account, not just a separate bucket: If the Wasabi account itself is compromised or suspended, a second bucket in the same account offers no protection. The secondary destination must sit under different credentials and ideally a different provider entirely.
|
||
Replication approach: a scheduled job (e.g., rclone) reads from the primary bucket using the restore/admin credential and writes to the secondary destination. Production servers never write directly to the secondary — this keeps the write-only credential model intact and avoids doubling the attack surface.
|
||
4. Local Standby Cache (app1-bu)
|
||
The warm standby must not depend on reaching S3 at the exact moment of failover. app1-bu keeps a locally cached copy of the most recent Hermes state, refreshed continuously (targeting the same 15-minute RPO as the S3 snapshots), so it can start service even if the S3 endpoint is temporarily unreachable during the outage that triggered failover in the first place.
|
||
5. Retention Model
|
||
Flat 90-day daily / 48-hour snapshot retention is fine for routine recovery but offers no protection against slow-developing corruption discovered after 90 days. Add tiered rollups:
|
||
Backup Type
|
||
Retention
|
||
15-minute Hermes snapshots
|
||
72 hours
|
||
Daily backups
|
||
90 days
|
||
Weekly rollups
|
||
6–12 months
|
||
Monthly rollups
|
||
12–36 months
|
||
Yearly archive
|
||
Optional — compliance-dependent
|
||
6. Per-Application Backup Inventory — Verification Required
|
||
The backup standard states Docker volumes and database dumps are covered daily. Confirm this is true for every host, not assumed. Track it in a living table:
|
||
Host
|
||
Docker Volumes Enumerated?
|
||
Included in Backup Job?
|
||
DB Dump Scheduled?
|
||
Last Verified
|
||
Core
|
||
☐
|
||
☐
|
||
☐
|
||
|
||
app1
|
||
☐
|
||
☐
|
||
☐
|
||
|
||
app2
|
||
☐
|
||
☐
|
||
☐
|
||
|
||
app3
|
||
☐
|
||
☐
|
||
☐
|
||
|
||
app1-bu
|
||
☐
|
||
☐
|
||
N/A
|
||
|
||
wphost02
|
||
☐
|
||
☐
|
||
☐ (MariaDB)
|
||
|
||
fleettracker360
|
||
☐
|
||
☐
|
||
☐ (MariaDB)
|
||
|
||
|
||
Known gap to close first: wphost02 and fleettracker360 both run MariaDB — confirm scheduled dumps exist, upload to S3, are non-empty, and are restorable. Add an alert: no successful DB backup for a service in more than 26 hours = critical.
|
||
7. Monitoring & Alerting Requirements
|
||
Backup job success/failure — alert same day on failure.
|
||
Backup age / RPO compliance — alert if the newest backup for any service exceeds its RPO target.
|
||
Standby sync freshness — alert if app1-bu's local cache falls more than 15 minutes stale.
|
||
Standby cannot reach S3 — alert immediately, since this affects both backup and failover readiness.
|
||
Disk usage on every host — warning at 80%, critical at 90%, emergency at 95%.
|
||
Mass delete / unusual access on the backup bucket.
|
||
|
||
Next: see Per-Server Runbooks for the exact restore commands per host, and the DR Testing & Validation Schedule to verify these controls are actually working. |