Two-Bucket DR Backup Strategy

Architecture

Layer 1: Provider-Level (automated snapshots)

Asset	Backup Method	Retention	Coverage
Hetzner Cloud VPS	Hetzner snapshot (API/CLI)	Keep latest + 7-day auto	Full OS recovery
Hetzner dedicated	Robot backup service	7 days	Full system
RunCloud servers	RunCloud built-in backup	Their retention	Web apps, WordPress, databases
CCR routers	RouterOS API export → S3 (daily)	365 days per lifecycle rule	Config + logs only

These protect against OS corruption, accidental config wipe, bad update — NOT data loss.

Layer 2: Application-Level (S3 — two-bucket DR)

Asset	Primary Bucket	DR Bucket	Frequency
Hermes (config, sessions, skills)	`hermes-vps-backups` (us-east-1)	`hermes-vps-backups-dr` (us-west-2)	Daily
Portal (DB, uploads, configs)	`itpropartner-backups` (us-east-1)	`itpropartner-backups-dr` (us-west-2)	Daily
CCR router configs	`mikrotik-ccr-backups` (us-east-1)	`mikrotik-ccr-backups-dr` (us-west-2)	Daily

Layer 3: Hudu (source of truth)

Hudu has its own backup export in Admin settings. Export to Wasabi bucket periodically. Hudu is the canonical record for passwords, configs, network diagrams, and procedures.

Two-Bucket Pattern

Primary bucket — versioning ON, 365-day lifecycle for router configs, 90-day for Hermes/portal
DR bucket (different Wasabi region) — Cross-Region Replication from primary, Object Lock (7 day immutable)
Same provider (Wasabi), different regions — two providers is overkill for MSPs
Cost: ~$5-8/mo per bucket for small MSP portal + config backups

Retention policy (from user decisions)

Router configs: 365 days (1 year). Versioning ON. Lifecycle: expire current versions after 365 days, expire delete markers after 30 days.
- Config files are small (~15-20KB each) — 365 exports per router = ~5-7MB/year. Storage cost at Wasabi is essentially zero.
Hermes backups: 90 days primary, 90 days DR (Object Lock 7-day immutability).
Portal backups: 90 days primary, 90 days DR (Object Lock 7-day immutability).
Hetzner snapshots: Keep latest weekly, auto-delete older than 7 days.
Provider snapshots are NOT a substitute for application S3 backups — they restore the OS, not the data.

Bucket naming convention

{service}-backups          # Primary (us-east-1)
{service}-backups-dr       # DR (us-west-2)

Wasabi-specific notes

Wasabi is 100% AWS S3 API compatible — standard aws s3 / boto3 commands with --endpoint-url
Must specify --endpoint-url https://s3.<region>.wasabisys.com for every call
Credentials in ~/.aws/credentials in standard format
Bucket names are globally unique — handle BucketAlreadyExists gracefully

Key practices

Versioning ON on all primary buckets (accidental deletion = recoverable)
- Verified working: aws s3api put-bucket-versioning --bucket name --versioning-configuration Status=Enabled --endpoint-url ...
Object Lock on DR buckets (7 day immutability — ransomware can't touch them)
Lifecycle: 30-90 day retention on primary, 90 days on DR, then delete or glacier
Restore drill: Periodically test that a backup download + extract + restore actually works
Staging directory: Use disk-backed path (e.g. /root/.hermes/.backups/), NOT /tmp (tmpfs fills up)

Hermes backup fixes (from experience)

/tmp is tmpfs (RAM-backed, ~1GB). Hermes state.db + profiles easily exceeds it.
Always verify with df -h /tmp if you see "No space left on device" but df -h / shows free space.
Fix: BACKUP_DIR and ARCHIVE both go to ~/.hermes/.backups/ on root disk.
Also check Python scripts' config.yaml temp_dir — they may independently write to /tmp.

3.8 KiB Raw Permalink Blame History