# Two-Bucket DR Backup Strategy

## Architecture

### Layer 1: Provider-Level (automated snapshots)
| Asset | Backup Method | Retention | Coverage |
|---|---|---|---|
| **Hetzner Cloud VPS** | Hetzner snapshot (API/CLI) | Keep latest + 7-day auto | Full OS recovery |
| **Hetzner dedicated** | Robot backup service | 7 days | Full system |
| **RunCloud servers** | RunCloud built-in backup | Their retention | Web apps, WordPress, databases |
| **CCR routers** | RouterOS API export → S3 (daily) | 365 days per lifecycle rule | Config + logs only |

These protect against OS corruption, accidental config wipe, bad update — NOT data loss.

### Layer 2: Application-Level (S3 — two-bucket DR)
| Asset | Primary Bucket | DR Bucket | Frequency |
|---|---|---|---|
| **Hermes** (config, sessions, skills) | `hermes-vps-backups` (us-east-1) | `hermes-vps-backups-dr` (us-west-2) | Daily |
| **Portal** (DB, uploads, configs) | `itpropartner-backups` (us-east-1) | `itpropartner-backups-dr` (us-west-2) | Daily |
| **CCR router configs** | `mikrotik-ccr-backups` (us-east-1) | `mikrotik-ccr-backups-dr` (us-west-2) | Daily |

### Layer 3: Hudu (source of truth)
Hudu has its own backup export in Admin settings. Export to Wasabi bucket periodically. Hudu is the canonical record for passwords, configs, network diagrams, and procedures.

## Two-Bucket Pattern

1. **Primary bucket** — versioning ON, 365-day lifecycle for router configs, 90-day for Hermes/portal
2. **DR bucket** (different Wasabi region) — Cross-Region Replication from primary, Object Lock (7 day immutable)
3. **Same provider** (Wasabi), different regions — two providers is overkill for MSPs
4. **Cost:** ~$5-8/mo per bucket for small MSP portal + config backups

### Retention policy (from user decisions)

- **Router configs**: 365 days (1 year). Versioning ON. Lifecycle: expire current versions after 365 days, expire delete markers after 30 days.
  - Config files are small (~15-20KB each) — 365 exports per router = ~5-7MB/year. Storage cost at Wasabi is essentially zero.
- **Hermes backups**: 90 days primary, 90 days DR (Object Lock 7-day immutability).
- **Portal backups**: 90 days primary, 90 days DR (Object Lock 7-day immutability).
- **Hetzner snapshots**: Keep latest weekly, auto-delete older than 7 days.
- Provider snapshots are NOT a substitute for application S3 backups — they restore the OS, not the data.

### Bucket naming convention
```
{service}-backups          # Primary (us-east-1)
{service}-backups-dr       # DR (us-west-2)
```

### Wasabi-specific notes
- Wasabi is 100% AWS S3 API compatible — standard `aws s3` / `boto3` commands with `--endpoint-url`
- Must specify `--endpoint-url https://s3.<region>.wasabisys.com` for every call
- Credentials in `~/.aws/credentials` in standard format
- Bucket names are globally unique — handle `BucketAlreadyExists` gracefully

## Key practices
- **Versioning ON** on all primary buckets (accidental deletion = recoverable)
  - Verified working: `aws s3api put-bucket-versioning --bucket name --versioning-configuration Status=Enabled --endpoint-url ...`
- **Object Lock** on DR buckets (7 day immutability — ransomware can't touch them)
- **Lifecycle**: 30-90 day retention on primary, 90 days on DR, then delete or glacier
- **Restore drill**: Periodically test that a backup download + extract + restore actually works
- **Staging directory**: Use disk-backed path (e.g. `/root/.hermes/.backups/`), NOT `/tmp` (tmpfs fills up)

## Hermes backup fixes (from experience)
- `/tmp` is tmpfs (RAM-backed, ~1GB). Hermes state.db + profiles easily exceeds it.
- Always verify with `df -h /tmp` if you see "No space left on device" but `df -h /` shows free space.
- Fix: `BACKUP_DIR` and `ARCHIVE` both go to `~/.hermes/.backups/` on root disk.
- Also check Python scripts' `config.yaml temp_dir` — they may independently write to `/tmp`.