66 lines
3.8 KiB
Markdown
66 lines
3.8 KiB
Markdown
# Two-Bucket DR Backup Strategy
|
|
|
|
## Architecture
|
|
|
|
### Layer 1: Provider-Level (automated snapshots)
|
|
| Asset | Backup Method | Retention | Coverage |
|
|
|---|---|---|---|
|
|
| **Hetzner Cloud VPS** | Hetzner snapshot (API/CLI) | Keep latest + 7-day auto | Full OS recovery |
|
|
| **Hetzner dedicated** | Robot backup service | 7 days | Full system |
|
|
| **RunCloud servers** | RunCloud built-in backup | Their retention | Web apps, WordPress, databases |
|
|
| **CCR routers** | RouterOS API export → S3 (daily) | 365 days per lifecycle rule | Config + logs only |
|
|
|
|
These protect against OS corruption, accidental config wipe, bad update — NOT data loss.
|
|
|
|
### Layer 2: Application-Level (S3 — two-bucket DR)
|
|
| Asset | Primary Bucket | DR Bucket | Frequency |
|
|
|---|---|---|---|
|
|
| **Hermes** (config, sessions, skills) | `hermes-vps-backups` (us-east-1) | `hermes-vps-backups-dr` (us-west-2) | Daily |
|
|
| **Portal** (DB, uploads, configs) | `itpropartner-backups` (us-east-1) | `itpropartner-backups-dr` (us-west-2) | Daily |
|
|
| **CCR router configs** | `mikrotik-ccr-backups` (us-east-1) | `mikrotik-ccr-backups-dr` (us-west-2) | Daily |
|
|
|
|
### Layer 3: Hudu (source of truth)
|
|
Hudu has its own backup export in Admin settings. Export to Wasabi bucket periodically. Hudu is the canonical record for passwords, configs, network diagrams, and procedures.
|
|
|
|
## Two-Bucket Pattern
|
|
|
|
1. **Primary bucket** — versioning ON, 365-day lifecycle for router configs, 90-day for Hermes/portal
|
|
2. **DR bucket** (different Wasabi region) — Cross-Region Replication from primary, Object Lock (7 day immutable)
|
|
3. **Same provider** (Wasabi), different regions — two providers is overkill for MSPs
|
|
4. **Cost:** ~$5-8/mo per bucket for small MSP portal + config backups
|
|
|
|
### Retention policy (from user decisions)
|
|
|
|
- **Router configs**: 365 days (1 year). Versioning ON. Lifecycle: expire current versions after 365 days, expire delete markers after 30 days.
|
|
- Config files are small (~15-20KB each) — 365 exports per router = ~5-7MB/year. Storage cost at Wasabi is essentially zero.
|
|
- **Hermes backups**: 90 days primary, 90 days DR (Object Lock 7-day immutability).
|
|
- **Portal backups**: 90 days primary, 90 days DR (Object Lock 7-day immutability).
|
|
- **Hetzner snapshots**: Keep latest weekly, auto-delete older than 7 days.
|
|
- Provider snapshots are NOT a substitute for application S3 backups — they restore the OS, not the data.
|
|
|
|
### Bucket naming convention
|
|
```
|
|
{service}-backups # Primary (us-east-1)
|
|
{service}-backups-dr # DR (us-west-2)
|
|
```
|
|
|
|
### Wasabi-specific notes
|
|
- Wasabi is 100% AWS S3 API compatible — standard `aws s3` / `boto3` commands with `--endpoint-url`
|
|
- Must specify `--endpoint-url https://s3.<region>.wasabisys.com` for every call
|
|
- Credentials in `~/.aws/credentials` in standard format
|
|
- Bucket names are globally unique — handle `BucketAlreadyExists` gracefully
|
|
|
|
## Key practices
|
|
- **Versioning ON** on all primary buckets (accidental deletion = recoverable)
|
|
- Verified working: `aws s3api put-bucket-versioning --bucket name --versioning-configuration Status=Enabled --endpoint-url ...`
|
|
- **Object Lock** on DR buckets (7 day immutability — ransomware can't touch them)
|
|
- **Lifecycle**: 30-90 day retention on primary, 90 days on DR, then delete or glacier
|
|
- **Restore drill**: Periodically test that a backup download + extract + restore actually works
|
|
- **Staging directory**: Use disk-backed path (e.g. `/root/.hermes/.backups/`), NOT `/tmp` (tmpfs fills up)
|
|
|
|
## Hermes backup fixes (from experience)
|
|
- `/tmp` is tmpfs (RAM-backed, ~1GB). Hermes state.db + profiles easily exceeds it.
|
|
- Always verify with `df -h /tmp` if you see "No space left on device" but `df -h /` shows free space.
|
|
- Fix: `BACKUP_DIR` and `ARCHIVE` both go to `~/.hermes/.backups/` on root disk.
|
|
- Also check Python scripts' `config.yaml temp_dir` — they may independently write to `/tmp`.
|