# S3 Backup Bucket Audit Procedure

A repeatable methodology for auditing all Wasabi S3 backup buckets — versioning status, file counts, total sizes, staleness checks, and fresh-object verification.

## Why audit

- Catch backup pipeline failures early (the MikroTik router backup pipeline stopped producing configs after Jul 5, 2026, and went unnoticed for 3 days)
- Verify the live-sync checkpoint is actually producing objects from today
- Confirm versioning wasn't accidentally disabled during a bucket config change
- Know your total storage consumption for cost tracking

## Prerequisites

```bash
# AWS CLI with Wasabi endpoint configured
/opt/awscli-venv/bin/aws configure  # ~/.aws/credentials must exist
ENDPOINT="https://s3.us-east-1.wasabisys.com"
```

## Quick audit script

Run this on all existing buckets at once:

```bash
ENDPOINT="https://s3.us-east-1.wasabisys.com"

for BUCKET in hermes-vps-backups itpropartner-backups mikrotik-ccr-backups itpropartner-system-configs itpropartner-docker-volumes; do
    echo "=== $BUCKET ==="
    
    # Versioning status
    echo "--- Versioning ---"
    /opt/awscli-venv/bin/aws s3api get-bucket-versioning --bucket "$BUCKET" --endpoint-url "$ENDPOINT" 2>&1
    
    # Top-level layout and latest object
    echo "--- Top-level listing (last 5 modified) ---"
    /opt/awscli-venv/bin/aws s3 ls s3://"$BUCKET"/ --endpoint-url "$ENDPOINT" --human-readable --recursive 2>&1 | sort -k1,2r | head -5
    
    # Summary (count + size)
    /opt/awscli-venv/bin/aws s3 ls s3://"$BUCKET"/ --endpoint-url "$ENDPOINT" --human-readable --recursive --summarize 2>&1 | tail -2
    
    # Staleness: count objects from today
    TODAY=$(date +%F)
    TODAY_COUNT=$(/opt/awscli-venv/bin/aws s3 ls s3://"$BUCKET"/ --endpoint-url "$ENDPOINT" --recursive 2>&1 | grep -c "$TODAY" || true)
    echo "Objects from today ($TODAY): $TODAY_COUNT"
    echo ""
done
```

## Detailed per-bucket checks

### 1. Versioning

```bash
aws s3api get-bucket-versioning --bucket <name> --endpoint-url https://s3.us-east-1.wasabisys.com
```

Expected: `{"Status": "Enabled", "MFADelete": "Disabled"}`
If `Status` is absent or `Suspended`, the bucket is no longer protected against accidental overwrites/deletes.

### 2. File count and total size

```bash
aws s3 ls s3://<bucket>/ --endpoint-url <endpoint> --recursive --human-readable --summarize
```

The summary lines at the bottom give you:
- `Total Objects: N`
- `Total Size: X.X GiB/MiB`

### 3. Staleness check (per top-level prefix)

Check whether any objects exist from today:

```bash
aws s3 ls s3://<bucket>/ --endpoint-url <endpoint> --recursive | grep "$(date +%F)" | sort -r | head -5
```

If zero objects from today, check yesterday:
```bash
aws s3 ls s3://<bucket>/ --endpoint-url <endpoint> --recursive | grep "$(date -d yesterday +%F)" | sort -r | head -5
```

**Staleness thresholds:**
| Type | Max acceptable gap |
|---|---|
| Live Hermes sync | < 30 minutes (expected every 15 min) |
| Full Hermes backup | < 48 hours (daily at 1 AM UTC) |
| Router config backup | < 48 hours (daily at 6 AM UTC) |

### 4. Live sync freshness (deep check for `hermes-vps-backups`)

The most recent object timestamp tells you when the 15-min live sync last ran:

```bash
aws s3 ls s3://hermes-vps-backups/live/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive 2>&1 | sort -k1,2r | head -5
```

Also count today's objects in the live prefix specifically:
```bash
aws s3 ls s3://hermes-vps-backups/live/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive 2>&1 | grep -c "$(date +%F)"
```

This should be a large number (hundreds to thousands) indicating the sync is actively pushing state changes.

### 5. Full backup archive check

```bash
aws s3 ls s3://hermes-vps-backups/hermes-full-backup/ --endpoint-url https://s3.us-east-1.wasabisys.com --human-readable
```

Check that a `.tar.gz` file exists from within the last 48 hours. Also check the manifest file is present beside each archive.

### 6. Caddyfile copy check

The live sync pushes `/etc/caddy/Caddyfile` to `s3://hermes-vps-backups/live/caddy/Caddyfile` every 15 min. Verify it's current:

```bash
aws s3 ls s3://hermes-vps-backups/live/caddy/ --endpoint-url https://s3.us-east-1.wasabisys.com
```

### 7. Empty purpose-specific bucket check

The `itpropartner-system-configs` and `itpropartner-docker-volumes` buckets exist on Wasabi but may contain 0 objects if their sync scripts are not running. Check both:

```bash
for BUCKET in itpropartner-system-configs itpropartner-docker-volumes; do
    echo "=== $BUCKET ==="
    /opt/awscli-venv/bin/aws s3 ls s3://"$BUCKET"/ --endpoint-url "$ENDPOINT" --recursive --summarize 2>&1 | tail -2
done
```

Expected: `Total Objects: > 0`. If 0 objects, the corresponding sync script (`hermes-system-config-sync.sh` or `hermes-docker-sync.sh`) is not running or failing silently.

## Known bucket inventory (as of Jul 11, 2026)

| Bucket | Purpose | Status |
|---|---|---|
| `hermes-vps-backups` | Hermes full backup + live sync + standby | ✅ Versioning ON, active objects |
| `itpropartner-backups` | Portal/ITP backups | ✅ Versioning ON, minimal objects |
| `mikrotik-ccr-backups` | Router config exports | ✅ Versioning ON, stale |
| `itpropartner-system-configs` | System configs | 🟡 **Bucket exists but EMPTY** — sync script not running or failing |
| `itpropartner-docker-volumes` | Docker volume data | 🟡 **Bucket exists but EMPTY** — sync script not running or failing |

## Common failure patterns

| Symptom | Likely cause | Fix |
|---|---|---|
| Zero objects from today on `live/` | Live sync cron job stopped or credentials rotated | Check `hermes cron list` for `hermes-live-sync` status; verify `~/.aws/credentials` |
| Bucket listing returns objects but versioning says `Suspended` | Someone ran `put-bucket-versioning` with `Status=Suspended` | Re-enable: `aws s3api put-bucket-versioning --bucket <name> --versioning-configuration Status=Enabled --endpoint-url <endpoint>` |
| Full backup archive is >48h old | Daily cron job stopped running; live sync masks the gap | Check system crontab; run `bash /root/.hermes/scripts/hermes-backup.sh` manually; verify `BACKUP_DIR` not on `/tmp` |
| Router config backups stopped | VPN tunnel down, SSH key rotated, or pipeline script error | SSH to gateway CCR and test manually; check the tower config YAML |
| `AccessDenied` on `ListBuckets` | Expected — IAM policy scopes to specific bucket ARNs | Test by bucket name directly, not `aws s3 ls` (no arg) |
| `InvalidAccessKeyId` | Stale credentials in `~/.aws/credentials` | Get current key from password manager and update the file |
| Bucket exists but has 0 objects | Sync script not running or failing silently | Check system crontab for the sync job; run the script manually and check output; verify `aws` is in PATH (venv activation) |
| `s3api get-bucket-versioning` returns empty | Versioning not yet enabled on bucket or `s3:GetBucketVersioning` missing from IAM policy | Enable with `put-bucket-versioning`; check IAM policy includes `s3:GetBucketVersioning` on the bucket ARN |
| AWS CLI `command not found` in cron | Backup/sync script missing venv activation | Add `source /opt/awscli-venv/bin/activate` near the top of the script (after `set -euo pipefail`, before first `aws` call) |
| Staleness >48h but watchdog didn't alert | Watchdog uses 36h threshold, so it reports OK for ~1.5 days after last successful backup | The watchdog is working as designed (grace period prevents false alarms). The real fix is preventing the backup from failing in the first place |

## Periodic audit cadence

- **Weekly**: Run the quick audit script on existing buckets
- **Monthly**: Review full backup vs live sync size trends for cost tracking
- **Post-migration**: Run the full audit procedure immediately after any server migration or credential rotation