hermes-skills/skills/devops/hermes-backup/references/backup-audit-procedure.md

S3 Backup Bucket Audit Procedure

A repeatable methodology for auditing all Wasabi S3 backup buckets — versioning status, file counts, total sizes, staleness checks, and fresh-object verification.

Why audit

• Catch backup pipeline failures early (the MikroTik router backup pipeline stopped producing configs after Jul 5, 2026, and went unnoticed for 3 days)
• Verify the live-sync checkpoint is actually producing objects from today
• Confirm versioning wasn't accidentally disabled during a bucket config change
• Know your total storage consumption for cost tracking

Prerequisites

# AWS CLI with Wasabi endpoint configured
/opt/awscli-venv/bin/aws configure  # ~/.aws/credentials must exist
ENDPOINT="https://s3.us-east-1.wasabisys.com"

Quick audit script

Run this on all existing buckets at once:

ENDPOINT="https://s3.us-east-1.wasabisys.com"

for BUCKET in hermes-vps-backups itpropartner-backups mikrotik-ccr-backups itpropartner-system-configs itpropartner-docker-volumes; do
    echo "=== $BUCKET ==="
    
    # Versioning status
    echo "--- Versioning ---"
    /opt/awscli-venv/bin/aws s3api get-bucket-versioning --bucket "$BUCKET" --endpoint-url "$ENDPOINT" 2>&1
    
    # Top-level layout and latest object
    echo "--- Top-level listing (last 5 modified) ---"
    /opt/awscli-venv/bin/aws s3 ls s3://"$BUCKET"/ --endpoint-url "$ENDPOINT" --human-readable --recursive 2>&1 | sort -k1,2r | head -5
    
    # Summary (count + size)
    /opt/awscli-venv/bin/aws s3 ls s3://"$BUCKET"/ --endpoint-url "$ENDPOINT" --human-readable --recursive --summarize 2>&1 | tail -2
    
    # Staleness: count objects from today
    TODAY=$(date +%F)
    TODAY_COUNT=$(/opt/awscli-venv/bin/aws s3 ls s3://"$BUCKET"/ --endpoint-url "$ENDPOINT" --recursive 2>&1 | grep -c "$TODAY" || true)
    echo "Objects from today ($TODAY): $TODAY_COUNT"
    echo ""
done

Detailed per-bucket checks

1. Versioning

aws s3api get-bucket-versioning --bucket <name> --endpoint-url https://s3.us-east-1.wasabisys.com

Expected: {"Status": "Enabled", "MFADelete": "Disabled"} If Status is absent or Suspended, the bucket is no longer protected against accidental overwrites/deletes.

2. File count and total size

aws s3 ls s3://<bucket>/ --endpoint-url <endpoint> --recursive --human-readable --summarize

The summary lines at the bottom give you:

• Total Objects: N
• Total Size: X.X GiB/MiB

3. Staleness check (per top-level prefix)

Check whether any objects exist from today:

aws s3 ls s3://<bucket>/ --endpoint-url <endpoint> --recursive | grep "$(date +%F)" | sort -r | head -5

If zero objects from today, check yesterday:

aws s3 ls s3://<bucket>/ --endpoint-url <endpoint> --recursive | grep "$(date -d yesterday +%F)" | sort -r | head -5

Staleness thresholds:

Type	Max acceptable gap
Live Hermes sync	< 30 minutes (expected every 15 min)
Full Hermes backup	< 48 hours (daily at 1 AM UTC)
Router config backup	< 48 hours (daily at 6 AM UTC)

4. Live sync freshness (deep check for hermes-vps-backups)

The most recent object timestamp tells you when the 15-min live sync last ran:

aws s3 ls s3://hermes-vps-backups/live/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive 2>&1 | sort -k1,2r | head -5

Also count today's objects in the live prefix specifically:

aws s3 ls s3://hermes-vps-backups/live/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive 2>&1 | grep -c "$(date +%F)"

This should be a large number (hundreds to thousands) indicating the sync is actively pushing state changes.

5. Full backup archive check

aws s3 ls s3://hermes-vps-backups/hermes-full-backup/ --endpoint-url https://s3.us-east-1.wasabisys.com --human-readable

Check that a .tar.gz file exists from within the last 48 hours. Also check the manifest file is present beside each archive.

6. Caddyfile copy check

The live sync pushes /etc/caddy/Caddyfile to s3://hermes-vps-backups/live/caddy/Caddyfile every 15 min. Verify it's current:

aws s3 ls s3://hermes-vps-backups/live/caddy/ --endpoint-url https://s3.us-east-1.wasabisys.com

7. Empty purpose-specific bucket check

The itpropartner-system-configs and itpropartner-docker-volumes buckets exist on Wasabi but may contain 0 objects if their sync scripts are not running. Check both:

for BUCKET in itpropartner-system-configs itpropartner-docker-volumes; do
    echo "=== $BUCKET ==="
    /opt/awscli-venv/bin/aws s3 ls s3://"$BUCKET"/ --endpoint-url "$ENDPOINT" --recursive --summarize 2>&1 | tail -2
done

Expected: Total Objects: > 0. If 0 objects, the corresponding sync script (hermes-system-config-sync.sh or hermes-docker-sync.sh) is not running or failing silently.

Known bucket inventory (as of Jul 11, 2026)

Bucket	Purpose	Status
hermes-vps-backups	Hermes full backup + live sync + standby	✅ Versioning ON, active objects
itpropartner-backups	Portal/ITP backups	✅ Versioning ON, minimal objects
mikrotik-ccr-backups	Router config exports	✅ Versioning ON, stale
itpropartner-system-configs	System configs	🟡 Bucket exists but EMPTY — sync script not running or failing
itpropartner-docker-volumes	Docker volume data	🟡 Bucket exists but EMPTY — sync script not running or failing

Common failure patterns

Symptom	Likely cause	Fix
Zero objects from today on live/	Live sync cron job stopped or credentials rotated	Check hermes cron list for hermes-live-sync status; verify ~/.aws/credentials
Bucket listing returns objects but versioning says Suspended	Someone ran put-bucket-versioning with Status=Suspended	Re-enable: aws s3api put-bucket-versioning --bucket <name> --versioning-configuration Status=Enabled --endpoint-url <endpoint>
Full backup archive is >48h old	Daily cron job stopped running; live sync masks the gap	Check system crontab; run bash /root/.hermes/scripts/hermes-backup.sh manually; verify BACKUP_DIR not on /tmp
Router config backups stopped	VPN tunnel down, SSH key rotated, or pipeline script error	SSH to gateway CCR and test manually; check the tower config YAML
AccessDenied on ListBuckets	Expected — IAM policy scopes to specific bucket ARNs	Test by bucket name directly, not aws s3 ls (no arg)
InvalidAccessKeyId	Stale credentials in ~/.aws/credentials	Get current key from password manager and update the file
Bucket exists but has 0 objects	Sync script not running or failing silently	Check system crontab for the sync job; run the script manually and check output; verify aws is in PATH (venv activation)
s3api get-bucket-versioning returns empty	Versioning not yet enabled on bucket or s3:GetBucketVersioning missing from IAM policy	Enable with put-bucket-versioning; check IAM policy includes s3:GetBucketVersioning on the bucket ARN
AWS CLI command not found in cron	Backup/sync script missing venv activation	Add source /opt/awscli-venv/bin/activate near the top of the script (after set -euo pipefail, before first aws call)
Staleness >48h but watchdog didn't alert	Watchdog uses 36h threshold, so it reports OK for ~1.5 days after last successful backup	The watchdog is working as designed (grace period prevents false alarms). The real fix is preventing the backup from failing in the first place

Periodic audit cadence

• Weekly: Run the quick audit script on existing buckets
• Monthly: Review full backup vs live sync size trends for cost tracking
• Post-migration: Run the full audit procedure immediately after any server migration or credential rotation