Initial skills documentation — 25 categories, all SKILL.md + references + scripts
This commit is contained in:
root
@@ -0,0 +1,163 @@
|
||||
# S3 Backup Bucket Audit Procedure
|
||||
|
||||
A repeatable methodology for auditing all Wasabi S3 backup buckets — versioning status, file counts, total sizes, staleness checks, and fresh-object verification.
|
||||
|
||||
## Why audit
|
||||
|
||||
- Catch backup pipeline failures early (the MikroTik router backup pipeline stopped producing configs after Jul 5, 2026, and went unnoticed for 3 days)
|
||||
- Verify the live-sync checkpoint is actually producing objects from today
|
||||
- Confirm versioning wasn't accidentally disabled during a bucket config change
|
||||
- Know your total storage consumption for cost tracking
|
||||
|
||||
## Prerequisites
|
||||
|
||||
```bash
|
||||
# AWS CLI with Wasabi endpoint configured
|
||||
/opt/awscli-venv/bin/aws configure # ~/.aws/credentials must exist
|
||||
ENDPOINT="https://s3.us-east-1.wasabisys.com"
|
||||
```
|
||||
|
||||
## Quick audit script
|
||||
|
||||
Run this on all existing buckets at once:
|
||||
|
||||
```bash
|
||||
ENDPOINT="https://s3.us-east-1.wasabisys.com"
|
||||
|
||||
for BUCKET in hermes-vps-backups itpropartner-backups mikrotik-ccr-backups itpropartner-system-configs itpropartner-docker-volumes; do
|
||||
echo "=== $BUCKET ==="
|
||||
|
||||
# Versioning status
|
||||
echo "--- Versioning ---"
|
||||
/opt/awscli-venv/bin/aws s3api get-bucket-versioning --bucket "$BUCKET" --endpoint-url "$ENDPOINT" 2>&1
|
||||
|
||||
# Top-level layout and latest object
|
||||
echo "--- Top-level listing (last 5 modified) ---"
|
||||
/opt/awscli-venv/bin/aws s3 ls s3://"$BUCKET"/ --endpoint-url "$ENDPOINT" --human-readable --recursive 2>&1 | sort -k1,2r | head -5
|
||||
|
||||
# Summary (count + size)
|
||||
/opt/awscli-venv/bin/aws s3 ls s3://"$BUCKET"/ --endpoint-url "$ENDPOINT" --human-readable --recursive --summarize 2>&1 | tail -2
|
||||
|
||||
# Staleness: count objects from today
|
||||
TODAY=$(date +%F)
|
||||
TODAY_COUNT=$(/opt/awscli-venv/bin/aws s3 ls s3://"$BUCKET"/ --endpoint-url "$ENDPOINT" --recursive 2>&1 | grep -c "$TODAY" || true)
|
||||
echo "Objects from today ($TODAY): $TODAY_COUNT"
|
||||
echo ""
|
||||
done
|
||||
```
|
||||
|
||||
## Detailed per-bucket checks
|
||||
|
||||
### 1. Versioning
|
||||
|
||||
```bash
|
||||
aws s3api get-bucket-versioning --bucket <name> --endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
```
|
||||
|
||||
Expected: `{"Status": "Enabled", "MFADelete": "Disabled"}`
|
||||
If `Status` is absent or `Suspended`, the bucket is no longer protected against accidental overwrites/deletes.
|
||||
|
||||
### 2. File count and total size
|
||||
|
||||
```bash
|
||||
aws s3 ls s3://<bucket>/ --endpoint-url <endpoint> --recursive --human-readable --summarize
|
||||
```
|
||||
|
||||
The summary lines at the bottom give you:
|
||||
- `Total Objects: N`
|
||||
- `Total Size: X.X GiB/MiB`
|
||||
|
||||
### 3. Staleness check (per top-level prefix)
|
||||
|
||||
Check whether any objects exist from today:
|
||||
|
||||
```bash
|
||||
aws s3 ls s3://<bucket>/ --endpoint-url <endpoint> --recursive | grep "$(date +%F)" | sort -r | head -5
|
||||
```
|
||||
|
||||
If zero objects from today, check yesterday:
|
||||
```bash
|
||||
aws s3 ls s3://<bucket>/ --endpoint-url <endpoint> --recursive | grep "$(date -d yesterday +%F)" | sort -r | head -5
|
||||
```
|
||||
|
||||
**Staleness thresholds:**
|
||||
| Type | Max acceptable gap |
|
||||
|---|---|
|
||||
| Live Hermes sync | < 30 minutes (expected every 15 min) |
|
||||
| Full Hermes backup | < 48 hours (daily at 1 AM UTC) |
|
||||
| Router config backup | < 48 hours (daily at 6 AM UTC) |
|
||||
|
||||
### 4. Live sync freshness (deep check for `hermes-vps-backups`)
|
||||
|
||||
The most recent object timestamp tells you when the 15-min live sync last ran:
|
||||
|
||||
```bash
|
||||
aws s3 ls s3://hermes-vps-backups/live/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive 2>&1 | sort -k1,2r | head -5
|
||||
```
|
||||
|
||||
Also count today's objects in the live prefix specifically:
|
||||
```bash
|
||||
aws s3 ls s3://hermes-vps-backups/live/ --endpoint-url https://s3.us-east-1.wasabisys.com --recursive 2>&1 | grep -c "$(date +%F)"
|
||||
```
|
||||
|
||||
This should be a large number (hundreds to thousands) indicating the sync is actively pushing state changes.
|
||||
|
||||
### 5. Full backup archive check
|
||||
|
||||
```bash
|
||||
aws s3 ls s3://hermes-vps-backups/hermes-full-backup/ --endpoint-url https://s3.us-east-1.wasabisys.com --human-readable
|
||||
```
|
||||
|
||||
Check that a `.tar.gz` file exists from within the last 48 hours. Also check the manifest file is present beside each archive.
|
||||
|
||||
### 6. Caddyfile copy check
|
||||
|
||||
The live sync pushes `/etc/caddy/Caddyfile` to `s3://hermes-vps-backups/live/caddy/Caddyfile` every 15 min. Verify it's current:
|
||||
|
||||
```bash
|
||||
aws s3 ls s3://hermes-vps-backups/live/caddy/ --endpoint-url https://s3.us-east-1.wasabisys.com
|
||||
```
|
||||
|
||||
### 7. Empty purpose-specific bucket check
|
||||
|
||||
The `itpropartner-system-configs` and `itpropartner-docker-volumes` buckets exist on Wasabi but may contain 0 objects if their sync scripts are not running. Check both:
|
||||
|
||||
```bash
|
||||
for BUCKET in itpropartner-system-configs itpropartner-docker-volumes; do
|
||||
echo "=== $BUCKET ==="
|
||||
/opt/awscli-venv/bin/aws s3 ls s3://"$BUCKET"/ --endpoint-url "$ENDPOINT" --recursive --summarize 2>&1 | tail -2
|
||||
done
|
||||
```
|
||||
|
||||
Expected: `Total Objects: > 0`. If 0 objects, the corresponding sync script (`hermes-system-config-sync.sh` or `hermes-docker-sync.sh`) is not running or failing silently.
|
||||
|
||||
## Known bucket inventory (as of Jul 11, 2026)
|
||||
|
||||
| Bucket | Purpose | Status |
|
||||
|---|---|---|
|
||||
| `hermes-vps-backups` | Hermes full backup + live sync + standby | ✅ Versioning ON, active objects |
|
||||
| `itpropartner-backups` | Portal/ITP backups | ✅ Versioning ON, minimal objects |
|
||||
| `mikrotik-ccr-backups` | Router config exports | ✅ Versioning ON, stale |
|
||||
| `itpropartner-system-configs` | System configs | 🟡 **Bucket exists but EMPTY** — sync script not running or failing |
|
||||
| `itpropartner-docker-volumes` | Docker volume data | 🟡 **Bucket exists but EMPTY** — sync script not running or failing |
|
||||
|
||||
## Common failure patterns
|
||||
|
||||
| Symptom | Likely cause | Fix |
|
||||
|---|---|---|
|
||||
| Zero objects from today on `live/` | Live sync cron job stopped or credentials rotated | Check `hermes cron list` for `hermes-live-sync` status; verify `~/.aws/credentials` |
|
||||
| Bucket listing returns objects but versioning says `Suspended` | Someone ran `put-bucket-versioning` with `Status=Suspended` | Re-enable: `aws s3api put-bucket-versioning --bucket <name> --versioning-configuration Status=Enabled --endpoint-url <endpoint>` |
|
||||
| Full backup archive is >48h old | Daily cron job stopped running; live sync masks the gap | Check system crontab; run `bash /root/.hermes/scripts/hermes-backup.sh` manually; verify `BACKUP_DIR` not on `/tmp` |
|
||||
| Router config backups stopped | VPN tunnel down, SSH key rotated, or pipeline script error | SSH to gateway CCR and test manually; check the tower config YAML |
|
||||
| `AccessDenied` on `ListBuckets` | Expected — IAM policy scopes to specific bucket ARNs | Test by bucket name directly, not `aws s3 ls` (no arg) |
|
||||
| `InvalidAccessKeyId` | Stale credentials in `~/.aws/credentials` | Get current key from password manager and update the file |
|
||||
| Bucket exists but has 0 objects | Sync script not running or failing silently | Check system crontab for the sync job; run the script manually and check output; verify `aws` is in PATH (venv activation) |
|
||||
| `s3api get-bucket-versioning` returns empty | Versioning not yet enabled on bucket or `s3:GetBucketVersioning` missing from IAM policy | Enable with `put-bucket-versioning`; check IAM policy includes `s3:GetBucketVersioning` on the bucket ARN |
|
||||
| AWS CLI `command not found` in cron | Backup/sync script missing venv activation | Add `source /opt/awscli-venv/bin/activate` near the top of the script (after `set -euo pipefail`, before first `aws` call) |
|
||||
| Staleness >48h but watchdog didn't alert | Watchdog uses 36h threshold, so it reports OK for ~1.5 days after last successful backup | The watchdog is working as designed (grace period prevents false alarms). The real fix is preventing the backup from failing in the first place |
|
||||
|
||||
## Periodic audit cadence
|
||||
|
||||
- **Weekly**: Run the quick audit script on existing buckets
|
||||
- **Monthly**: Review full backup vs live sync size trends for cost tracking
|
||||
- **Post-migration**: Run the full audit procedure immediately after any server migration or credential rotation
|
||||
Reference in New Issue
Block a user