hermes-skills/skills/devops/hermes-backup/references/backup-audit-watchdog.md

Backup Audit Watchdog

Since Jul 10, 2026: a system crontab watchdog checks daily whether the full backup is fresh.

Pattern

A no_agent script runs at 2 AM UTC (1 hour after the scheduled daily full backup at 1 AM). It checks S3 for the latest backup archive and reports freshness.

Script: /root/.hermes/scripts/backup-audit-check.sh

#!/bin/bash
# Checks S3 for latest full backup. Reports success if <36h old.

ENDPOINT="https://s3.us-east-1.wasabisys.com"
BUCKET="hermes-vps-backups"
PREFIX="hermes-full-backup"

source /opt/awscli-venv/bin/activate 2>/dev/null || true

latest=$(aws s3 ls "s3://$BUCKET/$PREFIX/" --endpoint-url "$ENDPOINT" 2>&1 | \
  grep "\.tar\.gz$" | sort | tail -1)

if [ -z "$latest" ]; then
    echo "🔴 No full backup archives found on S3!"
    exit 1
fi

date_str=$(echo "$latest" | awk '{print $1}')
size_bytes=$(echo "$latest" | awk '{print $3}')
file_name=$(echo "$latest" | awk '{print $4}')
age_hours=$(( ( $(date +%s) - $(date -d "$date_str" +%s) ) / 3600 ))

if [ "$age_hours" -lt 36 ]; then
    echo "✅ BACKUP OK — $file_name ($size_bytes bytes, ${age_hours}h ago)"
elif [ "$age_hours" -lt 60 ]; then
    echo "⚠️ BACKUP WARNING — $age_hours hours old: $file_name"
else
    echo "🔴 BACKUP STALE — $age_hours hours ago: $file_name"
fi

System crontab

0 2 * * * /root/.hermes/scripts/backup-audit-check.sh 2>&1 | logger -t backup-audit

Output goes to syslog with backup-audit tag. View with:

journalctl -t backup-audit