hermes-skills/skills/devops/docker-service-deployment/references/service-health-check-pattern.md

Service Health Check (no_agent watchdog)

A reusable pattern for no_agent cron jobs that monitor ALL critical services and only report on failure.

Design

#!/bin/bash
# silent on success, report failures only

CHECKS=()
FAILED=0

# Each check:
command || { CHECKS+=("Service Name"); FAILED=1; }

if [ $FAILED -eq 1 ]; then
    echo "⚠️  Service Health Alert — $(date)"
    for c in "${CHECKS[@]}"; do echo "  ❌ $c"; done
    exit 1
fi
exit 0

What to check

• Systemd services (caddy, mysql-tunnel)
• Docker containers (docker ps --format)
• Local HTTP endpoints (curl -sf)
• Remote reachability (ping through tunnel, SSH)
• Database connectivity (mysql -e "SELECT 1")
• API token validity (curl + verify response)

Pitfalls

• Must be no_agent: true in cron config — LLM processing is wasted on binary health checks and causes typing indicators
• Include the exit code — exit 0 = silent, exit 1 = output delivered. Without the right exit code, cron can't tell success from failure
• Test with bash script.sh and check echo $? before deploying
• MySQL checks need --skip-ssl for local tunnel connections
• Don't check too many things — 10-12 checks is the sweet spot. More than 15 and the script gets brittle
• SSH checks need -o BatchMode=yes -o ConnectTimeout=5 to avoid hanging on unreachable hosts