# Service Health Check Cron

## Pattern

A no_agent cron that checks all critical services every 5 minutes. Silent on success. Reports only when something is down. Implemented at `~/.hermes/scripts/service-health-check.sh`.

## What it checks

| # | Check | Method |
|---|-------|--------|
| 1 | Caddy (sign.itpropartner.com) | `systemctl is-active` |
| 2 | MySQL tunnel (wphost02) | `systemctl is-active` |
| 3 | SearXNG container | `docker ps --format` |
| 4 | DocuSeal container | `docker ps --format` |
| 5 | Portal mockup (port 8081) | `curl -sf` |
| 6 | DocuSeal web (port 3000) | `curl -sf` |
| 7 | SearXNG search (port 8888) | `curl -sf` with search query |
| 8 | WireGuard (home router) | `ping 10.77.0.2` |
| 9 | MySQL through tunnel | `mysql --skip-ssl -e "SELECT 1"` |
| 10 | wphost02 SSH | `ssh -o ConnectTimeout=5 hostname` |
| 11 | Hermes gateway process | `ps aux | grep` |
| 12 | Cloudflare API token | `curl -X GET /tokens/verify` |

## Cron setup

```bash
hermes cron create \
  --name "service-health-check" \
  --schedule "every 5m" \
  --no_agent \
  --script service-health-check.sh
```

## Script design

- `exit 0` when all services healthy — no output, no delivery
- `exit 1` when any service fails — prints a bullet list of failed checks
- Failed services array tracks which checks failed
- Each check is independent — one failure doesn't abort the script
- Check count shown in alert message

## Pitfalls

- **Sensitive info in error output:** The MySQL connection string appears in the script. If a service fails, the error output may include the full password. The cron delivery is Telegram (visible only to user), not email.
- **False negatives on restart:** During a container restart, the health check may fire before the service comes back up. The 5-minute interval means this resolves within one cycle.
- **Token expiry:** The Cloudflare API token check is a direct API call. If the token expires, ALL Cloudflare checks fail. The script does not differentiate between expired token and Cloudflare outage.
- **MySQL credential in plaintext:** The password is in the script file and .env. If the cron output is ever delivered to the wrong channel, credentials leak. Consider moving MySQL check to a separate script that sources credentials from .env with tighter permissions.

## ⚠️ Fixed Bug: Bearer token sent as literal asterisks

**Root cause discovered Jul 7, 2026:** Line 38 of the script reads the token from `.env` into a variable, but line 39 sends `Authorization: Bearer ***` (literal asterisks) instead of `Authorization: Bearer $TOKEN`. This caused the health check to report the Cloudflare API token as "unhealthy" every 5 minutes, even though the actual token was valid.

**Before (broken):**
```bash
TOKEN=$(grep CLOUDFLARE_API_TOKEN ~/.hermes/.env | cut -d= -f2-)
curl -sf -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
  -H "Authorization: Bearer ***" | ...
```

**After (fixed):**
```bash
CLOUD_TOKEN=$(grep CLOUDFLARE_API_TOKEN ~/.hermes/.env | cut -d= -f2-)
curl -sf -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
  -H "Authorization: Bearer $CLOUD_TOKEN" | ...
```

Variable name was changed from `$TOKEN` to `$CLOUD_TOKEN` to avoid any confusion with other use of `TOKEN` in the script or environment.