Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,67 @@
+# Service Health Check Cron
+
+## Pattern
+
+A no_agent cron that checks all critical services every 5 minutes. Silent on success. Reports only when something is down. Implemented at `~/.hermes/scripts/service-health-check.sh`.
+
+## What it checks
+
+| # | Check | Method |
+|---|-------|--------|
+| 1 | Caddy (sign.itpropartner.com) | `systemctl is-active` |
+| 2 | MySQL tunnel (wphost02) | `systemctl is-active` |
+| 3 | SearXNG container | `docker ps --format` |
+| 4 | DocuSeal container | `docker ps --format` |
+| 5 | Portal mockup (port 8081) | `curl -sf` |
+| 6 | DocuSeal web (port 3000) | `curl -sf` |
+| 7 | SearXNG search (port 8888) | `curl -sf` with search query |
+| 8 | WireGuard (home router) | `ping 10.77.0.2` |
+| 9 | MySQL through tunnel | `mysql --skip-ssl -e "SELECT 1"` |
+| 10 | wphost02 SSH | `ssh -o ConnectTimeout=5 hostname` |
+| 11 | Hermes gateway process | `ps aux | grep` |
+| 12 | Cloudflare API token | `curl -X GET /tokens/verify` |
+
+## Cron setup
+
+```bash
+hermes cron create \
+  --name "service-health-check" \
+  --schedule "every 5m" \
+  --no_agent \
+  --script service-health-check.sh
+```
+
+## Script design
+
+- `exit 0` when all services healthy — no output, no delivery
+- `exit 1` when any service fails — prints a bullet list of failed checks
+- Failed services array tracks which checks failed
+- Each check is independent — one failure doesn't abort the script
+- Check count shown in alert message
+
+## Pitfalls
+
+- **Sensitive info in error output:** The MySQL connection string appears in the script. If a service fails, the error output may include the full password. The cron delivery is Telegram (visible only to user), not email.
+- **False negatives on restart:** During a container restart, the health check may fire before the service comes back up. The 5-minute interval means this resolves within one cycle.
+- **Token expiry:** The Cloudflare API token check is a direct API call. If the token expires, ALL Cloudflare checks fail. The script does not differentiate between expired token and Cloudflare outage.
+- **MySQL credential in plaintext:** The password is in the script file and .env. If the cron output is ever delivered to the wrong channel, credentials leak. Consider moving MySQL check to a separate script that sources credentials from .env with tighter permissions.
+
+## ⚠️ Fixed Bug: Bearer token sent as literal asterisks
+
+**Root cause discovered Jul 7, 2026:** Line 38 of the script reads the token from `.env` into a variable, but line 39 sends `Authorization: Bearer ***` (literal asterisks) instead of `Authorization: Bearer $TOKEN`. This caused the health check to report the Cloudflare API token as "unhealthy" every 5 minutes, even though the actual token was valid.
+
+**Before (broken):**
+```bash
+TOKEN=$(grep CLOUDFLARE_API_TOKEN ~/.hermes/.env | cut -d= -f2-)
+curl -sf -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
+  -H "Authorization: Bearer ***" | ...
+```
+
+**After (fixed):**
+```bash
+CLOUD_TOKEN=$(grep CLOUDFLARE_API_TOKEN ~/.hermes/.env | cut -d= -f2-)
+curl -sf -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
+  -H "Authorization: Bearer $CLOUD_TOKEN" | ...
+```
+
+Variable name was changed from `$TOKEN` to `$CLOUD_TOKEN` to avoid any confusion with other use of `TOKEN` in the script or environment.