Initial skills documentation — 25 categories, all SKILL.md + references + scripts
This commit is contained in:
root
@@ -0,0 +1,197 @@
|
||||
# Home Router Watchdog — Always-On Monitoring with User Alert
|
||||
|
||||
A two-layer monitoring system for a VPN-connected remote router (or any critical network device behind a tunnel).
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
Layer 1: systemd keepalive ── every 2min ── auto-reconnect VPN if dropped ── silent
|
||||
↓
|
||||
Layer 2: cron watchdog ── every 5min ── 60s ping loop ── alert user on confirmed outage
|
||||
```
|
||||
|
||||
The keepalive handles transient flaps silently. The watchdog only messages the user when the device is truly unreachable after a full minute of consecutive failures and a final reconnect attempt.
|
||||
|
||||
## Layer 1 — Systemd Keepalive
|
||||
|
||||
**Service unit** (`home-router-keepalive.service`):
|
||||
```ini
|
||||
[Unit]
|
||||
Description=Home Router VPN Keepalive Check
|
||||
After=home-router-vpn.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/opt/home-router-keepalive.sh
|
||||
```
|
||||
|
||||
**Timer unit** (`home-router-keepalive.timer`):
|
||||
```ini
|
||||
[Timer]
|
||||
OnBootSec=2min
|
||||
OnUnitActiveSec=2min
|
||||
Persistent=true
|
||||
```
|
||||
|
||||
**Keepalive script** (`home-router-keepalive.sh`):
|
||||
```bash
|
||||
#!/bin/bash
|
||||
VPN_SCRIPT="/opt/home-router-vpn.sh"
|
||||
STATUS_FILE="/tmp/vpn-status"
|
||||
|
||||
if "$VPN_SCRIPT" status >/dev/null 2>&1; then
|
||||
date +%s > "$STATUS_FILE"
|
||||
exit 0
|
||||
else
|
||||
"$VPN_SCRIPT" up 2>&1
|
||||
if "$VPN_SCRIPT" status >/dev/null 2>&1; then
|
||||
echo "[+] VPN reconnected"
|
||||
date +%s > "$STATUS_FILE"
|
||||
exit 0
|
||||
else
|
||||
echo "[-] VPN reconnect failed"
|
||||
exit 1
|
||||
fi
|
||||
fi
|
||||
```
|
||||
|
||||
## Layer 2 — Cron Watchdog with 60s Ping Verification
|
||||
|
||||
**Key design:** The script pings the router's **WAN IP** (not LAN/VPN IP) to test the full internet path, not just the tunnel. The VPS must be able to ping the public IP — if the router blocks ICMP, add a firewall rule:
|
||||
|
||||
```routeros
|
||||
/ip firewall filter add chain=input protocol=icmp in-interface-list=WAN action=accept comment="Allow ICMP WAN Ping" place-before=[drop-rule-number]
|
||||
```
|
||||
|
||||
### Deployed script location
|
||||
|
||||
The actual watchdog script lives at `/root/.hermes/scripts/home-router-watchdog.sh`, NOT `/opt/`. The VPN script it calls lives at `/root/.hermes/scripts/wisp-backup/home-router-vpn.sh`. When deploying on a new box, adjust paths — the reference in this doc uses `/opt/` as the canonical example path.
|
||||
|
||||
### Watchdog Script (deployed version)
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
# home-router-watchdog.sh — Ping the home router for 60s to verify it's truly down
|
||||
# before notifying the user. Also triggers reconnect if VPN is just flapping.
|
||||
|
||||
VPN_SCRIPT="/root/.hermes/scripts/wisp-backup/home-router-vpn.sh"
|
||||
ROUTER_IP="76.195.7.60"
|
||||
PING_COUNT=6 # 6 pings, 10s apart = 60 seconds
|
||||
PING_INTERVAL=10
|
||||
|
||||
# If VPN isn't up, try reconnecting first
|
||||
if ! "$VPN_SCRIPT" status >/dev/null 2>&1; then
|
||||
echo "[-] VPN is down. Attempting reconnect..."
|
||||
"$VPN_SCRIPT" up >/dev/null 2>&1
|
||||
sleep 5
|
||||
fi
|
||||
|
||||
# Ping the router via WAN IP (firewall now allows ICMP)
|
||||
echo "[*] Pinging $ROUTER_IP for 60 seconds to verify connectivity..."
|
||||
SUCCESS=0
|
||||
for i in $(seq 1 $PING_COUNT); do
|
||||
if ping -c 1 -W 5 "$ROUTER_IP" >/dev/null 2>&1; then
|
||||
SUCCESS=$((SUCCESS + 1))
|
||||
echo " Ping $i/$PING_COUNT: OK ($SUCCESS so far)"
|
||||
# 2 successful pings = consider it good
|
||||
if [ "$SUCCESS" -ge 2 ]; then
|
||||
echo "[+] Router is reachable — connection is fine"
|
||||
exit 0
|
||||
fi
|
||||
else
|
||||
echo " Ping $i/$PING_COUNT: FAILED"
|
||||
fi
|
||||
sleep "$PING_INTERVAL"
|
||||
done
|
||||
|
||||
# All pings failed — do one final reconnect attempt before alerting
|
||||
echo "[-] Router not responding after 60 seconds of pings."
|
||||
echo "[*] Final reconnect attempt..."
|
||||
"$VPN_SCRIPT" up >/dev/null 2>&1
|
||||
sleep 5
|
||||
if "$VPN_SCRIPT" status >/dev/null 2>&1; then
|
||||
echo "[+] VPN reconnected — checking router..."
|
||||
if ping -c 2 -W 5 "76.195.7.60" >/dev/null 2>&1; then
|
||||
echo "[+] Router is back online — canceling alert"
|
||||
rm -f ~/.hermes/cron/output/home_router_alert.json
|
||||
exit 0
|
||||
fi
|
||||
fi
|
||||
|
||||
# Write alert JSON for cron job to read
|
||||
echo "[-] Alert remains — router confirmed offline"
|
||||
python3 -c "
|
||||
import json, os
|
||||
msg = {
|
||||
'alert': 'home_router_down',
|
||||
'subject': '🏠 Home Router Offline',
|
||||
'message': '''The home router WAN (76.195.7.60) is not responding.
|
||||
|
||||
The VPN tunnel was up and no ping replies came through for 60 seconds.
|
||||
|
||||
Actions taken:
|
||||
- VPN was auto-reconnected (if it was down)
|
||||
- 6 pings to WAN IP attempted over 60 seconds — all failed
|
||||
|
||||
Possible causes:
|
||||
- ISP outage at home
|
||||
- Power outage at home
|
||||
- MikroTik crashed/hung
|
||||
- RouterOS update rebooted
|
||||
|
||||
You may need to check from another network or wait for it to come back online.'''
|
||||
}
|
||||
path = os.path.expanduser('~/.hermes/cron/output/home_router_alert.json')
|
||||
os.makedirs(os.path.dirname(path), exist_ok=True)
|
||||
with open(path, 'w') as f: json.dump(msg, f)
|
||||
"
|
||||
exit 1
|
||||
```
|
||||
|
||||
**Key differences from minimal `/opt/` reference version:**
|
||||
- Prints status messages so cron stdout shows what happened (useful for LLM-driven cron that can see output)
|
||||
- Final reconnect attempt BEFORE writing the alert, not after — avoids false alerts from transient drops
|
||||
- Richer alert message with actionable troubleshooting steps in the JSON
|
||||
- Uses absolute paths under `/root/.hermes/scripts/` not `/opt/`
|
||||
|
||||
### Cron Job Setup
|
||||
|
||||
The cron job is **LLM-driven** (`no_agent=false`), not a pure script — so it can read the alert file and compose a Telegram message. Use restricted toolsets:
|
||||
|
||||
```bash
|
||||
hermes cron create \
|
||||
--name "Home Router Watchdog" \
|
||||
--schedule "every 5m" \
|
||||
--script "home-router-watchdog.sh" \
|
||||
--prompt "Run the watchdog script. If exit 0 = silent. If exit 1 = read alert file and message user." \
|
||||
--toolsets "terminal,file,search"
|
||||
```
|
||||
|
||||
**Pitfall:** An LLM-driven cron job expects the script path relative to `/root/.hermes/scripts/`. If you copy the script elsewhere, update the cron definition too.
|
||||
|
||||
### Naming Convention
|
||||
|
||||
Keep personal and business infrastructure separate:
|
||||
- Personal home router → `home-router-*` (service, keepalive, watchdog)
|
||||
- Business WISP infrastructure → `wisp-*` or per-client prefix
|
||||
- Backup directories match: `/root/.hermes/backups/home-router/`
|
||||
|
||||
## Alternative: Uptime Kuma Push Monitor
|
||||
|
||||
Instead of raw ping, you can set up an Uptime Kuma "Push" type monitor and have the watchdog hit its URL:
|
||||
|
||||
```
|
||||
https://your-kuma.com/api/push/YOUR_PUSH_TOKEN?status=up&ping=37
|
||||
```
|
||||
|
||||
Push the ping response time as the `ping` parameter. Kuma tracks uptime, graphs latency, and handles alerting. The cron job becomes simpler — just curl the URL.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **ICMP blocked on WAN by default** — Most routers (including MikroTik) block inbound ping on WAN. Add a firewall rule before the default drop rule or use push-based monitoring instead.
|
||||
- **Ping through VPN tests the tunnel + router, not the internet** — If you want to know whether the router's WAN is reachable, ping the public IP. If you want to know whether the VPN works, ping the LAN IP.
|
||||
- **2-out-of-6 threshold** — Using 2 successful pings out of 6 over 60 seconds prevents false positives from a single dropped packet. Adjust based on link quality.
|
||||
- **Alert file must be cleaned up** — If the watchdog writes an alert file but the router comes back before the cron job runs, stale alerts can fire. The watchdog script removes the file on successful reconnection.
|
||||
- **SSH line length limit on RouterOS** — Ed25519 keys (~92 chars) fit fine within the ~170 char SSH CLI limit. RSA keys will not — use hex encoding or `/file/add contents=` instead.
|
||||
- **`/tool/fetch` requires `ftp` policy** — Users in `read` or `write` groups cannot use this command. Use `/file/add contents=` for file creation.
|
||||
- **`/user ssh-keys import` works after `/file/add`** — The file-add approach works on RouterOS 7.18. Create the file first, then import, then clean up.
|
||||
Reference in New Issue
Block a user