hermes-skills/skills/devops/infrastructure-automation/references/home-router-watchdog.md

# Home Router Watchdog — Always-On Monitoring with User Alert

A two-layer monitoring system for a VPN-connected remote router (or any critical network device behind a tunnel).

## Architecture

```
Layer 1: systemd keepalive ── every 2min ── auto-reconnect VPN if dropped ── silent
     ↓
Layer 2: cron watchdog ── every 5min ── 60s ping loop ── alert user on confirmed outage
```

The keepalive handles transient flaps silently. The watchdog only messages the user when the device is truly unreachable after a full minute of consecutive failures and a final reconnect attempt.

## Layer 1 — Systemd Keepalive

**Service unit** (`home-router-keepalive.service`):
```ini
[Unit]
Description=Home Router VPN Keepalive Check
After=home-router-vpn.service

[Service]
Type=oneshot
ExecStart=/opt/home-router-keepalive.sh
```

**Timer unit** (`home-router-keepalive.timer`):
```ini
[Timer]
OnBootSec=2min
OnUnitActiveSec=2min
Persistent=true
```

**Keepalive script** (`home-router-keepalive.sh`):
```bash
#!/bin/bash
VPN_SCRIPT="/opt/home-router-vpn.sh"
STATUS_FILE="/tmp/vpn-status"

if "$VPN_SCRIPT" status >/dev/null 2>&1; then
    date +%s > "$STATUS_FILE"
    exit 0
else
    "$VPN_SCRIPT" up 2>&1
    if "$VPN_SCRIPT" status >/dev/null 2>&1; then
        echo "[+] VPN reconnected"
        date +%s > "$STATUS_FILE"
        exit 0
    else
        echo "[-] VPN reconnect failed"
        exit 1
    fi
fi
```

## Layer 2 — Cron Watchdog with 60s Ping Verification

**Key design:** The script pings the router's **WAN IP** (not LAN/VPN IP) to test the full internet path, not just the tunnel. The VPS must be able to ping the public IP — if the router blocks ICMP, add a firewall rule:

```routeros
/ip firewall filter add chain=input protocol=icmp in-interface-list=WAN action=accept comment="Allow ICMP WAN Ping" place-before=[drop-rule-number]
```

### Deployed script location

The actual watchdog script lives at `/root/.hermes/scripts/home-router-watchdog.sh`, NOT `/opt/`. The VPN script it calls lives at `/root/.hermes/scripts/wisp-backup/home-router-vpn.sh`. When deploying on a new box, adjust paths — the reference in this doc uses `/opt/` as the canonical example path.

### Watchdog Script (deployed version)

```bash
#!/bin/bash
# home-router-watchdog.sh — Ping the home router for 60s to verify it's truly down
# before notifying the user. Also triggers reconnect if VPN is just flapping.

VPN_SCRIPT="/root/.hermes/scripts/wisp-backup/home-router-vpn.sh"
ROUTER_IP="76.195.7.60"
PING_COUNT=6   # 6 pings, 10s apart = 60 seconds
PING_INTERVAL=10

# If VPN isn't up, try reconnecting first
if ! "$VPN_SCRIPT" status >/dev/null 2>&1; then
    echo "[-] VPN is down. Attempting reconnect..."
    "$VPN_SCRIPT" up >/dev/null 2>&1
    sleep 5
fi

# Ping the router via WAN IP (firewall now allows ICMP)
echo "[*] Pinging $ROUTER_IP for 60 seconds to verify connectivity..."
SUCCESS=0
for i in $(seq 1 $PING_COUNT); do
    if ping -c 1 -W 5 "$ROUTER_IP" >/dev/null 2>&1; then
        SUCCESS=$((SUCCESS + 1))
        echo "  Ping $i/$PING_COUNT: OK ($SUCCESS so far)"
        # 2 successful pings = consider it good
        if [ "$SUCCESS" -ge 2 ]; then
            echo "[+] Router is reachable — connection is fine"
            exit 0
        fi
    else
        echo "  Ping $i/$PING_COUNT: FAILED"
    fi
    sleep "$PING_INTERVAL"
done

# All pings failed — do one final reconnect attempt before alerting
echo "[-] Router not responding after 60 seconds of pings."
echo "[*] Final reconnect attempt..."
"$VPN_SCRIPT" up >/dev/null 2>&1
sleep 5
if "$VPN_SCRIPT" status >/dev/null 2>&1; then
    echo "[+] VPN reconnected — checking router..."
    if ping -c 2 -W 5 "76.195.7.60" >/dev/null 2>&1; then
        echo "[+] Router is back online — canceling alert"
        rm -f ~/.hermes/cron/output/home_router_alert.json
        exit 0
    fi
fi

# Write alert JSON for cron job to read
echo "[-] Alert remains — router confirmed offline"
python3 -c "
import json, os
msg = {
    'alert': 'home_router_down',
    'subject': '🏠 Home Router Offline',
    'message': '''The home router WAN (76.195.7.60) is not responding.

The VPN tunnel was up and no ping replies came through for 60 seconds.

Actions taken:
- VPN was auto-reconnected (if it was down)
- 6 pings to WAN IP attempted over 60 seconds — all failed

Possible causes:
- ISP outage at home
- Power outage at home
- MikroTik crashed/hung
- RouterOS update rebooted

You may need to check from another network or wait for it to come back online.'''
}
path = os.path.expanduser('~/.hermes/cron/output/home_router_alert.json')
os.makedirs(os.path.dirname(path), exist_ok=True)
with open(path, 'w') as f: json.dump(msg, f)
"
exit 1
```

**Key differences from minimal `/opt/` reference version:**
- Prints status messages so cron stdout shows what happened (useful for LLM-driven cron that can see output)
- Final reconnect attempt BEFORE writing the alert, not after — avoids false alerts from transient drops
- Richer alert message with actionable troubleshooting steps in the JSON
- Uses absolute paths under `/root/.hermes/scripts/` not `/opt/`

### Cron Job Setup

The cron job is **LLM-driven** (`no_agent=false`), not a pure script — so it can read the alert file and compose a Telegram message. Use restricted toolsets:

```bash
hermes cron create \
  --name "Home Router Watchdog" \
  --schedule "every 5m" \
  --script "home-router-watchdog.sh" \
  --prompt "Run the watchdog script. If exit 0 = silent. If exit 1 = read alert file and message user." \
  --toolsets "terminal,file,search"
```

**Pitfall:** An LLM-driven cron job expects the script path relative to `/root/.hermes/scripts/`. If you copy the script elsewhere, update the cron definition too.

### Naming Convention

Keep personal and business infrastructure separate:
- Personal home router → `home-router-*` (service, keepalive, watchdog)
- Business WISP infrastructure → `wisp-*` or per-client prefix
- Backup directories match: `/root/.hermes/backups/home-router/`

## Alternative: Uptime Kuma Push Monitor

Instead of raw ping, you can set up an Uptime Kuma "Push" type monitor and have the watchdog hit its URL:

```
https://your-kuma.com/api/push/YOUR_PUSH_TOKEN?status=up&ping=37
```

Push the ping response time as the `ping` parameter. Kuma tracks uptime, graphs latency, and handles alerting. The cron job becomes simpler — just curl the URL.

## Pitfalls

- **ICMP blocked on WAN by default** — Most routers (including MikroTik) block inbound ping on WAN. Add a firewall rule before the default drop rule or use push-based monitoring instead.
- **Ping through VPN tests the tunnel + router, not the internet** — If you want to know whether the router's WAN is reachable, ping the public IP. If you want to know whether the VPN works, ping the LAN IP.
- **2-out-of-6 threshold** — Using 2 successful pings out of 6 over 60 seconds prevents false positives from a single dropped packet. Adjust based on link quality.
- **Alert file must be cleaned up** — If the watchdog writes an alert file but the router comes back before the cron job runs, stale alerts can fire. The watchdog script removes the file on successful reconnection.
- **SSH line length limit on RouterOS** — Ed25519 keys (~92 chars) fit fine within the ~170 char SSH CLI limit. RSA keys will not — use hex encoding or `/file/add contents=` instead.
- **`/tool/fetch` requires `ftp` policy** — Users in `read` or `write` groups cannot use this command. Use `/file/add contents=` for file creation.
- **`/user ssh-keys import` works after `/file/add`** — The file-add approach works on RouterOS 7.18. Create the file first, then import, then clean up.