hermes-skills/skills/devops/reboot-with-health-check/SKILL.md

---
name: reboot-with-health-check
description: "Reboot any server with automated post-reboot verification — SSH availability, hostname check, and configurable service port health checks."
version: 1.2.0
author: ShoNuff
platforms: [linux]
metadata:
  hermes:
    tags: [reboot, maintenance, health-check, ssh, recovery]
---

# Reboot With Health Check

Never reboot a server blind. Always use this procedure to ensure the server and its services come back fully before reporting success or proceeding.

## Script

`/root/.hermes/scripts/reboot-with-check.sh`

Usage:
```bash
# Reboot a server, verify SSH comes back
bash reboot-with-check.sh <hostname|ip>

# Reboot a web server, verify SSH + HTTP + HTTPS come back
bash reboot-with-check.sh 5.161.62.38 80 443

# Reboot a database server, verify SSH + PostgreSQL
bash reboot-with-check.sh 178.156.130.130 5432

# Reboot with SSH + multiple ports
bash reboot-with-check.sh app1-bu.itpropartner.com 22
```

## Procedure

**HARD PRECONDITION — user-approved policy (Jul 5, 2026):**

Before any reboot, shutdown, or power-cycle command:
1. RESOLVE any opaque server identifier (Hetzner API ID, etc.) to hostname + IP
2. STATE both hostname and IP to the user
3. FLAG if the server runs active services (N8N, Postgres, LLM proxy, Docker containers)
4. WAIT for explicit confirmation before executing

Resolution example:
```bash
python3 -c "
import urllib.request, json
token = open('/root/.hermes/scripts/.hetzner_token').read().strip()
s = json.loads(urllib.request.urlopen(
    urllib.request.Request('https://api.hetzner.cloud/v1/servers/<ID>',
    headers={'Authorization': f'Bearer {token}'})
).read())['server']
print(f\"Target: {s['name']} ({s['public_net']['ipv4']['ip']})\")
print(f\"Status: {s['status']}\")
"
```

Then state: "Rebooting [HOSTNAME] ([IP])" and wait for user confirmation. Never rely on mental memory of server IDs — they are opaque integers that look nothing like hostnames.

1. **Declare intent.** State the target hostname AND IP. Do NOT rely on memory of server IDs.
2. **Confirm with user.** If the server runs active services (N8N, database, LLM proxy), flag it before proceeding.
3. **Trigger reboot** via SSH or Hetzner API
4. **Confirm server goes down** — 2 failed pings
5. **Wait for SSH** to come back (up to 120s timeout)
6. **Brief buffer** (5s) for post-reboot stabilization
7. **Verify hostname** — reject if still in rescue mode
8. **Verify each service port** specified (HTTP 80, HTTPS 443, etc.)
9. **Report result** — success or specific failure

## Prerequisites

Before invoking any reboot, you MUST have the target server's **hostname** and **IP address** directly verifiable — not just a server ID or mental note. Hetzner API server IDs are opaque integers that look nothing like their hostnames.

**Hard rule:** State the hostname AND IP to the user before triggering any reboot, shutdown, or power-cycle command. Wait for confirmation if the server runs active services (N8N, databases, LLM proxy, Docker containers with state).

See `references/destructive-action-protocol.md` in the `infrastructure-automation` umbrella for the full user-approved policy.

## Hard preconditions for ANY destructive action

Before ANY destructive or infrastructure-affecting operation (reboot, shutdown, power-cycle, DNS change, config deploy, service restart, firewall rule change, container destroy, API key rotation), you MUST:

1. **Verify the target** — resolve opaque identifiers to hostname + IP. For Hetzner API calls, run the server resolution before the action. Never use mental memory.
2. **State hostname + IP** to the user in the message before executing.
3. **Flag risk** — if the target runs active services (N8N, Postgres, LLM proxy, Docker containers with state), say so explicitly and wait for confirmation.
4. **Do not rely on server IDs alone** — they are visually indistinguishable from one another.

This applies to ALL tools: terminal, write_file (config files), patch (production configs), curl (API POST/DELETE), cronjob (modify), and any script that changes live state.

## Pitfalls

- **WRONG SERVER DISASTER.** Accidentally rebooted `app1.itpropartner.com` (N8N host, ID 127781917, IP 87.99.144.163) thinking it was `app1-bu` (standby, ID 125997675, IP 5.161.114.8). The IDs are close enough to confuse in memory (127781917 vs 125997675) but one runs N8N+Postgres and the other is a dormant standby. **What should have happened:** never reference servers by Hetzner API ID alone. Resolve the ID to hostname + IP, then state BOTH to the user before any power-cycle. Verifiable precondition: `python3 -c "import urllib.request,json; s=json.loads(urllib.request.urlopen(urllib.request.Request('https://api.hetzner.cloud/v1/servers/<ID>', headers={'Authorization':'Bearer ...'})).read())['server']; print(s['name'], s['public_net']['ipv4']['ip'])"` before any destructive action. Applies to app1-bu (standby), app1.itpropartner.com (N8N+Postgres), ai.itpropartner.com (LiteLLM proxy), and any future Hetzner box.
- **Always verify the server is NOT booted into rescue mode** (hostname check)
- **Don't assume SSH = all services** — specify critical service ports in the check
- **For rescue-mode reboots**, disable rescue FIRST, THEN reboot
- **The script uses `itpp-infra` key by default path** — if a different key is needed, specify it
- **AI/LLM proxy servers (admin-ai.itpropartner.com):** Rebooting your active LLM provider takes the agent offline. Before rebooting, either (a) have a local fallback model (ollama) installed and configured, or (b) have the user ready to restart the proxy manually.
- **app1.itpropartner.com (87.99.144.163)** has an intermittent SSH delay — port 80 responds quickly but SSH can take an extra 30-60s after the Hetzner API reports `running`.
- **Hetzner `enable_rescue` SSH key injection is server-type dependent.** Works on CPX11/CPX21, silently fails on CPX41. Workaround: use the rescue root password or have a user add the key manually.
- **Hostname briefly says `rescue`** after disabling rescue and rebooting. Wait for normal SSH to return.