hermes-skills/skills/devops/hermes-standby-deployment/references/bulk-hetzner-rescue.md

Bulk SSH Key Injection via Hetzner Rescue Mode

When deploying the itpp-infra key to multiple servers simultaneously, this pattern saves time over one-at-a-time rescue.

Process

1. Register the SSH key in the Hetzner project (one-time per key, SKILL.md step 1)
2. Enable rescue mode + reboot on all targets — batch via Python:

   targets = [(SERVER_ID, "hostname", "IP"), ...]
   for sid, name, ip in targets:
       data = json.dumps({"type": "linux64", "ssh_keys": [KEY_ID]})
       req = urllib.request.Request(
           f'https://api.hetzner.cloud/v1/servers/{sid}/actions/enable_rescue',
           data=data.encode(), headers=headers, method='POST')
       urllib.request.urlopen(req, timeout=15)
       req2 = urllib.request.Request(
           f'https://api.hetzner.cloud/v1/servers/{sid}/actions/reboot',
           data=b'{}', headers=headers, method='POST')
       urllib.request.urlopen(req2, timeout=15)
   

3. Wait 60-90s — poll SSH port on each:

   for ip in "ip1" "ip2" ...; do
       timeout 5 bash -c "echo > /dev/tcp/$ip/22" 2>/dev/null && echo "$ip: UP" || echo "$ip: WAITING"
   done
   

4. Inject key on all — loop over each rescue host:

   PUBKEY=$(cat ~/.ssh/itpp-infra.pub)
   for ip in ...; do
       ssh -o StrictHostKeyChecking=no -i ~/.ssh/itpp-infra root@$ip "
           mount /dev/sda1 /mnt 2>/dev/null || mount /dev/vda1 /mnt
           echo '$PUBKEY' >> /mnt/root/.ssh/authorized_keys
           chmod 600 /mnt/root/.ssh/authorized_keys
           umount /mnt
       "
   done
   

5. Disable rescue + reboot all (same pattern as step 2, but disable_rescue + reboot)
6. Wait 90s, then verify SSH access with hostname command

Pitfalls

• Not all servers use sda1 — Hetzner rescue only has access to the raw block device. Check with lsblk on first SSH. On CPX11/CPX21 it's typically sda1; on systems with NVMe it could be nvme0n1p1. Fall back to for part in sda1 vda1 nvme0n1p1; do mount /dev/$part /mnt 2>/dev/null && break; done.
• Remount after first mount attempt — If a partition is already auto-mounted from a previous rescue, the second mount fails. Check mountpoint -q /mnt before mounting.
• Boot order matters — Enable rescue FIRST, then reboot. Doing them in the wrong order leaves the server off.
• disable_rescue BEFORE reboot — Reboot with rescue still enabled just boots back into rescue.
• Server POST times vary — CPX11 (~40s) vs CPX41 (~60s). Wait for SSH port before attempting connection.
• app1.itpropartner.com (87.99.144.163) has intermittent SSH delay — port 80 responds quickly but SSH can take an extra 30-60s. Retry with longer timeout.
• ai.itpropartner.com (178.156.167.181, CPX41) hosts the LLM proxy — Rebooting it takes the agent offline. SSH key injection via rescue mode failed on this tier — the ssh_keys parameter was accepted (rescue enabled, rebooted) but the key was never authorized, despite the same procedure working on 8 other Hetzner boxes (CPX11, CPX21). Root cause unknown. The key IS registered in the project (ID 114709791) and will work if the server is ever rebuilt. A local ollama+Qwen2.5 7B fallback is installed on the netcup box for maintenance periods. Alternative injection method: use the rescue root_password from enable_rescue instead of key injection.