# Bulk SSH Key Injection via Hetzner Rescue Mode

When deploying the `itpp-infra` key to multiple servers simultaneously, this pattern saves time over one-at-a-time rescue.

## Process

1. **Register the SSH key** in the Hetzner project (one-time per key, SKILL.md step 1)
2. **Enable rescue mode + reboot on all targets** — batch via Python:
   ```python
   targets = [(SERVER_ID, "hostname", "IP"), ...]
   for sid, name, ip in targets:
       data = json.dumps({"type": "linux64", "ssh_keys": [KEY_ID]})
       req = urllib.request.Request(
           f'https://api.hetzner.cloud/v1/servers/{sid}/actions/enable_rescue',
           data=data.encode(), headers=headers, method='POST')
       urllib.request.urlopen(req, timeout=15)
       req2 = urllib.request.Request(
           f'https://api.hetzner.cloud/v1/servers/{sid}/actions/reboot',
           data=b'{}', headers=headers, method='POST')
       urllib.request.urlopen(req2, timeout=15)
   ```
3. **Wait 60-90s** — poll SSH port on each:
   ```bash
   for ip in "ip1" "ip2" ...; do
       timeout 5 bash -c "echo > /dev/tcp/$ip/22" 2>/dev/null && echo "$ip: UP" || echo "$ip: WAITING"
   done
   ```
4. **Inject key on all** — loop over each rescue host:
   ```bash
   PUBKEY=$(cat ~/.ssh/itpp-infra.pub)
   for ip in ...; do
       ssh -o StrictHostKeyChecking=no -i ~/.ssh/itpp-infra root@$ip "
           mount /dev/sda1 /mnt 2>/dev/null || mount /dev/vda1 /mnt
           echo '$PUBKEY' >> /mnt/root/.ssh/authorized_keys
           chmod 600 /mnt/root/.ssh/authorized_keys
           umount /mnt
       "
   done
   ```
5. **Disable rescue + reboot all** (same pattern as step 2, but `disable_rescue` + `reboot`)
6. **Wait 90s**, then verify SSH access with `hostname` command

## Pitfalls

- **Not all servers use `sda1`** — Hetzner rescue only has access to the raw block device. Check with `lsblk` on first SSH. On CPX11/CPX21 it's typically `sda1`; on systems with NVMe it could be `nvme0n1p1`. Fall back to `for part in sda1 vda1 nvme0n1p1; do mount /dev/$part /mnt 2>/dev/null && break; done`.
- **Remount after first mount attempt** — If a partition is already auto-mounted from a previous rescue, the second `mount` fails. Check `mountpoint -q /mnt` before mounting.
- **Boot order matters** — Enable rescue FIRST, then reboot. Doing them in the wrong order leaves the server off.
- **`disable_rescue` BEFORE `reboot`** — Reboot with rescue still enabled just boots back into rescue.
- **Server POST times vary** — CPX11 (~40s) vs CPX41 (~60s). Wait for SSH port before attempting connection.
- **app1.itpropartner.com (87.99.144.163) has intermittent SSH delay** — port 80 responds quickly but SSH can take an extra 30-60s. Retry with longer timeout.
- **ai.itpropartner.com (178.156.167.181, CPX41) hosts the LLM proxy** — Rebooting it takes the agent offline. SSH key injection via rescue mode failed on this tier — the `ssh_keys` parameter was accepted (rescue enabled, rebooted) but the key was never authorized, despite the same procedure working on 8 other Hetzner boxes (CPX11, CPX21). Root cause unknown. The key IS registered in the project (ID 114709791) and will work if the server is ever rebuilt. A local ollama+Qwen2.5 7B fallback is installed on the netcup box for maintenance periods. Alternative injection method: use the rescue `root_password` from `enable_rescue` instead of key injection.