52 lines
3.3 KiB
Markdown
52 lines
3.3 KiB
Markdown
# Bulk SSH Key Injection via Hetzner Rescue Mode
|
|
|
|
When deploying the `itpp-infra` key to multiple servers simultaneously, this pattern saves time over one-at-a-time rescue.
|
|
|
|
## Process
|
|
|
|
1. **Register the SSH key** in the Hetzner project (one-time per key, SKILL.md step 1)
|
|
2. **Enable rescue mode + reboot on all targets** — batch via Python:
|
|
```python
|
|
targets = [(SERVER_ID, "hostname", "IP"), ...]
|
|
for sid, name, ip in targets:
|
|
data = json.dumps({"type": "linux64", "ssh_keys": [KEY_ID]})
|
|
req = urllib.request.Request(
|
|
f'https://api.hetzner.cloud/v1/servers/{sid}/actions/enable_rescue',
|
|
data=data.encode(), headers=headers, method='POST')
|
|
urllib.request.urlopen(req, timeout=15)
|
|
req2 = urllib.request.Request(
|
|
f'https://api.hetzner.cloud/v1/servers/{sid}/actions/reboot',
|
|
data=b'{}', headers=headers, method='POST')
|
|
urllib.request.urlopen(req2, timeout=15)
|
|
```
|
|
3. **Wait 60-90s** — poll SSH port on each:
|
|
```bash
|
|
for ip in "ip1" "ip2" ...; do
|
|
timeout 5 bash -c "echo > /dev/tcp/$ip/22" 2>/dev/null && echo "$ip: UP" || echo "$ip: WAITING"
|
|
done
|
|
```
|
|
4. **Inject key on all** — loop over each rescue host:
|
|
```bash
|
|
PUBKEY=$(cat ~/.ssh/itpp-infra.pub)
|
|
for ip in ...; do
|
|
ssh -o StrictHostKeyChecking=no -i ~/.ssh/itpp-infra root@$ip "
|
|
mount /dev/sda1 /mnt 2>/dev/null || mount /dev/vda1 /mnt
|
|
echo '$PUBKEY' >> /mnt/root/.ssh/authorized_keys
|
|
chmod 600 /mnt/root/.ssh/authorized_keys
|
|
umount /mnt
|
|
"
|
|
done
|
|
```
|
|
5. **Disable rescue + reboot all** (same pattern as step 2, but `disable_rescue` + `reboot`)
|
|
6. **Wait 90s**, then verify SSH access with `hostname` command
|
|
|
|
## Pitfalls
|
|
|
|
- **Not all servers use `sda1`** — Hetzner rescue only has access to the raw block device. Check with `lsblk` on first SSH. On CPX11/CPX21 it's typically `sda1`; on systems with NVMe it could be `nvme0n1p1`. Fall back to `for part in sda1 vda1 nvme0n1p1; do mount /dev/$part /mnt 2>/dev/null && break; done`.
|
|
- **Remount after first mount attempt** — If a partition is already auto-mounted from a previous rescue, the second `mount` fails. Check `mountpoint -q /mnt` before mounting.
|
|
- **Boot order matters** — Enable rescue FIRST, then reboot. Doing them in the wrong order leaves the server off.
|
|
- **`disable_rescue` BEFORE `reboot`** — Reboot with rescue still enabled just boots back into rescue.
|
|
- **Server POST times vary** — CPX11 (~40s) vs CPX41 (~60s). Wait for SSH port before attempting connection.
|
|
- **app1.itpropartner.com (87.99.144.163) has intermittent SSH delay** — port 80 responds quickly but SSH can take an extra 30-60s. Retry with longer timeout.
|
|
- **ai.itpropartner.com (178.156.167.181, CPX41) hosts the LLM proxy** — Rebooting it takes the agent offline. SSH key injection via rescue mode failed on this tier — the `ssh_keys` parameter was accepted (rescue enabled, rebooted) but the key was never authorized, despite the same procedure working on 8 other Hetzner boxes (CPX11, CPX21). Root cause unknown. The key IS registered in the project (ID 114709791) and will work if the server is ever rebuilt. A local ollama+Qwen2.5 7B fallback is installed on the netcup box for maintenance periods. Alternative injection method: use the rescue `root_password` from `enable_rescue` instead of key injection.
|