Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,51 @@
+# Bulk SSH Key Injection via Hetzner Rescue Mode
+
+When deploying the `itpp-infra` key to multiple servers simultaneously, this pattern saves time over one-at-a-time rescue.
+
+## Process
+
+1. **Register the SSH key** in the Hetzner project (one-time per key, SKILL.md step 1)
+2. **Enable rescue mode + reboot on all targets** — batch via Python:
+   ```python
+   targets = [(SERVER_ID, "hostname", "IP"), ...]
+   for sid, name, ip in targets:
+       data = json.dumps({"type": "linux64", "ssh_keys": [KEY_ID]})
+       req = urllib.request.Request(
+           f'https://api.hetzner.cloud/v1/servers/{sid}/actions/enable_rescue',
+           data=data.encode(), headers=headers, method='POST')
+       urllib.request.urlopen(req, timeout=15)
+       req2 = urllib.request.Request(
+           f'https://api.hetzner.cloud/v1/servers/{sid}/actions/reboot',
+           data=b'{}', headers=headers, method='POST')
+       urllib.request.urlopen(req2, timeout=15)
+   ```
+3. **Wait 60-90s** — poll SSH port on each:
+   ```bash
+   for ip in "ip1" "ip2" ...; do
+       timeout 5 bash -c "echo > /dev/tcp/$ip/22" 2>/dev/null && echo "$ip: UP" || echo "$ip: WAITING"
+   done
+   ```
+4. **Inject key on all** — loop over each rescue host:
+   ```bash
+   PUBKEY=$(cat ~/.ssh/itpp-infra.pub)
+   for ip in ...; do
+       ssh -o StrictHostKeyChecking=no -i ~/.ssh/itpp-infra root@$ip "
+           mount /dev/sda1 /mnt 2>/dev/null || mount /dev/vda1 /mnt
+           echo '$PUBKEY' >> /mnt/root/.ssh/authorized_keys
+           chmod 600 /mnt/root/.ssh/authorized_keys
+           umount /mnt
+       "
+   done
+   ```
+5. **Disable rescue + reboot all** (same pattern as step 2, but `disable_rescue` + `reboot`)
+6. **Wait 90s**, then verify SSH access with `hostname` command
+
+## Pitfalls
+
+- **Not all servers use `sda1`** — Hetzner rescue only has access to the raw block device. Check with `lsblk` on first SSH. On CPX11/CPX21 it's typically `sda1`; on systems with NVMe it could be `nvme0n1p1`. Fall back to `for part in sda1 vda1 nvme0n1p1; do mount /dev/$part /mnt 2>/dev/null && break; done`.
+- **Remount after first mount attempt** — If a partition is already auto-mounted from a previous rescue, the second `mount` fails. Check `mountpoint -q /mnt` before mounting.
+- **Boot order matters** — Enable rescue FIRST, then reboot. Doing them in the wrong order leaves the server off.
+- **`disable_rescue` BEFORE `reboot`** — Reboot with rescue still enabled just boots back into rescue.
+- **Server POST times vary** — CPX11 (~40s) vs CPX41 (~60s). Wait for SSH port before attempting connection.
+- **app1.itpropartner.com (87.99.144.163) has intermittent SSH delay** — port 80 responds quickly but SSH can take an extra 30-60s. Retry with longer timeout.
+- **ai.itpropartner.com (178.156.167.181, CPX41) hosts the LLM proxy** — Rebooting it takes the agent offline. SSH key injection via rescue mode failed on this tier — the `ssh_keys` parameter was accepted (rescue enabled, rebooted) but the key was never authorized, despite the same procedure working on 8 other Hetzner boxes (CPX11, CPX21). Root cause unknown. The key IS registered in the project (ID 114709791) and will work if the server is ever rebuilt. A local ollama+Qwen2.5 7B fallback is installed on the netcup box for maintenance periods. Alternative injection method: use the rescue `root_password` from `enable_rescue` instead of key injection.