hermes-skills/skills/hetzner-server-provisioning/SKILL.md

---
name: hetzner-server-provisioning
description: "Provision Hetzner Cloud servers via API — create CPX/CX instances, inject SSH keys, choose datacenters, set up Ubuntu, and deliver credentials. Part of the IT Pro Partner infrastructure automation umbrella."
version: 1.2.0
author: Sho'Nuff
platforms: [linux]
metadata:
  hermes:
    tags: [hetzner, server, provisioning, cloud, vps]
---

# Hetzner Server Provisioning

Provision new Hetzner Cloud servers using the REST API. For IT Pro Partner infrastructure expansion (app3, Tony's VPS, etc.).

## Prerequisites

- Hetzner API token in `~/.hermes/.env` as `HETZNER_API_TOKEN`
- SSH key(s) registered in Hetzner project: `itpp-infra` (ID 114709791)
- `curl` + `python3` for API calls

## API endpoint

```
POST https://api.hetzner.cloud/v1/servers
GET  https://api.hetzner.cloud/v1/ssh_keys
GET  https://api.hetzner.cloud/v1/images?type=system
GET  https://api.hetzner.cloud/v1/datacenters
GET  https://api.hetzner.cloud/v1/locations
GET  https://api.hetzner.cloud/v1/servers/{id}
```

## Server types (common)

| Type | vCPU | RAM | Disk | Former price | Current API price (Jul 2026) |
|------|------|-----|------|-------------|------------------------------|
| CPX11 | 2 | 2 GB | 40 GB | ~€3.79 | **~€17.49/mo** |
| CPX21 | 3 | 4 GB | 80 GB | ~€7.85 | **~€31.99/mo** |
| CPX31 | 4 | 8 GB | 160 GB | ~€15.59 | **~€62.49/mo** |
| CPX32 | 4 | 8 GB | 160 GB | — | **~€35.49/mo** |
| CPX41 | 8 | 16 GB | 240 GB | — | **~€120.49/mo** |
| CPX42 | 8 | 16 GB | 320 GB | — | **~€69.49/mo** |

**⚠️ Prices shown are from live API — significantly inflated vs. historical rates.** For new servers, strongly consider netcup as a cheaper alternative. The consolidation plan (move Hetzner workloads to netcup) avoids inflated pricing entirely.

## Image IDs (Ubuntu)

- `161547269` — Ubuntu 24.04 (current)
- `161547270` — Ubuntu 24.04 (fallback)
- `103908130` — Ubuntu 22.04

## Locations

| Code | City | Latency to US East |
|------|------|-------------------|
| ash | Ashburn, VA | ~10ms |
| hil | Hillsboro, OR | ~60ms |
| nbg | Nuremberg, DE | ~90ms |
| fsn | Falkenstein, DE | ~100ms |
| hel | Helsinki, FI | ~120ms |
| sin | Singapore | ~200ms |

For US-based users, prefer **ash** (Ashburn). For European proximity, prefer **fsn** or **nbg**.

## Provision payload

```bash
curl -s -X POST "https://api.hetzner.cloud/v1/servers" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "server-name",
    "server_type": "cpx11",
    "image": 161547269,
    "location": "ash",
    "ssh_keys": [114709791]
  }'
```

**Important:** Use `"location": "ash"` not `"datacenter": "ash-dc1"`. The API rejects `datacenter` as a field. Use `location` with the two-letter city code.

## Server swap (delete + recreate)

To upgrade from one server type to another (e.g. CPX11 -> CPX21):

1. Call `DELETE /v1/servers/{id}` and note the response status
2. Wait at least **15 seconds** — the IP release is asynchronous and the Hetzner API enforces a per-project primary IP limit
3. If the create call returns "Primary IP limit exceeded", wait another 10 seconds and retry. The old IP may still be "in-flight"
4. The same IP address is typically reused once released — convenient for DNS, but SSH won't connect for another 20-30s while the new server boots

**Pitfall — timing is tight.** The "Primary IP limit exceeded" error is NOT an actual limit — it's a race condition where the old IP hasn't been released yet. Wait longer between delete and create. Do not change the IP or request a new one — reusing the freed IP is the goal.

## Post-provision steps

1. Wait ~20s for the server to initialize (up to 45s for the first SSH connection to succeed)
2. SSH in with `ssh -o StrictHostKeyChecking=no -i ~/.ssh/itpp-infra root@<ip>`
3. Verify hostname and uptime
4. Run initial setup:
   ```bash
   hostnamectl set-hostname <name>
   echo "<name>" > /etc/hostname
   sed -i "s/127.0.1.1.*/127.0.1.1 <name>/" /etc/hosts
   apt-get update -qq && apt-get upgrade -y -qq
   timedatectl set-timezone America/New_York
   ufw --force enable
   ufw default deny incoming
   ufw default allow outgoing
   ufw allow ssh
   useradd -m -s /bin/bash <username> 2>/dev/null || true
   ```
5. Verify: `hostnamectl`, `free -h`, `df -h /`, `ufw status verbose`

## Hermes deployment for a new user

When provisioning a VPS for someone else to use as a personal Hermes agent:

### Server spec considerations
- **CPX11 (2C/2GB)** is the cheapest option but has NO room for local models. Hermes + gateway fits, but any additional service (Docker, local LLM) will push it over.
- **CPX21 (3C/4GB)** is the correct choice if there's any chance of running a local fallback model or Docker services. The extra $4.50/mo removes the "no room for anything else" constraint.
- If in doubt, choose CPX21 over CPX11. The 4GB headroom is the difference between "runs Hermes" and "can experiment."

### Model architecture for a new user
- **Primary model:** An admin-ai API key provisioned by the main user (DeepSeek Chat for conversation, GPT-4o or Gemini for vision if needed)
- **Local fallback:** A quantized Llama 3.2 1B (Q4_K_M, ~771MB) running as a systemd service on `127.0.0.1:8080`
- **Failover:** A heartbeat script (system cron, every 60s) that checks admin-ai health. If down, switches Hermes' provider to local. When admin-ai comes back, switches back and alerts.

### Installing Hermes
```bash
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
```

Then configure `~/.hermes/config.yaml` with the admin-ai provider pointing at `https://admin-ai.itpropartner.com/v1` and the user's API key.

### Local Llama model setup
```bash
# Download model
mkdir -p /root/models
curl -sL "https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf" -o /root/models/llama-3.2-1b-q4.gguf

# Install Python deps (venv is required — Ubuntu 24 has PEP 668)
apt-get install -y python3.12-venv
python3 -m venv /opt/llama-venv
/opt/llama-venv/bin/pip3 install llama-cpp-python --break-system-packages

# Create server script at /usr/local/bin/llama-server.py
# (OpenAI-compatible HTTP server on :8080 using llama-cpp-python)

# Create systemd service
cat > /etc/systemd/system/llama-server.service << 'EOF'
[Unit]
Description=Local LLM fallback (Llama 3.2 1B)
After=network.target

[Service]
Type=simple
ExecStart=/opt/llama-venv/bin/python3 /usr/local/bin/llama-server.py
Restart=on-failure
RestartSec=5
User=root

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now llama-server
```

**Pitfall — PEP 668:** Ubuntu 24 prevents `pip install` outside a venv. Either create a venv or pass `--break-system-packages`. The venv approach is cleaner. The `python3.12-venv` package IS required on minimal Hetzner Ubuntu images — it is NOT installed by default.

**Pitfall — llama-server binary:** Do NOT download the pre-built `llama-server` binary from GitHub releases. It often fails silently (downloads a corrupt file). Use `pip install llama-cpp-python` instead and run a Python wrapper script (`/usr/local/bin/llama-server.py`) that creates an HTTP server using the `llama_cpp.Llama` class. This is more reliable, easier to restart as a systemd service, and simpler to debug.

### Heartbeat failover script
```bash
cat > /usr/local/bin/hermes-heartbeat.sh << 'EOF'
#!/bin/bash
# Checks admin-ai health. Falls back to local model if down.
CONFIG="$HOME/.hermes/config.yaml"
CURRENT=$(grep "^  provider:" "$CONFIG" | head -1 | awk "{print \$2}")

if curl -sf -o /dev/null --connect-timeout 5 https://admin-ai.itpropartner.com/v1/models -H "Authorization: Bearer $TOKEN"; then
  if [ "$CURRENT" != "admin-ai" ]; then
    sed -i "s/^  provider:.*/  provider: admin-ai/" "$CONFIG"
    echo "Switched back to admin-ai"
  fi
else
  if [ "$CURRENT" != "admin-ai" ]; then
    exit 0  # already on fallback
  fi
  if curl -sf -o /dev/null --connect-timeout 5 http://127.0.0.1:8080/v1/models; then
    sed -i "s/^  provider:.*/  provider: local/" "$CONFIG"
    echo "admin-ai down, switched to local Llama"
  else
    echo "BOTH models unreachable!"
  fi
fi
EOF
chmod +x /usr/local/bin/hermes-heartbeat.sh
echo "* * * * * /usr/local/bin/hermes-heartbeat.sh" | crontab -
```

### Access model for other-user instances

When provisioning a Hermes instance for someone OTHER than the main user:

- Their Hermes does NOT connect back to the main user's Hermes
- The main user CAN connect to theirs for emergency SSH access only
- They interact ONLY through Telegram — they never SSH in
- Their server credentials are stored for emergency-only use by the main user
- The welcome email should NOT include SSH IP, local model port, or server details
- Their server details file goes in `/root/.hermes/references/<name>-vps.txt` (main user's reference only)

### Welcome email pattern

When the main user asks you to draft and send a welcome email to a new Hermes user:

1. Send a draft to the main user for approval FIRST (BCC them)
2. After approval, send to the new user's email
3. Follow the email composition style guide from the `email-workflows` skill

The email should cover:
- What Hermes is (personal AI agent on their own server)
- Real examples of what the main user has accomplished with it
- Email section (ask if they want their email set up; explicit "permission required" statement)
- Backup & Safety (multi-layer, tested)
- Three-step Telegram setup guide (BotFather, userinfobot, reply with info)
- Commands reference (/queue, /retry, /undo, /clear, /title, /goal, /compress)
- Minimal server details (primary model provided by main user, backup built in)

## Verification

```bash
ssh -o StrictHostKeyChecking=no -i ~/.ssh/itpp-infra root@<ip> "hostname && uptime && echo OK"
```

## Pitfalls

- **`datacenter` is not a valid field** — Use `location` with the two-letter code. The API docs show datacenter but the create endpoint rejects it. Sending `"datacenter": "ash-dc1"` returns `"Error: invalid input in field 'datacenter'"`. Fix: use `"location": "ash"`.
- **Primary IP limit on re-provision** — If you delete a server and immediately recreate with the same project, you may hit "Primary IP limit exceeded" for a few seconds while the old IP is released. The Hetzner API enforces a hard limit on primary IPs per project. Deleting frees the IP, but the release and recreation cycle has a ~5-10s delay. Wait at least **15 seconds** (tested: 5s was too short, 10s was enough on retry) before retrying the create call, or the same IP may still be "in-flight" from the deletion. After a successful wait the same IP is typically reused.
- **Same IP reuse** — When you delete and recreate, Hetzner reuses the freed IP within seconds. The new server gets the same IP address, which is convenient for DNS but confusing during the initialization window (SSH will refuse connection for ~10-20s while the new server boots).
- **SSH key injection:** The API auto-injects keys by ID. No need for user-data scripts or post-provision key copying.
- **Server name uniqueness:** Hetzner enforces unique names within a project. Choose something that won't collide.
- **Image IDs change with new Ubuntu releases** — Verify the latest Ubuntu 24.04 image ID with `GET /v1/images?type=system` before provisioning.
- **No `file` command on fresh Ubuntu:** Use `ls -l` or `stat` instead.
- **First SSH attempt may fail:** Even after API says "running," the server may need 20-30s to boot and accept connections. Retry with a sleep loop.
- **`python3.12-venv` NOT installed on minimal Hetzner Ubuntu images** — This package is NOT present by default on Hetzner Cloud Ubuntu 24.04 images. Attempting `python3 -m venv` without it fails silently. Always install it: `apt-get install -y python3.12-venv`. Otherwise the venv is created without `pip` and the `bin/` directory has no `pip3` or `python3` symlinks.