# Hetzner VPS Provisioning Pattern

Created: 2026-07-06 during Tony's VPS setup.

## Overview

Pattern for provisioning a new Hetzner VPS from scratch, initializing it, and optionally installing Hermes for a user. Used for Tony's CPX21 (tony-vps).

## Steps

### 1. Verify SSH keys in Hetzner project

```bash
TOKEN=$(grep HETZNER_API_TOKEN ~/.hermes/.env | cut -d= -f2)
curl -s -H "Authorization: Bearer $TOKEN" "https://api.hetzner.cloud/v1/ssh_keys" | \
  python3 -c "import sys,json; [print(f'ID {k[\"id\"]}: {k[\"name\"]}') for k in json.load(sys.stdin).get('ssh_keys',[])]"
```

itpp-infra key = ID 114709791 (Ed25519).

### 2. Select server type and OS image

CPX11=2C/2G, CPX21=3C/4G, CPX41=4C/8G.

List Ubuntu 24.04 images: `curl -s -H "Authorization: Bearer $TOKEN" "https://api.hetzner.cloud/v1/images?type=system" | python3 -c "import sys,json; [print(f'ID {i[\"id\"]}: {i[\"name\"]}') for i in json.load(sys.stdin).get('images',[]) if 'ubuntu' in i['name'].lower()]"`

Ubuntu 24.04 = ID 161547269.

### 3. Choose a datacenter

Available: nbg1 (Nuremberg), hel1 (Helsinki), fsn1 (Falkenstein), ash (Ashburn, VA), hil (Hillsboro, OR), sin (Singapore).
Use `"location"` not `"datacenter"` in the API — `datacenter` is invalid.

### 4. Create the server

```bash
curl -s -X POST "https://api.hetzner.cloud/v1/servers" \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name":"tony-vps","server_type":"cpx21","image":161547269,"location":"ash","ssh_keys":[114709791]}'
```

### 5. Delete and recreate (if wrong tier)

```bash
curl -s -X DELETE "https://api.hetzner.cloud/v1/servers/<id>" -H "Authorization: Bearer $TOKEN"
sleep 15
# Then create with correct tier
```

**Pitfall:** "Primary IP limit exceeded" means the old IP hasn't been released yet. Wait 10-15s.

### 6. Wait for SSH

New servers take 20-40 seconds. Poll: `for i in 1 2 3; do timeout 10 ssh -o StrictHostKeyChecking=no -i ~/.ssh/itpp-infra root@<IP> "hostname" 2>&1 && break; sleep 15; done`

### 7. Initial setup

```bash
ssh root@<IP> '
hostnamectl set-hostname <name>
echo "<name>" > /etc/hostname
sed -i "s/127.0.1.1.*/127.0.1.1 <name>/" /etc/hosts
apt-get update -qq && apt-get upgrade -y -qq
timedatectl set-timezone America/New_York
ufw --force enable
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
useradd -m -s /bin/bash <name> 2>/dev/null || true
'
```

### 8. Install Hermes

```bash
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
```

Then configure admin-ai provider with user's API key.

### 9. Local model as backup (CPX21+ only)

Download Llama 3.2 1B Q4 (~771MB) and run a Python HTTP server on localhost:8080.

**Venv pitfall:** Ubuntu 24.04 doesn't ship ensurepip in python3-venv. Install `python3.12-venv` separately.

### 10. Failover heartbeat

Cron job every minute checking admin-ai health. Switches provider in config.yaml if admin-ai is down.

## Pitfalls

- **CPX11 cannot run local model.** Upgrade to CPX21 minimum for fallback.
- **"Primary IP limit exceeded"** means old IP not released yet.
- **User never logs in.** Everything through Telegram.
- **SMTP 587 blocked.** Use 2525.
- **WireGuard: UFW must allow 51820/udp** or no handshake.
- **Tony's setup:** IP 87.99.159.142, key `sk-mKoekhNQxSGda0kXPe_yjQ` with deepseek-chat/v4-pro/v4-flash/GPT-4o/Gemini-3-flash. Needs Telegram bot token to activate gateway.