hermes-recovery/references/server-provisioning-standard-v1.md

IT Pro Partner — Server Provisioning Standard v1.0

Author: Sho'Nuff / Network Services Team Date: July 9, 2026 Status: DRAFT — pending review

---

1. Server Tier Definitions

Tier	Spec	OS	Use Case	Monthly Cost
Standard	RS 4000 G12 (12C EPYC, 32GB RAM, 1TB NVMe)	Debian 13	app1, app2, app3	~$44/mo
Light	RS 2000 G12 (8C EPYC, 16GB RAM, 512GB NVMe)	Debian 13	Core (existing)	~$24/mo
Standby	CPX11 (2C/2G/40GB)	Debian 12	app1-bu (warm standby)	~$8/mo Hetzner
Legacy	CPX21 (3C/4G/80GB) or CPX41	Debian 12	Existing Hetzner boxes	~$14-44/mo

---

2. OS Provisioning

OS Selection

• netcup servers: Debian 13 (bookworm) — latest stable
• Hetzner servers: Debian 12 (existing, migrate to 13 on rebuild)
• Reasoning: Debian over Ubuntu for stability, no forced upgrades, smaller footprint

Base Install

# Set hostname
hostnamectl set-hostname <server-name>
echo "<server-name>" > /etc/hostname

# Update everything
apt update && apt upgrade -y

# Timezone
timedatectl set-timezone America/New_York

# Fail2ban
apt install -y fail2ban

# Monitoring agent
apt install -y prometheus-node-exporter

# Firewall
apt install -y ufw
ufw default deny incoming
ufw default allow outgoing
ufw allow ssh
ufw --force enable

Standard Users

# ippadmin — standard admin user for all servers
adduser ippadmin
usermod -aG sudo ippadmin
mkdir -p /home/ippadmin/.ssh

# Deploy itpp-infra SSH key (Hetzner project-wide key)
echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII4dxTH11aJkBqCY8lXl1kTfZ8yXWhTcthHnt1MtAuIE itpp-infra" \
  >> /home/ippadmin/.ssh/authorized_keys

# Disable password auth for SSH
sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart sshd

Key Deployment

• itpp-infra — standard admin key, deployed to all servers (Hetzner project-wide via SSH vault)
• Root SSH disabled after initial setup — use ippadmin with sudo

---

3. Standard Services

Every server gets:

Service	Purpose	Installed By
fail2ban	Brute force protection	Base install
prometheus-node-exporter	Metrics for monitoring	Base install
ufw	Host firewall	Base install
Docker (if needed)	Container runtime	Per-server
Caddy (if serving web)	Reverse proxy + TLS	Per-server

---

4. Docker Standard

Installation

curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin

Docker Compose

• All stacks at /opt/<service>/docker-compose.yml
• Standard network: bridge (default)
• Volume mounts under /opt/<service>/data/
• .env files at /opt/<service>/.env (chmod 600)

---

5. Backup Standard

Data	Frequency	Location	Retention
System config	Daily	s3://hermes-vps-backups//	90 days
Docker volumes	Daily	s3://hermes-vps-backups//	90 days
Database dumps	Daily	s3://hermes-vps-backups//	90 days
Application data	Per-application	s3://hermes-vps-backups//	90 days
Hermes state	Every 15 min	s3://hermes-vps-backups/live/	48h (snapshots)

---

6. DR Restore Procedure (Standard)

Standard Restore

# 1. Provision new server (same tier or better)
# 2. Run base install steps (Section 2)
# 3. Deploy SSH key
# 4. Restore from S3:
aws s3 cp s3://hermes-vps-backups/<server>/latest/ /opt/restore/ \
  --endpoint-url https://s3.us-east-1.wasabisys.com --recursive
# 5. Restore Docker stacks
cd /opt/<service> && docker compose up -d
# 6. Verify health endpoints

Emergency Restore (No S3 Access)

# Use Hetzner rescue mode / netcup recovery
# Reinstall OS, deploy SSH keys
# Rebuild from documented configs in Hudu