hermes-recovery/references/server-provisioning-standard-public.md

2. Server Standards

2.1 Tier Definitions

Tier	Spec	OS	Use	Cost
Standard	RS 4000 G12 (12C/32G/1TB)	Debian 13	Host A, Host B, Host C
Light	RS 2000 G12 (8C/16G/512GB)	Debian 13	Primary Host
Standby	CPX21 (4C/8G/80GB)	Debian 13	Standby Host
Legacy	Variable Secondary Provider	Debian 12	Existing (migrate on rebuild)	-44/mo

2.2 Base Install — Verified Checklist

# S1: hostname + timezone
hostnamectl set-hostname <name>
timedatectl set-timezone America/New_York

# S2: system update
apt update && apt upgrade -y

# S3: security baseline
apt install -y fail2ban ufw
ufw default deny incoming; ufw default allow outgoing
ufw allow ssh; ufw --force enable

# S4: monitoring
apt install -y prometheus-node-exporter

# S5: standard user + key
adduser [USER-REDACTED] && usermod -aG sudo [USER-REDACTED]
echo "ssh-ed25519 AAA... company-prefix-infra" >> /home/[USER-REDACTED]/.ssh/authorized_keys

# S6: harden SSH
sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
sed -i 's/^PermitRootLogin yes/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
systemctl restart sshd

# S7: Docker (if needed)
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin

# S8: Root essentials backup (daily, 3 AM UTC)
scp root-essentials-backup.sh [USER-REDACTED]@<new-server>:/root/
echo "0 3 * * * /root/root-essentials-backup.sh 2>&1 | logger -t root-essentials-backup" | crontab -

Backs up: configs, scripts, SSH keys, Caddyfile, systemd units, web files, and project documentation (/root/projects/). Excludes databases (covered by separate live sync). Uploads to S3 under s3://<backup-bucket>/<hostname>/.

2.3 S3 Backup Folder Structure

s3://<backup-bucket>/
  ├── <hostname>/              ← Per-server root essentials (daily, 42-100MB)
  ├── live/                     ← State sync (every 15 min)
  ├── hermes-full-backup/       ← Full Hermes tarballs (daily)
  ├── wphost02-backup/          ← Legacy web host (one-time)
  ├── siteground/               ← SFTP site backups
  └── root-backup/              ← Core root essentials (legacy path)

2.4 Application Data Paths — Per Server

Every server must document:

• What data lives where (volumes, bind mounts, databases, uploads, .env files)
• What can be rebuilt vs what must be restored
• Secrets location
• Caddy/nginx configs
• Cron jobs
• External API dependencies

(See Section 6 — Restore Runbooks for per-server details)

---

3. Backup Strategy

Exception: Root SSH on Admin Servers

Standard policy disables root SSH (PermitRootLogin no) on all servers. Exception granted for app1, app2, app3 — these are infrastructure/admin servers where automated provisioning requires root-level access.

Root access is still restricted to:

• Key-only authentication (no passwords)
• Only from the itpp-infra key
• AllowUsers ippadmin root — both users allowed
• Password authentication and root login with password remain disabled

This preserves the security baseline while allowing Sho'Nuff to automate deployments without sudo workarounds.

S7: Docker — Standard path /docker/<service>/

All Docker services must be deployed under /docker/<service>/, not /opt/ or /home/. This is the canonical path per the Docker Service Deployment standard.

mkdir -p /docker/<service>
cd /docker/<service>
# docker-compose.yml + .env live here
docker compose up -d