## 2. Server Standards

### 2.1 Tier Definitions

| Tier | Spec | OS | Use | Cost |
|---|---|---|---|---|
| **Standard** | RS 4000 G12 (12C/32G/1TB) | Debian 13 | Host A, Host B, Host C |  |
| **Light** | RS 2000 G12 (8C/16G/512GB) | Debian 13 | Primary Host |  |
| **Standby** | CPX21 (4C/8G/80GB) | Debian 13 | Standby Host |  |
| **Legacy** | Variable Secondary Provider | Debian 12 | Existing (migrate on rebuild) | -44/mo |

### 2.2 Base Install — Verified Checklist

```bash
# S1: hostname + timezone
hostnamectl set-hostname <name>
timedatectl set-timezone America/New_York

# S2: system update
apt update && apt upgrade -y

# S3: security baseline
apt install -y fail2ban ufw
ufw default deny incoming; ufw default allow outgoing
ufw allow ssh; ufw --force enable

# S4: monitoring
apt install -y prometheus-node-exporter

# S5: standard user + key
adduser [USER-REDACTED] && usermod -aG sudo [USER-REDACTED]
echo "ssh-ed25519 AAA... company-prefix-infra" >> /home/[USER-REDACTED]/.ssh/authorized_keys

# S6: harden SSH
sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
sed -i 's/^PermitRootLogin yes/PermitRootLogin prohibit-password/' /etc/ssh/sshd_config
systemctl restart sshd

# S7: Docker (if needed)
curl -fsSL https://get.docker.com | bash
apt install -y docker-compose-plugin

# S8: Root essentials backup (daily, 3 AM UTC)
scp root-essentials-backup.sh [USER-REDACTED]@<new-server>:/root/
echo "0 3 * * * /root/root-essentials-backup.sh 2>&1 | logger -t root-essentials-backup" | crontab -
```
Backs up: configs, scripts, SSH keys, Caddyfile, systemd units, web files, and project documentation (`/root/projects/`). Excludes databases (covered by separate live sync). Uploads to S3 under `s3://<backup-bucket>/<hostname>/`.

### 2.3 S3 Backup Folder Structure

```
s3://<backup-bucket>/
  ├── <hostname>/              ← Per-server root essentials (daily, 42-100MB)
  ├── live/                     ← State sync (every 15 min)
  ├── hermes-full-backup/       ← Full Hermes tarballs (daily)
  ├── wphost02-backup/          ← Legacy web host (one-time)
  ├── siteground/               ← SFTP site backups
  └── root-backup/              ← Core root essentials (legacy path)
```

### 2.4 Application Data Paths — Per Server

Every server must document:
- What data lives where (volumes, bind mounts, databases, uploads, .env files)
- What can be rebuilt vs what must be restored
- Secrets location
- Caddy/nginx configs
- Cron jobs
- External API dependencies

(See Section 6 — Restore Runbooks for per-server details)

---

## 3. Backup Strategy

### Exception: Root SSH on Admin Servers

Standard policy disables root SSH (`PermitRootLogin no`) on all servers. Exception granted for **app1, app2, app3** — these are infrastructure/admin servers where automated provisioning requires root-level access.

Root access is still restricted to:
- Key-only authentication (no passwords)
- Only from the `itpp-infra` key
- `AllowUsers ippadmin root` — both users allowed
- Password authentication and root login with password remain disabled

This preserves the security baseline while allowing Sho'Nuff to automate deployments without sudo workarounds.

### S7: Docker — Standard path `/docker/<service>/`

All Docker services must be deployed under `/docker/<service>/`, not `/opt/` or `/home/`. This is the canonical path per the Docker Service Deployment standard.

```
mkdir -p /docker/<service>
cd /docker/<service>
# docker-compose.yml + .env live here
docker compose up -d
```