Initial resurrection kit — 81 scripts, 62 references, configs, systemd units, crons, Docker compose files, Caddy config, master README

2026-07-15 17:45:36 -04:00
commit ae056eaf83
197 changed files with 26127 additions and 0 deletions
@@ -0,0 +1,137 @@
+# IT Pro Partner — Server Provisioning Standard v1.0
+
+**Author:** Sho'Nuff / Network Services Team
+**Date:** July 9, 2026
+**Status:** DRAFT — pending review
+
+---
+
+## 1. Server Tier Definitions
+
+| Tier | Spec | OS | Use Case | Monthly Cost |
+|------|------|----|----------|-------------|
+| **Standard** | RS 4000 G12 (12C EPYC, 32GB RAM, 1TB NVMe) | Debian 13 | app1, app2, app3 | ~$44/mo |
+| **Light** | RS 2000 G12 (8C EPYC, 16GB RAM, 512GB NVMe) | Debian 13 | Core (existing) | ~$24/mo |
+| **Standby** | CPX11 (2C/2G/40GB) | Debian 12 | app1-bu (warm standby) | ~$8/mo Hetzner |
+| **Legacy** | CPX21 (3C/4G/80GB) or CPX41 | Debian 12 | Existing Hetzner boxes | ~$14-44/mo |
+
+---
+
+## 2. OS Provisioning
+
+### OS Selection
+- **netcup servers:** Debian 13 (bookworm) — latest stable
+- **Hetzner servers:** Debian 12 (existing, migrate to 13 on rebuild)
+- **Reasoning:** Debian over Ubuntu for stability, no forced upgrades, smaller footprint
+
+### Base Install
+```bash
+# Set hostname
+hostnamectl set-hostname <server-name>
+echo "<server-name>" > /etc/hostname
+
+# Update everything
+apt update && apt upgrade -y
+
+# Timezone
+timedatectl set-timezone America/New_York
+
+# Fail2ban
+apt install -y fail2ban
+
+# Monitoring agent
+apt install -y prometheus-node-exporter
+
+# Firewall
+apt install -y ufw
+ufw default deny incoming
+ufw default allow outgoing
+ufw allow ssh
+ufw --force enable
+```
+
+### Standard Users
+```bash
+# ippadmin — standard admin user for all servers
+adduser ippadmin
+usermod -aG sudo ippadmin
+mkdir -p /home/ippadmin/.ssh
+
+# Deploy itpp-infra SSH key (Hetzner project-wide key)
+echo "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAII4dxTH11aJkBqCY8lXl1kTfZ8yXWhTcthHnt1MtAuIE itpp-infra" \
+  >> /home/ippadmin/.ssh/authorized_keys
+
+# Disable password auth for SSH
+sed -i 's/^#PasswordAuthentication yes/PasswordAuthentication no/' /etc/ssh/sshd_config
+systemctl restart sshd
+```
+
+### Key Deployment
+- **itpp-infra** — standard admin key, deployed to all servers (Hetzner project-wide via SSH vault)
+- **Root SSH disabled** after initial setup — use `ippadmin` with `sudo`
+
+---
+
+## 3. Standard Services
+
+Every server gets:
+
+| Service | Purpose | Installed By |
+|---------|---------|-------------|
+| fail2ban | Brute force protection | Base install |
+| prometheus-node-exporter | Metrics for monitoring | Base install |
+| ufw | Host firewall | Base install |
+| Docker (if needed) | Container runtime | Per-server |
+| Caddy (if serving web) | Reverse proxy + TLS | Per-server |
+
+---
+
+## 4. Docker Standard
+
+### Installation
+```bash
+curl -fsSL https://get.docker.com | bash
+apt install -y docker-compose-plugin
+```
+
+### Docker Compose
+- All stacks at `/opt/<service>/docker-compose.yml`
+- Standard network: bridge (default)
+- Volume mounts under `/opt/<service>/data/`
+- .env files at `/opt/<service>/.env` (chmod 600)
+
+---
+
+## 5. Backup Standard
+
+| Data | Frequency | Location | Retention |
+|------|-----------|----------|-----------|
+| System config | Daily | s3://hermes-vps-backups/<server>/ | 90 days |
+| Docker volumes | Daily | s3://hermes-vps-backups/<server>/ | 90 days |
+| Database dumps | Daily | s3://hermes-vps-backups/<server>/ | 90 days |
+| Application data | Per-application | s3://hermes-vps-backups/<server>/ | 90 days |
+| Hermes state | Every 15 min | s3://hermes-vps-backups/live/ | 48h (snapshots) |
+
+---
+
+## 6. DR Restore Procedure (Standard)
+
+### Standard Restore
+```bash
+# 1. Provision new server (same tier or better)
+# 2. Run base install steps (Section 2)
+# 3. Deploy SSH key
+# 4. Restore from S3:
+aws s3 cp s3://hermes-vps-backups/<server>/latest/ /opt/restore/ \
+  --endpoint-url https://s3.us-east-1.wasabisys.com --recursive
+# 5. Restore Docker stacks
+cd /opt/<service> && docker compose up -d
+# 6. Verify health endpoints
+```
+
+### Emergency Restore (No S3 Access)
+```bash
+# Use Hetzner rescue mode / netcup recovery
+# Reinstall OS, deploy SSH keys
+# Rebuild from documented configs in Hudu
+```