hermes-recovery/references/server-provisioning-standard-review.md

ITPP Server Provisioning Standard — Peer Review

Reviewer: Hermes Agent (subagent) Skill reviewed: devops/server-provisioning-standard (v1.0.0) Date: 2026-07-09

---

Executive Summary

The standard is well-structured, the ordering is logical, and the scope (OS provisioning → security hardening → base tooling → monitoring → networking → verification) is appropriate. It would work for someone experienced with the ITPP infrastructure, but has several gaps — some critical (commands that will fail in practice), some moderate (missing steps, incomplete coverage), and some cosmetic (documentation polish). Below is a detailed findings table followed by role-specific analyses.

---

Findings by Section

#	Section	Severity	Issue	Recommendation
F1	§7 — Base Tooling	High	wget URL node_exporter-*.linux-amd64.tar.gz uses a shell glob (*) in a URL. wget does not expand globs in URLs — it requests the literal asterisk, which 404s.	Use the GitHub API to resolve the latest version, or pin a known version (e.g., 1.8.2). Example fix: curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest | jq -r '.tag_name[1:]' to extract version, then construct URL. Or pin 1.8.2 and rely on periodic updates.
F2	§11 — Checklist	High	Checklist item "AWS CLI installed with S3 credentials" and "/root/.aws/credentials chmod 600" refer to S3 credentials, but no step in the standard creates AWS credentials. There is no aws configure, no credential file write, and no mkdir -p /root/.aws.	Add a step after the AWS CLI venv setup to configure credentials (or document that this is done post-provisioning via a secure channel). Also add mkdir -p /root/.aws before chmod.
F3	§7 — Base Tooling	High	apt install docker-compose-plugin depends on the Docker apt repository having been added by the convenience script. Recent versions of get.docker.com install docker-compose-plugin automatically, making this step redundant; but if the script didn't (e.g., older version or proxy issues), the apt install will fail because Docker's repo may not be the only source for this package on Debian 13.	After curl -fsSL https://get.docker.com | sh, run docker compose version to verify instead of a separate apt install. If the plugin is missing, the script get.docker.com can be run with --version flags, or add docker-ce-rootless-extras to the mix.
F4	§6 & §7 — Ordering	Medium	No apt update is run before the apt install commands in sections 6 (Fail2Ban, unattended-upgrades) and 7 (all base packages). On a minimal Debian 13 netcup install, the package cache may be stale or empty, causing apt install to fail or install outdated versions.	Add apt update at the start of §6 (or at the top of §3 before any package ops).
F5	§5, §6 — SSH config split	Medium	SSH password authentication is disabled in §5 (two sed commands on sshd_config), then systemctl restart sshd runs immediately. In §6, additional SSH hardening (PermitRootLogin no, AllowUsers, etc.) is listed as "In sshd_config" but there are no follow-up commands to apply those settings. This means the SSH daemon is restarted twice (once in §5, once in §6) with a configuration that's only partially applied after the first restart.	Consolidate all sshd_config settings into a single block in §6, then restart sshd once. Use a heredoc or sed to apply all settings in one pass, then restart.
F6	§5 — SSH key deploy	Medium	ssh-copy-id -i /root/.ssh/itpp-infra.pub ippadmin@<server-ip> assumes the command is run from Core. If a team member provisions from a different workstation, this path won't exist. Also, there's no instruction on how to obtain the itpp-infra private key onto the provisioning machine.	Either (a) document that provisioning must be done from Core, or (b) provide alternative instructions for agents/workstations that have the key loaded into ssh-agent.
F7	§8 — node_exporter	Medium	The wildcard in wget ... node_exporter-*.linux-amd64.tar.gz also means the tar line's glob pattern may not match the actual downloaded filename. And rm -rf /tmp/node_exporter* removes files matching * under /tmp, but the extractions went to a subdirectory — need to verify the cleanup path.	Combine with F1 fix: use a resolved version string for all three operations (download, extract, clean up).
F8	§9 — Tailscale	Medium	tailscale up --authkey <key> is documented with a placeholder, but there's no guidance on generating or rotating the auth key. Tailscale auth keys are one-time-use unless marked as reusable; if the key expires mid-provisioning, the step fails silently.	Add a prerequisite step: "In Tailscale admin console, generate an auth key tagged for the itpp-infra tag (or appropriate ACL tag) with 'reusable' checked if multiple servers share a key." Also verify with tailscale status afterward.
F9	§10 — DNS	Medium	"Cloudflare DNS: A record for server IP under itpropartner.com" is too terse for a reproducible procedure. No mention of API token, zone ID, or manual steps. Someone who's never done this will be stuck.	Add either (a) a Cloudflare API curl command to create the A record, or (b) a link to the Cloudflare dashboard with step-by-step instructions.
F10	§9 — WireGuard	Medium	"WireGuard configured per-server if direct tunnel to home network needed (10.77.0.0/24)" is mentioned but not implemented. No config file, no wg-quick setup, no key generation.	Either remove the statement (if WireGuard is out of scope for base provisioning) or add the full WireGuard setup procedure as a subsection.
F11	§8 — Monitoring	Low	"Prometheus server on Core auto-discovers via /opt/prometheus/prometheus.yml" is vague. Auto-discovery implies a file_sd_config, DNS SRV records, or a Consul service — none of which are documented. A new admin won't know how to add the new server to Prometheus.	Add a brief note: "On Core, add the new server's IP and port 9100 to the file_sd target list at /opt/prometheus/targets/<server-name>.yml and reload Prometheus with curl -X POST http://localhost:9090/-/reload." Or point to the relevant Prometheus config.
F12	§8 — UFW & node_exporter	Low	Port 9100 (node_exporter) is opened at the system level but there's no UFW rule to restrict it. By default, node_exporter binds to 0.0.0.0:9100, so it's accessible to anyone who knows the IP.	Add ufw allow from <core-ip> to any port 9100 proto tcp before enabling UFW. Or add --web.listen-address=127.0.0.1:9100 to the ExecStart line if only local Prometheus scraping is needed.
F13	§10 — Hostname	Low	hostnamectl set-hostname <server-name> changes the hostname, but /etc/hosts should also be updated so the hostname resolves to the server's IP (avoids sudo slow-DNS warnings and mailname issues).	Add: sed -i "s/127.0.1.1.*/127.0.1.1 <server-name>/" /etc/hosts or a similar fix.
F14	§6 — Unattended upgrades	Low	dpkg-reconfigure -plow unattended-upgrades is interactive and displays a dialog box. On a headless/minimal Debian without debconf-utils, this may hang waiting for user input.	Use echo unattended-upgrades unattended-upgrades/enable_auto_updates boolean true | debconf-set-selections before the install, or configure via /etc/apt/apt.conf.d/20auto-upgrades directly.
F15	§12 — Documentation	Low	§12 says update itpp-recovery-manual.md and app-inventory.csv after provisioning, but doesn't specify what fields to populate in each.	Add a template or field list: e.g., "Add server name, IP, hostname, purpose, provisioning date to app-inventory.csv." Add "Append server's IP, root password location, and SSH key fingerprint to itpp-recovery-manual.md."
F16	§4 — ippadmin creation	Low	adduser ippadmin is interactive (prompts for password, full name, etc.). In an automated/scripted context, this hangs.	Use adduser --disabled-password --gecos "" ippadmin and then passwd ippadmin separately, or use useradd -m -s /bin/bash ippadmin followed by explicit password setting.
F17	§6 — UFW enable	Low	ufw --force enable can still fail if ufw allow ssh wasn't successful (e.g., ssh service name not recognized). Also, UFW may need rsyslog to log denials.	Add ufw logging medium before enabling, and verify with ufw status verbose after enable. This helps debug connectivity issues later.
F18	§7 — NVMe tools on non-NVMe hardware	Low	nvme-cli is installed as a "system tool". The Standby tier (Hetzner CPX11, 40GB) may use standard SSD, not NVMe. This isn't harmful but is worth noting.	Recommend: either keep it (no-op on non-NVMe) or conditionally install based on storage type. Low-priority suggestion.
F19	§7 — Python section order	Low	python3-pip requires python3 (dependency), and python3-venv also depends on python3. Listing all three explicitly is fine, but there's a subtle issue: on Debian minimal, python3-pip may pull in python3 automatically. Not a bug, just slightly redundant.	No change needed — explicit is better than implicit for a standard.
F20	§2 — Ordering	Info	Hetzner CPX11 is listed for the Standby tier, but §2 says "All servers are ordered from netcup". Hetzner and netcup are different providers with different ordering processes.	Clarify that the Standby tier is ordered from Hetzner with a separate ordering process (or note that the source of truth for Hetzner provisioning lives elsewhere).
F21	§1 — Hardware Standard	Info	The RS 4000 G12 and RS 2000 G12 are netcup product names. If netcup changes their product line, this section will need updates.	Consider adding a netcup product page link so someone can verify current specs. No action needed now.
F22	§3 — Initial Access	Info	"servercontrolpanel.de/SCP/" — note this domain is servercontrolpanel.de, not a common domain. A new admin might think it's a typo.	Consider adding "(servercontrolpanel.de)" after the mention. No urgent fix.

---

Role-Specific Analyses

Server Admin — Assessment: Good with gaps

Positive:

• Ordering from hardware specs → ordering → initial access → user creation → SSH → security → tooling → monitoring → networking → verification is a logical, progressive flow.
• ippadmin with sudo, root-only-for-recovery pattern is sound.
• Docker via official convenience script is the recommended approach.

Issues to address (from finding table):

• F4 — Missing apt update before package installs will cause failures on a fresh minimal install.
• F5 — SSH config split across §5 and §6 with a redundant sshd restart means settings may not all land.
• F14 — Unattended upgrades interactive dialog will hang in non-interactive SSH sessions.
• F16 — adduser interactive prompts will block scripting.
• F7 — node_exporter download will fail (shell glob in URL).

Missing base packages to consider:

• ca-certificates (may not be pre-installed on minimal Debian; the Docker convenience script and HTTPS sources need it)
• gnupg (needed for apt-key operations if any)
• systemd-journal-remote or logwatch for centralized log shipping (out of scope but worth mentioning)
• cron — is it pre-installed on minimal Debian 13?

Verdict: Provisioning order is correct. The standard is missing a few critical commands (apt update, F14 fix) and has one major bug (F1/F7). With those fixed, the admin flow is solid.

---

Network Engineer — Assessment: Needs moderate improvements

Positive:

• Tailscale for overlay network is the right choice for a multi-server setup.
• UFW with default-deny and allow-outgoing is correct.
• IPv4 + IPv6 enabled at order time is forward-looking.
• netcat-openbsd, dnsutils, traceroute are the right network diagnostic tools.

Issues to address:

• F8 — Tailscale auth key lifecycle needs documentation.
• F9 — Cloudflare DNS step is too vague; needs reproducible instructions.
• F10 — WireGuard is mentioned but not implemented. Either do it or remove it.
• F12 — node_exporter on 0.0.0.0:9100 is an open port with no firewall restriction.
• F17 — UFW logging should be enabled for troubleshooting.
• F20 — Provider mismatch between "all from netcup" and Hetzner standby tier.

Missing firewall considerations:

• No ICMP rate limiting or consideration (ping flood protection).
• No mention of UFW application profiles vs raw port rules.
• No ufw limit ssh (rate-limiting SSH connections) as an alternative to ufw allow ssh.
• No mention of Docker's iptables interference with UFW (Docker manages iptables/nftables rules which can bypass UFW).

Verdict: The networking foundation is conceptually correct but lacks enough detail for a junior engineer to execute independently. The Cloudflare DNS step is the weakest link.

---

Backup Specialist — Assessment: Significant gaps

Positive:

• S3 credentials are mentioned (but not implemented — see F2).
• Recovery manual update is noted in §12.
• node_exporter provides CPU/memory/disk metrics for capacity planning.

Issues to address:

• F2 — Critical gap: S3 credentials step is completely missing despite being in the checklist.
• F11 — No guidance on how to add the new server to Prometheus discovery on Core.
• F13 — Hostname not in /etc/hosts can cause hostname -f failures that affect monitoring.

Missing backup considerations:

• No backup agent installed (e.g., restic, borg, duplicity, or aws s3 sync). While app-specific backups are out of scope for base provisioning, a note about the recovery strategy per server type would help.
• No log rotation policy for node_exporter or systemd journal. Without journalctl --vacuum-time=7d, logs grow unbounded on the 40GB standby tier.
• No cron job for nightly apt update && apt upgrade -y (the checklist runs it once manually).
• No monitoring threshold for disk space (node_exporter exposes it, but there's no alert rule documented).
• No mention of what data on this server needs backing up at the base OS level (e.g., /etc, /var/lib/docker, /opt).

Verdict: The backup specialist's concerns are the least addressed in this standard. The S3 credential gap (F2) is the most impactful — without it, the checklist is untestable and any S3-based backup plan fails.

---

General & Reproducibility — Assessment: Good skeleton, needs filling

Positive:

• The verification checklist is comprehensive (16 items covering SSH, services, tooling, networking, DNS).
• The standard clearly defines what's in scope ("BEFORE any Docker services, databases, or applications").
• Changelog and versioning are present.
• Exceptions process is documented.

Issues to address:

• Several commands have placeholders without explanation of where to get the values (<server-ip>, <key>, <server-name>). For most this is fine, but <key> in the Tailscale step is opaque.
• The standard assumes the operator is on Core for ssh-copy-id but doesn't say so explicitly until the verification phase.
• The Cloudflare DNS step (F9) is the weakest link for a new team member.
• The app-inventory.csv and recovery-manual updates (F15) lack templates or field definitions.
• No rollback instructions — what if a step fails mid-provisioning? A short "If something goes wrong" section would improve confidence.

Verdict: A senior team member who knows the ITPP infrastructure can successfully provision a server with this standard. A junior or external contractor needs more detail in several sections.

---

Recommended Priority Actions

Must fix before next provisioning (High severity):

1. F1/F7 — Fix node_exporter download to use a resolved version, not a shell glob in the URL.
2. F2 — Add AWS credential configuration step (or clearly document it as post-provisioning with a reference).
3. F4 — Add apt update before package installs.
4. F5 — Consolidate sshd_config changes into one section with one restart.

Should fix soon (Medium severity):

5. F8 — Document Tailscale auth key generation process.
6. F9 — Add Cloudflare DNS creation procedure (API or dashboard).
7. F10 — Either implement or remove WireGuard reference.
8. F12 — Restrict node_exporter port with UFW or bind to localhost.
9. F14 — Make unattended-upgrades config non-interactive.
10. F16 — Use non-interactive adduser flags.

Nice to have (Low severity):

11. F13 — Update /etc/hosts after hostname change.
12. F15 — Add field templates for post-provisioning documentation.
13. F17 — Enable UFW logging.
14. F11 — Document Prometheus target addition process.
15. F20 — Clarify Hetzner ordering vs netcup ordering.

---

Final Verdict

The standard is conceptually sound and covers the right domains, but has 5 high-severity issues (3 that will cause command failures, 1 missing step, 1 ordering bug) and several medium-severity gaps in reproducibility. None of the issues are architectural — they're all fixable with targeted edits. With the "must fix" items addressed, this standard is ready for production use. The verification checklist is a strong asset; once the S3 credential configuration is added, every checklist item maps to an actual provisioning step.