# ITPP Server Provisioning Standard — Peer Review **Reviewer:** Hermes Agent (subagent) **Skill reviewed:** `devops/server-provisioning-standard` (v1.0.0) **Date:** 2026-07-09 --- ## Executive Summary The standard is well-structured, the ordering is logical, and the scope (OS provisioning → security hardening → base tooling → monitoring → networking → verification) is appropriate. It would work for someone experienced with the ITPP infrastructure, but has several gaps — some critical (commands that will fail in practice), some moderate (missing steps, incomplete coverage), and some cosmetic (documentation polish). Below is a detailed findings table followed by role-specific analyses. --- ## Findings by Section | # | Section | Severity | Issue | Recommendation | |---|---------|----------|-------|---------------| | F1 | §7 — Base Tooling | **High** | `wget` URL `node_exporter-*.linux-amd64.tar.gz` uses a shell glob (`*`) in a URL. `wget` does not expand globs in URLs — it requests the literal asterisk, which 404s. | Use the GitHub API to resolve the latest version, or pin a known version (e.g., `1.8.2`). Example fix: `curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest \| jq -r '.tag_name[1:]'` to extract version, then construct URL. Or pin `1.8.2` and rely on periodic updates. | | F2 | §11 — Checklist | **High** | Checklist item "AWS CLI installed with S3 credentials" and "`/root/.aws/credentials chmod 600`" refer to S3 credentials, but **no step in the standard creates AWS credentials**. There is no `aws configure`, no credential file write, and no `mkdir -p /root/.aws`. | Add a step after the AWS CLI venv setup to configure credentials (or document that this is done post-provisioning via a secure channel). Also add `mkdir -p /root/.aws` before `chmod`. | | F3 | §7 — Base Tooling | **High** | `apt install docker-compose-plugin` depends on the Docker apt repository having been added by the convenience script. Recent versions of `get.docker.com` install `docker-compose-plugin` automatically, making this step redundant; but if the script didn't (e.g., older version or proxy issues), the `apt install` will fail because Docker's repo may not be the only source for this package on Debian 13. | After `curl -fsSL https://get.docker.com \| sh`, run `docker compose version` to verify instead of a separate `apt install`. If the plugin is missing, the script `get.docker.com` can be run with `--version` flags, or add `docker-ce-rootless-extras` to the mix. | | F4 | §6 & §7 — Ordering | **Medium** | No `apt update` is run before the `apt install` commands in sections 6 (Fail2Ban, unattended-upgrades) and 7 (all base packages). On a minimal Debian 13 netcup install, the package cache may be stale or empty, causing `apt install` to fail or install outdated versions. | Add `apt update` at the start of §6 (or at the top of §3 before any package ops). | | F5 | §5, §6 — SSH config split | **Medium** | SSH password authentication is disabled in §5 (two `sed` commands on sshd_config), then `systemctl restart sshd` runs immediately. In §6, additional SSH hardening (`PermitRootLogin no`, `AllowUsers`, etc.) is listed as "In sshd_config" but there are no follow-up commands to apply those settings. This means the SSH daemon is restarted twice (once in §5, once in §6) with a configuration that's only partially applied after the first restart. | Consolidate all sshd_config settings into a single block in §6, then restart `sshd` once. Use a heredoc or `sed` to apply all settings in one pass, then restart. | | F6 | §5 — SSH key deploy | **Medium** | `ssh-copy-id -i /root/.ssh/itpp-infra.pub ippadmin@` assumes the command is run **from Core**. If a team member provisions from a different workstation, this path won't exist. Also, there's no instruction on how to obtain the `itpp-infra` private key onto the provisioning machine. | Either (a) document that provisioning must be done from Core, or (b) provide alternative instructions for agents/workstations that have the key loaded into ssh-agent. | | F7 | §8 — node_exporter | **Medium** | The wildcard in `wget ... node_exporter-*.linux-amd64.tar.gz` also means the `tar` line's glob pattern may not match the actual downloaded filename. And `rm -rf /tmp/node_exporter*` removes files matching `*` under /tmp, but the extractions went to a subdirectory — need to verify the cleanup path. | Combine with F1 fix: use a resolved version string for all three operations (download, extract, clean up). | | F8 | §9 — Tailscale | **Medium** | `tailscale up --authkey ` is documented with a placeholder, but there's no guidance on generating or rotating the auth key. Tailscale auth keys are one-time-use unless marked as reusable; if the key expires mid-provisioning, the step fails silently. | Add a prerequisite step: "In Tailscale admin console, generate an auth key tagged for the `itpp-infra` tag (or appropriate ACL tag) with 'reusable' checked if multiple servers share a key." Also verify with `tailscale status` afterward. | | F9 | §10 — DNS | **Medium** | "Cloudflare DNS: A record for server IP under itpropartner.com" is too terse for a reproducible procedure. No mention of API token, zone ID, or manual steps. Someone who's never done this will be stuck. | Add either (a) a Cloudflare API curl command to create the A record, or (b) a link to the Cloudflare dashboard with step-by-step instructions. | | F10 | §9 — WireGuard | **Medium** | "WireGuard configured per-server if direct tunnel to home network needed (10.77.0.0/24)" is mentioned but not implemented. No config file, no `wg-quick` setup, no key generation. | Either remove the statement (if WireGuard is out of scope for base provisioning) or add the full WireGuard setup procedure as a subsection. | | F11 | §8 — Monitoring | **Low** | "Prometheus server on Core auto-discovers via /opt/prometheus/prometheus.yml" is vague. Auto-discovery implies a file_sd_config, DNS SRV records, or a Consul service — none of which are documented. A new admin won't know how to add the new server to Prometheus. | Add a brief note: "On Core, add the new server's IP and port 9100 to the file_sd target list at `/opt/prometheus/targets/.yml` and reload Prometheus with `curl -X POST http://localhost:9090/-/reload`." Or point to the relevant Prometheus config. | | F12 | §8 — UFW & node_exporter | **Low** | Port 9100 (node_exporter) is opened at the system level but there's no UFW rule to restrict it. By default, node_exporter binds to `0.0.0.0:9100`, so it's accessible to anyone who knows the IP. | Add `ufw allow from to any port 9100 proto tcp` before enabling UFW. Or add `--web.listen-address=127.0.0.1:9100` to the ExecStart line if only local Prometheus scraping is needed. | | F13 | §10 — Hostname | **Low** | `hostnamectl set-hostname ` changes the hostname, but `/etc/hosts` should also be updated so the hostname resolves to the server's IP (avoids `sudo` slow-DNS warnings and mailname issues). | Add: `sed -i "s/127.0.1.1.*/127.0.1.1 /" /etc/hosts` or a similar fix. | | F14 | §6 — Unattended upgrades | **Low** | `dpkg-reconfigure -plow unattended-upgrades` is interactive and displays a dialog box. On a headless/minimal Debian without `debconf-utils`, this may hang waiting for user input. | Use `echo unattended-upgrades unattended-upgrades/enable_auto_updates boolean true \| debconf-set-selections` before the install, or configure via `/etc/apt/apt.conf.d/20auto-upgrades` directly. | | F15 | §12 — Documentation | **Low** | §12 says update `itpp-recovery-manual.md` and `app-inventory.csv` after provisioning, but doesn't specify what fields to populate in each. | Add a template or field list: e.g., "Add server name, IP, hostname, purpose, provisioning date to `app-inventory.csv`." Add "Append server's IP, root password location, and SSH key fingerprint to `itpp-recovery-manual.md`." | | F16 | §4 — ippadmin creation | **Low** | `adduser ippadmin` is interactive (prompts for password, full name, etc.). In an automated/scripted context, this hangs. | Use `adduser --disabled-password --gecos "" ippadmin` and then `passwd ippadmin` separately, or use `useradd -m -s /bin/bash ippadmin` followed by explicit password setting. | | F17 | §6 — UFW enable | **Low** | `ufw --force enable` can still fail if `ufw allow ssh` wasn't successful (e.g., ssh service name not recognized). Also, UFW may need `rsyslog` to log denials. | Add `ufw logging medium` before enabling, and verify with `ufw status verbose` after enable. This helps debug connectivity issues later. | | F18 | §7 — NVMe tools on non-NVMe hardware | **Low** | `nvme-cli` is installed as a "system tool". The Standby tier (Hetzner CPX11, 40GB) may use standard SSD, not NVMe. This isn't harmful but is worth noting. | Recommend: either keep it (no-op on non-NVMe) or conditionally install based on storage type. Low-priority suggestion. | | F19 | §7 — Python section order | **Low** | `python3-pip` requires `python3` (dependency), and `python3-venv` also depends on `python3`. Listing all three explicitly is fine, but there's a subtle issue: on Debian minimal, `python3-pip` may pull in `python3` automatically. Not a bug, just slightly redundant. | No change needed — explicit is better than implicit for a standard. | | F20 | §2 — Ordering | **Info** | Hetzner CPX11 is listed for the Standby tier, but §2 says "All servers are ordered from **netcup**". Hetzner and netcup are different providers with different ordering processes. | Clarify that the Standby tier is ordered from Hetzner with a separate ordering process (or note that the source of truth for Hetzner provisioning lives elsewhere). | | F21 | §1 — Hardware Standard | **Info** | The RS 4000 G12 and RS 2000 G12 are netcup product names. If netcup changes their product line, this section will need updates. | Consider adding a netcup product page link so someone can verify current specs. No action needed now. | | F22 | §3 — Initial Access | **Info** | "servercontrolpanel.de/SCP/" — note this domain is `servercontrolpanel.de`, not a common domain. A new admin might think it's a typo. | Consider adding "(servercontrolpanel.de)" after the mention. No urgent fix. | --- ## Role-Specific Analyses ### Server Admin — Assessment: Good with gaps **Positive:** - Ordering from hardware specs → ordering → initial access → user creation → SSH → security → tooling → monitoring → networking → verification is a logical, progressive flow. - ippadmin with sudo, root-only-for-recovery pattern is sound. - Docker via official convenience script is the recommended approach. **Issues to address (from finding table):** - **F4** — Missing `apt update` before package installs will cause failures on a fresh minimal install. - **F5** — SSH config split across §5 and §6 with a redundant `sshd restart` means settings may not all land. - **F14** — Unattended upgrades interactive dialog will hang in non-interactive SSH sessions. - **F16** — `adduser` interactive prompts will block scripting. - **F7** — node_exporter download will fail (shell glob in URL). **Missing base packages to consider:** - `ca-certificates` (may not be pre-installed on minimal Debian; the Docker convenience script and HTTPS sources need it) - `gnupg` (needed for apt-key operations if any) - `systemd-journal-remote` or `logwatch` for centralized log shipping (out of scope but worth mentioning) - `cron` — is it pre-installed on minimal Debian 13? **Verdict:** Provisioning order is correct. The standard is missing a few critical commands (`apt update`, F14 fix) and has one major bug (F1/F7). With those fixed, the admin flow is solid. --- ### Network Engineer — Assessment: Needs moderate improvements **Positive:** - Tailscale for overlay network is the right choice for a multi-server setup. - UFW with default-deny and allow-outgoing is correct. - IPv4 + IPv6 enabled at order time is forward-looking. - `netcat-openbsd`, `dnsutils`, `traceroute` are the right network diagnostic tools. **Issues to address:** - **F8** — Tailscale auth key lifecycle needs documentation. - **F9** — Cloudflare DNS step is too vague; needs reproducible instructions. - **F10** — WireGuard is mentioned but not implemented. Either do it or remove it. - **F12** — node_exporter on 0.0.0.0:9100 is an open port with no firewall restriction. - **F17** — UFW logging should be enabled for troubleshooting. - **F20** — Provider mismatch between "all from netcup" and Hetzner standby tier. **Missing firewall considerations:** - No ICMP rate limiting or consideration (ping flood protection). - No mention of UFW application profiles vs raw port rules. - No `ufw limit ssh` (rate-limiting SSH connections) as an alternative to `ufw allow ssh`. - No mention of Docker's iptables interference with UFW (Docker manages iptables/nftables rules which can bypass UFW). **Verdict:** The networking foundation is conceptually correct but lacks enough detail for a junior engineer to execute independently. The Cloudflare DNS step is the weakest link. --- ### Backup Specialist — Assessment: Significant gaps **Positive:** - S3 credentials are mentioned (but not implemented — see F2). - Recovery manual update is noted in §12. - node_exporter provides CPU/memory/disk metrics for capacity planning. **Issues to address:** - **F2** — Critical gap: S3 credentials step is completely missing despite being in the checklist. - **F11** — No guidance on how to add the new server to Prometheus discovery on Core. - **F13** — Hostname not in `/etc/hosts` can cause `hostname -f` failures that affect monitoring. **Missing backup considerations:** - No backup agent installed (e.g., `restic`, `borg`, `duplicity`, or `aws s3 sync`). While app-specific backups are out of scope for base provisioning, a note about the recovery strategy per server type would help. - No log rotation policy for node_exporter or systemd journal. Without `journalctl --vacuum-time=7d`, logs grow unbounded on the 40GB standby tier. - No cron job for nightly `apt update && apt upgrade -y` (the checklist runs it once manually). - No monitoring threshold for disk space (node_exporter exposes it, but there's no alert rule documented). - No mention of what data on this server needs backing up at the base OS level (e.g., `/etc`, `/var/lib/docker`, `/opt`). **Verdict:** The backup specialist's concerns are the least addressed in this standard. The S3 credential gap (F2) is the most impactful — without it, the checklist is untestable and any S3-based backup plan fails. --- ### General & Reproducibility — Assessment: Good skeleton, needs filling **Positive:** - The verification checklist is comprehensive (16 items covering SSH, services, tooling, networking, DNS). - The standard clearly defines what's in scope ("BEFORE any Docker services, databases, or applications"). - Changelog and versioning are present. - Exceptions process is documented. **Issues to address:** - Several commands have placeholders without explanation of where to get the values (``, ``, ``). For most this is fine, but `` in the Tailscale step is opaque. - The standard assumes the operator is on Core for `ssh-copy-id` but doesn't say so explicitly until the verification phase. - The Cloudflare DNS step (F9) is the weakest link for a new team member. - The app-inventory.csv and recovery-manual updates (F15) lack templates or field definitions. - No rollback instructions — what if a step fails mid-provisioning? A short "If something goes wrong" section would improve confidence. **Verdict:** A senior team member who knows the ITPP infrastructure can successfully provision a server with this standard. A junior or external contractor needs more detail in several sections. --- ## Recommended Priority Actions ### Must fix before next provisioning (High severity): 1. **F1/F7** — Fix node_exporter download to use a resolved version, not a shell glob in the URL. 2. **F2** — Add AWS credential configuration step (or clearly document it as post-provisioning with a reference). 3. **F4** — Add `apt update` before package installs. 4. **F5** — Consolidate sshd_config changes into one section with one restart. ### Should fix soon (Medium severity): 5. **F8** — Document Tailscale auth key generation process. 6. **F9** — Add Cloudflare DNS creation procedure (API or dashboard). 7. **F10** — Either implement or remove WireGuard reference. 8. **F12** — Restrict node_exporter port with UFW or bind to localhost. 9. **F14** — Make unattended-upgrades config non-interactive. 10. **F16** — Use non-interactive `adduser` flags. ### Nice to have (Low severity): 11. **F13** — Update `/etc/hosts` after hostname change. 12. **F15** — Add field templates for post-provisioning documentation. 13. **F17** — Enable UFW logging. 14. **F11** — Document Prometheus target addition process. 15. **F20** — Clarify Hetzner ordering vs netcup ordering. --- ## Final Verdict **The standard is conceptually sound and covers the right domains, but has 5 high-severity issues (3 that will cause command failures, 1 missing step, 1 ordering bug) and several medium-severity gaps in reproducibility.** None of the issues are architectural — they're all fixable with targeted edits. With the "must fix" items addressed, this standard is ready for production use. The verification checklist is a strong asset; once the S3 credential configuration is added, every checklist item maps to an actual provisioning step.