Initial resurrection kit — 81 scripts, 62 references, configs, systemd units, crons, Docker compose files, Caddy config, master README
This commit is contained in:
root
@@ -0,0 +1,165 @@
|
||||
# ITPP Server Provisioning Standard — Peer Review
|
||||
|
||||
**Reviewer:** Hermes Agent (subagent)
|
||||
**Skill reviewed:** `devops/server-provisioning-standard` (v1.0.0)
|
||||
**Date:** 2026-07-09
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
The standard is well-structured, the ordering is logical, and the scope (OS provisioning → security hardening → base tooling → monitoring → networking → verification) is appropriate. It would work for someone experienced with the ITPP infrastructure, but has several gaps — some critical (commands that will fail in practice), some moderate (missing steps, incomplete coverage), and some cosmetic (documentation polish). Below is a detailed findings table followed by role-specific analyses.
|
||||
|
||||
---
|
||||
|
||||
## Findings by Section
|
||||
|
||||
| # | Section | Severity | Issue | Recommendation |
|
||||
|---|---------|----------|-------|---------------|
|
||||
| F1 | §7 — Base Tooling | **High** | `wget` URL `node_exporter-*.linux-amd64.tar.gz` uses a shell glob (`*`) in a URL. `wget` does not expand globs in URLs — it requests the literal asterisk, which 404s. | Use the GitHub API to resolve the latest version, or pin a known version (e.g., `1.8.2`). Example fix: `curl -s https://api.github.com/repos/prometheus/node_exporter/releases/latest \| jq -r '.tag_name[1:]'` to extract version, then construct URL. Or pin `1.8.2` and rely on periodic updates. |
|
||||
| F2 | §11 — Checklist | **High** | Checklist item "AWS CLI installed with S3 credentials" and "`/root/.aws/credentials chmod 600`" refer to S3 credentials, but **no step in the standard creates AWS credentials**. There is no `aws configure`, no credential file write, and no `mkdir -p /root/.aws`. | Add a step after the AWS CLI venv setup to configure credentials (or document that this is done post-provisioning via a secure channel). Also add `mkdir -p /root/.aws` before `chmod`. |
|
||||
| F3 | §7 — Base Tooling | **High** | `apt install docker-compose-plugin` depends on the Docker apt repository having been added by the convenience script. Recent versions of `get.docker.com` install `docker-compose-plugin` automatically, making this step redundant; but if the script didn't (e.g., older version or proxy issues), the `apt install` will fail because Docker's repo may not be the only source for this package on Debian 13. | After `curl -fsSL https://get.docker.com \| sh`, run `docker compose version` to verify instead of a separate `apt install`. If the plugin is missing, the script `get.docker.com` can be run with `--version` flags, or add `docker-ce-rootless-extras` to the mix. |
|
||||
| F4 | §6 & §7 — Ordering | **Medium** | No `apt update` is run before the `apt install` commands in sections 6 (Fail2Ban, unattended-upgrades) and 7 (all base packages). On a minimal Debian 13 netcup install, the package cache may be stale or empty, causing `apt install` to fail or install outdated versions. | Add `apt update` at the start of §6 (or at the top of §3 before any package ops). |
|
||||
| F5 | §5, §6 — SSH config split | **Medium** | SSH password authentication is disabled in §5 (two `sed` commands on sshd_config), then `systemctl restart sshd` runs immediately. In §6, additional SSH hardening (`PermitRootLogin no`, `AllowUsers`, etc.) is listed as "In sshd_config" but there are no follow-up commands to apply those settings. This means the SSH daemon is restarted twice (once in §5, once in §6) with a configuration that's only partially applied after the first restart. | Consolidate all sshd_config settings into a single block in §6, then restart `sshd` once. Use a heredoc or `sed` to apply all settings in one pass, then restart. |
|
||||
| F6 | §5 — SSH key deploy | **Medium** | `ssh-copy-id -i /root/.ssh/itpp-infra.pub ippadmin@<server-ip>` assumes the command is run **from Core**. If a team member provisions from a different workstation, this path won't exist. Also, there's no instruction on how to obtain the `itpp-infra` private key onto the provisioning machine. | Either (a) document that provisioning must be done from Core, or (b) provide alternative instructions for agents/workstations that have the key loaded into ssh-agent. |
|
||||
| F7 | §8 — node_exporter | **Medium** | The wildcard in `wget ... node_exporter-*.linux-amd64.tar.gz` also means the `tar` line's glob pattern may not match the actual downloaded filename. And `rm -rf /tmp/node_exporter*` removes files matching `*` under /tmp, but the extractions went to a subdirectory — need to verify the cleanup path. | Combine with F1 fix: use a resolved version string for all three operations (download, extract, clean up). |
|
||||
| F8 | §9 — Tailscale | **Medium** | `tailscale up --authkey <key>` is documented with a placeholder, but there's no guidance on generating or rotating the auth key. Tailscale auth keys are one-time-use unless marked as reusable; if the key expires mid-provisioning, the step fails silently. | Add a prerequisite step: "In Tailscale admin console, generate an auth key tagged for the `itpp-infra` tag (or appropriate ACL tag) with 'reusable' checked if multiple servers share a key." Also verify with `tailscale status` afterward. |
|
||||
| F9 | §10 — DNS | **Medium** | "Cloudflare DNS: A record for server IP under itpropartner.com" is too terse for a reproducible procedure. No mention of API token, zone ID, or manual steps. Someone who's never done this will be stuck. | Add either (a) a Cloudflare API curl command to create the A record, or (b) a link to the Cloudflare dashboard with step-by-step instructions. |
|
||||
| F10 | §9 — WireGuard | **Medium** | "WireGuard configured per-server if direct tunnel to home network needed (10.77.0.0/24)" is mentioned but not implemented. No config file, no `wg-quick` setup, no key generation. | Either remove the statement (if WireGuard is out of scope for base provisioning) or add the full WireGuard setup procedure as a subsection. |
|
||||
| F11 | §8 — Monitoring | **Low** | "Prometheus server on Core auto-discovers via /opt/prometheus/prometheus.yml" is vague. Auto-discovery implies a file_sd_config, DNS SRV records, or a Consul service — none of which are documented. A new admin won't know how to add the new server to Prometheus. | Add a brief note: "On Core, add the new server's IP and port 9100 to the file_sd target list at `/opt/prometheus/targets/<server-name>.yml` and reload Prometheus with `curl -X POST http://localhost:9090/-/reload`." Or point to the relevant Prometheus config. |
|
||||
| F12 | §8 — UFW & node_exporter | **Low** | Port 9100 (node_exporter) is opened at the system level but there's no UFW rule to restrict it. By default, node_exporter binds to `0.0.0.0:9100`, so it's accessible to anyone who knows the IP. | Add `ufw allow from <core-ip> to any port 9100 proto tcp` before enabling UFW. Or add `--web.listen-address=127.0.0.1:9100` to the ExecStart line if only local Prometheus scraping is needed. |
|
||||
| F13 | §10 — Hostname | **Low** | `hostnamectl set-hostname <server-name>` changes the hostname, but `/etc/hosts` should also be updated so the hostname resolves to the server's IP (avoids `sudo` slow-DNS warnings and mailname issues). | Add: `sed -i "s/127.0.1.1.*/127.0.1.1 <server-name>/" /etc/hosts` or a similar fix. |
|
||||
| F14 | §6 — Unattended upgrades | **Low** | `dpkg-reconfigure -plow unattended-upgrades` is interactive and displays a dialog box. On a headless/minimal Debian without `debconf-utils`, this may hang waiting for user input. | Use `echo unattended-upgrades unattended-upgrades/enable_auto_updates boolean true \| debconf-set-selections` before the install, or configure via `/etc/apt/apt.conf.d/20auto-upgrades` directly. |
|
||||
| F15 | §12 — Documentation | **Low** | §12 says update `itpp-recovery-manual.md` and `app-inventory.csv` after provisioning, but doesn't specify what fields to populate in each. | Add a template or field list: e.g., "Add server name, IP, hostname, purpose, provisioning date to `app-inventory.csv`." Add "Append server's IP, root password location, and SSH key fingerprint to `itpp-recovery-manual.md`." |
|
||||
| F16 | §4 — ippadmin creation | **Low** | `adduser ippadmin` is interactive (prompts for password, full name, etc.). In an automated/scripted context, this hangs. | Use `adduser --disabled-password --gecos "" ippadmin` and then `passwd ippadmin` separately, or use `useradd -m -s /bin/bash ippadmin` followed by explicit password setting. |
|
||||
| F17 | §6 — UFW enable | **Low** | `ufw --force enable` can still fail if `ufw allow ssh` wasn't successful (e.g., ssh service name not recognized). Also, UFW may need `rsyslog` to log denials. | Add `ufw logging medium` before enabling, and verify with `ufw status verbose` after enable. This helps debug connectivity issues later. |
|
||||
| F18 | §7 — NVMe tools on non-NVMe hardware | **Low** | `nvme-cli` is installed as a "system tool". The Standby tier (Hetzner CPX11, 40GB) may use standard SSD, not NVMe. This isn't harmful but is worth noting. | Recommend: either keep it (no-op on non-NVMe) or conditionally install based on storage type. Low-priority suggestion. |
|
||||
| F19 | §7 — Python section order | **Low** | `python3-pip` requires `python3` (dependency), and `python3-venv` also depends on `python3`. Listing all three explicitly is fine, but there's a subtle issue: on Debian minimal, `python3-pip` may pull in `python3` automatically. Not a bug, just slightly redundant. | No change needed — explicit is better than implicit for a standard. |
|
||||
| F20 | §2 — Ordering | **Info** | Hetzner CPX11 is listed for the Standby tier, but §2 says "All servers are ordered from **netcup**". Hetzner and netcup are different providers with different ordering processes. | Clarify that the Standby tier is ordered from Hetzner with a separate ordering process (or note that the source of truth for Hetzner provisioning lives elsewhere). |
|
||||
| F21 | §1 — Hardware Standard | **Info** | The RS 4000 G12 and RS 2000 G12 are netcup product names. If netcup changes their product line, this section will need updates. | Consider adding a netcup product page link so someone can verify current specs. No action needed now. |
|
||||
| F22 | §3 — Initial Access | **Info** | "servercontrolpanel.de/SCP/" — note this domain is `servercontrolpanel.de`, not a common domain. A new admin might think it's a typo. | Consider adding "(servercontrolpanel.de)" after the mention. No urgent fix. |
|
||||
|
||||
---
|
||||
|
||||
## Role-Specific Analyses
|
||||
|
||||
### Server Admin — Assessment: Good with gaps
|
||||
|
||||
**Positive:**
|
||||
- Ordering from hardware specs → ordering → initial access → user creation → SSH → security → tooling → monitoring → networking → verification is a logical, progressive flow.
|
||||
- ippadmin with sudo, root-only-for-recovery pattern is sound.
|
||||
- Docker via official convenience script is the recommended approach.
|
||||
|
||||
**Issues to address (from finding table):**
|
||||
- **F4** — Missing `apt update` before package installs will cause failures on a fresh minimal install.
|
||||
- **F5** — SSH config split across §5 and §6 with a redundant `sshd restart` means settings may not all land.
|
||||
- **F14** — Unattended upgrades interactive dialog will hang in non-interactive SSH sessions.
|
||||
- **F16** — `adduser` interactive prompts will block scripting.
|
||||
- **F7** — node_exporter download will fail (shell glob in URL).
|
||||
|
||||
**Missing base packages to consider:**
|
||||
- `ca-certificates` (may not be pre-installed on minimal Debian; the Docker convenience script and HTTPS sources need it)
|
||||
- `gnupg` (needed for apt-key operations if any)
|
||||
- `systemd-journal-remote` or `logwatch` for centralized log shipping (out of scope but worth mentioning)
|
||||
- `cron` — is it pre-installed on minimal Debian 13?
|
||||
|
||||
**Verdict:** Provisioning order is correct. The standard is missing a few critical commands (`apt update`, F14 fix) and has one major bug (F1/F7). With those fixed, the admin flow is solid.
|
||||
|
||||
---
|
||||
|
||||
### Network Engineer — Assessment: Needs moderate improvements
|
||||
|
||||
**Positive:**
|
||||
- Tailscale for overlay network is the right choice for a multi-server setup.
|
||||
- UFW with default-deny and allow-outgoing is correct.
|
||||
- IPv4 + IPv6 enabled at order time is forward-looking.
|
||||
- `netcat-openbsd`, `dnsutils`, `traceroute` are the right network diagnostic tools.
|
||||
|
||||
**Issues to address:**
|
||||
- **F8** — Tailscale auth key lifecycle needs documentation.
|
||||
- **F9** — Cloudflare DNS step is too vague; needs reproducible instructions.
|
||||
- **F10** — WireGuard is mentioned but not implemented. Either do it or remove it.
|
||||
- **F12** — node_exporter on 0.0.0.0:9100 is an open port with no firewall restriction.
|
||||
- **F17** — UFW logging should be enabled for troubleshooting.
|
||||
- **F20** — Provider mismatch between "all from netcup" and Hetzner standby tier.
|
||||
|
||||
**Missing firewall considerations:**
|
||||
- No ICMP rate limiting or consideration (ping flood protection).
|
||||
- No mention of UFW application profiles vs raw port rules.
|
||||
- No `ufw limit ssh` (rate-limiting SSH connections) as an alternative to `ufw allow ssh`.
|
||||
- No mention of Docker's iptables interference with UFW (Docker manages iptables/nftables rules which can bypass UFW).
|
||||
|
||||
**Verdict:** The networking foundation is conceptually correct but lacks enough detail for a junior engineer to execute independently. The Cloudflare DNS step is the weakest link.
|
||||
|
||||
---
|
||||
|
||||
### Backup Specialist — Assessment: Significant gaps
|
||||
|
||||
**Positive:**
|
||||
- S3 credentials are mentioned (but not implemented — see F2).
|
||||
- Recovery manual update is noted in §12.
|
||||
- node_exporter provides CPU/memory/disk metrics for capacity planning.
|
||||
|
||||
**Issues to address:**
|
||||
- **F2** — Critical gap: S3 credentials step is completely missing despite being in the checklist.
|
||||
- **F11** — No guidance on how to add the new server to Prometheus discovery on Core.
|
||||
- **F13** — Hostname not in `/etc/hosts` can cause `hostname -f` failures that affect monitoring.
|
||||
|
||||
**Missing backup considerations:**
|
||||
- No backup agent installed (e.g., `restic`, `borg`, `duplicity`, or `aws s3 sync`). While app-specific backups are out of scope for base provisioning, a note about the recovery strategy per server type would help.
|
||||
- No log rotation policy for node_exporter or systemd journal. Without `journalctl --vacuum-time=7d`, logs grow unbounded on the 40GB standby tier.
|
||||
- No cron job for nightly `apt update && apt upgrade -y` (the checklist runs it once manually).
|
||||
- No monitoring threshold for disk space (node_exporter exposes it, but there's no alert rule documented).
|
||||
- No mention of what data on this server needs backing up at the base OS level (e.g., `/etc`, `/var/lib/docker`, `/opt`).
|
||||
|
||||
**Verdict:** The backup specialist's concerns are the least addressed in this standard. The S3 credential gap (F2) is the most impactful — without it, the checklist is untestable and any S3-based backup plan fails.
|
||||
|
||||
---
|
||||
|
||||
### General & Reproducibility — Assessment: Good skeleton, needs filling
|
||||
|
||||
**Positive:**
|
||||
- The verification checklist is comprehensive (16 items covering SSH, services, tooling, networking, DNS).
|
||||
- The standard clearly defines what's in scope ("BEFORE any Docker services, databases, or applications").
|
||||
- Changelog and versioning are present.
|
||||
- Exceptions process is documented.
|
||||
|
||||
**Issues to address:**
|
||||
- Several commands have placeholders without explanation of where to get the values (`<server-ip>`, `<key>`, `<server-name>`). For most this is fine, but `<key>` in the Tailscale step is opaque.
|
||||
- The standard assumes the operator is on Core for `ssh-copy-id` but doesn't say so explicitly until the verification phase.
|
||||
- The Cloudflare DNS step (F9) is the weakest link for a new team member.
|
||||
- The app-inventory.csv and recovery-manual updates (F15) lack templates or field definitions.
|
||||
- No rollback instructions — what if a step fails mid-provisioning? A short "If something goes wrong" section would improve confidence.
|
||||
|
||||
**Verdict:** A senior team member who knows the ITPP infrastructure can successfully provision a server with this standard. A junior or external contractor needs more detail in several sections.
|
||||
|
||||
---
|
||||
|
||||
## Recommended Priority Actions
|
||||
|
||||
### Must fix before next provisioning (High severity):
|
||||
1. **F1/F7** — Fix node_exporter download to use a resolved version, not a shell glob in the URL.
|
||||
2. **F2** — Add AWS credential configuration step (or clearly document it as post-provisioning with a reference).
|
||||
3. **F4** — Add `apt update` before package installs.
|
||||
4. **F5** — Consolidate sshd_config changes into one section with one restart.
|
||||
|
||||
### Should fix soon (Medium severity):
|
||||
5. **F8** — Document Tailscale auth key generation process.
|
||||
6. **F9** — Add Cloudflare DNS creation procedure (API or dashboard).
|
||||
7. **F10** — Either implement or remove WireGuard reference.
|
||||
8. **F12** — Restrict node_exporter port with UFW or bind to localhost.
|
||||
9. **F14** — Make unattended-upgrades config non-interactive.
|
||||
10. **F16** — Use non-interactive `adduser` flags.
|
||||
|
||||
### Nice to have (Low severity):
|
||||
11. **F13** — Update `/etc/hosts` after hostname change.
|
||||
12. **F15** — Add field templates for post-provisioning documentation.
|
||||
13. **F17** — Enable UFW logging.
|
||||
14. **F11** — Document Prometheus target addition process.
|
||||
15. **F20** — Clarify Hetzner ordering vs netcup ordering.
|
||||
|
||||
---
|
||||
|
||||
## Final Verdict
|
||||
|
||||
**The standard is conceptually sound and covers the right domains, but has 5 high-severity issues (3 that will cause command failures, 1 missing step, 1 ordering bug) and several medium-severity gaps in reproducibility.** None of the issues are architectural — they're all fixable with targeted edits. With the "must fix" items addressed, this standard is ready for production use. The verification checklist is a strong asset; once the S3 credential configuration is added, every checklist item maps to an actual provisioning step.
|
||||
Reference in New Issue
Block a user