hermes-skills/skills/devops/infrastructure-automation/references/multi-server-inventory-audit.md

Multi-Server Inventory Audit Pattern

Walkthrough for performing a comprehensive SSH-based audit across a fleet of servers.

Approach

1. Create a self-contained audit script that collects everything in a single SSH session per host:

   • System info: hostname, OS, kernel, uptime, load, CPU cores/model, RAM, disk usage
   • Docker: docker ps (names, images, status, ports), docker images (repositories, tags, sizes), docker-compose.yml locations
   • Systemd: all running services, custom services under /etc/systemd/system/ (filter out base services like getty, sshd, cron, dbus, systemd-*, networking)
   • Listening ports: ss -tlnp (TCP), ss -ulnp (UDP) — note which services/pids own each port
   • Web servers: which binary exists (nginx/apache2/caddy), vhost/site configs, Caddyfile excerpts
   • Databases: which binaries exist (mysql, mysqld, psql, mongod, redis-server), which systemd services are active
   • Users: list /home/* directories, note non-standard users vs root/standard
   • Cron: crontab -l for root
   • Custom scripts: ls /root/*.sh /root/*.py, ls /usr/local/bin/*.sh /usr/local/bin/*.py

2. Run in parallel — all hosts are independent, so batch the SSH calls with background processes or a simple loop with & and wait. Use the common SSH key (itpp-infra at ~/.ssh/itpp-infra).

3. Compile the report as a single markdown file under /root/.hermes/references/<name>.md with:

   • Quick summary table (hostname, IP, specs, OS, Docker Y/N, key apps)
   • Per-server sections with the full field-by-field inventory
   • Cross-cutting summary tables (Docker hosts, non-Docker hosts, database engines across fleet, VPN types, disk warnings)

Audit Script

Location: /root/.hermes/scripts/audit-server.sh

# Usage
/root/.hermes/scripts/audit-server.sh <ip> [username]

The script collects all fields listed above in a single SSH call. Output is clearly delimited with --- SECTION --- headers for parsing.

Common Fields Per Server Entry

Each inventory entry should document:

Field	Source
IP	Known address
Hostname	hostname
Plan type	From known inventory (CPX11, CPX21, etc.)
CPU/RAM/Disk	nproc, free -h, df -h /
OS + Kernel	/etc/os-release, uname -a
Uptime + Load	uptime
Docker status	docker ps, docker images, compose file locations
Systemd services	systemctl list-units --type=service --state=running
Custom systemd	/etc/systemd/system/*.service minus base services
TCP listening	ss -tlnp
UDP listening	ss -ulnp
Web servers	binary check + config files
Databases	binary check + systemd status
Users	/home/* + /etc/passwd filtered for home dirs
Cron	crontab -l
Custom scripts	ls /root/*.sh /root/*.py /usr/local/bin/*.sh /usr/local/bin/*.py

Disk Warning Flags

Servers above 70% disk usage should be flagged. At 90%+, mark as critical.

Known disk warnings (as of 2026-07-09):

• ai.itpropartner.com (178.156.167.181): 92% — 199G of 226G used. Potential cause: large Ollama model images (10GB+), stale Docker images, old Open WebUI versions.
• docker (178.156.168.35): 73% — 26G of 38G used. Small disk (CPX11), trending up.

Full Audit Snapshot

The live full audit report lives at /root/.hermes/references/hetzner-server-audit.md — last refreshed 2026-07-09 covering all 10 servers.

Pitfalls

• Known hosts keys: The first SSH to a new server prompts for host key verification. Use -o StrictHostKeyChecking=no for automation, but be aware of the security trade-off.
• Large output: Some commands (like dpkg -l | grep) can produce long output. Keep queries targeted.
• Docker not installed: Check for docker ps returning "command not found" gracefully — don't treat it as a failure, just note "No Docker installed."
• Custom vs base services: Filter out standard systemd units (getty, sshd, cron, dbus, systemd-*, polkit, multipathd, etc.) to surface what's actually custom. Keep the full list in raw output but highlight custom ones separately.
• Traccar ports: The Traccar GPS tracking server uses a large range of ports (5001-5032 TCP+UDP) for various GPS protocols. Don't mistake these for security issues — they're expected for a GPS tracking platform.
• UniFi ports: UniFi controller uses 8080 (HTTP), 8443 (HTTPS), 8880/8843 (portal redirects), 6789 (speed test), 27117 (MongoDB local), 3478 (STUN UDP), 5514 (syslog UDP), 10001 (UBNT discovery UDP).