# Multi-Server Inventory Audit Pattern

Walkthrough for performing a comprehensive SSH-based audit across a fleet of servers.

## Approach

1. **Create a self-contained audit script** that collects everything in a single SSH session per host:
   - System info: hostname, OS, kernel, uptime, load, CPU cores/model, RAM, disk usage
   - Docker: `docker ps` (names, images, status, ports), `docker images` (repositories, tags, sizes), `docker-compose.yml` locations
   - Systemd: all running services, custom services under `/etc/systemd/system/` (filter out base services like getty, sshd, cron, dbus, systemd-*, networking)
   - Listening ports: `ss -tlnp` (TCP), `ss -ulnp` (UDP) — note which services/pids own each port
   - Web servers: which binary exists (nginx/apache2/caddy), vhost/site configs, Caddyfile excerpts
   - Databases: which binaries exist (mysql, mysqld, psql, mongod, redis-server), which systemd services are active
   - Users: list `/home/*` directories, note non-standard users vs root/standard
   - Cron: `crontab -l` for root
   - Custom scripts: `ls /root/*.sh /root/*.py`, `ls /usr/local/bin/*.sh /usr/local/bin/*.py`

2. **Run in parallel** — all hosts are independent, so batch the SSH calls with background processes or a simple loop with `&` and `wait`. Use the common SSH key (`itpp-infra` at `~/.ssh/itpp-infra`).

3. **Compile the report** as a single markdown file under `/root/.hermes/references/<name>.md` with:
   - Quick summary table (hostname, IP, specs, OS, Docker Y/N, key apps)
   - Per-server sections with the full field-by-field inventory
   - Cross-cutting summary tables (Docker hosts, non-Docker hosts, database engines across fleet, VPN types, disk warnings)

## Audit Script

Location: `/root/.hermes/scripts/audit-server.sh`

```bash
# Usage
/root/.hermes/scripts/audit-server.sh <ip> [username]
```

The script collects all fields listed above in a single SSH call. Output is clearly delimited with `--- SECTION ---` headers for parsing.

## Common Fields Per Server Entry

Each inventory entry should document:

| Field | Source |
|-------|--------|
| IP | Known address |
| Hostname | `hostname` |
| Plan type | From known inventory (CPX11, CPX21, etc.) |
| CPU/RAM/Disk | `nproc`, `free -h`, `df -h /` |
| OS + Kernel | `/etc/os-release`, `uname -a` |
| Uptime + Load | `uptime` |
| Docker status | `docker ps`, `docker images`, compose file locations |
| Systemd services | `systemctl list-units --type=service --state=running` |
| Custom systemd | `/etc/systemd/system/*.service` minus base services |
| TCP listening | `ss -tlnp` |
| UDP listening | `ss -ulnp` |
| Web servers | binary check + config files |
| Databases | binary check + systemd status |
| Users | `/home/*` + `/etc/passwd` filtered for home dirs |
| Cron | `crontab -l` |
| Custom scripts | `ls /root/*.sh /root/*.py /usr/local/bin/*.sh /usr/local/bin/*.py` |

## Disk Warning Flags

Servers above 70% disk usage should be flagged. At 90%+, mark as **critical**.

Known disk warnings (as of 2026-07-09):
- **ai.itpropartner.com (178.156.167.181)**: 92% — 199G of 226G used. Potential cause: large Ollama model images (10GB+), stale Docker images, old Open WebUI versions.
- **docker (178.156.168.35)**: 73% — 26G of 38G used. Small disk (CPX11), trending up.

## Full Audit Snapshot

The live full audit report lives at `/root/.hermes/references/hetzner-server-audit.md` — last refreshed 2026-07-09 covering all 10 servers.

## Pitfalls

- **Known hosts keys**: The first SSH to a new server prompts for host key verification. Use `-o StrictHostKeyChecking=no` for automation, but be aware of the security trade-off.
- **Large output**: Some commands (like `dpkg -l | grep`) can produce long output. Keep queries targeted.
- **Docker not installed**: Check for `docker ps` returning "command not found" gracefully — don't treat it as a failure, just note "No Docker installed."
- **Custom vs base services**: Filter out standard systemd units (getty, sshd, cron, dbus, systemd-*, polkit, multipathd, etc.) to surface what's actually custom. Keep the full list in raw output but highlight custom ones separately.
- **Traccar ports**: The Traccar GPS tracking server uses a large range of ports (5001-5032 TCP+UDP) for various GPS protocols. Don't mistake these for security issues — they're expected for a GPS tracking platform.
- **UniFi ports**: UniFi controller uses 8080 (HTTP), 8443 (HTTPS), 8880/8843 (portal redirects), 6789 (speed test), 27117 (MongoDB local), 3478 (STUN UDP), 5514 (syslog UDP), 10001 (UBNT discovery UDP).