79 lines
4.5 KiB
Markdown
79 lines
4.5 KiB
Markdown
# Multi-Server Inventory Audit Pattern
|
|
|
|
Walkthrough for performing a comprehensive SSH-based audit across a fleet of servers.
|
|
|
|
## Approach
|
|
|
|
1. **Create a self-contained audit script** that collects everything in a single SSH session per host:
|
|
- System info: hostname, OS, kernel, uptime, load, CPU cores/model, RAM, disk usage
|
|
- Docker: `docker ps` (names, images, status, ports), `docker images` (repositories, tags, sizes), `docker-compose.yml` locations
|
|
- Systemd: all running services, custom services under `/etc/systemd/system/` (filter out base services like getty, sshd, cron, dbus, systemd-*, networking)
|
|
- Listening ports: `ss -tlnp` (TCP), `ss -ulnp` (UDP) — note which services/pids own each port
|
|
- Web servers: which binary exists (nginx/apache2/caddy), vhost/site configs, Caddyfile excerpts
|
|
- Databases: which binaries exist (mysql, mysqld, psql, mongod, redis-server), which systemd services are active
|
|
- Users: list `/home/*` directories, note non-standard users vs root/standard
|
|
- Cron: `crontab -l` for root
|
|
- Custom scripts: `ls /root/*.sh /root/*.py`, `ls /usr/local/bin/*.sh /usr/local/bin/*.py`
|
|
|
|
2. **Run in parallel** — all hosts are independent, so batch the SSH calls with background processes or a simple loop with `&` and `wait`. Use the common SSH key (`itpp-infra` at `~/.ssh/itpp-infra`).
|
|
|
|
3. **Compile the report** as a single markdown file under `/root/.hermes/references/<name>.md` with:
|
|
- Quick summary table (hostname, IP, specs, OS, Docker Y/N, key apps)
|
|
- Per-server sections with the full field-by-field inventory
|
|
- Cross-cutting summary tables (Docker hosts, non-Docker hosts, database engines across fleet, VPN types, disk warnings)
|
|
|
|
## Audit Script
|
|
|
|
Location: `/root/.hermes/scripts/audit-server.sh`
|
|
|
|
```bash
|
|
# Usage
|
|
/root/.hermes/scripts/audit-server.sh <ip> [username]
|
|
```
|
|
|
|
The script collects all fields listed above in a single SSH call. Output is clearly delimited with `--- SECTION ---` headers for parsing.
|
|
|
|
## Common Fields Per Server Entry
|
|
|
|
Each inventory entry should document:
|
|
|
|
| Field | Source |
|
|
|-------|--------|
|
|
| IP | Known address |
|
|
| Hostname | `hostname` |
|
|
| Plan type | From known inventory (CPX11, CPX21, etc.) |
|
|
| CPU/RAM/Disk | `nproc`, `free -h`, `df -h /` |
|
|
| OS + Kernel | `/etc/os-release`, `uname -a` |
|
|
| Uptime + Load | `uptime` |
|
|
| Docker status | `docker ps`, `docker images`, compose file locations |
|
|
| Systemd services | `systemctl list-units --type=service --state=running` |
|
|
| Custom systemd | `/etc/systemd/system/*.service` minus base services |
|
|
| TCP listening | `ss -tlnp` |
|
|
| UDP listening | `ss -ulnp` |
|
|
| Web servers | binary check + config files |
|
|
| Databases | binary check + systemd status |
|
|
| Users | `/home/*` + `/etc/passwd` filtered for home dirs |
|
|
| Cron | `crontab -l` |
|
|
| Custom scripts | `ls /root/*.sh /root/*.py /usr/local/bin/*.sh /usr/local/bin/*.py` |
|
|
|
|
## Disk Warning Flags
|
|
|
|
Servers above 70% disk usage should be flagged. At 90%+, mark as **critical**.
|
|
|
|
Known disk warnings (as of 2026-07-09):
|
|
- **ai.itpropartner.com (178.156.167.181)**: 92% — 199G of 226G used. Potential cause: large Ollama model images (10GB+), stale Docker images, old Open WebUI versions.
|
|
- **docker (178.156.168.35)**: 73% — 26G of 38G used. Small disk (CPX11), trending up.
|
|
|
|
## Full Audit Snapshot
|
|
|
|
The live full audit report lives at `/root/.hermes/references/hetzner-server-audit.md` — last refreshed 2026-07-09 covering all 10 servers.
|
|
|
|
## Pitfalls
|
|
|
|
- **Known hosts keys**: The first SSH to a new server prompts for host key verification. Use `-o StrictHostKeyChecking=no` for automation, but be aware of the security trade-off.
|
|
- **Large output**: Some commands (like `dpkg -l | grep`) can produce long output. Keep queries targeted.
|
|
- **Docker not installed**: Check for `docker ps` returning "command not found" gracefully — don't treat it as a failure, just note "No Docker installed."
|
|
- **Custom vs base services**: Filter out standard systemd units (getty, sshd, cron, dbus, systemd-*, polkit, multipathd, etc.) to surface what's actually custom. Keep the full list in raw output but highlight custom ones separately.
|
|
- **Traccar ports**: The Traccar GPS tracking server uses a large range of ports (5001-5032 TCP+UDP) for various GPS protocols. Don't mistake these for security issues — they're expected for a GPS tracking platform.
|
|
- **UniFi ports**: UniFi controller uses 8080 (HTTP), 8443 (HTTPS), 8880/8843 (portal redirects), 6789 (speed test), 27117 (MongoDB local), 3478 (STUN UDP), 5514 (syslog UDP), 10001 (UBNT discovery UDP).
|