Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,78 @@
+# Multi-Server Inventory Audit Pattern
+
+Walkthrough for performing a comprehensive SSH-based audit across a fleet of servers.
+
+## Approach
+
+1. **Create a self-contained audit script** that collects everything in a single SSH session per host:
+   - System info: hostname, OS, kernel, uptime, load, CPU cores/model, RAM, disk usage
+   - Docker: `docker ps` (names, images, status, ports), `docker images` (repositories, tags, sizes), `docker-compose.yml` locations
+   - Systemd: all running services, custom services under `/etc/systemd/system/` (filter out base services like getty, sshd, cron, dbus, systemd-*, networking)
+   - Listening ports: `ss -tlnp` (TCP), `ss -ulnp` (UDP) — note which services/pids own each port
+   - Web servers: which binary exists (nginx/apache2/caddy), vhost/site configs, Caddyfile excerpts
+   - Databases: which binaries exist (mysql, mysqld, psql, mongod, redis-server), which systemd services are active
+   - Users: list `/home/*` directories, note non-standard users vs root/standard
+   - Cron: `crontab -l` for root
+   - Custom scripts: `ls /root/*.sh /root/*.py`, `ls /usr/local/bin/*.sh /usr/local/bin/*.py`
+
+2. **Run in parallel** — all hosts are independent, so batch the SSH calls with background processes or a simple loop with `&` and `wait`. Use the common SSH key (`itpp-infra` at `~/.ssh/itpp-infra`).
+
+3. **Compile the report** as a single markdown file under `/root/.hermes/references/<name>.md` with:
+   - Quick summary table (hostname, IP, specs, OS, Docker Y/N, key apps)
+   - Per-server sections with the full field-by-field inventory
+   - Cross-cutting summary tables (Docker hosts, non-Docker hosts, database engines across fleet, VPN types, disk warnings)
+
+## Audit Script
+
+Location: `/root/.hermes/scripts/audit-server.sh`
+
+```bash
+# Usage
+/root/.hermes/scripts/audit-server.sh <ip> [username]
+```
+
+The script collects all fields listed above in a single SSH call. Output is clearly delimited with `--- SECTION ---` headers for parsing.
+
+## Common Fields Per Server Entry
+
+Each inventory entry should document:
+
+| Field | Source |
+|-------|--------|
+| IP | Known address |
+| Hostname | `hostname` |
+| Plan type | From known inventory (CPX11, CPX21, etc.) |
+| CPU/RAM/Disk | `nproc`, `free -h`, `df -h /` |
+| OS + Kernel | `/etc/os-release`, `uname -a` |
+| Uptime + Load | `uptime` |
+| Docker status | `docker ps`, `docker images`, compose file locations |
+| Systemd services | `systemctl list-units --type=service --state=running` |
+| Custom systemd | `/etc/systemd/system/*.service` minus base services |
+| TCP listening | `ss -tlnp` |
+| UDP listening | `ss -ulnp` |
+| Web servers | binary check + config files |
+| Databases | binary check + systemd status |
+| Users | `/home/*` + `/etc/passwd` filtered for home dirs |
+| Cron | `crontab -l` |
+| Custom scripts | `ls /root/*.sh /root/*.py /usr/local/bin/*.sh /usr/local/bin/*.py` |
+
+## Disk Warning Flags
+
+Servers above 70% disk usage should be flagged. At 90%+, mark as **critical**.
+
+Known disk warnings (as of 2026-07-09):
+- **ai.itpropartner.com (178.156.167.181)**: 92% — 199G of 226G used. Potential cause: large Ollama model images (10GB+), stale Docker images, old Open WebUI versions.
+- **docker (178.156.168.35)**: 73% — 26G of 38G used. Small disk (CPX11), trending up.
+
+## Full Audit Snapshot
+
+The live full audit report lives at `/root/.hermes/references/hetzner-server-audit.md` — last refreshed 2026-07-09 covering all 10 servers.
+
+## Pitfalls
+
+- **Known hosts keys**: The first SSH to a new server prompts for host key verification. Use `-o StrictHostKeyChecking=no` for automation, but be aware of the security trade-off.
+- **Large output**: Some commands (like `dpkg -l | grep`) can produce long output. Keep queries targeted.
+- **Docker not installed**: Check for `docker ps` returning "command not found" gracefully — don't treat it as a failure, just note "No Docker installed."
+- **Custom vs base services**: Filter out standard systemd units (getty, sshd, cron, dbus, systemd-*, polkit, multipathd, etc.) to surface what's actually custom. Keep the full list in raw output but highlight custom ones separately.
+- **Traccar ports**: The Traccar GPS tracking server uses a large range of ports (5001-5032 TCP+UDP) for various GPS protocols. Don't mistake these for security issues — they're expected for a GPS tracking platform.
+- **UniFi ports**: UniFi controller uses 8080 (HTTP), 8443 (HTTPS), 8880/8843 (portal redirects), 6789 (speed test), 27117 (MongoDB local), 3478 (STUN UDP), 5514 (syslog UDP), 10001 (UBNT discovery UDP).