hermes-skills/skills/devops/infrastructure-automation/SKILL.md at master

ippadmin/hermes-skills

Fork 0

Files

T

root 1b4fcd4427 Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00

10 KiB

Raw Permalink Blame History

name, description, version, author, license, platforms, tags, related_skills

name

description

version

author

license

platforms

Infrastructure Inventory

Reference references/master-apps-services-inventory.md covers the master inventory document at /root/.hermes/references/master-apps-services.md — single source of truth for all servers, services, subscriptions, API keys, FOSS, domains, and backups. Update whenever a service is added, modified, or decommissioned.

Service Monitoring Patterns

Reference file references/service-monitoring-patterns.md documents all running monitoring cron patterns including:

Apex Mail Watchdog (WP SMTP health check via SSH)
Ops Data Collector (comprehensive JSON status)
Home Router Watchdog and Backup pipeline
Warm Standby Sync pattern Each has the trigger, script path, S3 target, and known failure modes.

WPForms Email Delivery

Reference references/wpforms-email-debugging.md covers PHP serialization mismatches in WP Mail SMTP, invalid sender address fields, and the workarounds used to fix SMTP auth and From: header issues on WordPress sites.

SiteGround SFTP Backup & Migration

Reference references/siteground-sftp-migration.md covers both manual migration and programmatic SFTP backup to S3.

RunCloud WordPress Backups

Reference references/runcloud-wordpress-backups.md covers the tar "file changed" pitfall when archiving live webapps with set -e, MariaDB dump tool deprecation (mariadb-dump), and RunCloud Nginx config backup paths.

Key constraints: SiteGround's SFTP server (port 18765) blocks SSH shell exec — only pure SFTP operations work. All sites live at /<domain>/<domain>/public_html. The programmatic approach uses paramiko's pure-SFTP recursive listing + local tarball creation, then uploads to Wasabi S3.

The backup script is at /root/backup_sites.py and must be run with /opt/awscli-venv/bin/python3 (paramiko lives in the awscli venv, not system Python).

S3 Log Shipping

Reference references/s3-log-shipping.md covers crontab-based log shipping for services (Hermes standby watchdog, apex mail watchdog) to Wasabi S3 for durable offsite retention and analysis.

WISP CCR Tower Backups

Reference references/wisp-ccr-backups.md covers the nightly backup pipeline for WISP MikroTik CCR towers via RouterOS SSH export and SCP fetch.

Home Router Backup Pipelines

Reference references/home-router-backup.md — WireGuard-tunneled MikroTik backup to S3. Covers xl2tpd/strongSwan dependency chain, .in_progress file stuck-state recovery, and the cron migration from old home-router-backup.sh to run-wisp-backup.sh.

MikroTik Router Security Hardening

Reference references/mikrotik-security-hardening.md — repeatable security audit procedure for RouterOS 7.x. Covers IP service restriction, firewall duplicate cleanup, L2TP auth hardening, stale user/PPP cleanup, and WireGuard key-mismatch diagnosis.

Reference references/caddy-proxy-port-mismatch.md — Caddy reverse proxy target port drifts from actual service port, causing 502s. Covers diagnosis (ss + journalctl grep) and the shark.iamgmb.com case (8081→8083 mismatch).

Netcup SCP API

Reference references/netcup-api-auth.md — correct Keycloak auth format. Critical pitfall: username is 389212 (customer number only), NOT customer#389212. The customer# prefix causes invalid_grant.

Cron Job Management

All crons live in root's crontab (crontab -l). Hermes cron entries are marked with a # Hermes crontab — managed entries header. System crons on remote boxes (app1-bu standby sync) use conventional system crontab.

Reference references/service-monitoring-patterns.md has the full cron schedule table including period, script path, and delivery method.

Pitfall: `#!/usr/bin/env python3` in cron scripts

Scripts using #!/usr/bin/env python3 resolve python3 from PATH, which differs between interactive shells and cron executors. On this system, interactive shells get /opt/awscli-venv/bin/python3 while cron gets /usr/bin/python3. This causes ModuleNotFoundError when a dependency exists only in one environment.

Fix: Use an absolute shebang — #!/usr/bin/python3 — which always has apt-installed packages available. Only use env shebangs when the script genuinely needs venv isolation and the venv is explicitly activated in the cron wrapper.

Postfix SMTP Relay for System/PHPMailer

Reference references/postfix-smtp-relay.md — when a RunCloud-managed server has stuck email in Postfix queue because port 25 is blocked, relay through mail.germainebrown.com:2525 with SASL auth and sender-domain rewriting.

Netcup UFW Port Management (Jul 12, 2026)

All netcup RS-series VPS ship with UFW enabled and only SSH (port 22) allowed inbound. Ports 80, 443, and any service ports are blocked by default. This causes:

Web services unreachable from the internet despite Docker being healthy
Let's Encrypt HTTP challenges timing out
Browsers showing ERR_CONNECTION_TIMED_OUT

Check on any new netcup deployment:

ufw status | grep -q '80.*ALLOW' || echo "Port 80 blocked — fix with: ufw allow 80/tcp"
ufw status | grep -q '443.*ALLOW' || echo "Port 443 blocked — fix with: ufw allow 443/tcp"

Fix:

ufw allow 80/tcp comment 'HTTP'
ufw allow 443/tcp comment 'HTTPS'
ufw reload

Reference: This was discovered on app2 (152.53.39.202) where UNMS/UISP Docker was running fine but unreachable from the internet. After opening UFW ports and obtaining Let's Encrypt certs, the service became accessible.

Infrastructure Paths

What	Path / Command
Crontab	`crontab -l` (system)
Ops dashboard JSON	`/var/www/ops/data/ops-status.json`
Ops collector script	`/root/.hermes/scripts/ops-data-collector.py`
Ops Caddyfile	`/etc/caddy/ops.itpropartner.com.Caddyfile`
Router backup scripts	`/root/.hermes/scripts/run-wisp-backup.sh`
Standby sync script	`/root/.hermes/scripts/hermes-standby-sync.sh` (on app1-bu)
Monitor scripts	`/root/.hermes/scripts/`

Testing Cron Changes

Edit crontab: crontab -e
Verify syntax: crontab -l

Test script standalone first (eliminate PATH vs different python env issues):

env -i PATH="/usr/local/bin:/usr/bin:/bin" HOME=/root python3 /path/to/script.py

After install, wait for next schedule interval or force run:
```
/path/to/script.py 2>&1 | logger -t test-run
```
Check syslog: journalctl -t test-run or grep 'test-run' /var/log/syslog

10 KiB Raw Permalink Blame History