---
name: infrastructure-automation
description: "Umbrella for infrastructure automation: network device backup pipelines, VPN-based remote access scripts, scheduled maintenance, and S3 log shipping."
version: 2.6.0
author: Hermes Agent
license: MIT
platforms: [linux]
tags: [devops, infrastructure, network, vpn, backup, s3, cron, automation]
related_skills: []
---

Reference file `references/vpn-fallback-wireguard.md` documents the WireGuard-preferred VPN strategy — backup scripts try WireGuard first, fall back to L2TP/IPsec only if unreachable.

Reference file `references/wireguard-backup-fallback.md` documents the exact code changes in wisp-backup.py that implement the WireGuard-first fallback — the `vpn_was_connected` tracking variable, ping check before VPN bringup, and conditional VPN teardown.

Reference file `references/daily-tech-digest.md` documents the RSS feed aggregation and email workflow — cron environment PATH quirks, `feedparser` system install note, `--email` delivery mode.

Reference `references/core-operating-boundary.md` documents the hands-off rule for Core's Hermes config — do not touch Core config.yaml unless explicitly asked. Created after the Jul 10 session where unauthorized config edits broke the session twice.

Reference `references/caddy-tailscale-port-conflict.md` documents the port 443 conflict between Caddy and Tailscale on Core — both want port 443, Tailscale binds `100.x.x.x:443`, Caddy tries to bind `:443` (all interfaces). Fix: add `default_bind <public-ip>` to the Caddyfile global options block. Created after the Jul 11 session where Caddy had been down for 4+ hours.

Reference `references/caddy-static-file-permissions.md` — when Caddy serves JS/CSS with wrong MIME types or 403, check file permissions (must be 644) and `handle_path` block nesting (must be top-level, not nested inside another handler).

Reference `references/mcp-rate-limiting-cache-pattern.md` — token-bucket rate limiter + TTL cache + fallback-chain pattern used by the Super Search MCP server for wrapping free/public-records APIs. Covers OpenCorporates, CourtListener, PACER integration for skip tracing.

Reference `references/ops-portal-pipeline.md` (v2.0.0) documents the full Ops Portal JS architecture, auth flow, nav loading pattern, and 11 common failure modes with fixes. Updated Jul 12 after a 42-bug audit.

Reference `references/hetzner-server-audit-2026-07-12.md` — complete Hetzner inventory audit: 10 servers, $160/mo, with cancellation order and DNS status tracking for UNMS, UniFi, hudu, fleettracker360.

Reference `references/fleettracker-traccar-deployment.md` — Traccar GPS tracking platform deployment: Docker setup, REST API reference, HERE Maps traffic-aware routing integration, iOS client setup, OBD2 dongle pairing, and dashboard concept.

Reference `references/hermes-vision-setup.md` — Hermes auxiliary vision configuration (auxiliary.vision section, NOT vision section). Covers admin-ai + gemini-flash-latest provider setup, common config mistakes, gateway restart requirement, and testing commands.

Reference `references/vaultwarden-deployment.md` — Vaultwarden password manager deployment: Docker compose, Tailscale-only exposure, admin setup, backup pattern, iOS Safari HTTP block workaround, and the vault.iamgmb.com Caddy reverse-proxy HTTPS endpoint used on Core.

Reference `references/uisp-vault-and-device-ws-recovery.md` — UISP credential vault key recovery, device-ws container startup fix after backup restore, backup locations and S3 sync cron, and TLS certificate renewal for UISP on app2.

`/root/.hermes/references/core-rebalance-plan.md` — master 6-phase migration plan: Core keeps Hermes + LiteLLM + Ollama; all other services move to app1; app1-bu renamed to core-bu and upgraded from CPX11 to CPX31.

| Ops dashboard JSON | `/var/www/ops/data/ops-status.json` |
| Ops collector script | `/root/.hermes/scripts/ops-data-collector.py` |
| Ops Caddyfile | `/etc/caddy/ops.itpropartner.com.Caddyfile` |

## Infrastructure Inventory

Reference `references/master-apps-services-inventory.md` covers the master inventory document at `/root/.hermes/references/master-apps-services.md` — single source of truth for all servers, services, subscriptions, API keys, FOSS, domains, and backups. Update whenever a service is added, modified, or decommissioned.

## Service Monitoring Patterns

Reference file `references/service-monitoring-patterns.md` documents all running monitoring cron patterns including:
- Apex Mail Watchdog (WP SMTP health check via SSH)
- Ops Data Collector (comprehensive JSON status)
- Home Router Watchdog and Backup pipeline
- Warm Standby Sync pattern
Each has the trigger, script path, S3 target, and known failure modes.

---

## WPForms Email Delivery

Reference `references/wpforms-email-debugging.md` covers PHP serialization mismatches in WP Mail SMTP, invalid sender address fields, and the workarounds used to fix SMTP auth and `From:` header issues on WordPress sites.

## SiteGround SFTP Backup & Migration

Reference `references/siteground-sftp-migration.md` covers both manual migration and programmatic SFTP backup to S3.

## RunCloud WordPress Backups

Reference `references/runcloud-wordpress-backups.md` covers the `tar` "file changed" pitfall when archiving live webapps with `set -e`, MariaDB dump tool deprecation (`mariadb-dump`), and RunCloud Nginx config backup paths.

**Key constraints**: SiteGround's SFTP server (port 18765) blocks SSH shell exec — only pure SFTP operations work. All sites live at `/<domain>/<domain>/public_html`. The programmatic approach uses paramiko's pure-SFTP recursive listing + local tarball creation, then uploads to Wasabi S3.

The backup script is at `/root/backup_sites.py` and must be run with `/opt/awscli-venv/bin/python3` (paramiko lives in the awscli venv, not system Python).

## S3 Log Shipping

Reference `references/s3-log-shipping.md` covers crontab-based log shipping for services (Hermes standby watchdog, apex mail watchdog) to Wasabi S3 for durable offsite retention and analysis.

## WISP CCR Tower Backups

Reference `references/wisp-ccr-backups.md` covers the nightly backup pipeline for WISP MikroTik CCR towers via RouterOS SSH export and SCP fetch.

## Home Router Backup Pipelines

Reference `references/home-router-backup.md` — WireGuard-tunneled MikroTik backup to S3. Covers xl2tpd/strongSwan dependency chain, `.in_progress` file stuck-state recovery, and the cron migration from old `home-router-backup.sh` to `run-wisp-backup.sh`.

## MikroTik Router Security Hardening

Reference `references/mikrotik-security-hardening.md` — repeatable security audit procedure for RouterOS 7.x. Covers IP service restriction, firewall duplicate cleanup, L2TP auth hardening, stale user/PPP cleanup, and WireGuard key-mismatch diagnosis.

Reference `references/home-router-backup.md` — WireGuard-tunneled MikroTik backup to S3. Covers xl2tpd/strongSwan dependency chain, `.in_progress` file stuck-state recovery, and the cron migration from old `home-router-backup.sh` to `run-wisp-backup.sh`.

Reference `references/caddy-proxy-port-mismatch.md` — Caddy reverse proxy target port drifts from actual service port, causing 502s. Covers diagnosis (ss + journalctl grep) and the shark.iamgmb.com case (8081→8083 mismatch).

## Netcup SCP API

Reference `references/netcup-api-auth.md` — correct Keycloak auth format. Critical pitfall: username is `389212` (customer number only), NOT `customer#389212`. The `customer#` prefix causes `invalid_grant`.

## Cron Job Management

All crons live in root's crontab (`crontab -l`). Hermes cron entries are marked with a `# Hermes crontab — managed entries` header. System crons on remote boxes (app1-bu standby sync) use conventional system crontab.

Reference `references/service-monitoring-patterns.md` has the full cron schedule table including period, script path, and delivery method.

### Pitfall: `#!/usr/bin/env python3` in cron scripts

Scripts using `#!/usr/bin/env python3` resolve `python3` from PATH, which differs between interactive shells and cron executors. On this system, interactive shells get `/opt/awscli-venv/bin/python3` while cron gets `/usr/bin/python3`. This causes `ModuleNotFoundError` when a dependency exists only in one environment.

**Fix:** Use an absolute shebang — `#!/usr/bin/python3` — which always has apt-installed packages available. Only use `env` shebangs when the script genuinely needs venv isolation and the venv is explicitly activated in the cron wrapper.

## Postfix SMTP Relay for System/PHPMailer

Reference `references/postfix-smtp-relay.md` — when a RunCloud-managed server has stuck email in Postfix queue because port 25 is blocked, relay through `mail.germainebrown.com:2525` with SASL auth and sender-domain rewriting.

## Netcup UFW Port Management (Jul 12, 2026)

All netcup RS-series VPS ship with UFW enabled and **only SSH (port 22) allowed inbound**. Ports 80, 443, and any service ports are blocked by default. This causes:
- Web services unreachable from the internet despite Docker being healthy
- Let's Encrypt HTTP challenges timing out
- Browsers showing ERR_CONNECTION_TIMED_OUT

**Check on any new netcup deployment:**
```bash
ufw status | grep -q '80.*ALLOW' || echo "Port 80 blocked — fix with: ufw allow 80/tcp"
ufw status | grep -q '443.*ALLOW' || echo "Port 443 blocked — fix with: ufw allow 443/tcp"
```

**Fix:**
```bash
ufw allow 80/tcp comment 'HTTP'
ufw allow 443/tcp comment 'HTTPS'
ufw reload
```

**Reference:** This was discovered on app2 (152.53.39.202) where UNMS/UISP Docker was running fine but unreachable from the internet. After opening UFW ports and obtaining Let's Encrypt certs, the service became accessible.

## Infrastructure Paths

| What | Path / Command |
|------|----------------|
| Crontab | `crontab -l` (system) |
| Ops dashboard JSON | `/var/www/ops/data/ops-status.json` |
| Ops collector script | `/root/.hermes/scripts/ops-data-collector.py` |
| Ops Caddyfile | `/etc/caddy/ops.itpropartner.com.Caddyfile` |
| Router backup scripts | `/root/.hermes/scripts/run-wisp-backup.sh` |
| Standby sync script | `/root/.hermes/scripts/hermes-standby-sync.sh` (on app1-bu) |
| Monitor scripts | `/root/.hermes/scripts/` |

## Testing Cron Changes

1. Edit crontab: `crontab -e`
2. Verify syntax: `crontab -l`
3. Test script standalone first (eliminate PATH vs different python env issues):
   ```bash
   env -i PATH="/usr/local/bin:/usr/bin:/bin" HOME=/root python3 /path/to/script.py
   ```
4. After install, wait for next schedule interval or force run:
   ```bash
   /path/to/script.py 2>&1 | logger -t test-run
   ```
5. Check syslog: `journalctl -t test-run` or `grep 'test-run' /var/log/syslog`
