162 lines
10 KiB
Markdown
162 lines
10 KiB
Markdown
---
|
|
name: infrastructure-automation
|
|
description: "Umbrella for infrastructure automation: network device backup pipelines, VPN-based remote access scripts, scheduled maintenance, and S3 log shipping."
|
|
version: 2.6.0
|
|
author: Hermes Agent
|
|
license: MIT
|
|
platforms: [linux]
|
|
tags: [devops, infrastructure, network, vpn, backup, s3, cron, automation]
|
|
related_skills: []
|
|
---
|
|
|
|
Reference file `references/vpn-fallback-wireguard.md` documents the WireGuard-preferred VPN strategy — backup scripts try WireGuard first, fall back to L2TP/IPsec only if unreachable.
|
|
|
|
Reference file `references/wireguard-backup-fallback.md` documents the exact code changes in wisp-backup.py that implement the WireGuard-first fallback — the `vpn_was_connected` tracking variable, ping check before VPN bringup, and conditional VPN teardown.
|
|
|
|
Reference file `references/daily-tech-digest.md` documents the RSS feed aggregation and email workflow — cron environment PATH quirks, `feedparser` system install note, `--email` delivery mode.
|
|
|
|
Reference `references/core-operating-boundary.md` documents the hands-off rule for Core's Hermes config — do not touch Core config.yaml unless explicitly asked. Created after the Jul 10 session where unauthorized config edits broke the session twice.
|
|
|
|
Reference `references/caddy-tailscale-port-conflict.md` documents the port 443 conflict between Caddy and Tailscale on Core — both want port 443, Tailscale binds `100.x.x.x:443`, Caddy tries to bind `:443` (all interfaces). Fix: add `default_bind <public-ip>` to the Caddyfile global options block. Created after the Jul 11 session where Caddy had been down for 4+ hours.
|
|
|
|
Reference `references/caddy-static-file-permissions.md` — when Caddy serves JS/CSS with wrong MIME types or 403, check file permissions (must be 644) and `handle_path` block nesting (must be top-level, not nested inside another handler).
|
|
|
|
Reference `references/mcp-rate-limiting-cache-pattern.md` — token-bucket rate limiter + TTL cache + fallback-chain pattern used by the Super Search MCP server for wrapping free/public-records APIs. Covers OpenCorporates, CourtListener, PACER integration for skip tracing.
|
|
|
|
Reference `references/ops-portal-pipeline.md` (v2.0.0) documents the full Ops Portal JS architecture, auth flow, nav loading pattern, and 11 common failure modes with fixes. Updated Jul 12 after a 42-bug audit.
|
|
|
|
Reference `references/hetzner-server-audit-2026-07-12.md` — complete Hetzner inventory audit: 10 servers, $160/mo, with cancellation order and DNS status tracking for UNMS, UniFi, hudu, fleettracker360.
|
|
|
|
Reference `references/fleettracker-traccar-deployment.md` — Traccar GPS tracking platform deployment: Docker setup, REST API reference, HERE Maps traffic-aware routing integration, iOS client setup, OBD2 dongle pairing, and dashboard concept.
|
|
|
|
Reference `references/hermes-vision-setup.md` — Hermes auxiliary vision configuration (auxiliary.vision section, NOT vision section). Covers admin-ai + gemini-flash-latest provider setup, common config mistakes, gateway restart requirement, and testing commands.
|
|
|
|
Reference `references/vaultwarden-deployment.md` — Vaultwarden password manager deployment: Docker compose, Tailscale-only exposure, admin setup, backup pattern, iOS Safari HTTP block workaround, and the vault.iamgmb.com Caddy reverse-proxy HTTPS endpoint used on Core.
|
|
|
|
Reference `references/uisp-vault-and-device-ws-recovery.md` — UISP credential vault key recovery, device-ws container startup fix after backup restore, backup locations and S3 sync cron, and TLS certificate renewal for UISP on app2.
|
|
|
|
`/root/.hermes/references/core-rebalance-plan.md` — master 6-phase migration plan: Core keeps Hermes + LiteLLM + Ollama; all other services move to app1; app1-bu renamed to core-bu and upgraded from CPX11 to CPX31.
|
|
|
|
| Ops dashboard JSON | `/var/www/ops/data/ops-status.json` |
|
|
| Ops collector script | `/root/.hermes/scripts/ops-data-collector.py` |
|
|
| Ops Caddyfile | `/etc/caddy/ops.itpropartner.com.Caddyfile` |
|
|
|
|
## Infrastructure Inventory
|
|
|
|
Reference `references/master-apps-services-inventory.md` covers the master inventory document at `/root/.hermes/references/master-apps-services.md` — single source of truth for all servers, services, subscriptions, API keys, FOSS, domains, and backups. Update whenever a service is added, modified, or decommissioned.
|
|
|
|
## Service Monitoring Patterns
|
|
|
|
Reference file `references/service-monitoring-patterns.md` documents all running monitoring cron patterns including:
|
|
- Apex Mail Watchdog (WP SMTP health check via SSH)
|
|
- Ops Data Collector (comprehensive JSON status)
|
|
- Home Router Watchdog and Backup pipeline
|
|
- Warm Standby Sync pattern
|
|
Each has the trigger, script path, S3 target, and known failure modes.
|
|
|
|
---
|
|
|
|
## WPForms Email Delivery
|
|
|
|
Reference `references/wpforms-email-debugging.md` covers PHP serialization mismatches in WP Mail SMTP, invalid sender address fields, and the workarounds used to fix SMTP auth and `From:` header issues on WordPress sites.
|
|
|
|
## SiteGround SFTP Backup & Migration
|
|
|
|
Reference `references/siteground-sftp-migration.md` covers both manual migration and programmatic SFTP backup to S3.
|
|
|
|
## RunCloud WordPress Backups
|
|
|
|
Reference `references/runcloud-wordpress-backups.md` covers the `tar` "file changed" pitfall when archiving live webapps with `set -e`, MariaDB dump tool deprecation (`mariadb-dump`), and RunCloud Nginx config backup paths.
|
|
|
|
**Key constraints**: SiteGround's SFTP server (port 18765) blocks SSH shell exec — only pure SFTP operations work. All sites live at `/<domain>/<domain>/public_html`. The programmatic approach uses paramiko's pure-SFTP recursive listing + local tarball creation, then uploads to Wasabi S3.
|
|
|
|
The backup script is at `/root/backup_sites.py` and must be run with `/opt/awscli-venv/bin/python3` (paramiko lives in the awscli venv, not system Python).
|
|
|
|
## S3 Log Shipping
|
|
|
|
Reference `references/s3-log-shipping.md` covers crontab-based log shipping for services (Hermes standby watchdog, apex mail watchdog) to Wasabi S3 for durable offsite retention and analysis.
|
|
|
|
## WISP CCR Tower Backups
|
|
|
|
Reference `references/wisp-ccr-backups.md` covers the nightly backup pipeline for WISP MikroTik CCR towers via RouterOS SSH export and SCP fetch.
|
|
|
|
## Home Router Backup Pipelines
|
|
|
|
Reference `references/home-router-backup.md` — WireGuard-tunneled MikroTik backup to S3. Covers xl2tpd/strongSwan dependency chain, `.in_progress` file stuck-state recovery, and the cron migration from old `home-router-backup.sh` to `run-wisp-backup.sh`.
|
|
|
|
## MikroTik Router Security Hardening
|
|
|
|
Reference `references/mikrotik-security-hardening.md` — repeatable security audit procedure for RouterOS 7.x. Covers IP service restriction, firewall duplicate cleanup, L2TP auth hardening, stale user/PPP cleanup, and WireGuard key-mismatch diagnosis.
|
|
|
|
Reference `references/home-router-backup.md` — WireGuard-tunneled MikroTik backup to S3. Covers xl2tpd/strongSwan dependency chain, `.in_progress` file stuck-state recovery, and the cron migration from old `home-router-backup.sh` to `run-wisp-backup.sh`.
|
|
|
|
Reference `references/caddy-proxy-port-mismatch.md` — Caddy reverse proxy target port drifts from actual service port, causing 502s. Covers diagnosis (ss + journalctl grep) and the shark.iamgmb.com case (8081→8083 mismatch).
|
|
|
|
## Netcup SCP API
|
|
|
|
Reference `references/netcup-api-auth.md` — correct Keycloak auth format. Critical pitfall: username is `389212` (customer number only), NOT `customer#389212`. The `customer#` prefix causes `invalid_grant`.
|
|
|
|
## Cron Job Management
|
|
|
|
All crons live in root's crontab (`crontab -l`). Hermes cron entries are marked with a `# Hermes crontab — managed entries` header. System crons on remote boxes (app1-bu standby sync) use conventional system crontab.
|
|
|
|
Reference `references/service-monitoring-patterns.md` has the full cron schedule table including period, script path, and delivery method.
|
|
|
|
### Pitfall: `#!/usr/bin/env python3` in cron scripts
|
|
|
|
Scripts using `#!/usr/bin/env python3` resolve `python3` from PATH, which differs between interactive shells and cron executors. On this system, interactive shells get `/opt/awscli-venv/bin/python3` while cron gets `/usr/bin/python3`. This causes `ModuleNotFoundError` when a dependency exists only in one environment.
|
|
|
|
**Fix:** Use an absolute shebang — `#!/usr/bin/python3` — which always has apt-installed packages available. Only use `env` shebangs when the script genuinely needs venv isolation and the venv is explicitly activated in the cron wrapper.
|
|
|
|
## Postfix SMTP Relay for System/PHPMailer
|
|
|
|
Reference `references/postfix-smtp-relay.md` — when a RunCloud-managed server has stuck email in Postfix queue because port 25 is blocked, relay through `mail.germainebrown.com:2525` with SASL auth and sender-domain rewriting.
|
|
|
|
## Netcup UFW Port Management (Jul 12, 2026)
|
|
|
|
All netcup RS-series VPS ship with UFW enabled and **only SSH (port 22) allowed inbound**. Ports 80, 443, and any service ports are blocked by default. This causes:
|
|
- Web services unreachable from the internet despite Docker being healthy
|
|
- Let's Encrypt HTTP challenges timing out
|
|
- Browsers showing ERR_CONNECTION_TIMED_OUT
|
|
|
|
**Check on any new netcup deployment:**
|
|
```bash
|
|
ufw status | grep -q '80.*ALLOW' || echo "Port 80 blocked — fix with: ufw allow 80/tcp"
|
|
ufw status | grep -q '443.*ALLOW' || echo "Port 443 blocked — fix with: ufw allow 443/tcp"
|
|
```
|
|
|
|
**Fix:**
|
|
```bash
|
|
ufw allow 80/tcp comment 'HTTP'
|
|
ufw allow 443/tcp comment 'HTTPS'
|
|
ufw reload
|
|
```
|
|
|
|
**Reference:** This was discovered on app2 (152.53.39.202) where UNMS/UISP Docker was running fine but unreachable from the internet. After opening UFW ports and obtaining Let's Encrypt certs, the service became accessible.
|
|
|
|
## Infrastructure Paths
|
|
|
|
| What | Path / Command |
|
|
|------|----------------|
|
|
| Crontab | `crontab -l` (system) |
|
|
| Ops dashboard JSON | `/var/www/ops/data/ops-status.json` |
|
|
| Ops collector script | `/root/.hermes/scripts/ops-data-collector.py` |
|
|
| Ops Caddyfile | `/etc/caddy/ops.itpropartner.com.Caddyfile` |
|
|
| Router backup scripts | `/root/.hermes/scripts/run-wisp-backup.sh` |
|
|
| Standby sync script | `/root/.hermes/scripts/hermes-standby-sync.sh` (on app1-bu) |
|
|
| Monitor scripts | `/root/.hermes/scripts/` |
|
|
|
|
## Testing Cron Changes
|
|
|
|
1. Edit crontab: `crontab -e`
|
|
2. Verify syntax: `crontab -l`
|
|
3. Test script standalone first (eliminate PATH vs different python env issues):
|
|
```bash
|
|
env -i PATH="/usr/local/bin:/usr/bin:/bin" HOME=/root python3 /path/to/script.py
|
|
```
|
|
4. After install, wait for next schedule interval or force run:
|
|
```bash
|
|
/path/to/script.py 2>&1 | logger -t test-run
|
|
```
|
|
5. Check syslog: `journalctl -t test-run` or `grep 'test-run' /var/log/syslog`
|