ippadmin/hermes-recovery
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial resurrection kit — 81 scripts, 62 references, configs, systemd units, crons, Docker compose files, Caddy config, master README

Browse Source
This commit is contained in:
root
2026-07-15 17:45:36 -04:00
commit ae056eaf83
197 changed files with 26127 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
references/master-apps-services.md
+340
View File
@@ -0,0 +1,340 @@
# IT Pro Partner — Master Apps & Services Inventory
> **Last Updated:** July 11, 2026
> **Maintainer:** Sho'Nuff
> **Purpose:** Complete registry of every installed application, subscription, API key, FOSS tool, and the server/service it lives on.
---
## Infrastructure Overview
| Server | IP | Specs | Provider | Role |
|---|---|---|---|---|
| **Core** | 152.53.192.33 | RS 2000 (8C/15G/512GB) | netcup | Ops hub |
| **app1** | 152.53.36.131 | RS 4000 (12C/32G/1TB) | netcup | AI/ML stack |
| **app2** | 152.53.39.202 | RS 4000 (12C/32G/1TB) | netcup | UISP/Infrastructure |
| **app3** | 152.53.241.111 | RS 4000 (12C/32G/1TB) | netcup | Web apps (spare) |
| **app1-bu** | 5.161.114.8 | CPX11 (2C/2G/40G) | Hetzner | Warm standby |
| **old-ai** | 178.156.167.181 | CPX41 (226G disk) | Hetzner | **Decommissioning** — admin-ai pending migration to app1 |
| **wphost02** | 5.161.62.38 | — | Hetzner | WordPress host (migrating to app3) |
| **Tony Hermes** | 87.99.159.142 | — | — | Anthony's personal Hermes |
**SSH Key:** `itpp-infra` (deployed to all servers, root SSH key-only)
**Base OS:** Debian 13 (all netcup boxes), Debian 12 (Hetzner boxes)
### Active Projects
| Project | Status | Server | Description |
|---|---|---|---|
| **Super Search** | ✅ Live v1.1.0 | Core | Multi-provider web search (SearXNG → Exa → Firecrawl) + extraction via FastMCP |
| **DRE / Debt Recovery Experts** | Active | Core + Cloudflare | Collections platform, CRM, approval workflows, letter templates |
| **Apex Track Experience** | Active | wphost02 (→ app3) | NASA Top Speed Event, WPForms, SMTP, PayPal |
| **Shark Attack Fantasy League** | Active | Core | shark.iamgmb.com, Discord bridge, changelog |
| **VoIPSimplicity** | Planning | — | RingLogix CPaaS, Yealink rentals |
| **Ops Portal** | Active | Core | ops.itpropartner.com, admin dashboards, customer portal |
| **admin-ai Migration** | Phase 0 Pending | old-ai → app1 | Move LiteLLM from Hetzner to netcup |
---
## Server-by-Server Inventory
### Core (netcup RS 2000 — 152.53.192.33)
#### System Services
| Service | Type | Purpose |
|---|---|---|
| **Hermes Agent** | systemd | AI assistant gateway (Telegram + cron) |
| **Caddy** | systemd | Reverse proxy (port 80/443) with Let's Encrypt |
| **Ops Portal** | systemd (uvicorn) | Internal admin dashboard at ops.itpropartner.com |
| **OSINT API** | systemd (uvicorn) | OSINT search API on port 8100 |
| **Hermes Assistant (PWA)** | systemd | PWA backend on port 8082 |
| **Shark Game** | systemd | Shark Attack Fantasy League backend on port 8083 |
| **Ollama** | systemd | Local LLM fallback (llama3.2:3b) |
| **Tailscale** | systemd | Mesh VPN for infra access |
| **StrongSwan + xl2tpd** | systemd | L2TP/IPsec VPN server |
| **WireGuard** | systemd | Tunnel to home router (10.77.0.0/24) |
| **MySQL Tunnel** | systemd | SSH tunnel to wphost02 MySQL (port 33060) |
| **Redis** | systemd | Cache layer |
| **Prometheus Node Exporter** | systemd | System metrics collection |
| **Hermes Browser** | systemd | Chromium headless for browser automation (9222) |
#### Docker Containers
| Container | Image | Port | Purpose |
|---|---|---|---|
| **Komodo Core** | ghcr.io/moghtech/komodo-core:2 | 9120 | Docker management UI |
| **Komodo Mongo** | mongo | — | Komodo database |
| **Prometheus** | prom/prometheus | 9090 (loopback) | Metrics aggregation |
| **Grafana** | grafana/grafana | 3002 (loopback) | Monitoring dashboards |
| **Twenty CRM** | twentycrm/twenty:latest | 3001 | DRE CRM (crm.debtrecoveryexperts.com) |
| **Twenty Worker** | twentycrm/twenty:latest | — | Twenty async worker |
| **Twenty DB** | postgres:16 | — | Twenty PostgreSQL |
| **Twenty Redis** | redis | — | Twenty cache |
| **DocuSeal** | docuseal/docuseal:latest | 3000 (loopback) | Document signing |
| **SearXNG** | searxng/searxng:latest | 8888 (loopback) | Private search engine |
| **Vaultwarden** | vaultwarden/server:1.33.2 | 8080 | Password manager (tailnet only) |
#### Caddy Sites
| Domain | Backend | Purpose |
|---|---|---|
| sign.itpropartner.com | → :3000 (DocuSeal) | Document signing |
| core.itpropartner.com | static files | JSON data, capabilities, portal |
| crm.debtrecoveryexperts.com | → :3001 (Twenty) | DRE customer CRM |
| portal.debtrecoveryexperts.com | static | DRE customer portal |
| internal.debtrecoveryexperts.com | static | DRE internal dashboard |
| app.itpropartner.com | static + → :8082 | Hermes Assistant PWA |
| shark.itpropartner.com | static + → :8083 | Shark game |
| shark.iamgmb.com | static + → :8083 | Shark game alias |
| ops.itpropartner.com | → :8090 + static | Ops admin portal |
#### Cron Jobs (Hermes-managed, 22 total)
| Job | Schedule | Purpose |
|---|---|---|
| hermes-live-sync | Every 15m | S3 backup sync |
| service-health-check | Every 5m | Service monitoring |
| Home-Router-Watchdog | Every 5m | CCR watchdog |
| vps-threshold-check | Every 15m | Disk/mem alerts |
| hermes-memory-consolidate | Every 10m | Memory pruning |
| IMAP-email-triage | Every 60m | Email classification |
| IMAP-triage-watchdog | Every 10m | IMAP health check |
| shonuff-inbox-agent | Every 15m | Inbox monitoring |
| daily-tech-digest | Daily 11 AM | Tech news summary |
| exotic-vehicle-scout | Daily 9 AM | New car alerts |
| shark-scraper | Every 60m | Shark sighting data |
| dre-mail-poller | Every 1m | DRE inbox polling |
| dre-pending-approval-reminder | Hourly | DRE approval reminders |
| boxpilot-triage | Every 10m | BoxPilot email triage |
| bounce-check | Every 60m | Email bounce detection |
| apex-mail-watchdog | Every 5m | Apex SMTP health |
| ops-data-collector | Every 5m | Ops portal data |
| lynis-scan | Daily 3 AM | Security audit |
| firecrawl-usage-check | Daily 9 AM | Web search credits |
| model-usage-check | Daily 8 AM | LLM availability |
| hetzner-weekly-snapshots | Weekly Mon 5 AM | Server snapshots |
| home-router-daily-backup | Daily 6 AM | CCR config backup |
---
### app1 (netcup RS 4000 — 152.53.36.131) — AI/ML Stack
#### Docker Containers
| Container | Image | Port | Purpose |
|---|---|---|---|
| **Ollama** | ollama/ollama:latest | 11434 (loopback) | Local LLM inference |
| **Open WebUI** | ghcr.io/open-webui/open-webui:latest | 3000 (loopback) | LLM chat UI (ai.itpropartner.com) |
| **n8n** | n8nio/n8n:latest | 5678 (loopback) | Workflow automation (n8n.itpropartner.com) |
| **n8n-postgres** | postgres:16-alpine | — | n8n database |
#### Caddy Sites
| Domain | Backend | Purpose |
|---|---|---|
| ai.itpropartner.com | → :3000 (Open WebUI) | LLM chat interface |
| n8n.itpropartner.com | → :5678 (n8n) | Automation workflows |
#### System Services
- Caddy, Docker, fail2ban, unattended-upgrades
#### Plans
- LiteLLM + Postgres (migrating from old-ai)
- Designated admin-ai.itpropartner.com target
---
### app2 (netcup RS 4000 — 152.53.39.202) — Infrastructure
#### Docker Containers
| Container | Image | Port | Purpose |
|---|---|---|---|
| **UNMS API** | ubnt/unms:latest | — | UISP core API |
| **UCRM** | ubnt/unms-crm:4.5.33 | 80/443 | Customer CRM |
| **UNMS Nginx** | ubnt/unms-nginx:latest | 80/443/8089 | UISP web proxy |
| **UNMS Postgres** | ubnt/unms-postgres:latest | 5432 | UISP database |
| **UNMS SiriDB** | ubnt/unms-siridb:latest | — | UISP time-series DB |
| **UNMS RabbitMQ** | rabbitmq:3.7.28-alpine | 5672 | UISP message queue |
| **UNMS Netflow** | ubnt/unms-netflow:latest | 2055/udp | Network flow data |
| **UNMS Fluentd** | ubnt/unms-fluentd:latest | 24224 | Log aggregation |
#### System Services
Docker, fail2ban, unattended-upgrades
---
### app3 (netcup RS 4000 — 152.53.241.111) — Web Apps
**Status:** Fresh server, Docker installed, no containers yet. Designated for WordPress migrations from wphost02/SiteGround.
---
### app1-bu (Hetzner CPX11 — 5.161.114.8) — Warm Standby
#### System Services
| Service | Purpose |
|---|---|
| **Hermes** | Gateway dormant — auto-activates if Core fails (5 min watchdog) |
| **Ollama** | Local LLM fallback |
| **StrongSwan** | L2TP/IPsec VPN |
| **Tailscale** | Mesh VPN |
| **Node Exporter** | Metrics |
| **apex-status-receiver** | Apex endpoint |
---
### wphost02 (Hetzner — 5.161.62.38) — WordPress
| Site | Purpose |
|---|---|
| apextrackexperience.com | Apex Track Experience (WordPress + WooCommerce + WPForms) |
| apex-status | Apex status receiver endpoint |
**MySQL:** WordPress DB at `apextrackexperience_1781549652` (via Core tunnel on port 33060)
---
### old-ai (Hetzner CPX41 — 178.156.167.181)
**Status: Decommissioning.** All services migrating to app1.
| Container | Image | Purpose |
|---|---|---|
| LiteLLM | ghcr.io/berriai/litellm:v1.84.0 | LLM proxy (admin-ai) |
| LiteLLM Postgres | postgres:16 | Model configs, API keys |
| Caddy | caddy:latest | TLS termination |
| Qdrant | qdrant/qdrant:latest | Vector DB |
| Ollama | ollama/ollama:latest | Local models |
---
## Subscriptions & API Keys
### Active Subscriptions
| Service | Plan | Cost | Billing | Purpose |
|---|---|---|---|---|
| **netcup** | RS 2000 + 4× RS 4000 | ~$156/mo | Monthly | All servers |
| **Hetzner** | CPX11 + CPX41 + misc | ~$50/mo | Monthly | app1-bu, old-ai (decomm), wphost02, old boxes (decomm) |
| **MXroute** | Email hosting | ~$15/yr? | Annual | email@redacted, DRE email, boys' email |
| **SiteGround** | WordPress hosting | ~$10/mo | Monthly | Apex Track Experience (migrating) |
| **Cloudflare** | Free (DNS + CDN + Access) | Free | — | All domains |
| **OpenRouter** | Pay-as-you-go | Variable | Prepaid | Model API fallback |
| **Firecrawl** | Pay-as-you-go | Variable | — | Web scraping for AI |
| **ScrapingAnt** (or similar) | Pay-as-you-go | Variable | — | Web extraction |
| **SyncroMSP** | RMM | ~$2/mo per endpoint | Monthly | 26 customers, ITPP endpoints |
| **Wasabi S3** | Hot cloud storage | ~$7/TB/mo | Monthly | Hermes backups, router configs |
| **Nous Research** | Hermes Agent license | Usage-based | Prepaid | Hermes platform |
### API Keys & Tokens
| Key | Location | Service | Status |
|---|---|---|---|
| FIRECRAWL_API_KEY | ~/.hermes/.env | Web scraping | ✅ Active |
| CLOUDFLARE_API_TOKEN | ~/.hermes/.env | DNS management | ✅ Active |
| NETCUP_API_KEY | ~/.hermes/.env | Server provisioning | ✅ Active |
| NETCUP_PASSWORD | ~/.hermes/.env | SCP API auth | ✅ Active |
| SYNCROMSP_API_TOKEN | ~/.hermes/.env | RMM platform | ✅ Active |
| TELEGRAM_BOT_TOKEN | ~/.hermes/.env | Telegram bot | ✅ Active |
| LITELLM_MASTER_KEY | ~/.hermes/config.yaml + /opt/ai/.env | admin-ai proxy | ✅ Active |
| LITELLM_SALT_KEY | /opt/ai/.env | Admin-ai encryption | ✅ Active |
| OpenRouter API Key | ~/.hermes/config.yaml | Model API + credits | ✅ Active |
| Hermes Nous Portal Key | — | Hermes updates | ❌ Not logged in |
| SERVER_PASSWORDS | ~/.hermes/.env | Server root passwords | ✅ Active |
| DeepSeek API Key | Litellm Postgres (encrypted) | DeepSeek LLM API | ✅ Active |
| Anthropic API Key | LiteLLM Postgres (encrypted) | Claude models | ✅ Active |
| Google/Gemini API Key | LiteLLM Postgres (encrypted) | Gemini vision + text | ✅ Active |
| OpenAI API Key | LiteLLM Postgres (encrypted) | GPT models | ✅ Active |
| Groq API Key | LiteLLM Postgres (encrypted) | Fast inference | ✅ Active |
| xAI API Key | LiteLLM Postgres (encrypted) | Grok models | ✅ Active |
---
## FOSS Software Stack
### Core Infrastructure
| Software | License | Server | Purpose |
|---|---|---|---|
| Debian 13 | GPL | All | Server OS |
| Docker + Compose | Apache 2.0 | All | Container runtime |
| Caddy | Apache 2.0 | Core, app1 | Reverse proxy + TLS |
| Tailscale | BSD | Core, app1-bu | Mesh VPN |
| WireGuard | GPL | Core | Site-to-site VPN |
| StrongSwan | GPL | Core, app1-bu | L2TP/IPsec VPN |
| Prometheus | Apache 2.0 | Core | Metrics |
| Grafana | AGPL | Core | Dashboards |
### Applications
| Software | License | Server | Purpose |
|---|---|---|---|
| **Hermes Agent** | Proprietary | Core, app1-bu | AI assistant |
| **Ollama** | MIT | Core, app1, app1-bu | Local LLM |
| **Open WebUI** | MIT | app1 | LLM chat UI |
| **n8n** | Sustainable Use License | app1 | Workflow automation |
| **Twenty CRM** | AGPL | Core | Customer CRM |
| **DocuSeal** | AGPL | Core | Document signing |
| **SearXNG** | AGPL | Core | Private search |
| **Vaultwarden** | GPL | Core | Password manager |
| **Komodo** | GPL | Core | Docker management |
| **LiteLLM** | MIT | old-ai (→ app1) | LLM proxy |
| **Qdrant** | Apache 2.0 | old-ai | Vector DB |
| **UISP/UNMS** | Proprietary | app2 | ISP management |
| **UCRM** | Proprietary | app2 | Customer management |
| **Ollama Models** | Various | Core, app1, old-ai | LLM weights |
| **Redis** | BSD | Core | Cache |
| **PostgreSQL** | PostgreSQL | Core, app1, app2 | Databases |
| **WordPress** | GPL | wphost02 (→ app3) | Web content |
| **WooCommerce** | GPL | wphost02 | E-commerce |
| **WPForms** | GPL | wphost02 | Form builder |
| **BlogWatcher** | — | Core cron | RSS monitoring |
| **Lynis** | GPL | Core | Security auditing |
---
## Domains & DNS
| Domain | Registrar | DNS | Purpose |
|---|---|---|---|
| itpropartner.com | — | Cloudflare | Primary |
| germainebrown.com | — | MXroute | Email |
| debtrecoveryexperts.com | — | Cloudflare | DRE |
| apextrackexperience.com | — | SiteGround/Cloudflare | Apex |
| iamgmb.com | — | MXroute | Boys' email |
| voipsimplicity.com | — | Cloudflare | VoIPSimplicity |
| tailc2f3b0.ts.net | — | Tailscale | Tailnet DNS (internal) |
---
## Backup & DR
| Backup | Schedule | Location | Retention |
|---|---|---|---|
| Hermes full backup | Daily 5 AM | s3://hermes-vps-backups/hermes-full-backup/ | — |
| Hermes live state | Every 15 min | s3://hermes-vps-backups/live/ | Rolling |
| Router config (home) | Daily 6 AM | s3://mikrotik-ccr-backups/ | — |
| Root essentials | Daily | S3 | — |
| Hetzner snapshots | Weekly Mon | Hetzner Cloud | — |
**DR Standby:** app1-bu (Hetzner) boots and auto-restores from S3 if Core is down for 2+ minutes.
---
## Email Accounts
| Address | Provider | Server | Purpose |
|---|---|---|---|
| email@redacted | MXroute | mail.germainebrown.com | Germaine's primary |
| email@redacted | MXroute | mail.germainebrown.com | Sho'Nuff's outbox |
| email@redacted | SiteGround | c1113726.sgvps.net:2525/993 | Apex business |
| email@redacted | MXroute | — | DRE client-facing |
| email@redacted | MXroute | — | DRE debtor-facing |
| email@redacted | MXroute | — | Garrison Brown |
| email@redacted | MXroute | — | Greyson Brown |