Files
hermes-recovery/references/master-apps-services.md
T

14 KiB
Raw Blame History

IT Pro Partner — Master Apps & Services Inventory

Last Updated: July 11, 2026 Maintainer: Sho'Nuff Purpose: Complete registry of every installed application, subscription, API key, FOSS tool, and the server/service it lives on.


Infrastructure Overview

Server IP Specs Provider Role
Core 152.53.192.33 RS 2000 (8C/15G/512GB) netcup Ops hub
app1 152.53.36.131 RS 4000 (12C/32G/1TB) netcup AI/ML stack
app2 152.53.39.202 RS 4000 (12C/32G/1TB) netcup UISP/Infrastructure
app3 152.53.241.111 RS 4000 (12C/32G/1TB) netcup Web apps (spare)
app1-bu 5.161.114.8 CPX11 (2C/2G/40G) Hetzner Warm standby
old-ai 178.156.167.181 CPX41 (226G disk) Hetzner Decommissioning — admin-ai pending migration to app1
wphost02 5.161.62.38 Hetzner WordPress host (migrating to app3)
Tony Hermes 87.99.159.142 Anthony's personal Hermes

SSH Key: itpp-infra (deployed to all servers, root SSH key-only) Base OS: Debian 13 (all netcup boxes), Debian 12 (Hetzner boxes)

Active Projects

Project Status Server Description
Super Search Live v1.1.0 Core Multi-provider web search (SearXNG → Exa → Firecrawl) + extraction via FastMCP
DRE / Debt Recovery Experts Active Core + Cloudflare Collections platform, CRM, approval workflows, letter templates
Apex Track Experience Active wphost02 (→ app3) NASA Top Speed Event, WPForms, SMTP, PayPal
Shark Attack Fantasy League Active Core shark.iamgmb.com, Discord bridge, changelog
VoIPSimplicity Planning RingLogix CPaaS, Yealink rentals
Ops Portal Active Core ops.itpropartner.com, admin dashboards, customer portal
admin-ai Migration Phase 0 Pending old-ai → app1 Move LiteLLM from Hetzner to netcup

Server-by-Server Inventory

Core (netcup RS 2000 — 152.53.192.33)

System Services

Service Type Purpose
Hermes Agent systemd AI assistant gateway (Telegram + cron)
Caddy systemd Reverse proxy (port 80/443) with Let's Encrypt
Ops Portal systemd (uvicorn) Internal admin dashboard at ops.itpropartner.com
OSINT API systemd (uvicorn) OSINT search API on port 8100
Hermes Assistant (PWA) systemd PWA backend on port 8082
Shark Game systemd Shark Attack Fantasy League backend on port 8083
Ollama systemd Local LLM fallback (llama3.2:3b)
Tailscale systemd Mesh VPN for infra access
StrongSwan + xl2tpd systemd L2TP/IPsec VPN server
WireGuard systemd Tunnel to home router (10.77.0.0/24)
MySQL Tunnel systemd SSH tunnel to wphost02 MySQL (port 33060)
Redis systemd Cache layer
Prometheus Node Exporter systemd System metrics collection
Hermes Browser systemd Chromium headless for browser automation (9222)

Docker Containers

Container Image Port Purpose
Komodo Core ghcr.io/moghtech/komodo-core:2 9120 Docker management UI
Komodo Mongo mongo Komodo database
Prometheus prom/prometheus 9090 (loopback) Metrics aggregation
Grafana grafana/grafana 3002 (loopback) Monitoring dashboards
Twenty CRM twentycrm/twenty:latest 3001 DRE CRM (crm.debtrecoveryexperts.com)
Twenty Worker twentycrm/twenty:latest Twenty async worker
Twenty DB postgres:16 Twenty PostgreSQL
Twenty Redis redis Twenty cache
DocuSeal docuseal/docuseal:latest 3000 (loopback) Document signing
SearXNG searxng/searxng:latest 8888 (loopback) Private search engine
Vaultwarden vaultwarden/server:1.33.2 8080 Password manager (tailnet only)

Caddy Sites

Domain Backend Purpose
sign.itpropartner.com → :3000 (DocuSeal) Document signing
core.itpropartner.com static files JSON data, capabilities, portal
crm.debtrecoveryexperts.com → :3001 (Twenty) DRE customer CRM
portal.debtrecoveryexperts.com static DRE customer portal
internal.debtrecoveryexperts.com static DRE internal dashboard
app.itpropartner.com static + → :8082 Hermes Assistant PWA
shark.itpropartner.com static + → :8083 Shark game
shark.iamgmb.com static + → :8083 Shark game alias
ops.itpropartner.com → :8090 + static Ops admin portal

Cron Jobs (Hermes-managed, 22 total)

Job Schedule Purpose
hermes-live-sync Every 15m S3 backup sync
service-health-check Every 5m Service monitoring
Home-Router-Watchdog Every 5m CCR watchdog
vps-threshold-check Every 15m Disk/mem alerts
hermes-memory-consolidate Every 10m Memory pruning
IMAP-email-triage Every 60m Email classification
IMAP-triage-watchdog Every 10m IMAP health check
shonuff-inbox-agent Every 15m Inbox monitoring
daily-tech-digest Daily 11 AM Tech news summary
exotic-vehicle-scout Daily 9 AM New car alerts
shark-scraper Every 60m Shark sighting data
dre-mail-poller Every 1m DRE inbox polling
dre-pending-approval-reminder Hourly DRE approval reminders
boxpilot-triage Every 10m BoxPilot email triage
bounce-check Every 60m Email bounce detection
apex-mail-watchdog Every 5m Apex SMTP health
ops-data-collector Every 5m Ops portal data
lynis-scan Daily 3 AM Security audit
firecrawl-usage-check Daily 9 AM Web search credits
model-usage-check Daily 8 AM LLM availability
hetzner-weekly-snapshots Weekly Mon 5 AM Server snapshots
home-router-daily-backup Daily 6 AM CCR config backup

app1 (netcup RS 4000 — 152.53.36.131) — AI/ML Stack

Docker Containers

Container Image Port Purpose
Ollama ollama/ollama:latest 11434 (loopback) Local LLM inference
Open WebUI ghcr.io/open-webui/open-webui:latest 3000 (loopback) LLM chat UI (ai.itpropartner.com)
n8n n8nio/n8n:latest 5678 (loopback) Workflow automation (n8n.itpropartner.com)
n8n-postgres postgres:16-alpine n8n database

Caddy Sites

Domain Backend Purpose
ai.itpropartner.com → :3000 (Open WebUI) LLM chat interface
n8n.itpropartner.com → :5678 (n8n) Automation workflows

System Services

  • Caddy, Docker, fail2ban, unattended-upgrades

Plans

  • LiteLLM + Postgres (migrating from old-ai)
  • Designated admin-ai.itpropartner.com target

app2 (netcup RS 4000 — 152.53.39.202) — Infrastructure

Docker Containers

Container Image Port Purpose
UNMS API ubnt/unms:latest UISP core API
UCRM ubnt/unms-crm:4.5.33 80/443 Customer CRM
UNMS Nginx ubnt/unms-nginx:latest 80/443/8089 UISP web proxy
UNMS Postgres ubnt/unms-postgres:latest 5432 UISP database
UNMS SiriDB ubnt/unms-siridb:latest UISP time-series DB
UNMS RabbitMQ rabbitmq:3.7.28-alpine 5672 UISP message queue
UNMS Netflow ubnt/unms-netflow:latest 2055/udp Network flow data
UNMS Fluentd ubnt/unms-fluentd:latest 24224 Log aggregation

System Services

Docker, fail2ban, unattended-upgrades


app3 (netcup RS 4000 — 152.53.241.111) — Web Apps

Status: Fresh server, Docker installed, no containers yet. Designated for WordPress migrations from wphost02/SiteGround.


app1-bu (Hetzner CPX11 — 5.161.114.8) — Warm Standby

System Services

Service Purpose
Hermes Gateway dormant — auto-activates if Core fails (5 min watchdog)
Ollama Local LLM fallback
StrongSwan L2TP/IPsec VPN
Tailscale Mesh VPN
Node Exporter Metrics
apex-status-receiver Apex endpoint

wphost02 (Hetzner — 5.161.62.38) — WordPress

Site Purpose
apextrackexperience.com Apex Track Experience (WordPress + WooCommerce + WPForms)
apex-status Apex status receiver endpoint

MySQL: WordPress DB at apextrackexperience_1781549652 (via Core tunnel on port 33060)


old-ai (Hetzner CPX41 — 178.156.167.181)

Status: Decommissioning. All services migrating to app1.

Container Image Purpose
LiteLLM ghcr.io/berriai/litellm:v1.84.0 LLM proxy (admin-ai)
LiteLLM Postgres postgres:16 Model configs, API keys
Caddy caddy:latest TLS termination
Qdrant qdrant/qdrant:latest Vector DB
Ollama ollama/ollama:latest Local models

Subscriptions & API Keys

Active Subscriptions

Service Plan Cost Billing Purpose
netcup RS 2000 + 4× RS 4000 ~$156/mo Monthly All servers
Hetzner CPX11 + CPX41 + misc ~$50/mo Monthly app1-bu, old-ai (decomm), wphost02, old boxes (decomm)
MXroute Email hosting ~$15/yr? Annual email@redacted, DRE email, boys' email
SiteGround WordPress hosting ~$10/mo Monthly Apex Track Experience (migrating)
Cloudflare Free (DNS + CDN + Access) Free All domains
OpenRouter Pay-as-you-go Variable Prepaid Model API fallback
Firecrawl Pay-as-you-go Variable Web scraping for AI
ScrapingAnt (or similar) Pay-as-you-go Variable Web extraction
SyncroMSP RMM ~$2/mo per endpoint Monthly 26 customers, ITPP endpoints
Wasabi S3 Hot cloud storage ~$7/TB/mo Monthly Hermes backups, router configs
Nous Research Hermes Agent license Usage-based Prepaid Hermes platform

API Keys & Tokens

Key Location Service Status
FIRECRAWL_API_KEY ~/.hermes/.env Web scraping Active
CLOUDFLARE_API_TOKEN ~/.hermes/.env DNS management Active
NETCUP_API_KEY ~/.hermes/.env Server provisioning Active
NETCUP_PASSWORD ~/.hermes/.env SCP API auth Active
SYNCROMSP_API_TOKEN ~/.hermes/.env RMM platform Active
TELEGRAM_BOT_TOKEN ~/.hermes/.env Telegram bot Active
LITELLM_MASTER_KEY ~/.hermes/config.yaml + /opt/ai/.env admin-ai proxy Active
LITELLM_SALT_KEY /opt/ai/.env Admin-ai encryption Active
OpenRouter API Key ~/.hermes/config.yaml Model API + credits Active
Hermes Nous Portal Key Hermes updates Not logged in
SERVER_PASSWORDS ~/.hermes/.env Server root passwords Active
DeepSeek API Key Litellm Postgres (encrypted) DeepSeek LLM API Active
Anthropic API Key LiteLLM Postgres (encrypted) Claude models Active
Google/Gemini API Key LiteLLM Postgres (encrypted) Gemini vision + text Active
OpenAI API Key LiteLLM Postgres (encrypted) GPT models Active
Groq API Key LiteLLM Postgres (encrypted) Fast inference Active
xAI API Key LiteLLM Postgres (encrypted) Grok models Active

FOSS Software Stack

Core Infrastructure

Software License Server Purpose
Debian 13 GPL All Server OS
Docker + Compose Apache 2.0 All Container runtime
Caddy Apache 2.0 Core, app1 Reverse proxy + TLS
Tailscale BSD Core, app1-bu Mesh VPN
WireGuard GPL Core Site-to-site VPN
StrongSwan GPL Core, app1-bu L2TP/IPsec VPN
Prometheus Apache 2.0 Core Metrics
Grafana AGPL Core Dashboards

Applications

Software License Server Purpose
Hermes Agent Proprietary Core, app1-bu AI assistant
Ollama MIT Core, app1, app1-bu Local LLM
Open WebUI MIT app1 LLM chat UI
n8n Sustainable Use License app1 Workflow automation
Twenty CRM AGPL Core Customer CRM
DocuSeal AGPL Core Document signing
SearXNG AGPL Core Private search
Vaultwarden GPL Core Password manager
Komodo GPL Core Docker management
LiteLLM MIT old-ai (→ app1) LLM proxy
Qdrant Apache 2.0 old-ai Vector DB
UISP/UNMS Proprietary app2 ISP management
UCRM Proprietary app2 Customer management
Ollama Models Various Core, app1, old-ai LLM weights
Redis BSD Core Cache
PostgreSQL PostgreSQL Core, app1, app2 Databases
WordPress GPL wphost02 (→ app3) Web content
WooCommerce GPL wphost02 E-commerce
WPForms GPL wphost02 Form builder
BlogWatcher Core cron RSS monitoring
Lynis GPL Core Security auditing

Domains & DNS

Domain Registrar DNS Purpose
itpropartner.com Cloudflare Primary
germainebrown.com MXroute Email
debtrecoveryexperts.com Cloudflare DRE
apextrackexperience.com SiteGround/Cloudflare Apex
iamgmb.com MXroute Boys' email
voipsimplicity.com Cloudflare VoIPSimplicity
tailc2f3b0.ts.net Tailscale Tailnet DNS (internal)

Backup & DR

Backup Schedule Location Retention
Hermes full backup Daily 5 AM s3://hermes-vps-backups/hermes-full-backup/
Hermes live state Every 15 min s3://hermes-vps-backups/live/ Rolling
Router config (home) Daily 6 AM s3://mikrotik-ccr-backups/
Root essentials Daily S3
Hetzner snapshots Weekly Mon Hetzner Cloud

DR Standby: app1-bu (Hetzner) boots and auto-restores from S3 if Core is down for 2+ minutes.


Email Accounts

Address Provider Server Purpose
email@redacted MXroute mail.germainebrown.com Germaine's primary
email@redacted MXroute mail.germainebrown.com Sho'Nuff's outbox
email@redacted SiteGround c1113726.sgvps.net:2525/993 Apex business
email@redacted MXroute DRE client-facing
email@redacted MXroute DRE debtor-facing
email@redacted MXroute Garrison Brown
email@redacted MXroute Greyson Brown