# ITPP Infrastructure Documentation

## Server Inventory

### Core Server
- **Hostname**: core
- **IP**: 152.53.192.33
- **Provider**: netcup RS 2000 G12
- **Specs**: RS 2000 G12 — 8 vCPU EPYC 9645, 16 GB DDR5 ECC, 512 GB NVMe
- **OS**: Debian 13
- **Role**: Primary infrastructure server
- **Key Services**:
  - Hermes Agent
  - LiteLLM
  - Vaultwarden
  - Uptime Kuma
  - ops-portal
  - Super Search MCP
  - Cron jobs

### App1 Server
- **Hostname**: app1
- **IP**: 152.53.36.131
- **Provider**: netcup RS 4000
- **Specs**: RS 4000 G12 — 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
- **OS**: Debian 13
- **Role**: Application server
- **Key Services**:
  - Ollama
  - Open WebUI
  - n8n
  - Postgres
  - Caddy
  - SearXNG

### App2 Server
- **Hostname**: app2
- **IP**: 152.53.39.202
- **Provider**: netcup RS 4000
- **Specs**: RS 4000 G12 — 12 vCPU EPYC 9645, 32 GB DDR5 ECC, 1 TB NVMe
- **OS**: Debian 13
- **Role**: Application server
- **Key Services**:
  - UNMS
  - UniFi Controller (v10.0.162, Docker: jacobalberty/unifi)
    - Source: unifi.itpropartner.com (Caddy → localhost:8443)
    - Old controller: 178.156.131.57 (v9.5.21) — scheduled to decommission after migration
    - Backup: /var/lib/unifi/backup/autobackup/ on old server → S3 daily 2 AM ET via unifi-backup-sync.sh
  - Hudu
  - Traccar
  - Caddy (4 domains)
  - Docker

### App3 Server
- **Hostname**: app3
- **IP**: 152.53.241.111
- **Provider**: netcup RS 2000
- **Specs**: RS 2000 G12 — 8 vCPU EPYC 9645, 16 GB DDR5 ECC, 512 GB NVMe
- **OS**: Debian 13
- **Role**: Application server
- **Key Services**:
  - WordPress
  - Apex Track

### WPHost02 Server
- **Hostname**: wphost02
- **IP**: 5.161.62.38
- **Provider**: RunCloud
- **Role**: WordPress hosting

### App1 Backup Server
- **Hostname**: app1-bu
- **IP**: 5.161.114.8
- **Provider**: Hetzner CPX11
- **Role**: Warm standby for app1

### Old Hudu Server
- **Hostname**: hudu (old)
- **IP**: 178.156.130.130
- **Provider**: Hetzner
- **Role**: Fallback for Hudu (7-day retention)

## Domain/DNS Map

### itpropartner.com
- **DNS Provider**: SiteGround
- **Subdomains**:
  - core.itpropartner.com → 152.53.192.33
  - hudu.itpropartner.com → 152.53.39.202
  - ops.itpropartner.com → 152.53.192.33
  - portal.itpropartner.com → 152.53.192.33
  - admin-ai.itpropartner.com → 152.53.36.131
  - uptimekuma.itpropartner.com → 152.53.192.33
  - unifi.itpropartner.com → 152.53.39.202

### fleettracker360.com
- **DNS Provider**: Cloudflare
- **Subdomains**:
  - gps.fleettracker360.com → 152.53.39.202 (Caddy)

### forefrontwireless.com
- **DNS Provider**: Unknown
- **Subdomains**:
  - unms.forefrontwireless.com → 152.53.39.202 (Caddy)

### debtrecoveryexperts.com
- **DNS Provider**: Cloudflare
- **Subdomains**:
  - portal.debtrecoveryexperts.com → 152.53.192.33
  - crm.debtrecoveryexperts.com → Cloudflare Access

### iamgmb.com
- **DNS Provider**: SiteGround
- **Subdomains**:
  - vault.iamgmb.com
  - shark.iamgmb.com
  - sign.iamgmb.com

## Backup Pipeline

- **hermes-live-sync**: Every 15 min → s3://hermes-vps-backups/live/
- **hermes-full-backup**: Daily 1 AM → s3://hermes-vps-backups/hermes-full-backup/
- **home-router-backup**: Daily 6 AM → s3://mikrotik-ccr-backups/ (currently empty)
- **unms-backup-sync**: Daily 6 AM
- **hetzner-weekly-snapshots**: Monday 5 AM
- **root-essentials-backup**: Daily 3 AM
- **backup-audit-check**: Daily 2 AM
- **Warm standby**: app1-bu checks Core every 10 min, auto-failover

## Caddy Config

### Core Server
- Proxied paths:
  - /vault/
  - /n8n/
  - /webui/
  - /litellm/

### App1 Server
- Proxied domains:
  - admin-ai.itpropartner.com

### App2 Server
- Proxied domains:
  - hudu.itpropartner.com
  - gps.fleettracker360.com
  - unms.forefrontwireless.com
  - unifi.itpropartner.com

## SSL

All SSL certificates are issued via Let's Encrypt through Caddy. No manual certificate management is required. Certificates auto-renew.

## Service Access

| Service | URL | Location | Notes |
|---|---|---|---|
| Open WebUI | http://ai.itpropartner.com | app1 | User-facing chat UI |
| Open WebUI Admin | https://admin-ai.itpropartner.com/ui | app1 | Admin panel (user: admin, password: LiteLLM master key from /root/docker/litellm/.env) |
| LiteLLM Proxy | https://admin-ai.itpropartner.com | app1 | API gateway for all models |
| Ops Portal | https://ops.itpropartner.com | Core | Internal operations dashboard |
| Uptime Kuma | https://uptimekuma.itpropartner.com | Core | Service monitoring |
| Vaultwarden | https://core.itpropartner.com/vault/ | Core | Password vault |
| n8n | https://core.itpropartner.com/n8n/ | Core | Workflow automation |
| CRM (DRE) | https://crm.debtrecoveryexperts.com | Cloudflare | TwentyCRM (Cloudflare Access) |

### MCP Access from Open WebUI

All app1 MCPs (Super Search, Browser, Filesystem, Email, Git) are wired into Open WebUI's tool system. See [mcp-servers/README.md](../mcp-servers/README.md#access) for details on adding and testing MCPs.

## SSH Access

- **Key**: itpp-infra key on all servers
- **User**: ippadmin (sudo privileges)
- **Root SSH**: Enabled on app1, app2, app3 (key-only exception)

## Firewall

UFW is enabled on all servers. Current rules for app2:
- Allowed ports: 22, 80, 443, 3000, 8080, 8082, 8089, 8443, 8843, 3478, 10001

## Notes

This document is a living record of the ITPP infrastructure. Update it as changes occur.