ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Model fallback chain: GPT-5.5 primary, 5 providers, Ollama removed, DNS updates

Browse Source
This commit is contained in:
root
2026-07-17 13:28:05 -04:00
parent 17e1fee30e
commit 3424b9013b
3 changed files with 43 additions and 12 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+21 -7
View File
@@ -13,22 +13,20 @@
- **Provider:** netcup RS 2000 G12 - **Provider:** netcup RS 2000 G12
- **Specs:** 8 vCPU EPYC 9645, 15 GB DDR5 ECC, 512 GB NVMe - **Specs:** 8 vCPU EPYC 9645, 15 GB DDR5 ECC, 512 GB NVMe
- **OS:** Debian 13 - **OS:** Debian 13
- **Role:** Hermes + Ollama fallback + Portals - **Role:** Hermes + Portals
- **Key Services:** - **Key Services:**
- Hermes Agent (Telegram + cron, 22 cron jobs) - Hermes Agent (Telegram + cron, 22 cron jobs)
- Caddy reverse proxy (10 domains, auto-TLS) - Caddy reverse proxy (12 domains, auto-TLS)
- Ops Portal (FastAPI, port 8090) - Ops Portal (FastAPI, port 8090)
- Prometheus (native, port 9090) + Grafana (native, port 3002) - Prometheus (native, port 9090) + Grafana (native, port 3002)
- Uptime Kuma (Docker, port 3001) — 9+ monitors - Uptime Kuma (Docker, port 3001) — 9+ monitors
- Vaultwarden (Docker, port 8080) — vault.iamgmb.com - Vaultwarden (Docker, port 8080) — vault.iamgmb.com
- Super Search MCP v2.0.0 — migrated to app1
- Twenty CRM (Docker) — crm.debtrecoveryexperts.com - Twenty CRM (Docker) — crm.debtrecoveryexperts.com
- DocuSeal (Docker, port 3000) — sign.core.itpropartner.com - DocuSeal (Docker, port 3000) — sign.core.itpropartner.com
- SearXNG (Docker, port 8888) - SearXNG (Docker, port 8888)
- Komodo (Docker, port 9120) - Komodo (Docker, port 9120)
- Tailscale, StrongSwan, WireGuard (home CCR tunnel 10.77.0.0/24) - Tailscale, StrongSwan, WireGuard (home CCR tunnel 10.77.0.0/24)
- Redis cache - Redis cache
- Ollama (systemd, port 11434)
### App1 Server ### App1 Server
- **Hostname:** app1 - **Hostname:** app1
@@ -38,7 +36,6 @@
- **OS:** Debian 13 - **OS:** Debian 13
- **Role:** AI/Service Hub - **Role:** AI/Service Hub
- **Key Services:** - **Key Services:**
- Ollama (Docker, port 11434)
- Open WebUI (Docker, port 3000) — ai.itpropartner.com - Open WebUI (Docker, port 3000) — ai.itpropartner.com
- n8n + Postgres (Docker, port 5678) — n8n.itpropartner.com - n8n + Postgres (Docker, port 5678) — n8n.itpropartner.com
- LiteLLM (Docker) + Postgres — admin-ai.itpropartner.com - LiteLLM (Docker) + Postgres — admin-ai.itpropartner.com
@@ -89,8 +86,25 @@
### Legacy / Decommissioned ### Legacy / Decommissioned
- **old-ai:** 178.156.167.181 (Hetzner CPX41) — **decommissioned** (LiteLLM migrated to app1) - **old-ai:** 178.156.167.181 (Hetzner CPX41) — **decommissioned** (LiteLLM migrated to app1)
- **old app1:** 87.99.144.163 (Hetzner CPX11) — **deleted** - **old app1:** 87.99.144.163 (Hetzner CPX11) — **deleted**
- **wphost02:** 5.161.62.38 (Hetzner CPX21) — **migrating to app3** - **wphost02:** 5.161.62.38 (Hetzner CPX21) — **migrated to app3**
- **Tony Hermes:** 87.99.159.142 (Hetzner CPX21) — Anthony's personal Hermes - **Ollama:** Removed from Core (systemd) and app1 (Docker) Jul 17
---
## Model Fallback Chain
All providers use direct API keys. GPT-5.5 quality survives through admin-ai → OpenRouter, then degrades through DeepSeek → Gemini → Grok.
| # | Model | Provider | Gateway |
|---|---|---|---|
| Primary | GPT-5.5 | admin-ai | Self-hosted LiteLLM (app1) |
| Fallback 1 | GPT-5.5 | OpenRouter | openrouter.ai |
| Fallback 2 | DeepSeek v4 Pro | DeepSeek | api.deepseek.com |
| Fallback 3 | Gemini 3.5 Flash | Google | generativelanguage.googleapis.com |
| Fallback 4 | Grok 4.5 | xAI | api.x.ai |
**Credits (Jul 17):** DeepSeek $58, OpenRouter ~$30 remaining, OpenAI/xAI/Google on pay-as-you-go
**Health check:** Daily 8 AM cron (`model-usage-check`)
--- ---
backup-plan.md
-1
View File
@@ -52,7 +52,6 @@
| LiteLLM | Postgres pg_dump + config.yaml → S3 (daily) | 24 hr | | LiteLLM | Postgres pg_dump + config.yaml → S3 (daily) | 24 hr |
| Open WebUI | /app/backend/data → S3 (daily) | 24 hr | | Open WebUI | /app/backend/data → S3 (daily) | 24 hr |
| n8n | Postgres pg_dump → S3 (daily) | 24 hr | | n8n | Postgres pg_dump → S3 (daily) | 24 hr |
| Ollama | Models preserved in Docker volume | 24 hr |
| MCP servers | Configs in Git + Docker volumes | 24 hr | | MCP servers | Configs in Git + Docker volumes | 24 hr |
| Mattermost | Postgres DB + data volume → S3 (daily) | 24 hr | | Mattermost | Postgres DB + data volume → S3 (daily) | 24 hr |
project-log-2026-07-17.md
+22 -4
View File
@@ -32,7 +32,25 @@
- DNS: Cloudflare, updated A record to 152.53.241.111 - DNS: Cloudflare, updated A record to 152.53.241.111
- Previously: was resolving to old IP, Cloudflare zone accessible - Previously: was resolving to old IP, Cloudflare zone accessible
## Model Credits Reliability ## Model Fallback Chain (Major Update)
- DeepSeek balance: $59.29 (not exhausted) - **Primary:** GPT-5.5 via admin-ai (self-hosted LiteLLM on app1)
- Cron fixed to track deepseek-v4-pro instead of deepseek-chat - **Fallback 1:** GPT-5.5 via OpenRouter (different gateway, same model)
- Local Ollama fallback: llama3.2:3b - **Fallback 2:** DeepSeek v4 Pro direct (DeepSeek API)
- **Fallback 3:** Gemini 3.5 Flash direct (Google API)
- **Fallback 4:** Grok 4.5 direct (xAI API)
- **Funded:** OpenRouter ($280 total, $250 used), DeepSeek ($58)
- **Removed:** Ollama uninstalled from Core (systemd) and app1 (Docker)
- **Health check:** Daily 8 AM cron checks all 5 providers
## Model Credits (Jul 17)
- DeepSeek: $58.06 remaining
- OpenRouter: $280 total, ~$30 remaining
- GPT-5.5, Gemini, Grok: pay-as-you-go via direct API keys
- OpenAI direct API key exists but not in fallback chain
## DNS Updates
- noc.itpropartner.com → 152.53.36.131 (Mattermost, app1)
- vault.itpropartner.com → 152.53.36.131 (Vaultwarden, app1)
- status.itpropartner.com → 152.53.192.33 (Status page, Core)
- my.itpropartner.com → moved from CREATE to PRODUCTION
- mainwp.itpropartner.com → moved from DELETE to PRODUCTION