ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Sync docs, audit artifacts, project notes, and VerdictTank proposal docs

Browse Source 
- audit/phase-one + phase-two: security audit briefs, findings, credential-rotation plan, Docker-USER hardening scripts, rollback refs
- disaster-recovery/restore-test-log.md + backup-dr-audit-2026-08-10.md
- clients/ (modelortho SEO audit, ai-biz-dev competitive landscape), notes/ (tiktok strategy)
- projects/: front-desk-voice-agent, seo-visibility-checker product plan, hotnow-savannah HTML, resend-transactional-email, backup-dashboard-enhancements, code-review-graph, seo-ci-architecture
- proposals/verdicttank/: architecture v4.0, methodology, judge-pool review, consolidation reasoning, cross-check review
- docs/super-search/firecrawl-provider-strategy.md
- updates: CHANGELOG, model-chain, projects-master-readme, intelsight.io
- .gitignore: exclude nested standalone repos (seo-tool, venturebuilt)
This commit is contained in:
root
2026-08-26 02:27:28 -04:00
parent 23e9751d38
commit f5175f1ce0
55 changed files with 14669 additions and 3 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/infrastructure/model-chain.md
+4 -3
View File
@@ -1,6 +1,6 @@
# AI Model Architecture — IT Pro Partner
**Updated:** August 8, 2026
**Updated:** August 17, 2026
Two separate concepts: **fallback chain** (survival — direct API keys) and **operational chain** (daily toolbox — admin-ai only). The two-key strategy means operational keys run through admin-ai/LiteLLM; fallback keys are direct provider API keys with daily limits.
@@ -19,7 +19,7 @@ Survives admin-ai outage. All direct provider keys have daily caps. Fires in ord
| **F2** | `gemini-3.6-flash` | `google` (direct) | fallback | $2 |
| **F3** | `grok-4.5` | `xai` (direct) | fallback | $2 |
| **F4** | `claude-sonnet-5` | `anthropic` (direct) | fallback | $5 |
| **F5** | `gpt-4.1-nano` | `openai` (direct) | fallback | $2 |
| **F5** | `gpt-5-mini` | `openai` (direct) | fallback | $2 |
Total emergency budget: **$14/day** — down from $45 single-leg burn on Aug 5.
@@ -78,4 +78,5 @@ All route through admin-ai. Shared budget via `hermes-agent-v5` key.
- **Jul 30, 2026:** LiteLLM key regenerated — hermes-agent-v3 created but config.yaml NOT updated. Hermes fell back to DeepSeek direct since Jul 30 19:25 UTC.
- **Jul 31, 2026:** Discovered stale key in config.yaml. Generated hermes-agent-v5 with same model list. Verified chat OK. Cleaned up stale keys (v3, v4, v4b).
- **Aug 5, 2026:** Admin-ai budget cap hit (~$20). Fallback chain exhausted 4 dead legs, landed on Anthropic direct. Burned $45 in 10 hours on claude-sonnet-5 via direct key. Anthropic key capped until Sep 1.
- **Aug 6, 2026:** Root cause of Aug 5 outage: 4 fallback legs dead simultaneously (admin-ai budget, DeepSeek balance $0, grok-4.6 404, Anthropic capped). Implemented two-key strategy (operational vs fallback). Rotated all 5 fallback keys. Added F5 (gpt-4.1-nano via OpenAI). Added haiku-4-5 and grok-4.5 to operational chain. Fixed grok-4.6 → grok-4.5. Synced Anita profile identically. Budget raised to $30.
- **Aug 6, 2026:** Root cause of Aug 5 outage: 4 fallback legs dead simultaneously (admin-ai budget, DeepSeek balance $0, grok-4.6 404, Anthropic capped). Implemented two-key strategy (operational vs fallback). Rotated all 5 fallback keys. Added F5 (gpt-5-mini via OpenAI). Added haiku-4-5 and grok-4.5 to operational chain. Fixed grok-4.6 → grok-4.5. Synced Anita profile identically. Budget raised to $30.
- **Aug 17, 2026:** Live verification of fallback chain against the LiteLLM model DB (admin-ai). Confirmed present: `deepseek-v4-flash`, `gemini-3.6-flash` (registered as `gemini/gemini-3.6-flash`), `grok-4.5` (`xai/grok-4.5`), `claude-sonnet-5`. Replaced F5 `gpt-4.1-mini` → `gpt-5-mini` — the gpt-4.1 series is no longer present in LiteLLM. Synced `config.yaml` `fallback_providers` and `model.fallbacks`.
docs/projects/projects-master-readme.md
+1
View File
@@ -10,3 +10,4 @@ Master index of all internal and client projects.
- **[BoxPilot Logistics](./boxpilot/README.md)**: Logistics and shipping management platform. (PLANNED)
- **[Open-Source SaaS Alternatives](../projects/oss-saas-alternatives.md)**: 10 self-hostable replacements for paid SaaS (AppFlowy, Immich, Documenso, Excalidraw, Penpot, Cal.DIY, ListMonk, Dub, RustDesk, FluidVoice). Future productize/host candidates. (FUTURE PROJECTS)
- **[Hosted AI Agent Platform](../projects/hosted-agent-platform.md)**: AgentThread-style hosted Hermes agents — each client/space gets a containerized agent with chat, live URL, and credit billing. Reference: agentthread.ai. (FUTURE PROJECTS)
- **[Code-Review Graph (Tooling)](../projects/code-review-graph.md)**: Adopt the code-review-graph structural knowledge pattern for our own repos, plus "everything is a plugin" skill discipline. Do NOT adopt the dsh harness itself — Hermes already covers ~80%. Reference: deepseek-ai/deepseek-harness. (FUTURE PROJECTS)
docs/super-search/firecrawl-provider-strategy.md
+95
View File
@@ -0,0 +1,95 @@
# Firecrawl — Provider Strategy & Configuration
**Created:** 2026-08-10 | **Status:** Active Burn Period → Hobby
---
## Plan Timeline
| Date | Event |
|---|---|
| 2026-08-10 | Standard plan (100k credits, $99/mo) — Firecrawl promoted to primary |
| 2026-09-10 | Billing period ends — Standard plan credits expire |
| 2026-09-11 | Downgrade to Hobby plan (5k credits, $16–19/mo) |
---
## Why We Burned the Standard Plan
Usage was ~1,400 credits/month — 1.4% of the Standard plan's 100k allocation. The $99/mo Standard plan was over-provisioned by 70x. Firecrawl was position #11 (dead last) in the search fallback chain, so it was almost never called. The decision: burn through Standard credits aggressively until the plan naturally expires on Sep 10, then let the Hobby downgrade take effect with 5k credits/month (~3.5x our actual needs).
---
## Rate Limit History
| When | Firecrawl Rate Limit | Context |
|---|---|---|
| Pre-Aug 10 | 5 tokens / 60s | Position #11 in search chain — rarely called |
| Aug 10 (15:45) | 20 tokens / 60s | Promoted to primary in both chains |
| Aug 10 (15:51) | 100 tokens / 5s | Maximized for burn period (20 req/s throughput) |
### Rate Limit Comparison (Pricing Page)
| Plan | Concurrent Requests | Credits/Month |
|---|---|---|
| Free | 2 | 1,000 |
| Hobby | 5 | 5,000 |
| Standard | 50 | 100,000 |
| Growth | 100 | 500,000 |
| Scale | 150 | 1,000,000 |
No per-second or per-minute rate caps published per tier. Concurrent request caps are the only documented limit, and Super Search runs calls sequentially — it never hits the 50 concurrent cap even on Standard.
---
## Provider Chain Order
### `web_search`
```
1. Firecrawl ← PRIMARY (promoted Aug 10 from #11)
2. SearXNG (local)
3. Exa
4. OpenCorporates
5. CourtListener
6. DuckDuckGo
7. Brave
8. Serper
9. Tavily
10. Perplexity
11. Parallel (Exa + Brave + DuckDuckGo)
```
### `web_extract`
```
1. Firecrawl ← PRIMARY (promoted Aug 10 from #4)
2. Jina Reader
3. Trafilatura
4. Browserless (CF bypass)
```
---
## Files Modified
| File | Change |
|---|---|
| `/root/docker/super-search/server.py` | Firecrawl moved to position #1 in both search + extract chains; docstrings updated |
| `/root/docker/super-search/ratelimit.py` | Firecrawl rate limit: 5/min → 20/min → 100/5s |
---
## Monitoring
| Job | Schedule | Description |
|---|---|---|
| `firecrawl-usage-check` (2a2f) | Every 2 hours | Queries Firecrawl API `/v1/team/credit-usage` — reports remaining credits to Telegram |
| Script: `firecrawl-credit-check.sh` | — | Curl-based, no agent needed |
---
## Post-Burn Reversion Plan (Sep 11)
After Hobby downgrade:
1. Revert rate limit to `{"tokens": 5, "interval": 60.0}` in `ratelimit.py`
2. Consider moving Firecrawl back to fallback position (TBD based on credit consumption patterns during burn period)
3. Keep 2-hour monitoring until burn period confirmed complete