verdicttank/dogfood-v4-report.md

VerdictTank v4.0 — Dogfood Verdict Report

Date: 2026-08-18 Subject: The v4.0 proposal itself, run through the v4.0 single-pass panel (10 seats: 9 scoring + synthesis gate) Verdict: CONDITIONAL

---

1. Headline Scores

Metric	Score
Proposal Strength	71
Investor Readiness	44
Composite	58
Divergence	27 (proposal ahead)
Blocking findings	Yes (6)

2. Dimension Means

Dimension	Score
Structure & Communication	93.8
Internal Consistency	86.2
Solution Coherence	84.9
Problem Clarity	81.2
Risk & Compliance	64.3
Financial Integrity	61.8
Execution Feasibility	50.0
Team & Capability	31.7
Evidence Quality	24.0
Market Reality	15.7

3. Panel Read

The panel's one-line take: a well-designed critique loop that reads as an internal blueprint, not a funded product. It scores 80+ on the four dimensions that measure writing quality, and collapses on the three that measure proof: market reality (15.7), evidence (24.0), team (31.7). Every margin claim, pricing scenario, and shipped-date assertion is unverified.

4. Six Blocking Conditions

1. Market evidence — bottom-up SAM, three named competitors with ARR/pricing, current paid account count by tier + trailing 30-day volume.
2. Production proof — anonymized logs: 500+ reviews processed, median latency <90s, uptime >99.5% over 30 days.
3. Verified COGS — vendor invoices/token logs from 100+ real reviews, reconciled to the $0.36 standard / $3.00 White-Label ceiling.
4. Zero data-retention agreements — executed with all nine vendors, or drop those vendors from the roster.
5. Senior technical hire — name a platform engineer/cofounder with multi-tenant SaaS track record.
6. Arithmetic fix — ceiling volume is 2,690 paid reviews, not 3,940 (a 1,250 gap), and the annual MRR reduction recomputes to $2,619, not $2,946.

---

5. Panel Integrity Findings (from running the panel itself)

5.1 glm-5.2 was not flaky — it was token-starved

Root cause: glm-5.2 is a reasoning model. On the financial seat (the heaviest brief), it consumed its entire token budget on hidden reasoning and returned zero visible tokens.

Metric	Before (12k floor)	After (24k floor)
finish_reason	length	stop
reasoning_tokens	11,998	14,635
visible content	0-3,015 chars	8,545 chars
JSON parsed	no	yes (all 10 keys)

Conclusion: glm-5.2 stays in the roster. The financial seat needs ~17k completion tokens (14.6k reasoning + 2k output). This was a budget bug, not a model defect.

5.2 Two production blockers found and fixed

1. VerdictTank-Key allowlist was stale (v3.5-era, 11 models). It 403'd on six v4.0 roster models: sonar, gpt-5.2, glm-5.2, claude-haiku-4-5, xai/grok-4.3, and bare gemini-pro-latest (it only knew gemini/gemini-pro-latest).
2. Reasoning-model token floors too low — deployed worker had glm-5.2 and deepseek-v4-pro at 8,000, which silently truncated reasoning models.

6. Deployment Fixes Applied

Fix A — VerdictTank-Key allowlist (LiteLLM)

Added the 6 missing v4 models, keeping the existing 11 for backward compatibility. Total now 17 models.

Fix B — MODEL_QUIRKS token floors (/opt/verdicttank/worker.py)

• glm-5.2: 8,000 → 24,000
• deepseek-v4-pro: 8,000 → 12,000
• gemini-pro-latest: added at 12,000
• kimi-k2.6: unchanged (already 16,000)

Fix C — Robust JSON extraction (/opt/verdicttank/worker.py)

Replaced the fragile non-greedy regex + rfind parser (which silently dropped the financial seat) with brace-matching that handles nested objects, strings containing braces, and trailing commas in arrays. Verified against 6 test cases (fenced nested, fenced explicit, truncated, plain nested, trailing comma, empty) — all pass.

7. Remaining Work

• Re-run the full panel against the deployed worker to confirm end-to-end parity with the runner
• Decide whether glm-5.2 stays as permanent financial seat or whether the 24k floor is acceptable long-term cost
• Clear the six blocking conditions in §4 before GA
• Rotate the temp key verdicttank-v4-test (or delete it) once the production key is confirmed working