root
3c07727f5c
Mark v5.0/v5.1 SUPERSEDED (error-detection thesis failed at -0.40 delta); v4.1 is canonical. Cut v4.1 proposal with 12 version strings bumped. Moonshot to Mistral across production seats; production worker de-kimi'd 2026-08-18. Data-retention posture corrected 7/9 to 8/9 no-training (DeepSeek sole exception). Reconciled COGS with measured Mistral spend. Committed deployed v4.0 content and research docs to resolve the repo/live fork.
204 lines
11 KiB
HTML
204 lines
11 KiB
HTML
|
|
<!-- ====== 5. JUDGE POOL v2.3 ====== -->
|
|
<section id="panel">
|
|
<h2><span class="n">05</span>Judge Pool v2.3: 11 Seats, 9 Vendors, Zero Double-Ups</h2>
|
|
<p class="lead">
|
|
The panel that produced the validation data ran at 8 of 11 seats because two seats hit a
|
|
provider credit wall mid-run and one model family could not be dispatched at all. v2.3 is the
|
|
spec written in response to that failure. Full detail lives in the
|
|
<a href="judge-pool-spec.md">judge pool specification v2.3</a>.
|
|
</p>
|
|
|
|
<h3>The roster</h3>
|
|
<table>
|
|
<thead><tr><th>Band</th><th>Seat</th><th>Model</th><th>Vendor</th><th>Scores</th></tr></thead>
|
|
<tbody>
|
|
<tr><td>0</td><td>Research Agent</td><td>Grok 4.5</td><td>xAI</td><td>No</td></tr>
|
|
<tr><td>A</td><td>Primary Reviewer</td><td>Claude Opus 5</td><td>Anthropic</td><td>Yes</td></tr>
|
|
<tr><td>A</td><td>Cross-Check A</td><td>DeepSeek V4 Flash</td><td>DeepSeek</td><td>Yes</td></tr>
|
|
<tr><td>A</td><td>Cross-Check B</td><td>Gemini Pro Latest</td><td>Google</td><td>Yes</td></tr>
|
|
<tr><td>A</td><td>Cross-Check C</td><td>DeepSeek V4 Pro</td><td>DeepSeek</td><td>Yes</td></tr>
|
|
<tr><td>A</td><td>Legal / Regulatory</td><td>Claude Sonnet 5</td><td>Anthropic</td><td>Yes</td></tr>
|
|
<tr><td>B</td><td>Financial Integrity</td><td>MiniMax-M3</td><td>MiniMax</td><td>Yes</td></tr>
|
|
<tr><td>B</td><td>Team / Founder</td><td>Claude Fable 5</td><td>Anthropic</td><td>Yes</td></tr>
|
|
<tr><td>B</td><td>Market Reality</td><td>Qwen3.7 Plus</td><td>Alibaba</td><td>Yes</td></tr>
|
|
<tr><td>B</td><td>Execution Feasibility</td><td>GPT-5.2 Pro</td><td>OpenAI</td><td>Yes</td></tr>
|
|
<tr><td>C</td><td>Synthesis & Integrity Gate</td><td>Kimi K2.6</td><td>Moonshot</td><td>No</td></tr>
|
|
</tbody>
|
|
</table>
|
|
<p class="table-note">
|
|
Nine distinct vendors across eleven seats. Nine distinct scoring models. Zero model double-ups:
|
|
no single model occupies two scoring seats, which is the constraint that keeps correlated
|
|
failure out of the panel mean. Maximum vendor concentration is Anthropic at 3 of 11 (27.3%),
|
|
comfortably inside the 40% ceiling. DeepSeek holds 2 of 11 (18.2%). Every remaining vendor holds
|
|
exactly one seat.
|
|
</p>
|
|
|
|
<h3>What changed in v2.3</h3>
|
|
<div class="defense-grid">
|
|
<div class="defense-item">
|
|
<h4 style="margin-top:0">Pre-flight health gate</h4>
|
|
<p>
|
|
Before any scoring begins, the orchestrator pings every rostered model with a 5-second
|
|
probe and writes the result to a per-model health file. Any model returning HTTP 400,
|
|
HTTP 429, or a no-healthy-deployments error is swapped for its pre-assigned failover before
|
|
a single scoring call is spent. The 2026-08-12 run burned roughly 12 dispatches discovering
|
|
dead models at runtime. That failure mode is now closed.
|
|
</p>
|
|
</div>
|
|
<div class="defense-item">
|
|
<h4 style="margin-top:0">Pre-baked failover roster</h4>
|
|
<p>
|
|
Every seat carries a named failover from a different vendor, resolved at gate time rather
|
|
than improvised mid-run. Failover selection preserves both the vendor-diversity ceiling and
|
|
the no-double-up rule, so a degraded panel is still a valid panel rather than an
|
|
accidentally correlated one.
|
|
</p>
|
|
</div>
|
|
<div class="defense-item">
|
|
<h4 style="margin-top:0">Credit-wall resilience</h4>
|
|
<p>
|
|
The provider credit exhaustion that cost two seats mid-run is now detected at the gate and
|
|
treated as an availability failure, not an error. Anthropic capacity has been restored and
|
|
the Primary Reviewer seat runs Claude Opus 5 as specified.
|
|
</p>
|
|
</div>
|
|
<div class="defense-item">
|
|
<h4 style="margin-top:0">Permanent exclusions</h4>
|
|
<p>
|
|
One frontier model family proved structurally incapable of running as a panel seat under
|
|
our orchestration and is permanently excluded from the roster, not merely deprioritized.
|
|
Excluded models cannot be selected as a failover target either.
|
|
</p>
|
|
</div>
|
|
</div>
|
|
|
|
<h3>Latency</h3>
|
|
<p>
|
|
v2.3 targets a critical path of roughly 113 seconds, against 218 seconds measured on the
|
|
v2.1 architecture. The improvement comes from band parallelism: Band A and Band B seats execute
|
|
concurrently rather than sequentially, and the Synthesis seat is the only stage that must wait
|
|
for all scoring seats to return.
|
|
</p>
|
|
|
|
<h3>The ten scored dimensions</h3>
|
|
<p>
|
|
Every scoring seat rates the proposal 1 to 10 on the same ten dimensions, so panel spread is
|
|
computed dimension by dimension and not only in aggregate:
|
|
</p>
|
|
<div class="arch-box">
|
|
Problem Clarity · Market Opportunity · Product Differentiation
|
|
Revenue Model Viability · Go-to-Market Strategy · Competitive Moat
|
|
Financial Projections · Team / Execution · Risk Mitigation · Legal / Compliance
|
|
</div>
|
|
|
|
<h3>The Synthesis and Integrity Gate</h3>
|
|
<p>
|
|
The Band C seat never scores. It reads all scoring output and performs a fixed checklist:
|
|
verify score arithmetic, compute panel means and per-dimension spread, compute the delta against
|
|
the solo baseline, flag every score more than 1.5 standard deviations from the panel mean with a
|
|
written rationale, and confirm the verdict follows from panel evidence rather than from the
|
|
Primary Reviewer alone. That gate is what turns eleven opinions into one auditable report.
|
|
</p>
|
|
</section>
|
|
|
|
<!-- ====== 6. WORKED EXAMPLE ====== -->
|
|
<section id="worked">
|
|
<h2><span class="n">06</span>Worked Example: VentureBuilt v2, Where the Score Said Nothing</h2>
|
|
<p class="lead">
|
|
This is the clearest case in the validation set, because it is the one where score elevation
|
|
delivered exactly zero and error detection delivered everything. Real scores from the
|
|
2026-08-12 run.
|
|
</p>
|
|
|
|
<div class="wt-scores">
|
|
<div class="score-block score-block--narrative">
|
|
<div class="score-label">Panel mean · 8 judges</div>
|
|
<div class="score-value">6.14<span>/10</span></div>
|
|
<p class="muted" style="margin-bottom:0">Median 6.45 · standard deviation 1.758 · spread 5.4 (min 2.8, max 8.2)</p>
|
|
</div>
|
|
<div class="score-block score-block--compliance" style="border-left-color:var(--muted)">
|
|
<div class="score-label">Solo baseline · Claude Opus 5</div>
|
|
<div class="score-value" style="color:var(--muted)">6.10<span>/10</span></div>
|
|
<p class="muted" style="margin-bottom:0">Spread 2.4 · delta +0.04 · statistically tied with the panel</p>
|
|
</div>
|
|
</div>
|
|
|
|
<h3>Individual seat scores</h3>
|
|
<table>
|
|
<thead><tr><th>Seat</th><th>Model</th><th>Score</th><th>Sigma from mean</th><th>Flag</th></tr></thead>
|
|
<tbody>
|
|
<tr><td>Band B · Financial Integrity</td><td>MiniMax</td><td><strong style="color:var(--accent2)">8.2</strong></td><td>+1.17</td><td>Within 1.5 sigma</td></tr>
|
|
<tr><td>Band B · Market Reality</td><td>Gemini</td><td>7.7</td><td>+0.89</td><td>Within 1.5 sigma</td></tr>
|
|
<tr><td>Band A · Primary Reviewer</td><td>Opus 5</td><td>6.9</td><td>+0.43</td><td>Within 1.5 sigma</td></tr>
|
|
<tr><td>Band A · Legal / Regulatory</td><td>Qwen</td><td>6.5</td><td>+0.21</td><td>Within 1.5 sigma</td></tr>
|
|
<tr><td>Band A · Cross-Check C</td><td>DeepSeek V4 Pro</td><td>6.4</td><td>+0.15</td><td>Within 1.5 sigma</td></tr>
|
|
<tr><td>Band A · Cross-Check B</td><td>Gemini</td><td>6.2</td><td>+0.04</td><td>Within 1.5 sigma</td></tr>
|
|
<tr><td>Band B · Team / Founder</td><td>Kimi</td><td>4.4</td><td>-0.99</td><td>Within 1.5 sigma</td></tr>
|
|
<tr><td>Band B · Execution Feasibility</td><td>DeepSeek V4 Pro</td><td><strong style="color:var(--bad)">2.8</strong></td><td>-1.91</td><td><span class="tag tag-red">OUTLIER</span></td>
|
|
</tr>
|
|
</tbody>
|
|
</table>
|
|
|
|
<div class="callout callout--fail">
|
|
<strong>The average is a lie of composition.</strong> A 6.14 reads as a solid, fundable
|
|
proposal with room to improve. The distribution says something completely different: the money
|
|
is excellent (8.2) and the delivery plan is close to unworkable (2.8). Those are not two
|
|
opinions about one thing. They are two accurate findings about two different things, and
|
|
averaging them produces a number that describes neither.
|
|
</div>
|
|
|
|
<h3>What the outlier actually found</h3>
|
|
<p>
|
|
The 2.8 was not a grumpy model. The Integrity Gate challenged it at 1.91 sigma and it survived
|
|
the challenge on evidence: a contractor budget broken 4x to 7x, an architecture that regressed
|
|
from v1 with no schema and no API contract, and a founder workload of 37 engagements plus 950
|
|
hours alongside an MSP day job. The Team seat (4.4) and the Band A generalists (6.2 to 6.9)
|
|
all acknowledged the same workload problem. They weighted it less severely. The Execution
|
|
specialist is the only seat that forced it into the verdict.
|
|
</p>
|
|
<div class="highlight">
|
|
Remove that one seat and the panel averages <strong>6.6</strong>, which reads as a clean GO and
|
|
ships a proposal with an 800-hour founder gap in it. The specialist seat cost a few cents of
|
|
inference and changed the verdict.
|
|
</div>
|
|
|
|
<h3>Fix-It items, ranked by materiality</h3>
|
|
<ol class="fixit-list">
|
|
<li>
|
|
<div class="fixit-head"><span class="fixit-tag fixit-tag--critical">Critical</span><span class="fixit-delta">Execution</span></div>
|
|
<strong>Fix the contractor budget or cut the scope.</strong> $1,500/mo buys 105 to 140 hours
|
|
at market rates, not the volume the plan assumes. Raise to roughly $7,500/mo or reduce scope
|
|
to fit the hours actually purchased.
|
|
</li>
|
|
<li>
|
|
<div class="fixit-head"><span class="fixit-tag fixit-tag--critical">Critical</span><span class="fixit-delta">Financial</span></div>
|
|
<strong>Restate Year 2 on a recognized-revenue basis.</strong> On run-rate the year looks
|
|
healthy. On recognized revenue it loses roughly $11K to $18K. Present both.
|
|
</li>
|
|
<li>
|
|
<div class="fixit-head"><span class="fixit-tag fixit-tag--high">High</span><span class="fixit-delta">Competitive</span></div>
|
|
<strong>Withdraw or qualify the uniqueness claim.</strong> LivePlan Plan Review and IdeaProof
|
|
already ship in this space. Reposition on a defensible axis.
|
|
</li>
|
|
<li>
|
|
<div class="fixit-head"><span class="fixit-tag fixit-tag--high">High</span><span class="fixit-delta">Team</span></div>
|
|
<strong>Name the contractor and the sourcing plan before Phase 2.</strong> A budget line with
|
|
no named person is not a capacity plan.
|
|
</li>
|
|
<li>
|
|
<div class="fixit-head"><span class="fixit-tag fixit-tag--medium">Medium</span><span class="fixit-delta">Go-to-market</span></div>
|
|
<strong>Map the Year 1 to Year 2 GTM bridge.</strong> Eight net-new signups per month appear
|
|
in the model with no acquisition mechanism behind them.
|
|
</li>
|
|
<li>
|
|
<div class="fixit-head"><span class="fixit-tag fixit-tag--medium">Medium</span><span class="fixit-delta">Legal</span></div>
|
|
<strong>Complete data protection and trademark clearance before Phase 0 to 1.</strong>
|
|
</li>
|
|
</ol>
|
|
<p class="table-note">
|
|
Panel verdict: CONDITIONAL GO. Estimated rework 15 to 20 hours. The solo baseline returned a
|
|
6.10 and none of the six conditions above.
|
|
</p>
|
|
</section>
|