v4.1 go-live cut, Moonshot to Mistral swap, v5.x superseded
Mark v5.0/v5.1 SUPERSEDED (error-detection thesis failed at -0.40 delta); v4.1 is canonical. Cut v4.1 proposal with 12 version strings bumped. Moonshot to Mistral across production seats; production worker de-kimi'd 2026-08-18. Data-retention posture corrected 7/9 to 8/9 no-training (DeepSeek sole exception). Reconciled COGS with measured Mistral spend. Committed deployed v4.0 content and research docs to resolve the repo/live fork.
This commit is contained in:
root
+177
@@ -0,0 +1,177 @@
|
||||
|
||||
<!-- ====== 2. PIPELINE STORY ====== -->
|
||||
<section id="pipeline">
|
||||
<h2><span class="n">02</span>How We Found This: Four Generations and a Self-Review</h2>
|
||||
<p class="lead">
|
||||
VerdictTank is a proposal review engine, not a proposal writer. It ingests a finished document
|
||||
and returns scored dimensions plus a ranked list of concrete Fix-It items. The pipeline was not
|
||||
designed in the abstract. It was hardened across four architectural generations, and then it
|
||||
was pointed at its own proposals.
|
||||
</p>
|
||||
|
||||
<div class="evolution-grid">
|
||||
<div class="evolution-card">
|
||||
<div class="version-badge">v2</div>
|
||||
<h3 style="margin-top:0">Single-Model Scorer</h3>
|
||||
<div class="breakthrough-label">Breakthrough: the dual-axis rubric</div>
|
||||
<p>
|
||||
v2 established the core insight: a proposal has two independent quality axes. Narrative
|
||||
quality (clarity, structure, persuasion) and compliance quality (does it actually answer
|
||||
the scored requirements). One model scored both from one prompt. It proved the concept and
|
||||
exposed the flaw: the axes bled together. A beautifully written section that missed a
|
||||
mandatory requirement scored too high, because the same reasoning pass that admired the
|
||||
prose also graded the compliance.
|
||||
</p>
|
||||
</div>
|
||||
<div class="evolution-card">
|
||||
<div class="version-badge">v3</div>
|
||||
<h3 style="margin-top:0">Separated Scoring Passes</h3>
|
||||
<div class="breakthrough-label">Breakthrough: axis isolation</div>
|
||||
<p>
|
||||
v3 split scoring into two independent passes with two purpose-built prompts. The narrative
|
||||
pass never sees the compliance rubric. The compliance pass never rewards eloquence. This is
|
||||
the decision that makes the dual score trustworthy: the two numbers can now disagree, and
|
||||
their disagreement carries information. A 9/10 narrative next to a 4/10 compliance is a
|
||||
proposal about to lose.
|
||||
</p>
|
||||
</div>
|
||||
<div class="evolution-card">
|
||||
<div class="version-badge">v4</div>
|
||||
<h3 style="margin-top:0">Multimodel Adversarial Review</h3>
|
||||
<div class="breakthrough-label">Breakthrough: cross-model verification</div>
|
||||
<p>
|
||||
A single model scoring in isolation is confidently wrong at a predictable rate. v4
|
||||
introduced a multimodel pipeline: a fast model produces first-pass scores and Fix-It
|
||||
candidates, then a stronger model reviews that output adversarially, challenging every
|
||||
deduction and confirming each Fix-It maps to real proposal text. Scores stopped drifting
|
||||
between runs. This is the generation that made the output defensible.
|
||||
</p>
|
||||
</div>
|
||||
<div class="evolution-card evolution-card--current">
|
||||
<div class="version-badge version-badge--current">v5 · Current</div>
|
||||
<h3 style="margin-top:0">Specialist Panel and the Integrity Gate</h3>
|
||||
<div class="breakthrough-label">Breakthrough: disagreement as output</div>
|
||||
<p>
|
||||
v5 replaces the adversarial pair with an 11-seat specialist panel across nine vendors, and
|
||||
adds a synthesis seat whose only job is to compute panel statistics, flag scores more than
|
||||
1.5 standard deviations from the mean, and reconcile the verdict against the evidence.
|
||||
The output is no longer a number. It is a number, a spread, an outlier list, and a ranked
|
||||
set of material errors with the judge that caught each one.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<h3>The self-review that broke the old thesis</h3>
|
||||
<p>
|
||||
Before selling a review engine we ran the engine on our own work. We assembled the panel and
|
||||
scored three real proposals, in full, with the same prompts and rubric a paying customer would
|
||||
get. One of the three was VerdictTank's own sibling product. The panel returned a NO GO on it.
|
||||
</p>
|
||||
<p>
|
||||
That run cost roughly $150 in inference and returned 8 of 11 seats. Two seats were lost to a
|
||||
provider credit wall hit mid-run and one to a model family that could not be dispatched at all.
|
||||
The incomplete panel is why v2.3 of the judge pool spec now requires a pre-flight health gate
|
||||
and a pre-baked failover roster, covered in section 5. The results below are what those 8 seats
|
||||
produced, and we report them at 8 seats rather than extrapolating to 11.
|
||||
</p>
|
||||
|
||||
<div class="callout callout--proof">
|
||||
<strong>The pipeline is its own reference implementation.</strong> The full architecture is
|
||||
documented in the companion
|
||||
<a href="architecture.html">technical architecture document</a> to a standard where an engineer
|
||||
can implement it from the spec alone. VerdictTank is pre-revenue. We make zero claims about
|
||||
users, beta cohorts, or external validation. What we claim is narrower and verifiable: the
|
||||
architecture is built, the pipeline runs, it was executed against three real proposals on
|
||||
2026-08-12, and it failed its own headline thesis in public.
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<!-- ====== 3. VALIDATION DATA ====== -->
|
||||
<section id="validation">
|
||||
<h2><span class="n">03</span>Validation Run: 3 Real Proposals, 8 Reporting Seats</h2>
|
||||
<p class="lead">
|
||||
Every figure in this section comes from the 2026-08-12 validation run. Nothing is modeled,
|
||||
projected, or illustrative. The solo baseline is Claude Opus 5 scoring the same documents
|
||||
against the same 10-dimension rubric.
|
||||
</p>
|
||||
|
||||
<h3>Panel mean vs solo baseline</h3>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>Proposal</th><th>Panel mean</th><th>Solo baseline</th><th>Delta</th><th>Panel spread</th><th>Solo spread</th><th>Verdict</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td><strong>RFP Tank v1.0</strong></td><td>4.40</td><td>4.93</td>
|
||||
<td style="color:var(--bad)">-0.53</td><td>3.7</td><td>1.4</td>
|
||||
<td><span class="tag tag-red">NO GO</span></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>VentureBuilt v2</strong></td><td>6.14</td><td>6.10</td>
|
||||
<td style="color:var(--accent2)">+0.04</td><td>5.4</td><td>2.4</td>
|
||||
<td><span class="tag tag-amber">CONDITIONAL GO</span></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><strong>CartMySupply</strong></td><td>4.29</td><td>5.00</td>
|
||||
<td style="color:var(--bad)">-0.71</td><td>2.6</td><td>1.8</td>
|
||||
<td><span class="tag tag-red">NO GO</span></td>
|
||||
</tr>
|
||||
<tr class="comp-table__us">
|
||||
<td><strong>Aggregate</strong></td><td><strong>4.94</strong></td><td><strong>5.34</strong></td>
|
||||
<td><strong>-0.40</strong></td><td colspan="2"><strong>Panel spread exceeded solo spread on all 3</strong></td>
|
||||
<td><strong>THESIS FAIL</strong></td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p class="table-note">
|
||||
Thesis under test: the panel must show a greater than 0.5 point advantage over the solo mean to
|
||||
justify premium pricing. Result: FAIL on all three proposals and FAIL in aggregate. Panel
|
||||
composition for this run was 8 reporting judges (4 Band A, 4 Band B) out of 11 specified seats,
|
||||
a 73% coverage rate.
|
||||
</p>
|
||||
|
||||
<h3>Why the panel scored lower</h3>
|
||||
<p>
|
||||
The panel does not elevate scores. It sharpens error detection, and error detection on a flawed
|
||||
document moves the number down. All three proposals contained severe cross-cutting defects that
|
||||
additional specialist scrutiny exposed more precisely: fatal execution gaps, competitive
|
||||
mispositioning, and legal blockers. The solo baseline was directionally correct on all three.
|
||||
The panel added precision, not points.
|
||||
</p>
|
||||
<p>
|
||||
That is the entire finding, and it inverts the sales pitch. If your proposal is sound, the panel
|
||||
will roughly agree with a good solo model and cost you more. If your proposal has a hole in it,
|
||||
the panel finds the hole and the solo model does not. You are not buying a score. You are buying
|
||||
the probability that a specific, expensive, named mistake gets caught before an evaluator or an
|
||||
investor finds it for you.
|
||||
</p>
|
||||
|
||||
<h3>Specialist divergence, measured</h3>
|
||||
<table>
|
||||
<thead>
|
||||
<tr><th>Observation</th><th>Evidence from the run</th><th>What it means</th></tr>
|
||||
</thead>
|
||||
<tbody>
|
||||
<tr>
|
||||
<td>Generalist seats run optimistic</td>
|
||||
<td>Gemini Pro scored RFP Tank 6.7 as a Band A generalist and 3.0 as the Band B Market specialist. Same model, same document, 3.7 points apart.</td>
|
||||
<td>Band A generalist scoring without specialist cross-check is systematically over-optimistic. The role, not the model, drives the score.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Role divergence beats model divergence</td>
|
||||
<td>DeepSeek V4 Pro scored VentureBuilt 6.4 as Cross-Check C and 2.8 as Execution Feasibility. A 3.6 point split inside one vendor.</td>
|
||||
<td>Panel diversity is not primarily about buying different vendors. It is about buying different questions.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>One seat can flip a verdict</td>
|
||||
<td>Remove the 2.8 Execution score from VentureBuilt and the panel averages 6.6, reading as a clean GO. With it, the verdict is CONDITIONAL GO with a named contractor-budget fix.</td>
|
||||
<td>The lowest score in the panel is frequently the only one doing work. Averaging is what a solo model already does.</td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td>Tight clustering is also a signal</td>
|
||||
<td>CartMySupply produced zero outliers beyond 1.5 sigma and the tightest spread of the three (sigma 0.89).</td>
|
||||
<td>Unanimity across nine vendors on a low score is a far stronger NO GO than one model's low score.</td>
|
||||
</tr>
|
||||
</tbody>
|
||||
</table>
|
||||
</section>
|
||||
Reference in New Issue
Block a user