verdicttank/parts/p2.html

<!-- ====== 2. PIPELINE STORY ====== -->
<section id="pipeline">
  <h2><span class="n">02</span>How We Found This: Four Generations and a Self-Review</h2>
  <p class="lead">
    VerdictTank is a proposal review engine, not a proposal writer. It ingests a finished document
    and returns scored dimensions plus a ranked list of concrete Fix-It items. The pipeline was not
    designed in the abstract. It was hardened across four architectural generations, and then it
    was pointed at its own proposals.
  </p>

  <div class="evolution-grid">
    <div class="evolution-card">
      <div class="version-badge">v2</div>
      <h3 style="margin-top:0">Single-Model Scorer</h3>
      <div class="breakthrough-label">Breakthrough: the dual-axis rubric</div>
      <p>
        v2 established the core insight: a proposal has two independent quality axes. Narrative
        quality (clarity, structure, persuasion) and compliance quality (does it actually answer
        the scored requirements). One model scored both from one prompt. It proved the concept and
        exposed the flaw: the axes bled together. A beautifully written section that missed a
        mandatory requirement scored too high, because the same reasoning pass that admired the
        prose also graded the compliance.
      </p>
    </div>
    <div class="evolution-card">
      <div class="version-badge">v3</div>
      <h3 style="margin-top:0">Separated Scoring Passes</h3>
      <div class="breakthrough-label">Breakthrough: axis isolation</div>
      <p>
        v3 split scoring into two independent passes with two purpose-built prompts. The narrative
        pass never sees the compliance rubric. The compliance pass never rewards eloquence. This is
        the decision that makes the dual score trustworthy: the two numbers can now disagree, and
        their disagreement carries information. A 9/10 narrative next to a 4/10 compliance is a
        proposal about to lose.
      </p>
    </div>
    <div class="evolution-card">
      <div class="version-badge">v4</div>
      <h3 style="margin-top:0">Multimodel Adversarial Review</h3>
      <div class="breakthrough-label">Breakthrough: cross-model verification</div>
      <p>
        A single model scoring in isolation is confidently wrong at a predictable rate. v4
        introduced a multimodel pipeline: a fast model produces first-pass scores and Fix-It
        candidates, then a stronger model reviews that output adversarially, challenging every
        deduction and confirming each Fix-It maps to real proposal text. Scores stopped drifting
        between runs. This is the generation that made the output defensible.
      </p>
    </div>
    <div class="evolution-card evolution-card--current">
      <div class="version-badge version-badge--current">v5 · Current</div>
      <h3 style="margin-top:0">Specialist Panel and the Integrity Gate</h3>
      <div class="breakthrough-label">Breakthrough: disagreement as output</div>
      <p>
        v5 replaces the adversarial pair with an 11-seat specialist panel across nine vendors, and
        adds a synthesis seat whose only job is to compute panel statistics, flag scores more than
        1.5 standard deviations from the mean, and reconcile the verdict against the evidence.
        The output is no longer a number. It is a number, a spread, an outlier list, and a ranked
        set of material errors with the judge that caught each one.
      </p>
    </div>
  </div>

  <h3>The self-review that broke the old thesis</h3>
  <p>
    Before selling a review engine we ran the engine on our own work. We assembled the panel and
    scored three real proposals, in full, with the same prompts and rubric a paying customer would
    get. One of the three was VerdictTank's own sibling product. The panel returned a NO GO on it.
  </p>
  <p>
    That run cost roughly $150 in inference and returned 8 of 11 seats. Two seats were lost to a
    provider credit wall hit mid-run and one to a model family that could not be dispatched at all.
    The incomplete panel is why v2.3 of the judge pool spec now requires a pre-flight health gate
    and a pre-baked failover roster, covered in section 5. The results below are what those 8 seats
    produced, and we report them at 8 seats rather than extrapolating to 11.
  </p>

  <div class="callout callout--proof">
    <strong>The pipeline is its own reference implementation.</strong> The full architecture is
    documented in the companion
    <a href="architecture.html">technical architecture document</a> to a standard where an engineer
    can implement it from the spec alone. VerdictTank is pre-revenue. We make zero claims about
    users, beta cohorts, or external validation. What we claim is narrower and verifiable: the
    architecture is built, the pipeline runs, it was executed against three real proposals on
    2026-08-12, and it failed its own headline thesis in public.
  </div>
</section>

<!-- ====== 3. VALIDATION DATA ====== -->
<section id="validation">
  <h2><span class="n">03</span>Validation Run: 3 Real Proposals, 8 Reporting Seats</h2>
  <p class="lead">
    Every figure in this section comes from the 2026-08-12 validation run. Nothing is modeled,
    projected, or illustrative. The solo baseline is Claude Opus 5 scoring the same documents
    against the same 10-dimension rubric.
  </p>

  <h3>Panel mean vs solo baseline</h3>
  <table>
    <thead>
      <tr><th>Proposal</th><th>Panel mean</th><th>Solo baseline</th><th>Delta</th><th>Panel spread</th><th>Solo spread</th><th>Verdict</th></tr>
    </thead>
    <tbody>
      <tr>
        <td><strong>RFP Tank v1.0</strong></td><td>4.40</td><td>4.93</td>
        <td style="color:var(--bad)">-0.53</td><td>3.7</td><td>1.4</td>
        <td><span class="tag tag-red">NO GO</span></td>
      </tr>
      <tr>
        <td><strong>VentureBuilt v2</strong></td><td>6.14</td><td>6.10</td>
        <td style="color:var(--accent2)">+0.04</td><td>5.4</td><td>2.4</td>
        <td><span class="tag tag-amber">CONDITIONAL GO</span></td>
      </tr>
      <tr>
        <td><strong>CartMySupply</strong></td><td>4.29</td><td>5.00</td>
        <td style="color:var(--bad)">-0.71</td><td>2.6</td><td>1.8</td>
        <td><span class="tag tag-red">NO GO</span></td>
      </tr>
      <tr class="comp-table__us">
        <td><strong>Aggregate</strong></td><td><strong>4.94</strong></td><td><strong>5.34</strong></td>
        <td><strong>-0.40</strong></td><td colspan="2"><strong>Panel spread exceeded solo spread on all 3</strong></td>
        <td><strong>THESIS FAIL</strong></td>
      </tr>
    </tbody>
  </table>
  <p class="table-note">
    Thesis under test: the panel must show a greater than 0.5 point advantage over the solo mean to
    justify premium pricing. Result: FAIL on all three proposals and FAIL in aggregate. Panel
    composition for this run was 8 reporting judges (4 Band A, 4 Band B) out of 11 specified seats,
    a 73% coverage rate.
  </p>

  <h3>Why the panel scored lower</h3>
  <p>
    The panel does not elevate scores. It sharpens error detection, and error detection on a flawed
    document moves the number down. All three proposals contained severe cross-cutting defects that
    additional specialist scrutiny exposed more precisely: fatal execution gaps, competitive
    mispositioning, and legal blockers. The solo baseline was directionally correct on all three.
    The panel added precision, not points.
  </p>
  <p>
    That is the entire finding, and it inverts the sales pitch. If your proposal is sound, the panel
    will roughly agree with a good solo model and cost you more. If your proposal has a hole in it,
    the panel finds the hole and the solo model does not. You are not buying a score. You are buying
    the probability that a specific, expensive, named mistake gets caught before an evaluator or an
    investor finds it for you.
  </p>

  <h3>Specialist divergence, measured</h3>
  <table>
    <thead>
      <tr><th>Observation</th><th>Evidence from the run</th><th>What it means</th></tr>
    </thead>
    <tbody>
      <tr>
        <td>Generalist seats run optimistic</td>
        <td>Gemini Pro scored RFP Tank 6.7 as a Band A generalist and 3.0 as the Band B Market specialist. Same model, same document, 3.7 points apart.</td>
        <td>Band A generalist scoring without specialist cross-check is systematically over-optimistic. The role, not the model, drives the score.</td>
      </tr>
      <tr>
        <td>Role divergence beats model divergence</td>
        <td>DeepSeek V4 Pro scored VentureBuilt 6.4 as Cross-Check C and 2.8 as Execution Feasibility. A 3.6 point split inside one vendor.</td>
        <td>Panel diversity is not primarily about buying different vendors. It is about buying different questions.</td>
      </tr>
      <tr>
        <td>One seat can flip a verdict</td>
        <td>Remove the 2.8 Execution score from VentureBuilt and the panel averages 6.6, reading as a clean GO. With it, the verdict is CONDITIONAL GO with a named contractor-budget fix.</td>
        <td>The lowest score in the panel is frequently the only one doing work. Averaging is what a solo model already does.</td>
      </tr>
      <tr>
        <td>Tight clustering is also a signal</td>
        <td>CartMySupply produced zero outliers beyond 1.5 sigma and the tightest spread of the three (sigma 0.89).</td>
        <td>Unanimity across nine vendors on a low score is a far stronger NO GO than one model's low score.</td>
      </tr>
    </tbody>
  </table>
</section>