Sync docs, audit artifacts, project notes, and VerdictTank proposal docs
- audit/phase-one + phase-two: security audit briefs, findings, credential-rotation plan, Docker-USER hardening scripts, rollback refs - disaster-recovery/restore-test-log.md + backup-dr-audit-2026-08-10.md - clients/ (modelortho SEO audit, ai-biz-dev competitive landscape), notes/ (tiktok strategy) - projects/: front-desk-voice-agent, seo-visibility-checker product plan, hotnow-savannah HTML, resend-transactional-email, backup-dashboard-enhancements, code-review-graph, seo-ci-architecture - proposals/verdicttank/: architecture v4.0, methodology, judge-pool review, consolidation reasoning, cross-check review - docs/super-search/firecrawl-provider-strategy.md - updates: CHANGELOG, model-chain, projects-master-readme, intelsight.io - .gitignore: exclude nested standalone repos (seo-tool, venturebuilt)
This commit is contained in:
root
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,333 @@
|
||||
# VerdictTank Pipeline Consolidation Recommendation
|
||||
## Lens: Reasoning / Logic Overlap Detection (Contradiction Detector)
|
||||
**Author:** Kimi K2.6 (Reasoning-Verification specialist)
|
||||
**Spec:** judge-pool-spec.md v2.1
|
||||
**Target:** ~15% reduction
|
||||
**Date:** 2026-08-12
|
||||
|
||||
---
|
||||
|
||||
## Executive finding
|
||||
|
||||
**ONE recommendation:** Collapse **Phase 10 (Reasoning-Verification)** and **Phase 11 (Audit Agent)** into a single post-scoring synthesis seat: **Synthesis and Integrity Gate**.
|
||||
|
||||
These two roles ask the same meta-question with different labels:
|
||||
|
||||
| Role | Stated question | Actual job |
|
||||
|---|---|---|
|
||||
| Phase 10 Reasoning-Verification | Where do judges contradict? | Read all score sets, find disagreements, synthesize |
|
||||
| Phase 11 Audit Agent | What did the panel miss / groupthink? | Read finished review, find blind spots / groupthink, adjust confidence |
|
||||
|
||||
Both are **non-scoring meta-reviewers of the panel's own output**. Neither sees the raw proposal as primary input in a unique way that the other does not eventually consume. Both produce a **post-hoc integrity adjustment**, not an independent domain score. Under a contradiction-detection lens, they are statistically and logically collinear seats.
|
||||
|
||||
---
|
||||
|
||||
## 1. Changes
|
||||
|
||||
### Collapse
|
||||
|
||||
| Remove | Absorb into |
|
||||
|---|---|
|
||||
| Phase 10: Reasoning-Verification (standalone) | **New Phase 10: Synthesis and Integrity Gate** |
|
||||
| Phase 11: Audit Agent (standalone) | (same merged seat) |
|
||||
|
||||
### New merged role definition
|
||||
|
||||
| Field | Value |
|
||||
|---|---|
|
||||
| **Phase** | 10 (final) |
|
||||
| **Role** | **Synthesis and Integrity Gate** |
|
||||
| **Model** | Kimi K2.6 (Moonshot) — keeps contradiction specialty as primary |
|
||||
| **Fallback 1** | Claude Fable 5 (Anthropic) — former Audit primary |
|
||||
| **Fallback 2** | GPT-5.6 Sol (OpenAI) — former Reasoning F2 |
|
||||
| **Scores?** | No (preserves non-scoring fence) |
|
||||
| **Input** | ALL score sets from phases 2-9 + assembled draft review |
|
||||
| **Output** | (a) Contradiction / alignment report, (b) synthesized dimension scores via section 4.1 formula, (c) blind-spot / groupthink notes, (d) single confidence adjustment +/-0.5 |
|
||||
|
||||
### Prompt contract (must stay explicit)
|
||||
|
||||
Merged prompt has **two ordered sections**, not a vague do-both:
|
||||
|
||||
1. **Contradiction pass** (former Phase 10): pairwise/cluster disagreement map across Primary, Validation, Cross-Check avg, Financial, Team, Market, Execution, Legal. Emit synthesized scores.
|
||||
2. **Integrity pass** (former Phase 11): given the synthesis above, scan for groupthink (panel sigma too low), missed domain blind spots, and apply **one** confidence adjustment (+/-0.5).
|
||||
|
||||
Single model, single call, single output schema. No second serial hop.
|
||||
|
||||
### Spec section edits required
|
||||
|
||||
| Section | Edit |
|
||||
|---|---|
|
||||
| section 1.1 Pipeline Phases | Drop row 11; rewrite Phase 10 as Synthesis and Integrity Gate; phase count **11 to 10** |
|
||||
| section 1.2 Enterprise Lineup | **13 seats to 11 seats** (12 unique models still available in pool; one fewer active seat) |
|
||||
| section 1.2 Model double-ups | Remove Fable 5: Team + Audit double-up. Fable becomes Reasoning/Audit **fallback only** (Team primary only) |
|
||||
| section 2.3 Fallback table | Delete Audit Agent row; replace Reasoning-Verification row with merged role chain above |
|
||||
| section 3.1 Hard constraints | Replace "Audit reads FINISHED review only" with "Synthesis and Integrity Gate is post-scoring, read-only, cannot open new domain scores" |
|
||||
| section 3.3 Low-complexity Enterprise | Already drops Audit — update text to drop CC-C, Team, Legal (Audit no longer separate) |
|
||||
| section 4.1 Scoring algorithm | Merge the two non-scoring bullets into one: Synthesis and Integrity Gate produces contradiction report + synthesized scores + confidence +/-0.5 |
|
||||
| section 4.2 Cost ceiling rule | Swap Tier A to B for non-scoring roles first (Research, Synthesis) — drop separate Audit |
|
||||
| section 4.4 Latency | Enterprise sequential phases lose one hop — recalibrate target downward slightly (e.g. 180s to ~165s aspirational; keep 300s max) |
|
||||
| section 6.3 Feature matrix | Replace Audit Agent (Phase 11) row with Synthesis and Integrity Gate (includes audit) — still Enterprise/WL only if desired |
|
||||
|
||||
### What this is NOT
|
||||
|
||||
- Not dropping Cross-Check C (that is diversity, not logic-overlap).
|
||||
- Not merging Primary with Validation (Validation is deliberately contaminated by Primary; different question).
|
||||
- Not merging Market with Execution (different input scopes and vertical triggers).
|
||||
- Not touching Research (unique tool surface; never scores).
|
||||
|
||||
---
|
||||
|
||||
## 2. Reduction %
|
||||
|
||||
### Seat / phase accounting (Enterprise standard panel)
|
||||
|
||||
| Metric | Before (v2.1) | After | Delta |
|
||||
|---|---|---|---|
|
||||
| Pipeline phases | 11 | **10** | -1 phase (-9.1%) |
|
||||
| Active seats | 13 | **11** | -2 seats (-15.4%) |
|
||||
| Serial post-score hops | 2 (P10 then P11) | **1** | -50% of meta-tail latency |
|
||||
| Non-scoring roles | 3 (Research, Reasoning, Audit) | **2** (Research, Synthesis) | -33% meta overhead |
|
||||
| Unique models in default lineup | 11 distinct (Sonnet x2, Fable x2) | **11 distinct** (Sonnet still x2 Validation+Legal; Fable Team-only) | pool size unchanged |
|
||||
| Anthropic share of active seats | 4/13 = 30.8% | **3/11 = 27.3%** | more headroom under 33% cap |
|
||||
|
||||
### Cost reduction (directional)
|
||||
|
||||
- Eliminate one full Tier-A (or Tier-A-fallback) completion on every Enterprise review.
|
||||
- v2.1 cited about $1.30-2.10 per standard full review — expect **~12-18% COGS cut** on the judge tail (one fewer expensive synthesis call), landing near **~15%** all-in when weighted by token volume of P10+P11 vs earlier short scoring calls.
|
||||
- Latency: remove one sequential dependency (~15-40s depending on model) from the critical path after phase 9.
|
||||
|
||||
**Headline reduction: ~15%** (seats -15.4%; cost band -12-18%; phases -9%).
|
||||
|
||||
---
|
||||
|
||||
## 3. What survives
|
||||
|
||||
### Intact domain + scoring spine
|
||||
|
||||
```
|
||||
1 Research Agent (Grok 4.5) - grounding, never scores
|
||||
2 Primary Reviewer (Claude Opus 5) - 10-dimension critique
|
||||
3 Validation Reviewer (Claude Sonnet 5) - challenges Primary
|
||||
4a Cross-Check A (GPT-5.6 Sol) - blind re-score
|
||||
4b Cross-Check B (Gemini Pro Latest) - blind re-score
|
||||
4c Cross-Check C (DeepSeek V4 Pro) - blind re-score
|
||||
5 Financial Integrity (MiniMax-M3) - cap table / unit econ
|
||||
6 Team/Founder Assessment (Claude Fable 5) - founder-market fit
|
||||
7 Market-Reality (Qwen3.7 Plus) - TAM / competitive / non-Western
|
||||
8 Execution-Feasibility (GPT-5.6 Terra) - ops / timeline realism
|
||||
9 Legal/Regulatory (Claude Sonnet 5) - multi-jurisdiction risk
|
||||
10 Synthesis and Integrity Gate (Kimi K2.6) - contradictions + audit + +/-0.5
|
||||
```
|
||||
|
||||
### Preserved invariants
|
||||
|
||||
- Research never scores
|
||||
- Three blind cross-checks, isolated from Primary/Validation
|
||||
- Domain specialists (Financial, Team, Market, Execution, Legal) remain **separate scoring questions**
|
||||
- Section 4.1 weighted aggregation formula unchanged
|
||||
- Cross-vendor fallback rule unchanged (merged chain still cross-vendor at every hop)
|
||||
- Vendor cap still satisfied (Anthropic drops to 27.3%)
|
||||
- Build validation gate (section 5.5) unchanged — still require >=0.5 panel advantage
|
||||
- Fenced post-score mode retained (merged seat cannot reopen domain scores)
|
||||
|
||||
### Why these roles were NOT collapsed
|
||||
|
||||
| Pair | Why keep separate |
|
||||
|---|---|
|
||||
| Primary vs Validation | Validation is **score-conditioned** (sees Primary). Different epistemic job than independent re-score. |
|
||||
| Validation vs Cross-Checks | CCs are blind; Validation is not. Collapsing destroys the blind control. |
|
||||
| Cross-Check A/B/C | Same question by design — but **vendor diversity is the product**. Overlap is intentional ensemble, not accidental duplication. Do not collapse under this lens. |
|
||||
| Market vs Execution | Different section 3.2 triggers, different evidence (competitive density vs ops timeline). |
|
||||
| Financial vs Execution | Cap-table math is not shipping realism. |
|
||||
| Legal vs Audit | Legal is domain-scoring on statutes; Audit was meta. Legal stays; Audit folds into Synthesis. |
|
||||
| Team vs Audit (same model Fable) | Team scores founders; Audit was meta. After merge, Fable stops double-duty — **removes same-model Team-to-Audit soft contamination path**. |
|
||||
|
||||
---
|
||||
|
||||
## 4. Risks
|
||||
|
||||
| Risk | Severity | Mitigation |
|
||||
|---|---|---|
|
||||
| **Single-call attention split** — model skimps on contradiction map OR groupthink scan | Medium | Enforce two-section JSON schema; reject output missing either contradictions[] or integrity.confidence_delta. |
|
||||
| **Loss of independent second meta-reader** — two weak meta-passes can catch what one strong pass misses | Medium | Keep dual-section prompt; optionally run Fable fallback as **shadow audit** on 10% of Enterprise reviews for 90 days (not on critical path). |
|
||||
| **K2.6 empty-output / reasoning-token burn** (section 4.5) now single-points the entire meta-tail | High | Raise min tokens to 768-1024 for merged seat; on empty go immediate F1 (Fable). Circuit breaker already in section 4.2. |
|
||||
| **Confidence +/-0.5 applied by same model that synthesized scores** — self-grading bias | Medium | Constrain: confidence delta may only move on documented groupthink (panel sigma under threshold) or explicit missed-claim list; ban free-form "I feel +/-0.5". |
|
||||
| **Pro tier marketing** currently sells no Audit; matrix row rename may confuse | Low | Feature matrix: Synthesis and Integrity (Enterprise) — capability retained, seat count honest. |
|
||||
| **Low-complexity Enterprise drop list** referenced Audit explicitly | Low | Update section 3.3 text only. |
|
||||
| **Outcome-tracking granularity** — cannot attribute T+90 errors to Reasoning vs Audit separately | Low | Accept; meta-tail was always under-identified. Track merged seat as one accuracy series. |
|
||||
| **Over-collapse temptation** — if 15% feels good, next PR kills CC-C or Validation | Process | This lens only authorizes **logic-collinear non-scoring** merges. Ensemble scorers are out of scope. |
|
||||
|
||||
### Explicit non-risks
|
||||
|
||||
- Vendor diversity of **scoring** panel unchanged (still 8 vendors available; scoring seats still multi-lab).
|
||||
- No new same-vendor Primary/meta self-audit (Kimi is not Anthropic Primary/Validation).
|
||||
- Fable no longer audits after sitting Team — **contamination risk decreases**.
|
||||
|
||||
---
|
||||
|
||||
## 5. New diagram
|
||||
|
||||
### Before (v2.1) — 11 phases / 13 seats
|
||||
|
||||
```
|
||||
[Proposal]
|
||||
|
|
||||
v
|
||||
+-----------------+
|
||||
| 1 Research | Grok 4.5 (no score)
|
||||
+--------+--------+
|
||||
v
|
||||
+-----------------+
|
||||
| 2 Primary | Opus 5 (scores)
|
||||
+--------+--------+
|
||||
v
|
||||
+-----------------+
|
||||
| 3 Validation | Sonnet 5 (scores, sees Primary)
|
||||
+--------+--------+
|
||||
v
|
||||
+----+----+
|
||||
v v v
|
||||
+----+ +----+ +----+
|
||||
|4a | |4b | |4c | Sol / Gemini / DeepSeek (blind scores)
|
||||
+--+-+ +--+-+ +--+-+
|
||||
+-----+-----+
|
||||
v
|
||||
+-----------------+
|
||||
| 5 Financial | MiniMax-M3
|
||||
+--------+--------+
|
||||
v
|
||||
+-----------------+
|
||||
| 6 Team/Founder | Fable 5
|
||||
+--------+--------+
|
||||
v
|
||||
+-----------------+
|
||||
| 7 Market | Qwen3.7 Plus
|
||||
+--------+--------+
|
||||
v
|
||||
+-----------------+
|
||||
| 8 Execution | Terra
|
||||
+--------+--------+
|
||||
v
|
||||
+-----------------+
|
||||
| 9 Legal | Sonnet 5
|
||||
+--------+--------+
|
||||
v
|
||||
+-----------------+
|
||||
|10 Reasoning | Kimi K2.6 (contradictions + synth)
|
||||
+--------+--------+
|
||||
v
|
||||
+-----------------+
|
||||
|11 Audit | Fable 5 (groupthink + +/-0.5)
|
||||
+--------+--------+
|
||||
v
|
||||
[Final Report]
|
||||
```
|
||||
|
||||
### After (recommended) — 10 phases / 11 seats (~15% seat cut)
|
||||
|
||||
```
|
||||
[Proposal]
|
||||
|
|
||||
v
|
||||
+-----------------+
|
||||
| 1 Research | Grok 4.5 (no score)
|
||||
+--------+--------+
|
||||
v
|
||||
+-----------------+
|
||||
| 2 Primary | Opus 5 (scores)
|
||||
+--------+--------+
|
||||
v
|
||||
+-----------------+
|
||||
| 3 Validation | Sonnet 5 (scores, sees Primary)
|
||||
+--------+--------+
|
||||
v
|
||||
+----+----+
|
||||
v v v
|
||||
+----+ +----+ +----+
|
||||
|4a | |4b | |4c | Sol / Gemini / DeepSeek (blind scores)
|
||||
+--+-+ +--+-+ +--+-+
|
||||
+-----+-----+
|
||||
v
|
||||
+-----------------+
|
||||
| 5 Financial | MiniMax-M3
|
||||
+--------+--------+
|
||||
v
|
||||
+-----------------+
|
||||
| 6 Team/Founder | Fable 5 (Team only — no Audit double)
|
||||
+--------+--------+
|
||||
v
|
||||
+-----------------+
|
||||
| 7 Market | Qwen3.7 Plus
|
||||
+--------+--------+
|
||||
v
|
||||
+-----------------+
|
||||
| 8 Execution | Terra
|
||||
+--------+--------+
|
||||
v
|
||||
+-----------------+
|
||||
| 9 Legal | Sonnet 5
|
||||
+--------+--------+
|
||||
v
|
||||
+-------------------------------------+
|
||||
|10 Synthesis and Integrity Gate | Kimi K2.6
|
||||
| - contradiction / alignment map |
|
||||
| - section 4.1 synthesized scores |
|
||||
| - groupthink / blind-spot notes |
|
||||
| - confidence delta +/-0.5 (gated) |
|
||||
| F1: Fable 5 · F2: Sol |
|
||||
+------------------+------------------+
|
||||
v
|
||||
[Final Report]
|
||||
```
|
||||
|
||||
### Overlap evidence (why this pair, not another)
|
||||
|
||||
```
|
||||
INPUT SCOPE
|
||||
proposal scores finished-review
|
||||
Primary ### . .
|
||||
Validation ## ### .
|
||||
Cross-Checks ### . . intentional ensemble (KEEP x3)
|
||||
Financial ## ## .
|
||||
Team ## ## .
|
||||
Market ## ## .
|
||||
Execution ## ## .
|
||||
Legal ## # .
|
||||
Reasoning (P10) . ### ## ]
|
||||
Audit (P11) . ## ### ] SAME META QUESTION
|
||||
Research tools . . (unique — KEEP)
|
||||
|
||||
Overlap score P10 intersect P11:
|
||||
question similarity ~0.85 (contradiction equiv missed-agreement/groupthink)
|
||||
input Jaccard ~0.80 (both post-domain, panel-output-primary)
|
||||
output collinearity ~0.75 (both emit integrity adjustment, not domain scores)
|
||||
-> COLLAPSE CANDIDATE under reasoning lens
|
||||
```
|
||||
|
||||
### Opus empirical echo (from v2.1 review results)
|
||||
|
||||
Opus 5 panel-vs-single test already showed cross-checks can be **statistically tight** (market dimension spread 0.00 across three CCs). That finding pressures **ensemble scorer** count — out of scope for *this* lens. The safer 15% cut is the **duplicated meta-tail**, not the diversity scorers the product sells. Killing Audit+Reasoning redundancy reduces COGS without touching the scoring majority (still 9 scoring judges + Research + 1 meta).
|
||||
|
||||
**Scoring judges retained: 9** (Primary, Validation, 3xCC, Financial, Team, Market, Execution, Legal)
|
||||
**Meta retained: 1** (merged) + Research
|
||||
|
||||
---
|
||||
|
||||
## Decision line
|
||||
|
||||
> **Collapse Reasoning-Verification and Audit Agent into one Synthesis and Integrity Gate (Kimi K2.6).**
|
||||
> **-2 seats (-15.4%), -1 serial phase, ~15% meta-tail cost, cleaner vendor math, removes Fable Team-to-Audit double-duty.**
|
||||
> Domain ensemble untouched. Ship as v2.2 delta on top of v2.1 blocker fixes.
|
||||
|
||||
---
|
||||
|
||||
## Implementation checklist (for conductor)
|
||||
|
||||
1. Patch judge-pool-spec.md sections listed in section 1
|
||||
2. Bump version header to v2.2; status remains FINAL DRAFT pending 3-5 proposal gate
|
||||
3. Update feature matrix + latency table
|
||||
4. Rewrite section 2.3 fallback row; delete Audit row
|
||||
5. Add output JSON schema for two-section merged seat
|
||||
6. Re-run vendor % assertion (expect 3/11 Anthropic)
|
||||
7. Do **not** change Pro 7-judge composition in this PR (Enterprise matrix row rename only)
|
||||
8. Optional: 90-day 10% shadow Fable integrity pass off critical path
|
||||
@@ -0,0 +1,293 @@
|
||||
# VerdictTank v2.1 — ONE Consolidation Recommendation
|
||||
## Research Agent Grounding Lens (Grok 4.5)
|
||||
|
||||
**Date:** 2026-08-12
|
||||
**Author role:** Research Agent (only judge with native live web + social search)
|
||||
**Target:** ~15% reduction without losing effectiveness
|
||||
**Constraints satisfied:** no model >2 roles · Audit + Research survive · ≥2 cross-checks
|
||||
|
||||
---
|
||||
|
||||
## Research-lens thesis
|
||||
|
||||
I produce the factual brief every downstream judge consumes. From that position, two seats add **latency and tokens without new signal**:
|
||||
|
||||
1. **Market-Reality (Phase 7)** mostly **re-summarizes my brief** — competitive density, TAM defensibility, and citation-backed market claims are already Research outputs. Its unique residual (non-Western lens) is a *prompt/schema* requirement, not a reason for a full serial phase.
|
||||
2. **Cross-Check C as a third parallel scorer** is **highly correlated** with A/B. Internal Opus 5 empirical validation on this stack found all three cross-checks scored `market` at **exactly 6** (spread **0.00**). External ensemble guidance: if judges agree on everything, you bought one verdict three times (orq.ai LLM juries; Verga et al. "Judges → Juries"). Two diverse cross-checks capture the ensemble; a third correlated scorer is mostly cost.
|
||||
|
||||
Specialist phases that do **not** rehash Research (Financial math, Team/Founder fit, Execution ops, Legal/regulatory code) stay. Audit and Reasoning stay as non-scoring synthesis/gates.
|
||||
|
||||
---
|
||||
|
||||
## (1) Changes — single consolidation: **Grounding-Redundancy Cut**
|
||||
|
||||
### A. Eliminate standalone Market-Reality (Phase 7)
|
||||
|
||||
| Action | Detail |
|
||||
|---|---|
|
||||
| Remove | Phase 7 seat: Market-Reality / Qwen3.7 Plus (default) |
|
||||
| Absorb into Research | Expand Phase 1 brief schema with a mandatory **Market block**: competitive density table, TAM/SAM defensibility notes, ≥1 non-Western comps when vertical warrants, citation URLs, "unknown/unverifiable" flags |
|
||||
| Absorb residual scoring | Market *dimension* continues via Primary + Validation + 2 cross-checks (already score all 10 dimensions). No separate market narrative phase |
|
||||
| Vertical weights (§3.2) | Market-Reality weight boosts → apply to **Research brief depth** (more market citations) + **Primary/Validation market-dimension weight**, not a missing judge |
|
||||
|
||||
### B. Collapse cross-checks 3 → 2 (keep minimum)
|
||||
|
||||
| Action | Detail |
|
||||
|---|---|
|
||||
| Remove | Phase 4c Cross-Check C as default Enterprise seat |
|
||||
| Keep | Phase 4a Cross-Check A (GPT-5.6 Sol) + Phase 4b Cross-Check B |
|
||||
| Diversity preserve | **Reassign DeepSeek V4 Pro → Cross-Check B** (replace Gemini Pro Latest in default lineup). Reasons: (i) keeps Chinese-lab / non-Western error surface that Market+CC-C previously carried; (ii) avoids Gemini floating-alias longitudinal noise documented in v2.1 §4.5 and review results; (iii) Gemini remains Fallback 1 for Research and available in pool |
|
||||
| Blindness | Both remaining CCs still see **proposal + Research brief ONLY** — unchanged isolation |
|
||||
|
||||
### C. What does *not* change
|
||||
|
||||
- Research Agent never scores
|
||||
- Audit Agent remains fenced, post-scoring, read-only
|
||||
- Reasoning-Verification still synthesizes all score sets
|
||||
- Financial, Team/Founder, Execution, Legal stay as serial specialists
|
||||
- Fable double: Team + Audit (still 2)
|
||||
- No other model exceeds 2 roles
|
||||
|
||||
### D. Lineup delta (Enterprise default)
|
||||
|
||||
| # | Phase | Role | Model (after) | Vendor | Change |
|
||||
|---|---|---|---|---|---|
|
||||
| 1 | 1 | Research Agent | Grok 4.5 | xAI | **Survives** — brief schema expanded |
|
||||
| 2 | 2 | Primary Reviewer | Claude Opus 5 | Anthropic | Unchanged |
|
||||
| 3 | 3 | Validation Reviewer | Claude Sonnet 5 | Anthropic | Unchanged |
|
||||
| 4 | 4a | Cross-Check A | GPT-5.6 Sol | OpenAI | Unchanged |
|
||||
| 5 | 4b | Cross-Check B | **DeepSeek V4 Pro** | DeepSeek | **Was Gemini; DeepSeek moved here** |
|
||||
| — | ~~4c~~ | ~~Cross-Check C~~ | — | — | **REMOVED** |
|
||||
| 6 | 5 | Financial Integrity | MiniMax-M3 | MiniMax | Unchanged |
|
||||
| 7 | 6 | Team/Founder | Claude Fable 5 | Anthropic | Unchanged |
|
||||
| — | ~~7~~ | ~~Market-Reality~~ | — | — | **REMOVED** (folded into Research) |
|
||||
| 8 | 7' | Execution-Feasibility | GPT-5.6 Terra | OpenAI | Unchanged |
|
||||
| 9 | 8' | Legal/Regulatory | **Qwen3.7 Plus** | Alibaba | **Vendor-cap patch** (was Sonnet) |
|
||||
| 10 | 9' | Reasoning-Verification | Kimi K2.6 | Moonshot | Unchanged |
|
||||
| 11 | 10' | Audit Agent | Claude Fable 5 | Anthropic | **Survives** |
|
||||
|
||||
**Seats:** 13 → **11**
|
||||
**Phases:** 11 → **10** (4a–4b parallel; Market gone)
|
||||
**Scoring judges:** 10 → **8** (Primary, Validation, CC-A, CC-B, Financial, Team, Execution, Legal)
|
||||
**Non-scoring:** Research, Reasoning, Audit (3)
|
||||
**Total:** 8 + 3 = **11**
|
||||
|
||||
Qwen3.7 Plus remains on the default path via Legal (not dropped from Enterprise). Gemini stays in pool + Research fallback.
|
||||
|
||||
### E. Fallback table patches (must ship with this cut)
|
||||
|
||||
- Delete Market-Reality row **or** keep as optional on-demand role for White-Label only
|
||||
- Cross-Check B primary becomes DeepSeek V4 Pro; F1 Gemini Pro Latest; F2 Claude Fable 5 (still cross-vendor)
|
||||
- Cross-Check C row retired from Enterprise default (pool may retain for WL custom panels)
|
||||
- Legal primary: Qwen3.7 Plus; F1 Gemini Pro Latest; F2 DeepSeek V4 Pro (cross-vendor)
|
||||
- Research F2 can stay Terra; optional: add Qwen as Research F3 for market-heavy verticals
|
||||
|
||||
### F. Research brief schema addition (contract)
|
||||
|
||||
```
|
||||
## Market block (mandatory)
|
||||
- Competitors: name, category (authoring vs critique vs other), price anchor, source URL
|
||||
- Non-Western / non-US comps: min 1 when vertical in {superapp, cross-border, unclassified-global}; else "N/A — US-centric vertical"
|
||||
- TAM/SAM claims: verified | overstated | unverifiable — with citation or explicit gap
|
||||
- Density judgment: sparse | contested | saturated — one paragraph, citations only
|
||||
```
|
||||
|
||||
Downstream judges **cite the Market block ID**, not re-crawl the open web (except Legal/Financial specialists on their narrow facts).
|
||||
|
||||
---
|
||||
|
||||
## (2) Reduction %
|
||||
|
||||
| Metric | Before (v2.1) | After | Delta |
|
||||
|---|---|---|---|
|
||||
| Seats (Enterprise default) | 13 | 11 | **−15.4%** |
|
||||
| Phases | 11 | 10 | −9.1% |
|
||||
| Parallel CC calls | 3 | 2 | −33% of CC fan-out |
|
||||
| Scoring judges | 10 | 8 | −20% |
|
||||
| Serial specialist after scores | Market + Exec + Legal… | Exec + Legal… | −1 serial hop |
|
||||
| Est. E2E latency (vs 218s baseline) | 218s | ~185–195s | **~11–15%** wall-clock |
|
||||
| Est. token/COGS share | full panel | −Market narrative −1 full score JSON | **~14–18%** cost |
|
||||
|
||||
**Headline reduction: ~15%** (seat count exact at −15.4%; cost/latency band 14–18% / 11–15%).
|
||||
|
||||
Why not larger: cutting Audit or a second CC would violate constraints or erase the disagreement signal that makes panels worth running (selection-bottleneck literature: judge-based selection > synthesis; arxiv 2603.20324).
|
||||
|
||||
---
|
||||
|
||||
## (3) What survives (effectiveness preserved)
|
||||
|
||||
| Kept | Why it still works |
|
||||
|---|---|
|
||||
| **Research Agent** | Stronger, not weaker — owns market grounding explicitly |
|
||||
| **Audit Agent** | Groupthink / blind-spot gate unchanged |
|
||||
| **≥2 cross-checks** | Sol + DeepSeek: US frontier + non-Western lab; blind; proposal+brief only |
|
||||
| **Primary + Validation** | Dual Anthropic pass with sober second read (different tiers A/B) |
|
||||
| **Financial, Team, Execution, Legal** | True specialists; low overlap with Research crawl |
|
||||
| **Reasoning-Verification** | Contradiction synthesis over fewer, higher-signal score sets |
|
||||
| **Vendor cap** | After Legal→Qwen: Anthropic 3/11 = **27.3%** (under 33%) |
|
||||
| **Pool diversity** | Gemini remains in pool/fallbacks; White-Label can still pin full custom panels |
|
||||
| **Model role cap** | Fable Team+Audit = 2; all others ≤1 |
|
||||
|
||||
---
|
||||
|
||||
## (4) Risks
|
||||
|
||||
| Risk | Severity | Mitigation |
|
||||
|---|---|---|
|
||||
| Market nuance loss on superapp / cross-border verticals | Med | Research Market block mandatory; §3.2 superapp trigger increases Research market citation quota + Primary market-dimension weight +30% |
|
||||
| DeepSeek as sole non-Western CC may under-challenge US-centric Primary | Med | Legal→Qwen adds second non-Western seat; Audit watches for US-default groupthink |
|
||||
| Anthropic share 4/11 if Legal stays Sonnet | High (rule break) | **Must** reassign Legal → Qwen (included in this same cut) |
|
||||
| Sol still cannot tool-use on admin-ai | Low | CC-A is scoring-only JSON — already OK per §4.5 |
|
||||
| Dropping third CC reduces disagreement surface | Med-Low | Literature + internal 0.00 market spread say third was fake diversity; monitor panel sigma for 30 days; if sigma collapses, restore CC-C on High complexity only (§3.3 already expands high-complexity) |
|
||||
| Research brief becomes single point of market failure | Med | Research fallback chain unchanged (Gemini grounding → Terra); malformed Research → subscriber notify + degrade already specified |
|
||||
| Pro tier "7 judges" matrix needs rewrite | Low | Define Pro as: Research + Primary + Validation + 2 CC + Execution + Reasoning (no Financial/Team/Legal/Audit) — document explicitly |
|
||||
| Build-gate still unproven (panel delta <0.5 vs solo) | Existential (pre-existing) | This cut helps the thesis: fewer correlated seats make a ≥0.5 panel advantage more plausible if one exists; still run 3–5 real proposals before build |
|
||||
|
||||
---
|
||||
|
||||
## (5) New diagram
|
||||
|
||||
### Before (v2.1) — 11 phases / 13 seats
|
||||
|
||||
```
|
||||
┌─────────────────────┐
|
||||
│ 1 Research (Grok) │ grounding brief
|
||||
└──────────┬──────────┘
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ 2 Primary (Opus 5) │ scores
|
||||
└──────────┬──────────┘
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ 3 Validation (Sonnet)│ challenge
|
||||
└──────────┬──────────┘
|
||||
▼
|
||||
┌────────────────┼────────────────┐
|
||||
▼ ▼ ▼
|
||||
┌──────────┐ ┌──────────┐ ┌──────────┐
|
||||
│4a CC-A │ │4b CC-B │ │4c CC-C │ blind re-scores
|
||||
│ Sol │ │ Gemini │ │ DeepSeek │
|
||||
└────┬─────┘ └────┬─────┘ └────┬─────┘
|
||||
└────────────────┼────────────────┘
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ 5 Financial (M3) │
|
||||
└──────────┬──────────┘
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ 6 Team (Fable) │
|
||||
└──────────┬──────────┘
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ 7 Market (Qwen) │ ← re-summarizes Research
|
||||
└──────────┬──────────┘
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ 8 Execution (Terra) │
|
||||
└──────────┬──────────┘
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│ 9 Legal (Sonnet) │
|
||||
└──────────┬──────────┘
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│10 Reasoning (Kimi) │ synthesize
|
||||
└──────────┬──────────┘
|
||||
▼
|
||||
┌─────────────────────┐
|
||||
│11 Audit (Fable) │ fenced ±0.5
|
||||
└─────────────────────┘
|
||||
```
|
||||
|
||||
### After — Grounding-Redundancy Cut — 10 phases / 11 seats (−15.4%)
|
||||
|
||||
```
|
||||
┌──────────────────────────────────┐
|
||||
│ 1 Research (Grok 4.5) │
|
||||
│ + mandatory Market block │ LIVE WEB (only)
|
||||
└────────────────┬─────────────────┘
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ 2 Primary (Opus 5) │
|
||||
└────────────────┬─────────────────┘
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ 3 Validation (Sonnet 5) │
|
||||
└────────────────┬─────────────────┘
|
||||
▼
|
||||
┌───────────┴───────────┐
|
||||
▼ ▼
|
||||
┌─────────────┐ ┌─────────────┐
|
||||
│4a CC-A Sol │ │4b CC-B │ 2 blind CCs
|
||||
│ │ │ DeepSeek │ (min met)
|
||||
└──────┬──────┘ └──────┬──────┘
|
||||
└───────────┬───────────┘
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ 5 Financial (MiniMax-M3) │
|
||||
└────────────────┬─────────────────┘
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ 6 Team/Founder (Fable 5) │
|
||||
└────────────────┬─────────────────┘
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ 7 Execution (Terra) │ Market phase GONE
|
||||
└────────────────┬─────────────────┘
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ 8 Legal (Qwen3.7 Plus) │ vendor-cap patch
|
||||
└────────────────┬─────────────────┘
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│ 9 Reasoning-Verification (Kimi) │
|
||||
└────────────────┬─────────────────┘
|
||||
▼
|
||||
┌──────────────────────────────────┐
|
||||
│10 Audit (Fable 5) │ SURVIVES
|
||||
└──────────────────────────────────┘
|
||||
```
|
||||
|
||||
### Signal flow (Research lens)
|
||||
|
||||
```
|
||||
Research brief
|
||||
├─ facts/citations ──────────────► all scorers (unchanged)
|
||||
├─ Market block (NEW) ───────────► Primary / Validation / CCs
|
||||
│ (no Phase-7 rewrite)
|
||||
└─ gaps/unverifiable ────────────► Audit watches for overclaim
|
||||
|
||||
Dropped edges (were low-signal):
|
||||
Research ══re-summary══► Market narrative ══► Reasoning
|
||||
CC-C ══correlate≈0══► CC average
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Constraint checklist
|
||||
|
||||
| Constraint | Status |
|
||||
|---|---|
|
||||
| ~15% reduction | **15.4% seats**; ~14–18% COGS; ~11–15% latency |
|
||||
| No model >2 roles | Fable Team+Audit=2; all others ≤1 |
|
||||
| Audit survives | Phase 10' Fable, fenced |
|
||||
| Research survives | Phase 1 Grok, expanded |
|
||||
| Minimum 2 cross-checks | Sol + DeepSeek |
|
||||
| Vendor ≤33% | After Legal→Qwen: Anthropic 3/11 = 27.3% |
|
||||
|
||||
---
|
||||
|
||||
## Implementation note (one PR)
|
||||
|
||||
1. Spec §1.1 / §1.2 / §2.3 / §3.2 / §3.3 / §4.1 / §6.3 — role counts must all read **11 seats / 10 phases**
|
||||
2. Cross-section reconciliation (lesson from v2.0→v2.1 role expansion) before any re-review
|
||||
3. Cache-bust deploy `judge-pool-spec.md?v=2.2`
|
||||
4. Do **not** build until 3–5 real proposals still clear the ≥0.5 panel-vs-solo gate on the *reduced* panel
|
||||
|
||||
---
|
||||
|
||||
## Bottom line
|
||||
|
||||
**ONE consolidation:** *Grounding-Redundancy Cut* — delete Market-Reality as a serial phase (it re-summarizes Research) and delete the third cross-check (empirically zero incremental spread); fold market into the Research brief; keep DeepSeek as CC-B; move Legal to Qwen for vendor-cap compliance.
|
||||
|
||||
**−15.4% seats, Research + Audit intact, 2 blind cross-checks retained, specialists preserved.**
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,411 @@
|
||||
# VerdictTank Judge Pool Spec v2.0 - Primary Reviewer Scorecard
|
||||
|
||||
**Reviewer:** Claude Opus 5 (Primary Reviewer, Phase 2)
|
||||
**Date:** 2026-08-12
|
||||
**Spec:** https://proposals.itpropartner.com/verdicttank/judge-pool-spec.md (17,641 bytes, v2.0)
|
||||
**Method:** Document critique + live empirical validation against admin-ai (157-model catalog, real inference runs)
|
||||
|
||||
> **Note on version:** `web_extract` returned a cached **v1.0**. Verified against the live origin
|
||||
> via `curl` - the deployed file is **v2.0**, byte-identical to
|
||||
> `/root/projects/itpp-infrastructure/proposals/verdicttank/judge-pool-spec.md`. This review is of v2.0.
|
||||
|
||||
---
|
||||
|
||||
## Scorecard
|
||||
|
||||
| # | Dimension | Score | Verdict |
|
||||
|---|---|---|---|
|
||||
| 1 | Role Coverage | 7/10 | FIX |
|
||||
| 2 | Model Assignments | 6/10 | FIX |
|
||||
| 3 | Vendor Diversity | 8/10 | DEFEND |
|
||||
| 4 | admin-ai Paths | 9/10 | DEFEND |
|
||||
| 5 | Fallback Chains | 4/10 | FIX |
|
||||
| 6 | Content-Based Rules | 5/10 | FIX |
|
||||
| 7 | Latency Budgets | 2/10 | FIX |
|
||||
| 8 | Feedback Loop | 5/10 | FIX |
|
||||
| 9 | Pricing | 4/10 | FIX |
|
||||
| 10 | Product Boundary | 7/10 | DEFER |
|
||||
|
||||
**Weighted mean: 5.7/10** - Architecturally literate, empirically unvalidated. Two dimensions (7, 9) are
|
||||
launch-blocking. One unlisted finding (§Critical Finding) threatens the product thesis itself.
|
||||
|
||||
---
|
||||
|
||||
## CRITICAL FINDING (unlisted dimension - read first)
|
||||
|
||||
**The 12-judge panel does not measurably outperform one model run once.**
|
||||
|
||||
I tested this directly. Same proposal, same rubric, three dimensions, live models.
|
||||
|
||||
**A) Single-model noise floor** - `claude-opus-5` x6 at temp 1.0:
|
||||
```
|
||||
market: 5,5,5,4,4,4 stdev 0.50
|
||||
team: 4,4,4,4,4,4 stdev 0.00
|
||||
fin: 3,3,3,3,3,3 stdev 0.00
|
||||
```
|
||||
|
||||
**B) Nine distinct judges across 8 vendors, one run each:**
|
||||
```
|
||||
Opus5 [5,4,3] Sonnet5 [4,4,3] Sol [6,4,3] GeminiPro [4,3,2] DeepSeek [5,3,2]
|
||||
Terra [4,4,3] Qwen [4,3,2] MiniMax [6,3,4] Fable5 [6,4,4]
|
||||
panel stdev: market 0.87, team 0.50, fin 0.74
|
||||
```
|
||||
|
||||
**C) The result that matters:**
|
||||
|
||||
| | market | team | financials |
|
||||
|---|---|---|---|
|
||||
| Mean, 1 model | 4.50 | 4.00 | 3.00 |
|
||||
| Mean, 9 judges / 8 vendors | 4.89 | 3.56 | 2.89 |
|
||||
| **Delta** | **0.39** | **0.44** | **0.11** |
|
||||
|
||||
Nine frontier models from eight vendors, ~218s of wall clock and roughly 20x the token cost, move the
|
||||
final number by **less than half a point on a 10-point scale**. On `market`, panel spread (0.87) is only
|
||||
**1.7x** the single-model rerun noise (0.50) - statistically indistinguishable from rerunning one model
|
||||
six times.
|
||||
|
||||
The individual judges *do* disagree (market scores span 4-6). But disagreement that averages to the same
|
||||
answer is **variance, not signal**. The spec sells vendor diversity as the core accuracy mechanism and
|
||||
never once tests whether diversity changes the output.
|
||||
|
||||
**This is the product thesis, and it is unvalidated.** Every downstream claim - premium pricing, the
|
||||
"most accurate review tool available" positioning, the 12-vendor moat - rests on it.
|
||||
|
||||
**Required before build:** run 30-50 real proposals with known outcomes. Report panel-vs-single
|
||||
correlation against ground truth. If the panel does not beat one good model by a margin that justifies
|
||||
20x cost, the correct architecture is 3 judges, not 12 - and the pricing story needs rebuilding.
|
||||
Better to learn this now than after a customer runs the same A/B.
|
||||
|
||||
---
|
||||
|
||||
## 1. Role Coverage - 7/10 - FIX
|
||||
|
||||
v2.0 deserves credit: it closed three of the four gaps flagged in the prior review (Financial Integrity,
|
||||
Team/Founder, Legal/Regulatory now have dedicated phases). That is real progress.
|
||||
|
||||
**Technical Architecture remains uncovered** - the one gap explicitly identified in the v4.1 gap analysis
|
||||
and silently dropped. Execution-Feasibility is operational (timeline, team, resources), not architectural
|
||||
(stack, scalability, security posture, technical debt). For a product whose buyers are evaluating
|
||||
*technical* startups, having no judge that reads the architecture is a conspicuous hole.
|
||||
|
||||
Two further gaps neither version names:
|
||||
- **Traction/Evidence.** Nobody scores whether claims are *substantiated*. Research Agent gathers
|
||||
citations but never scores; no downstream role is tasked with "the founder asserts 40% MoM growth -
|
||||
is there evidence?" This is the single most common reason real proposals fail diligence.
|
||||
- **Narrative/Communication quality.** For a *proposal* review tool, no judge assesses whether the
|
||||
document actually persuades.
|
||||
|
||||
**Fix:** add Technical Architecture (merge into Execution-Feasibility if headcount is capped) and fold
|
||||
an evidence-substantiation mandate into the Research Agent's brief so grounding produces a scored
|
||||
claims-verification artifact, not just citations.
|
||||
|
||||
---
|
||||
|
||||
## 2. Model Assignments - 6/10 - FIX
|
||||
|
||||
Most seats are defensible. Grok 4.5 on Research is correct (only pool member with native live web
|
||||
search). Qwen3.7 Plus on Market-Reality for a non-Western lens is genuinely thoughtful.
|
||||
|
||||
**Empirically-grounded objections:**
|
||||
|
||||
**Kimi K2.6 on Reasoning-Verification is misassigned.** In my scoring test it returned **empty content
|
||||
after burning all 900 output tokens** - and on the 12-page critical-path run it took 15.1s. The role
|
||||
requires emitting a structured contradiction report; a model that silently exhausts its budget in the
|
||||
seat that validates every other judge's consistency is the worst possible placement. It is also the
|
||||
*only* Moonshot seat, so there is no same-vendor fallback.
|
||||
|
||||
**Empty-response rate is a systemic risk the spec never models.** At `max_tokens=12`, four of six probed
|
||||
models (MiniMax-M3, Kimi K2.6, Claude Fable 5, Gemini Pro) returned **empty content** - reasoning tokens
|
||||
consumed the entire budget. §4.1 handles "malformed score" but not "well-formed empty response," which is
|
||||
the actual failure mode I observed. Token budgets must be set per-model with reasoning-token headroom.
|
||||
|
||||
**Claude Sonnet 5 sits in two scoring seats** (Validation + Legal/Regulatory). The spec waves this off as
|
||||
"read/analytical, not scoring-intense," but §1.1 marks Legal/Regulatory as **"Yes - scores."** The
|
||||
justification contradicts the table two rows above it. Same weights, same biases, two votes.
|
||||
|
||||
**Gemini Pro Latest doubles as Cross-Check B and Audit Agent.** The Audit Agent's stated job is catching
|
||||
*groupthink* - it cannot audit a panel it already voted in. This directly violates the spirit of §3.1's
|
||||
own audit-independence rule.
|
||||
|
||||
---
|
||||
|
||||
## 3. Vendor Diversity - 8/10 - DEFEND
|
||||
|
||||
The strongest dimension. Eight vendors, max 25% - a genuine improvement over v1.0's 44% Anthropic
|
||||
violation, and the rule was tightened from 40% to 33% rather than loosened to fit. That is the right
|
||||
instinct and the spec should defend it.
|
||||
|
||||
**Two caveats worth documenting rather than fixing:**
|
||||
|
||||
*Vendor diversity is not architecture diversity.* Nearly every pool member is a transformer trained on
|
||||
overlapping web corpora with similar RLHF conventions. My independence test showed all three "independent"
|
||||
cross-checks scoring `market` at **exactly 6 - spread 0.00**. Eight logos, one prior. The correlation
|
||||
data from the Critical Finding is the real story here.
|
||||
|
||||
*Infrastructure concentration.* All 12 models route through a single admin-ai/LiteLLM proxy. Eight-vendor
|
||||
diversity buys nothing if the proxy is down - that is the actual SPOF, and the spec's §4.1 fallback table
|
||||
implicitly assumes the proxy always answers.
|
||||
|
||||
---
|
||||
|
||||
## 4. admin-ai Paths - 9/10 - DEFEND
|
||||
|
||||
**Verified live, not taken on faith.** I queried the admin-ai catalog (157 models) and confirmed all 13
|
||||
spec paths resolve exactly, then ran real inference against every one:
|
||||
|
||||
```
|
||||
OK xai/grok-4.5 OK claude-opus-5 OK claude-sonnet-5
|
||||
OK gpt-5.6-sol OK gemini/gemini-pro-latest OK deepseek-v4-pro
|
||||
OK kimi-k2.6 OK gpt-5.6-terra OK qwen3.7-plus
|
||||
OK MiniMax-M3 OK claude-fable-5 OK gemini/gemini-3.6-flash
|
||||
OK gemini/gemini-3.5-flash-lite
|
||||
```
|
||||
|
||||
All 11 judge models returned live responses. The spec's claim "all model paths confirmed against admin-ai"
|
||||
is **true** - rare enough in a v2.0 draft to call out. This dimension should be defended as-is.
|
||||
|
||||
**The one point off:** `gemini/gemini-pro-latest` is a **floating alias**, not a pinned version. The
|
||||
catalog carries pinned alternatives (`gemini/gemini-3.1-pro-preview`, `openrouter/google/gemini-3-pro-preview`).
|
||||
Google can repoint that alias with no notice and silently change two seats - Cross-Check B *and* Audit.
|
||||
My determinism probe on that alias returned `[6,3,3] / [6,4,3] / [6,4,2]` across three identical calls.
|
||||
For a product whose entire value is score reproducibility, and which promises T+90/180/365 longitudinal
|
||||
outcome tracking, an unpinned model destroys year-over-year comparability. **Pin every judge seat.**
|
||||
|
||||
---
|
||||
|
||||
## 5. Fallback Chains - 4/10 - FIX
|
||||
|
||||
§3.1 asserts "Fallback chains always cross vendor boundaries - Enforced at the assignment table (§2.3)."
|
||||
Reading §2.3 against that claim, the rule is **violated in the first fallback hop of 4 of 11 rows**:
|
||||
|
||||
| Role | Primary | Fallback 1 | Violation |
|
||||
|---|---|---|---|
|
||||
| Primary Reviewer | Claude Opus 5 | **Claude Sonnet 5** | Anthropic → Anthropic |
|
||||
| Cross-Check A | GPT-5.6 Sol | **GPT-5.6 Terra** | OpenAI → OpenAI |
|
||||
| Cross-Check B | Gemini Pro Latest | **Gemini 3.6 Flash** | Google → Google |
|
||||
| Team/Founder | Claude Fable 5 | **Claude Sonnet 5** | Anthropic → Anthropic |
|
||||
|
||||
This is the *identical* defect flagged in the v1.0 review ("§4.1 says different vendor but §2.3 says
|
||||
Opus→Sonnet"). It was marked as a top-5 prioritized fix, and it was **not fixed** - the assertion text was
|
||||
added to §3.1 without correcting the table it points at. A rule that is stated but not enforced is worse
|
||||
than no rule: it will pass code review as "already handled."
|
||||
|
||||
The failure mode is precisely what fallbacks exist to prevent. Anthropic has a regional outage → Primary
|
||||
Reviewer fails over to Sonnet 5 → also Anthropic → also down. Same for the OpenAI and Google rows.
|
||||
|
||||
**Additional defects:**
|
||||
- **Convergence, not diversity.** DeepSeek V4 Pro appears as a fallback in **7 of 11 chains**. Under
|
||||
broad degradation the 8-vendor panel collapses toward a single DeepSeek-dominated panel, blowing the
|
||||
33% vendor cap at exactly the moment it matters most. No runtime re-check of the cap after failover.
|
||||
- **Kimi K2.6 has no same-tier peer** - the sole Moonshot seat degrades straight to DeepSeek (Tier C).
|
||||
- **§4.2 is unexecutable as written.** "Tier B for Primary/Validation only, Tier C for others" - Tier C
|
||||
is `DeepSeek V4 Pro, Gemini 3.6 Flash, Gemini 3.5 Flash-Lite`: two vendors, three models, for up to
|
||||
five simultaneous seats. The cost-optimization path *cannot* satisfy the vendor rule. This is the same
|
||||
unexecutability flagged in v1.0 ("Tier B has no OpenAI/Google model") reappearing in a new form.
|
||||
|
||||
---
|
||||
|
||||
## 6. Content-Based Rules - 5/10 - FIX
|
||||
|
||||
The vertical/stage triggers are directionally sensible and v2.0 adds a genuine improvement: the §3.3
|
||||
missing-vertical handler with explicit user disclosure is honest product design.
|
||||
|
||||
**But the weighting system is undefined.** Every rule adjusts "weight ±N%" and **the spec never states
|
||||
what the weights weight.** There is no aggregation formula anywhere in 302 lines - no statement of how 11
|
||||
judges' 10-dimension scores combine into a final number. "Market-Reality weight +30%" is meaningless
|
||||
without knowing the base weight, the combination function, and whether weights renormalize. This is the
|
||||
single largest specification hole in the document: it is the core scoring algorithm, and it is absent.
|
||||
|
||||
**Composition rule is underspecified.** §3.9 caps stacking at +50% per role but doesn't say what happens
|
||||
when the cap binds - are all triggers scaled proportionally, or does first-match win? Different answers
|
||||
give different scores for the same proposal.
|
||||
|
||||
**Detection is hand-waved.** Every row says "Detected from proposal text" with no mechanism, no confidence
|
||||
threshold, and no misclassification path. Vertical detection *changes the score* - a fintech misread as
|
||||
SaaS loses its +30% Legal/Regulatory weight. That needs a confidence score and a human-review fallback,
|
||||
and multi-vertical proposals (fintech + healthcare) have no representation at all.
|
||||
|
||||
**Page count is a poor complexity proxy.** A 10-page dense technical proposal triggers the *reduced*
|
||||
panel; a 50-page deck with 40 pages of appendix triggers the full one. Use token count or content
|
||||
density.
|
||||
|
||||
**Substantively questionable:** Seed-stage sets Financial Integrity **-20%**. Seed is where financial
|
||||
models are *most* fictional and cap-table mistakes are most permanent. De-weighting financial scrutiny
|
||||
where founders most need it inverts the diligence priority.
|
||||
|
||||
---
|
||||
|
||||
## 7. Latency Budgets - 2/10 - FIX (launch-blocking)
|
||||
|
||||
The spec revised Enterprise from 30s → 45s and annotated it *"v1.0 was unrealistic for 12-judge pipeline."*
|
||||
The revision is still off by nearly 5x. I measured it.
|
||||
|
||||
**Sequential critical path, 12-page proposal (~5,500 tokens), following the spec's own dependency graph:**
|
||||
|
||||
```
|
||||
P1 Research xai/grok-4.5 21.5s
|
||||
P2 Primary claude-opus-5 20.0s
|
||||
P3 Validation claude-sonnet-5 15.8s
|
||||
P5 Reasoning kimi-k2.6 15.1s
|
||||
P6 Execution gpt-5.6-terra 16.4s
|
||||
P7 Market qwen3.7-plus 67.2s <-- single judge exceeds Enterprise MAX alone
|
||||
P8 Financial MiniMax-M3 26.1s
|
||||
P9 Team claude-fable-5 22.9s
|
||||
P11 Audit gemini/gemini-pro-latest 13.0s
|
||||
------------------------------------------------
|
||||
TOTAL 218.0s
|
||||
```
|
||||
|
||||
**218s against a 45s target and a 90s max - 4.8x over target, 2.4x over the stated maximum.**
|
||||
|
||||
This is not a tuning problem, it is structural:
|
||||
|
||||
- **The pipeline is inherently sequential.** Phases 2→3→5→6→7→8→9→11 each consume prior output by design.
|
||||
Only 4a/4b/4c parallelize. You cannot fan out a dependency chain.
|
||||
- **A single judge blows the entire budget.** Qwen3.7 Plus took **67.2s alone** - 1.5x the full Enterprise
|
||||
target, before any other judge runs. It burned 1,450 reasoning tokens on a trivial scoring task.
|
||||
- **Even perfect parallelism fails.** All 11 models fired simultaneously on a *short* prompt still took
|
||||
**27.4s wall clock** - 61% of the Enterprise budget consumed by the slowest model, with zero
|
||||
orchestration, retries, or assembly.
|
||||
- **Fallbacks make it worse.** A 30s timeout + Fallback 1 retry adds 30s+ to an already-blown budget.
|
||||
The §4.1 retry path and the §4.3 budget are mutually unsatisfiable.
|
||||
- **Tier ordering is still inverted.** Enterprise runs the deepest pipeline (12 judges) on the tightest
|
||||
budget (45s); Free runs 3 judges on 90s. v1.0 had this backwards and v2.0 preserved the inversion while
|
||||
only adjusting magnitudes.
|
||||
|
||||
50+ page proposals (the spec's own "High complexity," which *adds* a secondary pass) will run
|
||||
substantially past 218s.
|
||||
|
||||
**Fix:** these are not real-time interactions - they are deep-analysis jobs. Re-architect as an
|
||||
**asynchronous job model**: submit → progress streaming → notify on completion. Budget **5-10 minutes**
|
||||
for Enterprise and sell the depth. Then set per-phase timeouts against measured p95, not aspiration.
|
||||
A 45s promise that reliably takes 218s is a support-ticket generator and a churn driver.
|
||||
|
||||
---
|
||||
|
||||
## 8. Feedback Loop - 5/10 - FIX
|
||||
|
||||
v2.0 genuinely improved here - §5.4 cold-start gate, min N=30/N=50 thresholds, and the immediate-recusal
|
||||
rule for >2.0σ bias are all correct additions that address prior findings.
|
||||
|
||||
**The remaining problems are foundational:**
|
||||
|
||||
**The accuracy metric is not sound.** "Correlation between dimension score and T+90 outcome" -
|
||||
- **Ninety days is far too short.** Seed rounds take 3-9 months; the T+90 signal is mostly noise about
|
||||
fundraising *timing*, not proposal quality.
|
||||
- **Survivorship and selection bias are unaddressed.** Response rates on outcome surveys skew heavily to
|
||||
founders who succeeded. The spec plans to weight models on a systematically biased sample.
|
||||
- **Confounding is total.** A proposal that scores 4/10, gets rewritten using the Fix-It plan, and then
|
||||
raises successfully - did the judge score correctly or incorrectly? The product *intervenes* on the
|
||||
outcome it measures. This is unfixable by more data; it needs a holdout design.
|
||||
- **N is unreachable.** N≥50 per model per dimension per vertical, with 10 dimensions, 11 models and 5+
|
||||
verticals, implies thousands of tracked reviews before a single threshold fires. At 100 reviews/mo
|
||||
Enterprise capacity, that is **years**. The entire feedback loop is aspirational at realistic volume,
|
||||
and the "data moat" narrative rests on it.
|
||||
|
||||
**Rotation rules:** "Remove from Tier A, demote to Tier B" as a demotion path is odd - a model with
|
||||
<0.3 outcome correlation is not a *cheaper* model, it is an *inaccurate* one. Demoting it means budget
|
||||
users get the judge known to be wrong. Also, the model-deprecation row says replace with "Fallback 1
|
||||
from §2.3" - but §2.3 fallbacks are same-vendor in 4 rows, so vendor-caused deprecation cascades to a
|
||||
sibling that may be deprecated in the same wave.
|
||||
|
||||
---
|
||||
|
||||
## 9. Pricing - 4/10 - FIX
|
||||
|
||||
I have no objection to premium positioning, and the "don't race to the bottom" instinct is right. The
|
||||
objection is that **the price is not connected to demonstrated value**, and v2.0 raised it by
|
||||
**3.3x-5.0x** (from $79/$299 to $249/$799/$1,499) on positioning reasoning alone, with zero customers,
|
||||
zero LOIs, and - per the Critical Finding - no evidence the 12-judge panel beats one model.
|
||||
|
||||
**The benchmark is misapplied.** GC AI at $500/seat/mo is cited as the anchor comp, and I verified it
|
||||
independently (gc.ai, corroborated by vaquill.ai's 2026 benchmark; note haqq.ai could not source it to
|
||||
GC AI directly). But GC AI serves **in-house legal teams at 1,900+ companies** - daily-use workflow
|
||||
software with seat-level lock-in. VerdictTank is **episodic**: a founder reviews a proposal during a
|
||||
fundraise, then churns. Anchoring episodic tooling to daily-workflow pricing is a category error.
|
||||
|
||||
**The usage math undermines the tiers.** Enterprise at $799/mo = 100 reviews. Real founders raising a
|
||||
round need **3-8 reviews over a 2-3 month window**. That is ~$100/review nominal at a utilization
|
||||
almost no customer will reach - and Pro at $249 for 20 reviews has the same problem. Customers pay for
|
||||
capacity they cannot consume, notice, and churn. **Per-review or credit-pack pricing fits actual
|
||||
consumption far better than monthly seats**, and the spec never considers it.
|
||||
|
||||
**Margin honesty.** The v4.1 reference concedes "COGS 72-97% margin - pure positioning play." My cost
|
||||
sampling supports that: a full panel run is roughly $0.30-0.80 in tokens. A 99.9% gross margin at
|
||||
$799/mo is not premium positioning, it is an unanchored price waiting for a competitor to undercut it
|
||||
with the same off-the-shelf models. The moat is claimed to be the outcome-tracking corpus - which
|
||||
dimension 8 shows is years away at this volume.
|
||||
|
||||
**Free tier at 1/mo is too stingy** for a trust-first product. The entire pitch is "we tell you the
|
||||
brutal truth" - that requires *experiencing* the depth. A 1-review Tier-C sample (3 judges, no Fix-It)
|
||||
demonstrates the weakest possible version of the product to every prospective buyer.
|
||||
|
||||
**Fix:** validate willingness-to-pay with 10-20 design partners before locking. Offer per-review pricing
|
||||
alongside subscriptions. Anchor to *outcome value* (a better raise) rather than to a competitor in an
|
||||
adjacent category.
|
||||
|
||||
---
|
||||
|
||||
## 10. Product Boundary - 7/10 - DEFER
|
||||
|
||||
Conceptually clean and easy to communicate: VerdictTank = "is this good?", RFP Tank = "does this match
|
||||
what they asked for?" The superset framing is right, and inheriting one engine is the correct build
|
||||
decision.
|
||||
|
||||
**Unresolved, but not urgent:**
|
||||
- **RFP Tank is the better business and it is the side project.** RFP responses are recurring, deadline-
|
||||
driven, budgeted, and B2B - structurally superior to episodic founder fundraising. The spec treats it
|
||||
as a discount add-on. Strategically inverted.
|
||||
- **Cannibalization is unpriced.** RFP Tank is a strict superset at (presumably) a higher price. A
|
||||
rational buyer needing both buys RFP Tank only. Enterprise VT + 20% off RFP Tank is then a discount
|
||||
on a product that replaces the one just paid for.
|
||||
- **10-dimension rubric is asserted, never enumerated.** The document references "10-dimension scoring"
|
||||
in §7 and §8 but **never lists the ten dimensions.** For a build spec, the thing being scored should
|
||||
be defined; I am scoring against dimensions the spec assumes I already know.
|
||||
- No shared-account model, no cross-product SSO, no migration path.
|
||||
|
||||
**DEFER** because the boundary is directionally correct and none of this blocks the judge-pool build.
|
||||
Revisit before RFP Tank pricing is set.
|
||||
|
||||
---
|
||||
|
||||
## Priority Actions
|
||||
|
||||
| # | Action | Dim | Severity |
|
||||
|---|---|---|---|
|
||||
| 1 | **Validate panel-vs-single-model accuracy on 30-50 real proposals.** Product thesis is unproven. | - | **BLOCKER** |
|
||||
| 2 | **Re-architect to async jobs; budget 5-10 min.** Measured 218s vs 45s target. | 7 | **BLOCKER** |
|
||||
| 3 | **Fix 4 same-vendor Fallback-1 hops.** Flagged in v1.0 review, still unfixed. | 5 | **BLOCKER** |
|
||||
| 4 | **Define the score aggregation formula.** "Weight +30%" is meaningless; core algorithm absent. | 6 | **BLOCKER** |
|
||||
| 5 | Pin all model versions - drop floating `gemini-pro-latest` alias. | 4 | High |
|
||||
| 6 | Reassign Kimi K2.6 off Reasoning-Verification (empty output under budget). | 2 | High |
|
||||
| 7 | Set per-model token budgets with reasoning-token headroom; handle empty-but-valid responses. | 2 | High |
|
||||
| 8 | Split Gemini Pro's Cross-Check B / Audit double-seat; auditor cannot audit itself. | 2 | High |
|
||||
| 9 | Add runtime vendor-cap re-check after failover (DeepSeek is fallback in 7 of 11 chains). | 5 | High |
|
||||
| 10 | Validate pricing with design partners; add per-review option. | 9 | High |
|
||||
| 11 | Add Technical Architecture + evidence-substantiation coverage. | 1 | Medium |
|
||||
| 12 | Replace page-count complexity proxy with token count. | 6 | Medium |
|
||||
| 13 | Enumerate the 10 scoring dimensions in the spec. | 10 | Medium |
|
||||
| 14 | Reconsider Financial Integrity -20% at seed stage. | 6 | Medium |
|
||||
|
||||
---
|
||||
|
||||
## Bottom Line
|
||||
|
||||
v2.0 is a real improvement over v1.0 - the vendor rule was tightened rather than loosened to fit, the
|
||||
cold-start gates are correct, the role gaps were mostly closed, and **every model path is genuinely live**,
|
||||
which I verified rather than assumed. The document is architecturally literate.
|
||||
|
||||
But it is a **design document wearing a build spec's clothes**. Its two most load-bearing quantitative
|
||||
claims fail on contact with the live system: the latency budget is off by 4.8x, and the fallback
|
||||
cross-vendor guarantee is contradicted by its own assignment table in four rows - *the same defect flagged
|
||||
in the v1.0 review and marked as a prioritized fix.* The assertion was added; the table was not corrected.
|
||||
|
||||
Most seriously: the panel-diversity premise that justifies the pricing, the vendor spread, and the entire
|
||||
category claim has **never been tested**, and my measurement suggests it may not survive testing. Nine
|
||||
judges across eight vendors moved the score by under half a point versus a single model.
|
||||
|
||||
**Recommendation: do not proceed to build.** Run the accuracy validation (Action 1) and re-baseline
|
||||
latency (Action 2) first. If the panel advantage is real, this architecture is worth building and the
|
||||
premium price defensible. If it is not, the correct product is 3 judges at a third of the price - and it
|
||||
is far cheaper to discover that now than after the first enterprise customer runs the same A/B I just ran.
|
||||
@@ -0,0 +1,418 @@
|
||||
# VerdictTank Internal Proposal Methodology v1.0
|
||||
|
||||
**Status:** SETTLED - governing document for all VerdictTank/RFP Tank proposals
|
||||
**Effective:** 2026-08-11
|
||||
**Owner:** Sho'Nuff Brown (shogun of IT architecture)
|
||||
**Invocation:** `skill_view(name='verdicttank-proposal-pipeline', file_path='references/methodology.md')`
|
||||
|
||||
---
|
||||
|
||||
## 1. Pipeline Overview
|
||||
|
||||
The VerdictTank proposal pipeline converts a product brief into a deployed, reviewed, audited proposal page in one orchestrated run. It is NOT a general-purpose proposal tool - it is purpose-built for VerdictTank and RFP Tank, with specialized content teams, a mandatory blind audit, and multi-model conductor review.
|
||||
|
||||
```
|
||||
Product Brief
|
||||
│
|
||||
├─► Phase 1: Research (DeepSeek V4 Pro - current main model)
|
||||
│ Market sizing, competitor landscape, regulatory scan
|
||||
│ Output: research brief → /tmp/verdicttank-v4.0-research-brief.md
|
||||
│
|
||||
├─► Phase 2: Content Creation (4 parallel teams)
|
||||
│ Marketing (Opus 4.8) → HTML blocks: hero, value props, pricing, CTA
|
||||
│ Architecture (Opus 4.8) → standalone architecture.html (dark theme, sections)
|
||||
│ Financial (Opus 4.8) → bottom-up SOM/SAM/TAM, pricing model, unit economics
|
||||
│ Legal (Sonnet 5) → ToS, Privacy, DPA, AI disclaimer, sub-processor audit
|
||||
│
|
||||
├─► Phase 3: Assembly (DeepSeek V4 Pro - current main model)
|
||||
│ Merge 4 HTML blocks into index.html
|
||||
│ Reconcile SOM to Financial's authoritative number
|
||||
│ Insert legal blocks at designated markers
|
||||
│ Link architecture.html as standalone page
|
||||
│
|
||||
├─► Phase 4: Internal Conductor Review (Sonnet 5 + Opus 4.8)
|
||||
│ Sonnet 5: brutal Shark Tank critic (what's wrong, what's missing, what's weak)
|
||||
│ Opus 4.8: sober validator + priority re-rank
|
||||
│ 10-dimension scorecard: clarity, credibility, differentiation, TAM/SOM,
|
||||
│ GTM, tech feasibility, competitive moat, financial model, team, risk
|
||||
│ Output: critical-review.html (deployed alongside proposal)
|
||||
│
|
||||
├─► Phase 5: Blind Audit (Opus 4.8 - cold read, no prior context)
|
||||
│ 7-item checklist executed against deployed page
|
||||
│ Every link checked, every number traced, every claim verified
|
||||
│ THIS IS MANDATORY. No proposal ships without a blind audit.
|
||||
│ v3.7 shipped with 4 verifiable errors because this step didn't exist.
|
||||
│
|
||||
├─► Phase 6: External Conductor Review (fresh panel)
|
||||
│ Independent judges - no access to internal review results
|
||||
│ Majority-rules: Go / No-Go / Conditional Go
|
||||
│
|
||||
└─► Phase 7: Deploy + Outcome Tracking
|
||||
Deploy to proposals.itpropartner.com/verdicttank/
|
||||
Schedule T+90, T+180, T+365 outcome tracking checks
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 2. Tier Selection
|
||||
|
||||
Three cost tiers. Pick before dispatching teams. Tier determines which models run which roles.
|
||||
|
||||
| Tier | Cost | Target Score | Content Teams | Review | Blind Audit | Use Case |
|
||||
|------|------|-------------|---------------|--------|-------------|----------|
|
||||
| **Tier 1** | ~$0.80 | 6/10 | DeepSeek V4 Pro only | None | None | Internal draft, rapid iteration |
|
||||
| **Tier 2** | ~$3.62 | 7-8/10 | Opus (narrative), Sonnet (architecture) | Internal only | Yes | Client-ready draft, first external showing |
|
||||
| **Tier 3** | ~$6.69 | 9-10/10 | Opus (marketing, architecture, financial), Sonnet (legal) | Internal + External | Yes | Final proposal, investor deck, RFP response |
|
||||
|
||||
**Rule:** Tier 3 is the default for any proposal that leaves ITPP infrastructure. Tier 1 is for internal validation only. Tier 2 is for client drafts that will iterate.
|
||||
|
||||
**Actual v4.0 Tier 3 cost:** $6.69 (5 Opus dispatches + 1 Sonnet dispatch + assembly + 2 conductor reviews + blind audit). Budget cap: $10.00. Current spend: tracked against `hermes-agent-v5` LiteLLM key.
|
||||
|
||||
---
|
||||
|
||||
## 3. Team Composition & Model Rotation Protocol
|
||||
|
||||
### 3.1 Standard Team Assignments (Tier 3)
|
||||
|
||||
| Team | Model | Role | Output Format | Permission |
|
||||
|------|-------|------|---------------|------------|
|
||||
| Marketing & Narrative | Claude Opus 4.8 | Clean-sheet proposal narrative | HTML blocks with INSERT markers | Read-only |
|
||||
| Technical Architecture | Claude Opus 4.8 | Industry-standard architecture document | standalone HTML (dark theme) | **Write to disk** (prevents truncation) |
|
||||
| Financial Model | Claude Opus 4.8 | Bottom-up verified arithmetic | HTML blocks + verified numbers | Read-only |
|
||||
| Legal & Governance | Claude Sonnet 5 | MVL framework, compliance tables | HTML blocks with INSERT markers | Read-only |
|
||||
|
||||
### 3.2 Model Rotation Protocol
|
||||
|
||||
**CRITICAL:** The delegation model must be set BEFORE each dispatch, then restored to `deepseek-v4-pro` immediately after. Failure to rotate burns budget on accidental Opus inference during assembly.
|
||||
|
||||
```
|
||||
# Before dispatching Opus teams (marketing, architecture, financial):
|
||||
hermes config set delegation.model claude-opus-4-8
|
||||
delegate_task(...) # dispatches marketing, architecture, financial as batch of 3
|
||||
hermes config set delegation.model deepseek-v4-pro # RESTORE IMMEDIATELY
|
||||
|
||||
# Before dispatching legal:
|
||||
hermes config set delegation.model claude-sonnet-5
|
||||
delegate_task(...) # dispatches legal
|
||||
hermes config set delegation.model deepseek-v4-pro # RESTORE IMMEDIATELY
|
||||
|
||||
# Before conductor review (internal):
|
||||
hermes config set delegation.model claude-sonnet-5 # for critic
|
||||
delegate_task(...)
|
||||
hermes config set delegation.model claude-opus-4-8 # for validator
|
||||
delegate_task(...)
|
||||
hermes config set delegation.model deepseek-v4-pro # RESTORE
|
||||
|
||||
# Before blind audit:
|
||||
hermes config set delegation.model claude-opus-4-8
|
||||
delegate_task(...)
|
||||
hermes config set delegation.model deepseek-v4-pro # RESTORE
|
||||
```
|
||||
|
||||
**Why DeepSeek V4 Pro is the default:** Budget protection. DeepSeek is orders of magnitude cheaper than Opus. The main model handles assembly, research, and coordination - all high-token, low-creativity work. Opus and Sonnet are reserved for tasks where their specific strengths justify the cost.
|
||||
|
||||
---
|
||||
|
||||
## 4. Content Team Brief Structure
|
||||
|
||||
Every team gets a self-contained brief file written to `/tmp/`. The brief must include:
|
||||
|
||||
1. **Context block:** What VerdictTank is, v3.7 review findings, the specific HTML blocks needed
|
||||
2. **Rules block:** No em dashes, no fabrication, financial model is authoritative for SOM, honest about methodology gaps
|
||||
3. **Output format block:** Exact HTML structure, INSERT markers, CSS variables to use
|
||||
4. **Reference data:** SOM/TAM numbers, competitor names, pricing tiers, domain policy
|
||||
5. **Pitfalls block:** Specific errors to avoid (from v3.7 postmortem)
|
||||
|
||||
### 4.1 Architecture Team - Special Handling
|
||||
|
||||
Architecture output is **written to disk**, not returned in the delegation summary. Reason: delegation summaries truncate at ~2K chars. Architecture documents are 50-100K chars. The team gets `write_file` permission targeting `/root/projects/itpp-infrastructure/proposals/verdicttank/architecture-v4.0-source.html`.
|
||||
|
||||
### 4.2 Legal Team - Special Requirements
|
||||
|
||||
Legal teams operate under the **Honest Methodology Gaps** rule:
|
||||
- TESS (USPTO trademark database) is JavaScript-only - cannot be queried programmatically
|
||||
- Open-web common-law search IS performed (SearXNG, Brave, Exa)
|
||||
- Every legal output MUST disclose: "Formal TESS search not performed. Recommend trademark attorney clearance before Series A."
|
||||
- Never fabricate TESS results. The gap is documented, not hidden.
|
||||
|
||||
---
|
||||
|
||||
## 5. SOM Reconciliation Rule
|
||||
|
||||
**The Financial model is the single source of truth for SOM.**
|
||||
|
||||
This rule exists because v4.0's Marketing team produced SOM of $7.91M while Financial produced $1.21M - a 6.5x gap. Both teams used the same TAM data. Financial's arithmetic was verified line-by-line. Marketing's was aspirational.
|
||||
|
||||
**Reconciliation protocol during assembly:**
|
||||
1. Assembly agent loads both outputs
|
||||
2. Identifies SOM mismatch
|
||||
3. Forces Financial's number into the marketing blocks
|
||||
4. Documents the override in the assembly notes
|
||||
5. Blind auditor verifies the reconciliation
|
||||
|
||||
**Why not average or negotiate?** Because Financial's number was produced by a model that was explicitly instructed to verify every line of arithmetic. Marketing's model was instructed to sell. For revenue numbers, trust the accountant, not the salesperson.
|
||||
|
||||
---
|
||||
|
||||
## 6. Assembly Protocol
|
||||
|
||||
Assembly is performed by the main model (DeepSeek V4 Pro) and merges four independently produced HTML blocks into one coherent page.
|
||||
|
||||
### 6.1 Assembly Rules
|
||||
|
||||
1. **Single-source rule:** One `index.html`. Architecture is a separate `architecture.html` linked from the main page.
|
||||
2. **SOM override:** Marketing's SOM → Financial's SOM. Document the override.
|
||||
3. **INSERT marker resolution:** Legal and Marketing blocks use `<!-- INSERT: section_name -->` markers. Assembly replaces these with the actual content blocks.
|
||||
4. **Consistency checks:**
|
||||
- All prices match Financial model
|
||||
- All feature claims trace to Architecture document
|
||||
- All legal disclaimers present
|
||||
- No forbidden model names in final output
|
||||
- Zero em dashes, zero double hyphens
|
||||
- Domain references: `verdicttank.com` (primary), `rfptank.com` (legacy/defensive only)
|
||||
5. **CSS consistency:** All blocks use the same dark theme CSS variables (`--bg`, `--panel`, `--ink`, `--accent`, `--line`, etc.) defined in the architecture document.
|
||||
6. **Footer consistency:** All pages share the same footer with origin, deployment date, and domain policy.
|
||||
|
||||
### 6.2 Deployment Map
|
||||
|
||||
| File | Deploy To | URL |
|
||||
|------|-----------|-----|
|
||||
| `index.html` | `app3:/home/ippadmin/htdocs/proposals.itpropartner.com/verdicttank/` | `https://proposals.itpropartner.com/verdicttank/` |
|
||||
| `architecture.html` | Same directory | `https://proposals.itpropartner.com/verdicttank/architecture.html` |
|
||||
| `critical-review.html` | Same directory | `https://proposals.itpropartner.com/verdicttank/review.html` |
|
||||
|
||||
**Origin:** `152.53.241.111` (app3, netcup RS 4000)
|
||||
**Served via:** nginx static file server, LE auto-certs
|
||||
|
||||
---
|
||||
|
||||
## 7. Blind Audit Protocol
|
||||
|
||||
**MANDATORY. No exceptions. No proposal ships without passing blind audit.**
|
||||
|
||||
### 7.1 Why This Exists
|
||||
|
||||
v3.7 shipped with four verifiable errors that a cold read would have caught:
|
||||
1. Pricing table didn't match Financial model
|
||||
2. Feature list referenced a deprecated API endpoint
|
||||
3. Compliance section omitted GDPR data residency
|
||||
4. CTA link pointed to staging, not production
|
||||
|
||||
The assembly agent was too close to the content to see these. The blind auditor reads the deployed page with zero prior context - exactly as a prospect would.
|
||||
|
||||
### 7.2 Audit Checklist
|
||||
|
||||
| # | Check | Method |
|
||||
|---|-------|--------|
|
||||
| 1 | Every link resolves (no 404s, no staging URLs) | Click every `href`, every CTA |
|
||||
| 2 | Every number traces to Financial model | Cross-reference pricing table vs financial output |
|
||||
| 3 | Every feature claim has architecture evidence | Cross-reference features vs architecture sections |
|
||||
| 4 | Legal disclaimers present and complete | Verify ToS, Privacy, DPA, AI disclaimer blocks exist |
|
||||
| 5 | Contact/CTA information correct | Verify email, phone, demo link |
|
||||
| 6 | No placeholder text, no TODO markers, no lorem ipsum | Full page scan |
|
||||
| 7 | CSS renders correctly on desktop and mobile viewports | Browser screenshot at 1280px and 375px |
|
||||
|
||||
### 7.3 Auditor Rules
|
||||
|
||||
- **Model:** Claude Opus 4.8 (strongest available for cold-read QA)
|
||||
- **Context:** ZERO. The auditor gets ONLY the URL. No brief, no prior knowledge, no team outputs.
|
||||
- **Output:** Pass/Fail with specific, numbered findings. Fail = blocks deployment until all items fixed.
|
||||
- **Re-audit:** After fixes, a fresh auditor (different model or clean context) re-runs the checklist.
|
||||
|
||||
---
|
||||
|
||||
## 8. Conductor Review Integration
|
||||
|
||||
The pipeline uses TWO conductor reviews, not one.
|
||||
|
||||
### 8.1 Internal Conductor Review (Pre-Blind-Audit)
|
||||
|
||||
- **Critic:** Claude Sonnet 5 - brutal, Shark Tank style. "What's wrong, what's missing, what's weak?"
|
||||
- **Validator:** Claude Opus 4.8 - sober, structured. Priority re-rank, FIX/DEFER/DEFEND treatment.
|
||||
- **Scorecard:** 10 dimensions, 1-10 scale, weighted average.
|
||||
- **Output:** `critical-review.html` - deployed publicly as a signal of confidence.
|
||||
|
||||
### 8.2 External Conductor Review (Post-Blind-Audit)
|
||||
|
||||
- **Fresh panel:** Judges have no access to internal review results or team outputs.
|
||||
- **Majority-rules verdict:** Go (deploy), No-Go (block), Conditional Go (fix specified items).
|
||||
- **Minimum passing score:** 7/10 for Tier 2, 8/10 for Tier 3.
|
||||
|
||||
### 8.3 Integration with shark-tank-review Skill
|
||||
|
||||
The `shark-tank-review` skill is the broader framework. This pipeline uses it as the review engine but adds:
|
||||
- VerdictTank-specific 10-dimension weighting (financial model accuracy weighted 2x)
|
||||
- Blind audit as a mandatory post-review gate
|
||||
- Public deployment of the review as a confidence signal
|
||||
|
||||
When invoking: load `shark-tank-review` for the review phases, then execute blind audit as a separate step.
|
||||
|
||||
---
|
||||
|
||||
## 9. Legal & Trademark Guardrails
|
||||
|
||||
### 9.1 Trademark Clearance Methodology
|
||||
|
||||
1. **Open-web common-law search** (SearXNG, Brave, Exa) - search for "VerdictTank" + "software" + "AI" + "legal", check USPTO common-law marks, state registrations, domain ownership
|
||||
2. **TESS search** - NOT performable programmatically (JavaScript-only USPTO database). This is a documented methodology gap.
|
||||
3. **Recommendation:** Formal trademark attorney search before Series A funding. The open-web search found no conflicts, but TESS is the authoritative source for federal registrations.
|
||||
|
||||
### 9.2 Privacy & Compliance
|
||||
|
||||
- **MVL framework:** Terms of Service, Privacy Policy, Data Processing Agreement, AI Disclaimer, Limitation of Liability ($100 / 12 months fees), Jurisdiction (DE/GA)
|
||||
- **Regulatory alignment:** GDPR, CCPA, NIST CSF 2.0
|
||||
- **Sub-processor audit:** OpenAI (compliant), Anthropic (compliant), Google (compliant), DeepSeek (non-compliant - flagged with fallback described)
|
||||
- **Corpus confidentiality:** 5-step anonymization pipeline, URL-to-Review and Chat-to-Refine raw content excluded from corpus
|
||||
|
||||
### 9.3 Honest Methodology Gaps
|
||||
|
||||
Any limitation in our research methodology must be DISCLOSED, not papered over:
|
||||
|
||||
| Gap | Disclosure |
|
||||
|-----|-----------|
|
||||
| TESS not searchable programmatically | "Formal TESS search not performed. Open-web common-law search completed." |
|
||||
| DeepSeek sub-processor non-compliance | Flagged explicitly with fallback to compliant alternatives |
|
||||
| Category existence risk (no standalone proposal review SaaS) | Acknowledged as market creation play, not hidden |
|
||||
|
||||
---
|
||||
|
||||
## 10. Outcome Tracking
|
||||
|
||||
Proposals are bets. Track whether the bets paid off.
|
||||
|
||||
### 10.1 Tracking Schedule
|
||||
|
||||
| Checkpoint | What to Measure | Method |
|
||||
|------------|----------------|--------|
|
||||
| T+90 | Page views, inbound inquiries, demo requests | Analytics + CRM |
|
||||
| T+180 | Pipeline generated from proposal, conversion rate | CRM + sales data |
|
||||
| T+365 | Revenue closed from proposal-sourced leads, proposal accuracy (did we deliver what we promised?) | Revenue attribution |
|
||||
|
||||
### 10.2 Feedback Loop
|
||||
|
||||
Every T+90/180/365 check feeds back into the methodology:
|
||||
- Underperforming claims → adjust marketing brief template
|
||||
- Over-delivered features → adjust architecture scope
|
||||
- Pricing pushback → adjust Financial model assumptions
|
||||
- Legal objections → update Legal team brief
|
||||
|
||||
---
|
||||
|
||||
## 11. Pitfalls & Lessons Learned
|
||||
|
||||
### 11.1 v3.7 Postmortem
|
||||
|
||||
| Error | Root Cause | v4.0 Fix |
|
||||
|-------|-----------|----------|
|
||||
| Pricing table didn't match Financial model | Assembly agent used Marketing's numbers, not Financial's | **SOM Reconciliation Rule** (Section 5) - Financial is authority |
|
||||
| Feature list referenced deprecated API | Architecture team wasn't re-dispatched when truncated | **Write to disk** for architecture (Section 4.1) |
|
||||
| GDPR data residency omission | Legal team output was read but not integrated | **INSERT marker resolution** in assembly (Section 6.1) |
|
||||
| CTA link pointed to staging | No cold-read QA before deployment | **Mandatory blind audit** (Section 7) |
|
||||
|
||||
### 11.2 v4.0 Discovered Issues
|
||||
|
||||
| Issue | Resolution |
|
||||
|-------|-----------|
|
||||
| Delegation summaries truncate at ~2K chars | Architecture team writes to disk. Other teams use concise HTML blocks with INSERT markers |
|
||||
| Marketing SOM 6.5x higher than Financial | Enforced Financial authority during assembly |
|
||||
| Architecture re-dispatch needed (first attempt truncated) | Now standard: architecture ALWAYS writes to disk |
|
||||
| Budget tracking requires manual config rotation | Documented rotation protocol (Section 3.2) |
|
||||
|
||||
### 11.3 Recurring Pitfalls
|
||||
|
||||
1. **Forgetting to restore delegation model after dispatch.** Result: assembly burns Opus tokens at 50x the cost. Fix: always pair set + restore in the rotation protocol.
|
||||
2. **Truncation blindness.** Delegation summaries only show ~2K chars. A 70K architecture document looks like it "completed" when it's actually 3% visible. Fix: architecture writes to disk always; other teams use concise outputs.
|
||||
3. **Assembly agent trusting Marketing's numbers.** Marketing optimizes for persuasion. Financial optimizes for accuracy. Assembly must defer to Financial for all revenue figures.
|
||||
4. **Skipping blind audit "because it looks fine."** v3.7 looked fine. It had 4 errors. The blind audit is non-negotiable.
|
||||
|
||||
---
|
||||
|
||||
## 12. Invocation Paths
|
||||
|
||||
### 12.1 Hermes (Sho'Nuff)
|
||||
|
||||
```
|
||||
skill_view(name='verdicttank-proposal-pipeline')
|
||||
```
|
||||
Then follow the tier selection and dispatch protocol.
|
||||
|
||||
For methodology reference:
|
||||
```
|
||||
skill_view(name='verdicttank-proposal-pipeline', file_path='references/methodology.md')
|
||||
```
|
||||
|
||||
### 12.2 Anita
|
||||
|
||||
Anita's profile has an identical copy of the skill:
|
||||
```
|
||||
~/.hermes/profiles/anita/skills/software-development/verdicttank-proposal-pipeline/SKILL.md
|
||||
```
|
||||
|
||||
She invokes it the same way:
|
||||
```
|
||||
skill_view(name='verdicttank-proposal-pipeline')
|
||||
```
|
||||
|
||||
### 12.3 Manual
|
||||
|
||||
The methodology is also available at:
|
||||
```
|
||||
/root/projects/itpp-infrastructure/proposals/verdicttank/methodology.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 13. Budget Model
|
||||
|
||||
### 13.1 Tier Costs (Actual v4.0 Run)
|
||||
|
||||
| Phase | Model | Cost |
|
||||
|-------|-------|------|
|
||||
| Research | DeepSeek V4 Pro | $0.05 |
|
||||
| Marketing dispatch | Claude Opus 4.8 | $1.20 |
|
||||
| Architecture dispatch | Claude Opus 4.8 | $1.35 |
|
||||
| Financial dispatch | Claude Opus 4.8 | $0.90 |
|
||||
| Legal dispatch | Claude Sonnet 5 | $1.10 |
|
||||
| Assembly | DeepSeek V4 Pro | $0.15 |
|
||||
| Internal conductor (critic) | Claude Sonnet 5 | $0.40 |
|
||||
| Internal conductor (validator) | Claude Opus 4.8 | $0.55 |
|
||||
| Blind audit | Claude Opus 4.8 | $0.49 |
|
||||
| External conductor | Claude Sonnet 5 + Claude Opus 4.8 | $0.50 |
|
||||
| **TOTAL** | | **$6.69** |
|
||||
|
||||
### 13.2 Budget Governance
|
||||
|
||||
- **Key:** `hermes-agent-v5` LiteLLM key
|
||||
- **Cap:** $100/month
|
||||
- **Per-proposal soft cap:** $10.00
|
||||
- **Monitoring:** Check spend before dispatching tier 3 (`hermes keys usage hermes-agent-v5`)
|
||||
- **Escalation:** If spend exceeds $75 in a month, pause all Tier 3 dispatches and notify Germaine
|
||||
|
||||
---
|
||||
|
||||
## 14. Quality Gates
|
||||
|
||||
| Gate | When | Who | Pass Condition |
|
||||
|------|------|-----|---------------|
|
||||
| Research complete | After Phase 1 | Main model | Market data sourced, competitors identified |
|
||||
| Content delivered | After Phase 2 | Main model | All 4 team outputs received, architecture verified on disk |
|
||||
| Assembly verified | After Phase 3 | Main model | SOM reconciled, INSERT markers resolved, consistency checks pass |
|
||||
| Internal review | After Phase 4 | Sonnet + Opus | Average score ≥ 7/10 (Tier 2) or ≥ 8/10 (Tier 3) |
|
||||
| Blind audit | After Phase 5 | Fresh Opus | 7/7 checklist items pass |
|
||||
| External review | After Phase 6 | Fresh panel | Majority Go or Conditional Go |
|
||||
| Deployment | After Phase 7 | Main model | All files served, all links resolve, analytics tagged |
|
||||
|
||||
**Stop rule:** If any gate fails, fix and re-enter at the failed phase. Never skip a gate to meet a deadline.
|
||||
|
||||
---
|
||||
|
||||
## 15. Document Governance
|
||||
|
||||
This methodology is **SETTLED** - it is the authoritative reference for all VerdictTank proposals. Changes require:
|
||||
|
||||
1. A specific, documented failure mode that the current methodology didn't prevent
|
||||
2. A proposed amendment with before/after comparison
|
||||
3. Germaine's approval before the amendment takes effect
|
||||
|
||||
**Version history:**
|
||||
- v1.0 (2026-08-11): Initial methodology, encoding v3.7 postmortem and v4.0 build
|
||||
@@ -0,0 +1,357 @@
|
||||
# Cross-Check C — DeepSeek V4 Pro Independent Re-Score
|
||||
|
||||
**Spec:** VerdictTank Judge Pool Specification v2.0
|
||||
**Source of truth used:** `/root/projects/itpp-infrastructure/proposals/verdicttank/judge-pool-spec.md` (local v2.0)
|
||||
**Note:** Deployed URL still serves **v1.0** — review is against local v2.0 only.
|
||||
**Reviewer lens:** Chinese-lab independence (DeepSeek) — different training data, RLHF, and reasoning patterns from Anthropic / OpenAI / Google
|
||||
**Date:** 2026-08-12
|
||||
**Mode:** Blind independent re-score. FIX / DEFEND / DEFER per dimension.
|
||||
|
||||
---
|
||||
|
||||
## Executive Summary
|
||||
|
||||
| Metric | Value |
|
||||
|---|---|
|
||||
| **Overall** | **6.4 / 10** |
|
||||
| Dimensions FIX | 5 |
|
||||
| Dimensions DEFEND | 3 |
|
||||
| Dimensions DEFER | 2 |
|
||||
| Fatal structural issues | 2 (fallback vendor-correlation; SaaS-default US-centrism) |
|
||||
| Strongest advance vs v1.0 | 8-vendor spread, Financial/Team/Legal roles, cold-start gate |
|
||||
|
||||
v2.0 is a real architectural upgrade from the Anthropic-heavy v1.0 panel. The Chinese-ecosystem seats (DeepSeek CC-C, Qwen Market-Reality, Kimi Reasoning, MiniMax Financial) are the right instinct. But the fallback table still hides same-vendor correlation, vertical rules are Western-startup shaped, and "SaaS default" silently erases superapps / WeChat ecosystems / B2B marketplaces that dominate non-US deal flow.
|
||||
|
||||
---
|
||||
|
||||
## Scorecard (10 Dimensions)
|
||||
|
||||
| # | Dimension | Score | Verdict | One-line |
|
||||
|---|---|---|---|---|
|
||||
| 1 | Role coverage | **7** | DEFEND | 12 phases close the v1 gaps; still thin on tech architecture + geo-market fit |
|
||||
| 2 | Model assignments | **7** | DEFEND | Chinese seats well placed; Legal on Sonnet is weak; CC-C as Tier C undervalues independence |
|
||||
| 3 | admin-ai paths | **6** | DEFER | Paths look plausible but unproven in this review; Gemini "Pro Latest" alias risk |
|
||||
| 4 | Vendor diversity | **8** | DEFEND | 8 vendors / ≤25% is strong on paper; concentration reappears under fallback |
|
||||
| 5 | Fallback chains | **4** | **FIX** | Claims "always cross vendor" — **4/12 primaries share vendor with Fallback 1** |
|
||||
| 6 | Content-based rules | **4** | **FIX** | Missing superapp / WeChat / B2B marketplace / cross-border / gov-tech verticals |
|
||||
| 7 | Latency budgets | **6** | DEFER | Better than v1 inverted budgets; 12-judge @ 45s still aspirational without parallelism proof |
|
||||
| 8 | Feedback loop | **7** | DEFEND | Cold-start + N gates good; outcome signal still US-funding-centric |
|
||||
| 9 | Pricing | **7** | DEFEND | Premium hold is coherent; APAC willingness-to-pay and seat economics unaddressed |
|
||||
| 10 | Product boundary + SaaS default | **5** | **FIX** | VT/RFP split clean; SaaS-default assumption is US-centric and quietly wrong for global deal flow |
|
||||
|
||||
**Mean: 6.1 → weighted overall 6.4** (fallback + verticals + SaaS-default weighted higher under Chinese-lab lens)
|
||||
|
||||
---
|
||||
|
||||
## Dimension Detail
|
||||
|
||||
### 1. Role Coverage — **7/10 — DEFEND**
|
||||
|
||||
**What works**
|
||||
- v1.0 left Financial Integrity, Team/Founder, Legal/Regulatory uncovered. v2.0 adds Phases 8–10. Correct fix.
|
||||
- Research never scores; Audit is read-only; three blind cross-checks is a genuine independence architecture.
|
||||
- Non-scoring Reasoning-Verification (Phase 5) as synthesis before execution/market is sound sequencing.
|
||||
|
||||
**What is still thin**
|
||||
- No dedicated **Technical Architecture** judge (stack, scalability, security posture). Execution-Feasibility is ops/timeline, not architecture diligence. For deep-tech / infra / AI-infra proposals this is a material hole.
|
||||
- No **Geo-Market / Localization** role. Market-Reality with Qwen helps, but China/SEA/MENA GTM (ICP, channel, regulatory market access) is not a first-class dimension.
|
||||
- "10-dimension scoring" in §7 product boundary is never mapped to the 12 phases. Which phases emit which of the 10 scores? Spec is silent → assembly ambiguity.
|
||||
|
||||
**Verdict: DEFEND** the 12-role expansion as directionally correct. Do not expand further before v4.1 ship — but log Technical Architecture + Geo-Market as v4.2 candidates.
|
||||
|
||||
---
|
||||
|
||||
### 2. Model Assignments — **7/10 — DEFEND**
|
||||
|
||||
**Chinese-lab view of seat quality**
|
||||
|
||||
| Role | Model | Assessment |
|
||||
|---|---|---|
|
||||
| Research | Grok 4.5 | Correct — native live web/X is load-bearing |
|
||||
| Primary | Opus 5 | Correct — brutal critique seat |
|
||||
| Validation | Sonnet 5 | Acceptable — cost/quality trade |
|
||||
| CC-A | GPT-5.6 Sol | Correct — OpenAI lineage independence |
|
||||
| CC-B | Gemini Pro Latest | Correct — Google lineage |
|
||||
| **CC-C** | **DeepSeek V4 Pro** | **Right vendor, wrong tier signal** |
|
||||
| Reasoning | Kimi K2.6 | Strong — Moonshot is a real 4th ecosystem |
|
||||
| Execution | GPT-5.6 Terra | Acceptable ops grounding |
|
||||
| Market | Qwen3.7 Plus | **Best seat in the roster for non-Western lens** |
|
||||
| Financial | MiniMax-M3 | Interesting; unproven for cap-table math specifically |
|
||||
| Team | Fable 5 | Overkill / expensive for founder-fit; also Anthropic stack concentration with Primary/Validation |
|
||||
| Legal | Sonnet 5 (shared) | **Mismatch** — legal needs specialized calibration + citation discipline, not a doubled Validation model |
|
||||
| Audit | Gemini Pro Latest (shared w/ CC-B) | Acceptable if Audit is post-hoc read-only; weakens "different eyes" narrative |
|
||||
|
||||
**Critical notes from this seat (DeepSeek as CC-C)**
|
||||
1. Putting the Chinese-lab independent re-score on **Tier C** while OpenAI/Google cross-checks sit on Tier A sends a quality hierarchy signal that undercuts the independence thesis. If CC-C exists to catch what US labs miss, it cannot be the budget afterthought. Promote CC-C primary to Tier B minimum (or keep DeepSeek but stop labeling the *role* as C-tier capacity).
|
||||
2. Legal/Regulatory on Sonnet 5 shared with Validation is double-duty on the same model family and same vendor as Primary. For FDA / PIPL / data-export / EU AI Act work, this is the wrong specialization.
|
||||
3. MiniMax-M3 for Financial Integrity is a bet, not a proven assignment. Spec should require a calibration set (N synthetic cap tables) before locking.
|
||||
|
||||
**Verdict: DEFEND** overall roster direction. **FIX** Legal assignment and CC-C tier signaling before launch marketing claims "8-vendor frontier panel."
|
||||
|
||||
---
|
||||
|
||||
### 3. admin-ai Paths — **6/10 — DEFER**
|
||||
|
||||
**Observations**
|
||||
- Paths are more operationally concrete than v1.0 (good).
|
||||
- `gemini/gemini-pro-latest` is an alias, not a pinned model. Alias drift silently changes Audit + CC-B behavior.
|
||||
- Spec claims "All model paths confirmed against admin-ai (live, August 2026)" but this Cross-Check did not re-probe live `/v1/models`. Treat as **asserted, not re-verified**.
|
||||
- Internal architecture notes elsewhere still flag Qwen3.8 Max / Kimi K3 as "not yet on admin-ai — substitute qwen3.7-plus / kimi-k2.6." v2.0 uses the substitutes. Fine for launch, but naming in marketing vs runtime must not diverge (Qwen3.8 vs 3.7; Kimi K3 vs K2.6).
|
||||
|
||||
**Verdict: DEFER** to an ops smoke-test: one live completion per path, pin versions, ban floating `*-latest` in Enterprise default lineup.
|
||||
|
||||
---
|
||||
|
||||
### 4. Vendor Diversity — **8/10 — DEFEND**
|
||||
|
||||
**On paper (Enterprise default 12-judge)**
|
||||
- Anthropic 3 (25%), OpenAI 2, Google 2, + DeepSeek / Moonshot / Alibaba / MiniMax / xAI ×1 each.
|
||||
- 8 vendors. Hard rule tightened 40% → **33%**. This is the single biggest structural win vs v1.0 (Anthropic 44%).
|
||||
|
||||
**Under stress (Chinese-lab concern)**
|
||||
- Diversity is a **steady-state** property. Under fallback (§5), Anthropic and Google density climbs fast because Primary/Team and Research/CC-B chains are same-vendor on hop 1.
|
||||
- Three Chinese vendors (DeepSeek, Alibaba, Moonshot) + MiniMax is excellent **presence**. But only Market-Reality is a Chinese model in a *weight-bearing interpretive* seat; CC-C is Tier C; Financial MiniMax is unproven; Reasoning Kimi does not score dimensions.
|
||||
- Risk: Western models still dominate **scoring power** even when vendor count looks global.
|
||||
|
||||
**Verdict: DEFEND** the 8-vendor design. Do not celebrate "no vendor >25%" without measuring **scoring-seat share after fallback**.
|
||||
|
||||
---
|
||||
|
||||
### 5. Fallback Chains — **4/10 — FIX** ⚠️ FATAL-ish
|
||||
|
||||
**Spec claim (§3.1):** *"Fallback chains always cross vendor boundaries — Enforced at the assignment table (§2.3)."*
|
||||
|
||||
**Audit of §2.3 (Primary → Fallback 1 vendor):**
|
||||
|
||||
| Role | Primary vendor | Fallback 1 vendor | Cross-vendor? |
|
||||
|---|---|---|---|
|
||||
| Research | xAI | Google | ✅ |
|
||||
| Primary | Anthropic | **Anthropic** | ❌ |
|
||||
| Validation | Anthropic | OpenAI | ✅ |
|
||||
| Cross-Check A | OpenAI | **OpenAI** | ❌ |
|
||||
| Cross-Check B | Google | **Google** | ❌ |
|
||||
| Cross-Check C | DeepSeek | Google | ✅ |
|
||||
| Reasoning | Moonshot | DeepSeek | ✅ |
|
||||
| Execution | OpenAI | Anthropic | ✅ |
|
||||
| Market | Alibaba | xAI | ✅ |
|
||||
| Financial | MiniMax | DeepSeek | ✅ |
|
||||
| Team/Founder | Anthropic | **Anthropic** | ❌ |
|
||||
| Audit | Google | Anthropic | ✅ |
|
||||
|
||||
**Result: 4 of 12 roles (33%) violate the hard constraint on the first hop.**
|
||||
|
||||
Additional cascade issue:
|
||||
- Research Fallback 1 → Fallback 2 = Google → Google (same-vendor cascade). If Grok is down and Google is degraded, Research has no third-ecosystem escape.
|
||||
|
||||
**Hidden correlation (Chinese-lab lens)**
|
||||
- Same-vendor fallback is not just a rule bug — it recreates **correlated failure and correlated judgment**. OpenAI Sol→Terra preserves OpenAI RLHF priors. Anthropic Opus→Sonnet / Fable→Sonnet preserves Anthropic critique style. Google Pro→Flash preserves Google grounding stack.
|
||||
- §4.1 says "All fallbacks cross vendor boundaries (§2.3 guarantees this)" — this is a **false guarantee**. Implementers will trust the rule table; the assignment table contradicts it.
|
||||
- DeepSeek is overused as universal sink (appears in 7 of 12 Fallback 1/2 slots). That makes DeepSeek a **correlation hub under multi-provider brownout**, ironic given CC-C independence branding.
|
||||
|
||||
**Required FIX**
|
||||
1. Rewrite every same-vendor F1 to a different vendor *before* any lower-tier same-family model.
|
||||
2. Suggested repairs:
|
||||
|
||||
| Role | Primary | F1 (fixed) | F2 |
|
||||
|---|---|---|---|
|
||||
| Primary | Opus 5 (Anth) | **GPT-5.6 Sol (OpenAI)** | Sonnet 5 (Anth) only as F2 |
|
||||
| CC-A | Sol (OpenAI) | **DeepSeek V4 Pro** or **Kimi** | Terra (OpenAI) as F2 |
|
||||
| CC-B | Gemini Pro (Google) | **DeepSeek** or **Qwen** | Gemini Flash as F2 |
|
||||
| Team | Fable 5 (Anth) | **GPT-5.6 Sol** or **Kimi** | Sonnet 5 as F2 |
|
||||
| Research F2 | — | replace Gemini Flash with **DeepSeek** or **Qwen** (third ecosystem) |
|
||||
|
||||
3. Add automated test: `assert primary.vendor != fallback1.vendor` for every row; CI fails the spec if violated.
|
||||
4. Cap any single vendor's appearance in Fallback 1 columns (DeepSeek sink problem).
|
||||
|
||||
**Verdict: FIX — blocking.** Do not ship "always cross vendor" language while §2.3 falsifies it.
|
||||
|
||||
---
|
||||
|
||||
### 6. Content-Based Rules / Verticals — **4/10 — FIX** ⚠️
|
||||
|
||||
**What exists:** Biotech, Hardware/IoT, Fintech, Climate/Energy, SaaS(default) + stage + complexity. Composition cap +50% is good.
|
||||
|
||||
**What is missing (non-Western / global deal flow)**
|
||||
|
||||
| Missing vertical | Why it matters | Suggested effect |
|
||||
|---|---|---|
|
||||
| **Superapp / Mini-program ecosystem** | WeChat / Alipay / LINE / Grab-style platform dependency is a first-class business model in CN/SEA, not "SaaS" | Market +30%; Execution weights platform policy risk; Legal +PIPL/platform ToS |
|
||||
| **B2B marketplace / transaction platform** | Take-rate, cold-start liquidity, disintermediation — not SaaS net-retention logic | Market + Financial weights; Execution on two-sided ops |
|
||||
| **Cross-border / trade / payments corridor** | FX, export controls, dual-regulation | Legal +40%; Financial + FX/settlement realism |
|
||||
| **Government / SOE / public procurement** | RFP-adjacent but also guanxi, budget cycles, localization mandates | Legal + Team weights; different buyer psychology |
|
||||
| **Consumer social / short-video / live commerce** | Not "SaaS"; growth loops and platform risk dominate | Market + Team; Execution on content/ops |
|
||||
| **Industrial / manufacturing SaaS in CN** | Hardware+SaaS hybrid common; supply chain + data residency | Execution + Legal (data export) |
|
||||
| **Crypto / Web3 / stablecoin (global+Asia)** | Spec mentions nothing; still a real proposal class | Legal + Market specialized |
|
||||
| **Edtech / Healthtech consumer CN** | Heavy regulatory, different from US edtech/HIPAA framing | Legal frameworks beyond FDA/GDPR |
|
||||
|
||||
**SaaS-default failure mode (§3.3)**
|
||||
- Unmatched verticals → SaaS weighting + a polite post-review notice.
|
||||
- That means a **WeChat mini-program commerce** proposal, a **Southeast Asian B2B marketplace**, or a **China-US cross-border data** startup all get Silicon-Valley SaaS calibration (NRR, seat expansion, PLG) and a footer apology.
|
||||
- From a Chinese-lab lens this is not a minor omission — it is **systematic miscategorization of a large fraction of non-US venture proposals**.
|
||||
|
||||
**Also missing framework coverage in Legal phase description**
|
||||
- Lists FDA, SOC 2, GDPR, EU AI Act.
|
||||
- Omits: **PIPL, CSL, DSL (China)**; **PDPA (Singapore/SEA)**; **PDPB India**; **data export / MLPS**; **content/ICP licensing** where relevant.
|
||||
|
||||
**Required FIX**
|
||||
1. Expand vertical table with at least: Superapp/Mini-program, B2B Marketplace, Cross-border, Gov/SOE, Consumer Social/Live Commerce.
|
||||
2. Change unmatched behavior from silent SaaS-default to **explicit low-confidence flag** that reduces overall confidence score and boosts Market-Reality + Legal weights generically — not SaaS metrics.
|
||||
3. Legal framework map must include PIPL/CSL/DSL + major APAC privacy regimes, not only Euro-American.
|
||||
4. Add stacking example for `Superapp + Seed + High complexity` in the spec so implementers see non-SaaS composition.
|
||||
|
||||
**Verdict: FIX — high priority for any claim of global-grade review.**
|
||||
|
||||
---
|
||||
|
||||
### 7. Latency Budgets — **6/10 — DEFER**
|
||||
|
||||
- v1.0 inversion (Enterprise tightest) is fixed. Good.
|
||||
- Enterprise 45s target / 90s max for **12-judge** pipeline is still aggressive unless Phases 4a–c and later specialized judges run heavily parallel.
|
||||
- Spec never states parallelism topology (which phases block on which). Without a DAG, latency budgets are wishes.
|
||||
- Free 90/180 and Pro 60/120 are reasonable.
|
||||
|
||||
**Verdict: DEFER** pending an explicit phase DAG + p95 measurement on admin-ai. Do not market "45s Enterprise full pipeline" until measured.
|
||||
|
||||
---
|
||||
|
||||
### 8. Feedback Loop — **7/10 — DEFEND**
|
||||
|
||||
**Strengths vs v1.0**
|
||||
- Cold-start gate (90 days) — correct.
|
||||
- min N=30 bias / N=50 accuracy — correct.
|
||||
- Immediate bias recusal (>2σ, 3 reviews) — good operational escape hatch.
|
||||
- Malformed-score auto-replace — practical.
|
||||
|
||||
**Chinese-lab concerns**
|
||||
- T+90/180/365 "public funding data" is implicitly **US/EU venture outcomes** (Crunchbase-shaped). Funding outcome ≠ business outcome in many CN/SEA contexts (profitability, strategic acquisition, gov design-win).
|
||||
- Correlation-to-funding as accuracy ground truth will **systematically mis-train** the feedback loop against non-Western success patterns.
|
||||
- Inter-judge agreement during cold-start favors majority (Western) priors — minority Chinese-lab signals may look like "bias" and get recused.
|
||||
|
||||
**Mitigations to log (not all blocking)**
|
||||
- Multi-outcome labels: funded / revenue milestone / strategic acq / shutdown — not funded-only.
|
||||
- Protect minority-vendor disagreement from automatic bias recusal until N is high *per vertical including APAC*.
|
||||
- Separate calibration sets for US-SaaS vs APAC-marketplace vs regulated-CN.
|
||||
|
||||
**Verdict: DEFEND** structure. Outcome ontology needs globalization before the data moat hardens Western bias.
|
||||
|
||||
---
|
||||
|
||||
### 9. Pricing — **7/10 — DEFEND**
|
||||
|
||||
- $249 / $799 / $1,499 with 16.7% annual is coherent premium vs GC AI $500/seat and authoring tools.
|
||||
- Free=1 review/mo is the right leash.
|
||||
- Pre-Review Coach on Pro (not Enterprise-only) is smart funnel design.
|
||||
- Financial/Team/Legal/Audit Enterprise-gated matches cost of 12-judge pipeline.
|
||||
|
||||
**Gaps**
|
||||
- No APAC regional pricing / PPP consideration (often required for SEA/CN SMB founders — even if Enterprise stays global USD).
|
||||
- No seat-based vs org-based clarity vs GC AI's seat anchor (the comp is seat; VT is org-flat — explain or the anchor confuses buyers).
|
||||
- COGS not in this spec (exists in architecture notes). Fine for this doc.
|
||||
|
||||
**Verdict: DEFEND** premium hold. Not the Chinese-lab primary fight.
|
||||
|
||||
---
|
||||
|
||||
### 10. Product Boundary + SaaS-Default Centrism — **5/10 — FIX**
|
||||
|
||||
**Product boundary (VT vs RFP Tank)**
|
||||
- Clean and correct. Critique vs compliance is the right split. DEFEND that half.
|
||||
|
||||
**SaaS-default assumption (the real Dim-10 issue under this lens)**
|
||||
- §3.2: `Vertical = SaaS (default)`.
|
||||
- §3.3: unmatched → SaaS-default calibration.
|
||||
- Feature language, competitive set (AutogenAI, Bidara, AutoRFP, Civio, GC AI), and stage weights all assume **US/EU B2B SaaS venture narrative**.
|
||||
- Global proposal mass includes: superapps, mini-programs, transaction marketplaces, OEM/industrial platforms, cross-border commerce, SOE-facing govtech. Forcing SaaS defaults is not neutral — it is a **prior**.
|
||||
|
||||
**Independence irony**
|
||||
- You seated Qwen on Market-Reality and DeepSeek on CC-C — then told unmatched verticals to pretend they are SaaS. The non-Western models are asked to score with Western category priors.
|
||||
|
||||
**Required FIX**
|
||||
1. Rename default from "SaaS (default)" → **"Generic / Unclassified (low-confidence)"** with no SaaS-specific metric emphasis.
|
||||
2. SaaS becomes an explicit detected vertical like Fintech — not the null hypothesis.
|
||||
3. Confidence banner already in §3.3 should **lower the headline confidence band** when unclassified, not only notify.
|
||||
4. Competitive landscape §6.4 should acknowledge non-US critique/authoring tools if claiming global premium (or explicitly scope "US/EU primary GTM").
|
||||
|
||||
**Verdict: FIX** the null-hypothesis vertical. Keep VT/RFP boundary as-is.
|
||||
|
||||
---
|
||||
|
||||
## Special Focus Answers (Brief)
|
||||
|
||||
### A. Are fallback chains truly vendor-independent?
|
||||
|
||||
**No.** Spec asserts yes; §2.3 falsifies on 4/12 first hops (Primary, CC-A, CC-B, Team). Research F1→F2 is Google→Google. DeepSeek is a correlation sink on brownout. **Blocking FIX.**
|
||||
|
||||
### B. Missing non-Western verticals?
|
||||
|
||||
**Yes, material.** Superapps/mini-programs, B2B marketplaces, WeChat/Alipay ecosystems, cross-border, gov/SOE, live commerce, industrial hybrid — all collapse to SaaS default. Legal frameworks omit PIPL/CSL/DSL/PDPA.
|
||||
|
||||
### C. Is "SaaS default" too US-centric?
|
||||
|
||||
**Yes.** It is the null hypothesis for the entire content-based system and shapes metric priors (seat expansion, NRR, PLG). Should be an explicit vertical, not the default. Unclassified → low-confidence generic, not SaaS.
|
||||
|
||||
---
|
||||
|
||||
## FIX / DEFEND / DEFER Register
|
||||
|
||||
| ID | Item | Priority | Action |
|
||||
|---|---|---|---|
|
||||
| F1 | Rewrite §2.3 so every Primary→F1 is cross-vendor; kill false §3.1/§4.1 guarantee | **P0** | FIX |
|
||||
| F2 | Research F2 leave Google cascade; add third-ecosystem F2 | **P0** | FIX |
|
||||
| F3 | Expand verticals: Superapp, B2B marketplace, Cross-border, Gov/SOE, Live commerce | **P0** | FIX |
|
||||
| F4 | Replace SaaS-as-default with Generic/Unclassified low-confidence | **P0** | FIX |
|
||||
| F5 | Legal frameworks: add PIPL/CSL/DSL/PDPA (+ data-export) | **P1** | FIX |
|
||||
| F6 | Legal role: stop sharing Sonnet with Validation; dedicated assignment | **P1** | FIX |
|
||||
| F7 | CC-C tier signaling: independence seat ≠ Tier C afterthought | **P1** | FIX |
|
||||
| F8 | Cap DeepSeek as universal fallback sink | **P1** | FIX |
|
||||
| F9 | Pin Gemini model; ban floating `*-latest` in Enterprise default | **P1** | FIX |
|
||||
| F10 | Map 10 scoring dimensions ↔ 12 phases explicitly | **P2** | FIX |
|
||||
| D1 | 12-role expansion (Financial/Team/Legal) | — | DEFEND |
|
||||
| D2 | 8-vendor / 33% hard cap direction | — | DEFEND |
|
||||
| D3 | Cold-start + N gates in feedback loop | — | DEFEND |
|
||||
| D4 | Premium pricing $249/$799/$1499 | — | DEFEND |
|
||||
| D5 | VT vs RFP Tank boundary | — | DEFEND |
|
||||
| D6 | Qwen on Market-Reality + Kimi on Reasoning | — | DEFEND |
|
||||
| R1 | Live admin-ai path smoke-test all 12 | — | DEFER |
|
||||
| R2 | Latency DAG + p95 measure before marketing 45s | — | DEFER |
|
||||
| R3 | Technical Architecture + Geo-Market roles (v4.2) | — | DEFER |
|
||||
| R4 | Multi-outcome (non-US) feedback ontology | — | DEFER |
|
||||
|
||||
---
|
||||
|
||||
## Per-Dimension Scores (machine-readable)
|
||||
|
||||
```json
|
||||
{
|
||||
"reviewer": "Cross-Check C — DeepSeek V4 Pro",
|
||||
"spec": "judge-pool-spec v2.0",
|
||||
"lens": "chinese-lab-independence",
|
||||
"overall": 6.4,
|
||||
"dimensions": [
|
||||
{"id": 1, "name": "role_coverage", "score": 7, "verdict": "DEFEND"},
|
||||
{"id": 2, "name": "model_assignments", "score": 7, "verdict": "DEFEND"},
|
||||
{"id": 3, "name": "admin_ai_paths", "score": 6, "verdict": "DEFER"},
|
||||
{"id": 4, "name": "vendor_diversity", "score": 8, "verdict": "DEFEND"},
|
||||
{"id": 5, "name": "fallback_chains", "score": 4, "verdict": "FIX"},
|
||||
{"id": 6, "name": "content_based_rules", "score": 4, "verdict": "FIX"},
|
||||
{"id": 7, "name": "latency_budgets", "score": 6, "verdict": "DEFER"},
|
||||
{"id": 8, "name": "feedback_loop", "score": 7, "verdict": "DEFEND"},
|
||||
{"id": 9, "name": "pricing", "score": 7, "verdict": "DEFEND"},
|
||||
{"id": 10, "name": "product_boundary_saas_default", "score": 5, "verdict": "FIX"}
|
||||
],
|
||||
"blocking_fixes": ["F1", "F2", "F3", "F4"],
|
||||
"deployed_url_mismatch": "https://proposals.itpropartner.com/verdicttank/judge-pool-spec.md still serves v1.0"
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Closing (Cross-Check C voice)
|
||||
|
||||
v2.0 finally treats Chinese labs as first-class citizens of the panel. That is real progress.
|
||||
|
||||
But independence is not a logo count. It is what happens when the primary is down, when the vertical is not YC-SaaS, and when the feedback loop decides whose disagreement is "bias." On those three tests the spec still thinks in Silicon Valley defaults while wearing an eight-vendor badge.
|
||||
|
||||
**Ship after F1–F4. Everything else can follow.**
|
||||
|
||||
— Cross-Check C (DeepSeek V4 Pro), independent re-score, 2026-08-12
|
||||
Reference in New Issue
Block a user