root
f5175f1ce0
- audit/phase-one + phase-two: security audit briefs, findings, credential-rotation plan, Docker-USER hardening scripts, rollback refs - disaster-recovery/restore-test-log.md + backup-dr-audit-2026-08-10.md - clients/ (modelortho SEO audit, ai-biz-dev competitive landscape), notes/ (tiktok strategy) - projects/: front-desk-voice-agent, seo-visibility-checker product plan, hotnow-savannah HTML, resend-transactional-email, backup-dashboard-enhancements, code-review-graph, seo-ci-architecture - proposals/verdicttank/: architecture v4.0, methodology, judge-pool review, consolidation reasoning, cross-check review - docs/super-search/firecrawl-provider-strategy.md - updates: CHANGELOG, model-chain, projects-master-readme, intelsight.io - .gitignore: exclude nested standalone repos (seo-tool, venturebuilt)
1116 lines
69 KiB
HTML
1116 lines
69 KiB
HTML
<!DOCTYPE html>
|
||
<html lang="en">
|
||
<head>
|
||
<meta charset="UTF-8">
|
||
<meta name="viewport" content="width=device-width, initial-scale=1.0">
|
||
<title>VerdictTank v4.0 — Technical Architecture</title>
|
||
<style>
|
||
:root{
|
||
--bg:#0b0e14;--panel:#131824;--panel2:#1a2030;--ink:#e6edf3;--muted:#8b97a8;
|
||
--line:#232b3a;--accent:#4da3ff;--accent2:#7ee787;--warn:#f0a45c;--bad:#ff6b6b;
|
||
--code:#0d1117;--codeink:#c9d1d9;--pass:#2ea043;--below:#d29922;
|
||
}
|
||
*{box-sizing:border-box}
|
||
html{scroll-behavior:smooth}
|
||
body{margin:0;background:var(--bg);color:var(--ink);
|
||
font:15px/1.65 -apple-system,BlinkMacSystemFont,"Segoe UI",Roboto,Helvetica,Arial,sans-serif}
|
||
.wrap{max-width:1180px;margin:0 auto;padding:0 24px}
|
||
header.hero{background:linear-gradient(135deg,#0b0e14,#131824 60%,#0f1a2e);
|
||
border-bottom:1px solid var(--line);padding:48px 0 36px}
|
||
.badge{display:inline-block;font-size:12px;letter-spacing:.14em;text-transform:uppercase;
|
||
color:var(--accent);border:1px solid var(--accent);border-radius:999px;padding:4px 12px;margin-bottom:16px}
|
||
h1{font-size:34px;margin:6px 0 8px;letter-spacing:-.5px}
|
||
.sub{color:var(--muted);font-size:16px;max-width:760px}
|
||
.meta{margin-top:18px;display:flex;flex-wrap:wrap;gap:10px}
|
||
.meta span{font-size:12px;background:var(--panel2);border:1px solid var(--line);
|
||
border-radius:6px;padding:5px 10px;color:var(--muted)}
|
||
.meta b{color:var(--ink)}
|
||
nav.toc{position:sticky;top:0;z-index:20;background:rgba(11,14,20,.92);
|
||
backdrop-filter:blur(8px);border-bottom:1px solid var(--line);padding:10px 0}
|
||
nav.toc ol{list-style:none;margin:0;padding:0;display:flex;flex-wrap:wrap;gap:6px;font-size:12.5px}
|
||
nav.toc a{color:var(--muted);text-decoration:none;padding:4px 9px;border-radius:6px;border:1px solid transparent}
|
||
nav.toc a:hover{color:var(--ink);border-color:var(--line);background:var(--panel)}
|
||
section{padding:40px 0;border-bottom:1px solid var(--line)}
|
||
h2{font-size:24px;margin:0 0 6px;letter-spacing:-.3px}
|
||
h2 .n{color:var(--accent);font-variant-numeric:tabular-nums;margin-right:10px}
|
||
h3{font-size:17px;margin:26px 0 8px;color:var(--accent2)}
|
||
h4{font-size:14px;margin:18px 0 6px;color:var(--warn);text-transform:uppercase;letter-spacing:.06em}
|
||
p{margin:10px 0}
|
||
.lead{color:var(--muted);font-size:15.5px;margin-bottom:8px}
|
||
ul,ol{margin:10px 0;padding-left:22px}
|
||
li{margin:5px 0}
|
||
code{background:var(--code);color:var(--codeink);padding:2px 6px;border-radius:4px;
|
||
font:13px/1.5 "SF Mono",ui-monospace,Menlo,Consolas,monospace}
|
||
pre{background:var(--code);color:var(--codeink);border:1px solid var(--line);border-radius:10px;
|
||
padding:16px;overflow:auto;font:12.5px/1.55 "SF Mono",ui-monospace,Menlo,Consolas,monospace;margin:14px 0}
|
||
pre.ascii{line-height:1.35;font-size:12px;color:#9fb3c8}
|
||
table{width:100%;border-collapse:collapse;margin:14px 0;font-size:13.5px}
|
||
th,td{border:1px solid var(--line);padding:8px 10px;text-align:left;vertical-align:top}
|
||
th{background:var(--panel2);color:var(--ink);font-weight:600}
|
||
td{background:var(--panel);color:var(--ink)}
|
||
.card{background:var(--panel);border:1px solid var(--line);border-radius:12px;padding:18px 20px;margin:14px 0}
|
||
.grid2{display:grid;grid-template-columns:1fr 1fr;gap:14px}
|
||
.grid3{display:grid;grid-template-columns:1fr 1fr 1fr;gap:14px}
|
||
@media(max-width:820px){.grid2,.grid3{grid-template-columns:1fr}}
|
||
.tag{display:inline-block;font-size:11px;padding:2px 8px;border-radius:999px;margin:2px 4px 2px 0}
|
||
.t-get{background:#132a1c;color:#7ee787;border:1px solid #1f6f3f}
|
||
.t-post{background:#12233a;color:#4da3ff;border:1px solid #1f4f7f}
|
||
.t-put{background:#2a2312;color:#f0a45c;border:1px solid #7f5f1f}
|
||
.t-del{background:#2a1216;color:#ff6b6b;border:1px solid #7f1f2a}
|
||
.pill-pass{background:#0f2a17;color:#2ea043;border:1px solid #2ea043;padding:2px 9px;border-radius:999px;font-size:12px;font-weight:600}
|
||
.pill-below{background:#2a2410;color:#d29922;border:1px solid #d29922;padding:2px 9px;border-radius:999px;font-size:12px;font-weight:600}
|
||
.note{border-left:3px solid var(--accent);background:var(--panel);padding:10px 14px;margin:14px 0;border-radius:0 8px 8px 0}
|
||
.warnbox{border-left:3px solid var(--warn);background:var(--panel);padding:10px 14px;margin:14px 0;border-radius:0 8px 8px 0}
|
||
.legacy{border-left:3px solid var(--bad);background:var(--panel);padding:10px 14px;margin:14px 0;border-radius:0 8px 8px 0}
|
||
.mono{font-family:"SF Mono",ui-monospace,Menlo,Consolas,monospace}
|
||
.kv{font-size:13px}.kv b{color:var(--accent2)}
|
||
footer{padding:36px 0;color:var(--muted);font-size:13px}
|
||
hr{border:0;border-top:1px solid var(--line);margin:22px 0}
|
||
.eplist{list-style:none;padding:0}
|
||
.eplist li{border:1px solid var(--line);border-radius:8px;padding:8px 12px;margin:6px 0;background:var(--panel)}
|
||
.small{font-size:12px;color:var(--muted)}
|
||
</style>
|
||
</head>
|
||
<body>
|
||
<header class="hero">
|
||
<div class="wrap">
|
||
<span class="badge">Technical Architecture · v4.0</span>
|
||
<h1>VerdictTank — Multimodel Proposal Review Platform</h1>
|
||
<p class="sub">Complete implementation reference for the VerdictTank consensus-scoring engine.
|
||
Any software engineer can build the system from this document alone: pipeline topology, scoring
|
||
schemas, fallback chains, data model DDL, API surface, security controls, and deployment topology.</p>
|
||
<div class="meta">
|
||
<span>Version <b>v4.0</b></span>
|
||
<span>Domain <b>verdicttank.com</b></span>
|
||
<span>Status <b>Implementation-Ready</b></span>
|
||
<span>Origin <b>netcup RS 4000 (app3)</b></span>
|
||
<span>Judges <b>3 via LiteLLM</b></span>
|
||
<span>Verdict <b>Binary PASS / BELOW</b></span>
|
||
</div>
|
||
</div>
|
||
</header>
|
||
|
||
<nav class="toc"><div class="wrap"><ol>
|
||
<li><a href="#overview">1 · System Overview</a></li>
|
||
<li><a href="#pipeline">2 · Scoring Pipeline</a></li>
|
||
<li><a href="#fixit">3 · Fix-It Engine</a></li>
|
||
<li><a href="#dataflow">4 · Data Flow</a></li>
|
||
<li><a href="#api">5 · API Surface</a></li>
|
||
<li><a href="#datamodel">6 · Data Model</a></li>
|
||
<li><a href="#security">7 · Security</a></li>
|
||
<li><a href="#wlgate">8 · White-Label Gate</a></li>
|
||
<li><a href="#deploy">9 · Deployment</a></li>
|
||
<li><a href="#infra">10 · Infrastructure</a></li>
|
||
</ol></div></nav>
|
||
|
||
<main class="wrap">
|
||
<section id="overview">
|
||
<h2><span class="n">1</span>System Overview</h2>
|
||
<p class="lead">VerdictTank scores an uploaded proposal or RFP response against a rubric using a
|
||
panel of independent LLM judges, then aggregates their verdicts into a single, defensible
|
||
<b>PASS / BELOW</b> decision with per-criterion breakdowns and actionable fixes.</p>
|
||
|
||
<h3>The Consensus Problem</h3>
|
||
<p>A single LLM asked "is this proposal good?" produces a fragile answer: score drift between
|
||
runs, sycophancy toward confident prose, sensitivity to prompt phrasing, and no way to detect
|
||
when the model is simply wrong. VerdictTank treats scoring as a <b>consensus problem</b>, not a
|
||
single-inference problem. Three heterogeneous judges (different model families, different
|
||
training corpora) score the same artifact under an identical rubric. Divergence between judges is
|
||
a signal — it flags criteria where the artifact is genuinely ambiguous — and agreement is the
|
||
measure of confidence. The platform's core value is not any one model's opinion; it is the
|
||
<b>reproducible, auditable aggregation</b> of independent opinions.</p>
|
||
|
||
<div class="card">
|
||
<b>Design thesis.</b> A verdict is trustworthy when (a) it is produced by judges that do not
|
||
share failure modes, (b) the aggregation rule is deterministic and published, (c) every score
|
||
carries a rationale traceable to rubric criteria, and (d) the decision boundary is a single,
|
||
documented threshold rather than a black-box regressor.
|
||
</div>
|
||
|
||
<h3>Architecture Decisions</h3>
|
||
<table>
|
||
<tr><th>#</th><th>Decision</th><th>Rationale</th></tr>
|
||
<tr><td>AD-01</td><td><b>Three-judge heterogeneous panel</b> via LiteLLM (Anthropic, OpenAI, DeepSeek families).</td>
|
||
<td>Different corpora and RLHF regimes decorrelate errors. Odd count avoids ties in criterion-level majority checks.</td></tr>
|
||
<tr><td>AD-02</td><td><b>LiteLLM proxy as the single model gateway.</b> No direct provider SDK calls from app code.</td>
|
||
<td>Uniform auth, per-key budgets, request logging, and hot-swappable model routing without redeploying the app.</td></tr>
|
||
<tr><td>AD-03</td><td><b>Parallel judge dispatch</b> with per-judge timeout, not sequential.</td>
|
||
<td>Wall-clock latency is bounded by the slowest judge, not the sum. Enables a strict end-to-end latency budget.</td></tr>
|
||
<tr><td>AD-04</td><td><b>Binary verdict (PASS / BELOW)</b> against one published threshold, not a letter grade.</td>
|
||
<td>A single decision boundary is auditable and defensible to a buyer. Numeric sub-scores are retained for detail.</td></tr>
|
||
<tr><td>AD-05</td><td><b>6-stage fallback chain</b> for every judge slot.</td>
|
||
<td>Provider outages, rate limits, and malformed output must degrade gracefully to a still-valid verdict, never a hard failure.</td></tr>
|
||
<tr><td>AD-06</td><td><b>PII sanitization before any model call</b> (deterministic redaction pass).</td>
|
||
<td>No customer PII crosses a sub-processor boundary. Sanitization is idempotent and logged per session.</td></tr>
|
||
<tr><td>AD-07</td><td><b>Per-tenant corpus isolation</b> at the row and object-store prefix level.</td>
|
||
<td>One tenant's reference corpus can never leak into another's scoring context. Enforced in query layer and S3 prefix ACLs.</td></tr>
|
||
<tr><td>AD-08</td><td><b>Sub-processor training guard:</b> all provider calls set no-train / zero-retention flags where offered; DeepSeek routed through a no-retention endpoint.</td>
|
||
<td>Contractual and technical guarantee that customer content is never used to train third-party models.</td></tr>
|
||
<tr><td>AD-09</td><td><b>Review session as an explicit state machine</b> persisted in Postgres.</td>
|
||
<td>Every transition is durable and replayable. Crash recovery resumes from the last committed state, never re-charges a completed judge call.</td></tr>
|
||
<tr><td>AD-10</td><td><b>Fix-It rescore delta is measured, not promised.</b> Re-scoring after fixes produces a real before/after delta.</td>
|
||
<td>The product proves its own value: the customer sees the verdict actually move.</td></tr>
|
||
<tr><td>AD-11</td><td><b>White-Label access gated by a 5-condition ledger</b> returning HTTP 423 until all conditions clear.</td>
|
||
<td>Processor-role features cannot be enabled until legal, billing, and isolation prerequisites are verifiably satisfied.</td></tr>
|
||
<tr><td>AD-12</td><td><b>Stateless app tier, stateful data tier.</b> App containers hold no session state; Postgres + Redis + Wasabi S3 hold all durable state.</td>
|
||
<td>Horizontal scale and zero-downtime redeploys. Any app replica can serve any request.</td></tr>
|
||
</table>
|
||
|
||
<h3>Component Map</h3>
|
||
<pre class="ascii">
|
||
┌──────────────────────────────────────────────────────────────────────────┐
|
||
│ verdicttank.com (Caddy) │
|
||
│ TLS termination · HTTP/2 · automatic certs · reverse proxy · rate limit │
|
||
└───────────────┬───────────────────────────────────────────┬──────────────┘
|
||
│ │
|
||
┌─────────▼─────────┐ ┌─────────▼─────────┐
|
||
│ API (FastAPI) │ ◀── stateless ──▶ │ Worker (arq/RQ) │
|
||
│ auth · sessions │ │ judge dispatch │
|
||
│ webhooks · gate │ │ aggregate · fixit │
|
||
└───┬───────┬───────┘ └───┬───────────┬────┘
|
||
│ │ │ │
|
||
┌─────▼──┐ ┌──▼─────┐ ┌─────▼───┐ ┌────▼─────────┐
|
||
│Postgres│ │ Redis │ │ LiteLLM │ │ Wasabi S3 │
|
||
│ state │ │queue + │ │ gateway │ │ artifacts + │
|
||
│ + DDL │ │ cache │ │ 3 judges│ │ corpus (iso) │
|
||
└────────┘ └────────┘ └────┬────┘ └──────────────┘
|
||
│
|
||
┌────────────────────────────┼────────────────────────┐
|
||
▼ ▼ ▼
|
||
Anthropic family OpenAI family DeepSeek family
|
||
(judge slot A) (judge slot B) (judge slot C)
|
||
|
||
Observability: Prometheus scrapes API+Worker+LiteLLM /metrics → Grafana dashboards + alerts.
|
||
</pre>
|
||
|
||
<div class="note"><b>Tech stack.</b> FastAPI (Python 3.11) API tier · arq worker on Redis ·
|
||
PostgreSQL 16 (state + rubrics + verdicts) · Redis 7 (queue + rate-limit + short-cache) ·
|
||
LiteLLM proxy (model gateway) · Wasabi S3 (artifact + corpus object store) ·
|
||
Caddy 2 (TLS + reverse proxy) · Prometheus + Grafana (observability) ·
|
||
Docker Compose on netcup RS 4000.</div>
|
||
</section>
|
||
|
||
<section id="pipeline">
|
||
<h2><span class="n">2</span>Multimodel Scoring Pipeline</h2>
|
||
<p class="lead">Three independent judges score the same sanitized artifact in parallel through the
|
||
LiteLLM gateway. Per-criterion scores are aggregated deterministically into an overall score and a
|
||
binary verdict.</p>
|
||
|
||
<h3>Judge Roster (via LiteLLM)</h3>
|
||
<table>
|
||
<tr><th>Slot</th><th>Family</th><th>LiteLLM model alias</th><th>Role</th><th>Weight</th></tr>
|
||
<tr><td>A</td><td>Anthropic</td><td><code>judge-anthropic</code></td><td>Reasoning depth, requirement traceability</td><td>1.0</td></tr>
|
||
<tr><td>B</td><td>OpenAI</td><td><code>judge-openai</code></td><td>Structure, clarity, completeness</td><td>1.0</td></tr>
|
||
<tr><td>C</td><td>DeepSeek</td><td><code>judge-deepseek</code></td><td>Cost-anchor, adversarial skepticism</td><td>1.0</td></tr>
|
||
</table>
|
||
<p class="small">Aliases resolve to concrete provider models inside LiteLLM's <code>config.yaml</code>.
|
||
Swapping a judge's underlying model is a LiteLLM config change plus a reload — no application deploy.
|
||
Weights are configurable per rubric; default is equal (1.0) so all three judges count identically.</p>
|
||
|
||
<h3>Parallel Dispatch</h3>
|
||
<pre class="ascii">
|
||
sanitized_artifact + rubric
|
||
│
|
||
├──────────────┬──────────────┐ all three fire concurrently
|
||
▼ ▼ ▼ (asyncio.gather, per-judge timeout)
|
||
┌────────┐ ┌────────┐ ┌────────┐
|
||
│Judge A │ │Judge B │ │Judge C │ each: prompt → LiteLLM → JSON score
|
||
│ 8s TO │ │ 8s TO │ │ 8s TO │ on timeout/error → fallback chain
|
||
└───┬────┘ └───┬────┘ └───┬────┘
|
||
└─────────────┼─────────────┘
|
||
▼
|
||
┌──────────────┐
|
||
│ Aggregator │ weighted mean per criterion → overall → PASS/BELOW
|
||
└──────┬───────┘
|
||
▼
|
||
verdict record (persisted)
|
||
</pre>
|
||
<p>Dispatch uses <code>asyncio.gather(return_exceptions=True)</code>. A judge that raises or
|
||
times out does not abort the panel; its slot enters the fallback chain independently. The panel is
|
||
<b>valid</b> if at least 2 of 3 judges return a well-formed score (quorum = 2). With only 1 valid
|
||
judge the session transitions to <code>DEGRADED</code> and the verdict is flagged low-confidence.</p>
|
||
|
||
<h3>Scoring Schema (judge output contract)</h3>
|
||
<p>Every judge is instructed to return strict JSON matching this schema. The response is parsed and
|
||
validated; a schema violation triggers the fallback chain (stage 4, reformat).</p>
|
||
<pre>{
|
||
"$schema": "verdicttank/judge-score/v4",
|
||
"type": "object",
|
||
"required": ["judge","criteria","overall","rationale_summary"],
|
||
"properties": {
|
||
"judge": { "type": "string", "enum": ["judge-anthropic","judge-openai","judge-deepseek"] },
|
||
"criteria": {
|
||
"type": "array",
|
||
"items": {
|
||
"type": "object",
|
||
"required": ["id","score","weight","rationale"],
|
||
"properties": {
|
||
"id": { "type": "string" }, // rubric criterion id, e.g. "req_coverage"
|
||
"score": { "type": "number", "minimum": 0, "maximum": 100 },
|
||
"weight": { "type": "number", "minimum": 0, "maximum": 1 },
|
||
"rationale": { "type": "string", "maxLength": 600 },
|
||
"evidence": { "type": "array", "items": { "type": "string" } } // quoted spans from artifact
|
||
}
|
||
}
|
||
},
|
||
"overall": { "type": "number", "minimum": 0, "maximum": 100 },
|
||
"rationale_summary": { "type": "string", "maxLength": 1200 },
|
||
"flags": { "type": "array", "items": { "type": "string" } } // e.g. "missing_pricing"
|
||
}
|
||
}</pre>
|
||
|
||
<h3>Aggregation</h3>
|
||
<p>Aggregation is deterministic and published so a customer can reproduce the verdict by hand.</p>
|
||
<ol>
|
||
<li><b>Per-criterion aggregate.</b> For criterion <code>c</code>, compute the judge-weighted mean:
|
||
<code>agg_c = Σ(judge_weight_j × score_jc) / Σ(judge_weight_j)</code> over judges that returned a
|
||
valid score for <code>c</code>.</li>
|
||
<li><b>Criterion-weighted overall.</b> <code>OVERALL = Σ(rubric_weight_c × agg_c) / Σ(rubric_weight_c)</code>.</li>
|
||
<li><b>Agreement metric.</b> <code>spread_c = max_j(score_jc) − min_j(score_jc)</code>. Any criterion
|
||
with <code>spread_c ≥ 25</code> is tagged <code>contested</code> and surfaced in the report.</li>
|
||
<li><b>Confidence.</b> <code>confidence = clamp(1 − mean(spread_c)/100, 0, 1)</code>, degraded by
|
||
0.15 per missing judge below quorum.</li>
|
||
<li><b>Verdict.</b> <code>OVERALL ≥ threshold → PASS</code> else <code>BELOW</code>. Default
|
||
<code>threshold = 75</code>; configurable per rubric, stored on the verdict for auditability.</li>
|
||
</ol>
|
||
<div class="note"><b>Determinism guarantee.</b> Given the same three judge JSON outputs, the same
|
||
rubric weights, and the same threshold, the aggregator always produces the identical overall score
|
||
and verdict. Judge sampling temperature is pinned to <code>0.2</code> and <code>seed</code> is set
|
||
where the provider supports it; raw judge outputs are persisted so a verdict is fully replayable.</div>
|
||
|
||
<h3>6-Stage Fallback Chain (per judge slot)</h3>
|
||
<p>Each judge slot independently walks this chain. The chain guarantees a slot either yields a valid
|
||
score or is cleanly marked <code>UNAVAILABLE</code> (counting against quorum) — it never hangs.</p>
|
||
<table>
|
||
<tr><th>Stage</th><th>Trigger</th><th>Action</th><th>Budget</th></tr>
|
||
<tr><td>1 · Primary</td><td>—</td><td>Call the slot's configured model through LiteLLM.</td><td>8s</td></tr>
|
||
<tr><td>2 · Retry</td><td>5xx / network / timeout</td><td>Single retry with 500ms jitter backoff, same model.</td><td>+8s</td></tr>
|
||
<tr><td>3 · Sibling model</td><td>429 rate-limit or repeated 5xx</td><td>LiteLLM routes to the same family's alternate deployment (fallback map in config.yaml).</td><td>+8s</td></tr>
|
||
<tr><td>4 · Reformat</td><td>200 but invalid/non-JSON body</td><td>Re-prompt the model with the exact schema and the offending output, demanding valid JSON only.</td><td>+6s</td></tr>
|
||
<tr><td>5 · Cross-family substitute</td><td>Family fully unavailable</td><td>Borrow a spare deployment from another family, tagged <code>substituted</code> in the verdict (weight halved).</td><td>+8s</td></tr>
|
||
<tr><td>6 · Mark unavailable</td><td>All above exhausted</td><td>Slot → <code>UNAVAILABLE</code>. Panel proceeds on remaining judges if quorum (≥2) holds; else session → <code>DEGRADED</code>.</td><td>0s</td></tr>
|
||
</table>
|
||
|
||
<h3>Latency Budgets</h3>
|
||
<table>
|
||
<tr><th>Path</th><th>Target (p50)</th><th>Ceiling (p95)</th><th>Notes</th></tr>
|
||
<tr><td>Ingest + sanitize</td><td>400 ms</td><td>1.2 s</td><td>Deterministic redaction; scales with artifact size.</td></tr>
|
||
<tr><td>Parallel judge panel (happy path)</td><td>4.5 s</td><td>8 s</td><td>Bounded by slowest judge, not the sum. Per-judge timeout 8s.</td></tr>
|
||
<tr><td>Judge panel with one fallback hop</td><td>—</td><td>16 s</td><td>Stage 2/3 adds one budget window to the affected slot only.</td></tr>
|
||
<tr><td>Aggregation + persist</td><td>60 ms</td><td>200 ms</td><td>Pure CPU + one DB write.</td></tr>
|
||
<tr><td><b>End-to-end (submit → verdict)</b></td><td><b>~5 s</b></td><td><b>18 s</b></td><td>Hard wall-clock cap at 25 s → session marked <code>TIMEOUT</code>, partial verdict if quorum met.</td></tr>
|
||
<tr><td>Fix-It rescore (single criterion)</td><td>3 s</td><td>9 s</td><td>Re-runs only affected criteria across the panel when possible.</td></tr>
|
||
</table>
|
||
|
||
<h4>Error handling — pipeline</h4>
|
||
<ul>
|
||
<li><b>All judges fail (quorum lost):</b> session → <code>FAILED</code>; API returns <code>503</code>
|
||
with <code>Retry-After</code>; no charge recorded against the session's review credit.</li>
|
||
<li><b>Partial panel (2 of 3):</b> verdict produced, <code>confidence</code> reduced, response
|
||
includes <code>"degraded": true</code> and the unavailable slot id.</li>
|
||
<li><b>Malformed after reformat (stage 4 fails):</b> slot treated as <code>UNAVAILABLE</code>; raw
|
||
body stored for post-mortem in <code>judge_raw_output</code>.</li>
|
||
<li><b>Wall-clock cap hit:</b> in-flight judge calls are cancelled; if quorum already met the
|
||
partial verdict is committed as <code>TIMEOUT_PARTIAL</code>, else <code>FAILED</code>.</li>
|
||
</ul>
|
||
</section>
|
||
|
||
<section id="fixit">
|
||
<h2><span class="n">3</span>Fix-It Engine</h2>
|
||
<p class="lead">After a verdict, the Fix-It Engine converts each weak criterion into a ranked,
|
||
actionable fix, then re-scores the revised artifact to produce a measured before/after delta.</p>
|
||
|
||
<h3>Failure Modes (what Fix-It detects)</h3>
|
||
<table>
|
||
<tr><th>Mode</th><th>Detection signal</th><th>Example fix</th></tr>
|
||
<tr><td><code>MISSING_REQUIREMENT</code></td><td>Criterion <code>req_coverage</code> score < 60 or judge <code>flags</code> contains a requirement id.</td><td>"RFP §4.2 asks for SOC 2 evidence — no security section addresses it. Add a compliance subsection citing your SOC 2 report."</td></tr>
|
||
<tr><td><code>WEAK_EVIDENCE</code></td><td>High-weight criterion scored mid-range with judge rationale citing "unsupported" / "no data".</td><td>"The ROI claim has no figures. Add a quantified savings table."</td></tr>
|
||
<tr><td><code>STRUCTURAL</code></td><td><code>structure</code> criterion low; ordering/section flags raised.</td><td>"Executive summary appears after pricing. Move it to the front."</td></tr>
|
||
<tr><td><code>CLARITY</code></td><td><code>clarity</code> low; long-sentence / jargon flags.</td><td>"Section 3 averages 40-word sentences. Break into shorter statements."</td></tr>
|
||
<tr><td><code>CONTESTED</code></td><td><code>spread_c ≥ 25</code> (judges disagree).</td><td>"Judges split on differentiation. Make the unique-value claim explicit and defensible."</td></tr>
|
||
<tr><td><code>COMPLIANCE_GAP</code></td><td>Mandatory rubric criterion below its hard floor.</td><td>"Insurance certificate is required and absent. This alone forces a BELOW verdict."</td></tr>
|
||
</table>
|
||
|
||
<h3>Prioritization Formula</h3>
|
||
<p>Fixes are ranked by expected verdict impact per unit of effort, so the customer fixes the things
|
||
that move the needle first.</p>
|
||
<div class="card mono">
|
||
priority_c = ( rubric_weight_c × gap_c × contested_bonus_c ) / effort_c<br><br>
|
||
where<br>
|
||
gap_c = max(0, target_score − agg_c) // distance to a passing sub-score, target default 80<br>
|
||
contested_bonus_c = 1 + (spread_c ≥ 25 ? 0.5 : 0) // disagreement is high-value to resolve<br>
|
||
effort_c = { STRUCTURAL:1, CLARITY:1, WEAK_EVIDENCE:2, MISSING_REQUIREMENT:3, COMPLIANCE_GAP:3 }
|
||
</div>
|
||
<p>Fixes are returned sorted by <code>priority_c</code> descending. Each carries an
|
||
<code>estimated_lift</code> = <code>rubric_weight_c × gap_c</code> — the maximum points the overall
|
||
score can gain if that criterion reaches <code>target_score</code>. The UI shows a running
|
||
"projected verdict if all applied" sum, clamped so <code>OVERALL ≤ 100</code>.</p>
|
||
|
||
<h3>Rescore Delta</h3>
|
||
<p>When the customer submits a revised artifact (or accepts inline suggestions), the engine re-runs
|
||
the panel and reports a real, measured delta — never a promise.</p>
|
||
<pre>{
|
||
"$schema": "verdicttank/rescore-delta/v4",
|
||
"session_id": "rs_9f3a...",
|
||
"before": { "overall": 68.4, "verdict": "BELOW", "confidence": 0.71 },
|
||
"after": { "overall": 81.2, "verdict": "PASS", "confidence": 0.83 },
|
||
"delta": { "overall": 12.8, "verdict_changed": true },
|
||
"per_criterion": [
|
||
{ "id": "req_coverage", "before": 55.0, "after": 84.0, "delta": 29.0, "fix_applied": "fix_01" },
|
||
{ "id": "clarity", "before": 72.0, "after": 78.0, "delta": 6.0, "fix_applied": "fix_03" }
|
||
],
|
||
"rescored_at": "2026-08-11T14:22:08Z"
|
||
}</pre>
|
||
<p class="small">Optimization: if only a subset of criteria changed (detected by artifact diff), the
|
||
engine re-scores just those criteria across all three judges and reuses unchanged criterion scores —
|
||
cutting rescore latency and cost. Full re-scores are forced when the artifact diff exceeds 30% of tokens.</p>
|
||
|
||
<h3>Single-Tier Binary Verdict — PASS / BELOW</h3>
|
||
<p>VerdictTank issues exactly one of two verdicts. There is no letter grade, no star rating, and no
|
||
intermediate "maybe" tier. This is deliberate (AD-04): a buyer needs a defensible go/no-go.</p>
|
||
<div class="grid2">
|
||
<div class="card"><span class="pill-pass">PASS</span>
|
||
<p style="margin-top:10px"><code>OVERALL ≥ threshold</code> AND every mandatory criterion is at
|
||
or above its hard floor. The artifact meets the bar defined by the rubric.</p></div>
|
||
<div class="card"><span class="pill-below">BELOW</span>
|
||
<p style="margin-top:10px"><code>OVERALL < threshold</code> OR any mandatory criterion is under
|
||
its hard floor (a single compliance gap forces BELOW regardless of overall score).</p></div>
|
||
</div>
|
||
<p class="small">The numeric <code>OVERALL</code>, per-criterion sub-scores, confidence, and contested
|
||
flags are always returned alongside the binary verdict for transparency — the binary is the decision,
|
||
the numbers are the justification.</p>
|
||
|
||
<h4>Error handling — Fix-It</h4>
|
||
<ul>
|
||
<li><b>No weak criteria (already PASS):</b> engine returns an empty fix list with
|
||
<code>"status":"no_action_needed"</code>; UI shows the passing report only.</li>
|
||
<li><b>Rescore panel loses quorum:</b> the <code>before</code> verdict is preserved unchanged;
|
||
response carries <code>"rescore_status":"degraded"</code> and does not overwrite the stored verdict.</li>
|
||
<li><b>Artifact unchanged from prior submission (hash match):</b> rescore is skipped;
|
||
returns <code>409 Conflict</code> with the existing verdict to avoid a redundant paid run.</li>
|
||
<li><b>Compliance floor still breached after fixes:</b> verdict stays <code>BELOW</code> even if
|
||
<code>OVERALL</code> crosses threshold; the blocking criterion is named in <code>blocking_criteria</code>.</li>
|
||
</ul>
|
||
</section>
|
||
|
||
<section id="dataflow">
|
||
<h2><span class="n">4</span>Data Flow</h2>
|
||
<p class="lead">A review moves through six durable stages. Each stage has an input contract, an output
|
||
contract, and a persisted state transition. The pipeline is crash-safe: recovery resumes at the last
|
||
committed stage.</p>
|
||
|
||
<h3>6-Stage Pipeline</h3>
|
||
<pre class="ascii">
|
||
①INGEST → ②SANITIZE → ③DISPATCH → ④AGGREGATE → ⑤FIXIT → ⑥PERSIST/DELIVER
|
||
│ │ │ │ │ │
|
||
upload redact PII 3 judges weighted ranked fixes verdict +
|
||
+ parse (idempotent) parallel mean → + rescore webhook +
|
||
artifact (LiteLLM) PASS/BELOW delta report URL
|
||
</pre>
|
||
|
||
<h4>① INGEST — input contract</h4>
|
||
<pre>{ "$schema":"verdicttank/ingest/v4",
|
||
"tenant_id":"t_8821", "rubric_id":"rub_default_v4",
|
||
"artifact":{ "filename":"proposal.pdf", "mime":"application/pdf",
|
||
"bytes_b64":"JVBERi0x...", "sha256":"a91f..." },
|
||
"options":{ "threshold":75, "webhook_url":"https://buyer.example/hooks/vt" } }</pre>
|
||
<p class="small">Parser extracts plain text (PDF/DOCX/MD/TXT). Artifact stored to Wasabi at
|
||
<code>s3://vt-artifacts/{tenant_id}/{session_id}/original</code>. Output: <code>artifact_text</code>
|
||
+ <code>token_count</code>.</p>
|
||
|
||
<h4>② SANITIZE — output contract</h4>
|
||
<pre>{ "$schema":"verdicttank/sanitized/v4",
|
||
"session_id":"rs_9f3a", "artifact_text":"...[REDACTED:EMAIL]... [REDACTED:PHONE]...",
|
||
"redactions":[ {"type":"EMAIL","count":3}, {"type":"PHONE","count":1}, {"type":"SSN","count":0} ],
|
||
"sanitize_hash":"7c2e..." }</pre>
|
||
<p class="small">Deterministic regex + NER redaction. Idempotent: re-running yields the identical
|
||
<code>sanitize_hash</code>. Only the sanitized text is ever sent to a judge (AD-06).</p>
|
||
|
||
<h4>③ DISPATCH — output contract (per judge, see §2 scoring schema)</h4>
|
||
<pre>{ "session_id":"rs_9f3a", "judge_results":[
|
||
{"judge":"judge-anthropic","status":"ok","overall":66,"criteria":[...]},
|
||
{"judge":"judge-openai","status":"ok","overall":71,"criteria":[...]},
|
||
{"judge":"judge-deepseek","status":"substituted","overall":68,"criteria":[...]} ],
|
||
"quorum_met":true, "degraded":false }</pre>
|
||
|
||
<h4>④ AGGREGATE — output contract</h4>
|
||
<pre>{ "$schema":"verdicttank/verdict/v4",
|
||
"session_id":"rs_9f3a", "overall":68.4, "verdict":"BELOW", "threshold":75,
|
||
"confidence":0.71,
|
||
"criteria":[ {"id":"req_coverage","agg":55.0,"spread":18,"contested":false},
|
||
{"id":"clarity","agg":72.0,"spread":9,"contested":false},
|
||
{"id":"differentiation","agg":60.0,"spread":31,"contested":true} ],
|
||
"blocking_criteria":[], "computed_at":"2026-08-11T14:20:55Z" }</pre>
|
||
|
||
<h4>⑤ FIXIT — output contract (see §3)</h4>
|
||
<pre>{ "session_id":"rs_9f3a", "status":"action_needed",
|
||
"fixes":[ {"id":"fix_01","criterion":"req_coverage","mode":"MISSING_REQUIREMENT",
|
||
"priority":24.8,"estimated_lift":9.0,"text":"Add SOC 2 evidence section..."} ],
|
||
"projected_overall":81.2 }</pre>
|
||
|
||
<h4>⑥ PERSIST / DELIVER — output contract</h4>
|
||
<pre>{ "session_id":"rs_9f3a", "state":"COMPLETED",
|
||
"report_url":"https://verdicttank.com/r/rs_9f3a",
|
||
"webhook_delivered":true, "verdict":"BELOW" }</pre>
|
||
|
||
<h3>Review Session State Machine</h3>
|
||
<pre class="ascii">
|
||
┌───────────────────────────────────────────────┐
|
||
│ │
|
||
(POST /reviews) ▼ │
|
||
─────────────▶ [ RECEIVED ] │
|
||
│ ingest+parse ok │
|
||
▼ │
|
||
[ SANITIZED ] │
|
||
│ redaction committed │
|
||
▼ │
|
||
[ DISPATCHING ] ──quorum lost──▶ [ FAILED ] ◀──wall-clock cap (no quorum)
|
||
│ ≥2 judges ok │
|
||
▼ │ (terminal, 503 to caller)
|
||
[ AGGREGATED ] ──1 judge only──▶ [ DEGRADED ]
|
||
│ verdict computed │ (low-confidence verdict stored)
|
||
▼ ▼
|
||
[ SCORED ] ◀──────────────────────────┘
|
||
│ │
|
||
fixes needed│ │already PASS / no action
|
||
▼ ▼
|
||
[ FIXIT_READY ] [ COMPLETED ]───(webhook + report)───▶ (terminal)
|
||
│ rescore submitted ▲
|
||
▼ │ verdict updated
|
||
[ RESCORING ] ──quorum lost──▶ keeps prior verdict, back to [ SCORED ]
|
||
│ new verdict computed │
|
||
└────────────────────────────────────┘
|
||
|
||
Timeout guard: any non-terminal state exceeding 25s wall-clock → [ TIMEOUT_PARTIAL ]
|
||
(if quorum already met, commit partial verdict → SCORED) else → [ FAILED ].
|
||
Crash recovery: worker restart reloads last committed state; completed judge calls
|
||
are cached by (session_id, judge, sanitize_hash) and never re-billed.
|
||
</pre>
|
||
|
||
<table>
|
||
<tr><th>State</th><th>Meaning</th><th>Terminal?</th></tr>
|
||
<tr><td><code>RECEIVED</code></td><td>Upload accepted, parse pending.</td><td>no</td></tr>
|
||
<tr><td><code>SANITIZED</code></td><td>PII redacted, ready for judges.</td><td>no</td></tr>
|
||
<tr><td><code>DISPATCHING</code></td><td>Judge panel in flight.</td><td>no</td></tr>
|
||
<tr><td><code>AGGREGATED</code></td><td>Scores combined, verdict pending write.</td><td>no</td></tr>
|
||
<tr><td><code>SCORED</code></td><td>Verdict committed, report available.</td><td>no</td></tr>
|
||
<tr><td><code>FIXIT_READY</code></td><td>Fixes generated, awaiting revised artifact.</td><td>no</td></tr>
|
||
<tr><td><code>RESCORING</code></td><td>Revised artifact re-scoring.</td><td>no</td></tr>
|
||
<tr><td><code>DEGRADED</code></td><td>Verdict from <quorum judges; low confidence.</td><td>no</td></tr>
|
||
<tr><td><code>COMPLETED</code></td><td>Delivered (webhook + report).</td><td>yes</td></tr>
|
||
<tr><td><code>TIMEOUT_PARTIAL</code></td><td>Wall-clock cap hit; partial verdict committed.</td><td>yes*</td></tr>
|
||
<tr><td><code>FAILED</code></td><td>Quorum never met; no charge.</td><td>yes</td></tr>
|
||
</table>
|
||
<p class="small">*<code>TIMEOUT_PARTIAL</code> is terminal for the original run but a new rescore may
|
||
reopen the session into <code>RESCORING</code>.</p>
|
||
</section>
|
||
|
||
<section id="api">
|
||
<h2><span class="n">5</span>API Surface</h2>
|
||
<p class="lead">REST/JSON over HTTPS. Base URL <code>https://api.verdicttank.com/v4</code>. All bodies
|
||
UTF-8 JSON. All timestamps RFC 3339 UTC. All IDs are prefixed opaque strings.</p>
|
||
|
||
<h3>Authentication</h3>
|
||
<ul>
|
||
<li><b>API keys</b> — <code>Authorization: Bearer vt_live_XXXX</code> for server-to-server. Keys are
|
||
tenant-scoped, hashed at rest (argon2id), and carry a role (<code>viewer</code>,
|
||
<code>reviewer</code>, <code>admin</code>). Test keys prefixed <code>vt_test_</code>.</li>
|
||
<li><b>Session JWT</b> — short-lived (15 min) bearer for browser clients, issued by
|
||
<code>POST /auth/token</code>, refreshed via <code>POST /auth/refresh</code>. Contains
|
||
<code>tenant_id</code>, <code>role</code>, <code>exp</code>. Signed HS256 with a rotated secret.</li>
|
||
<li><b>Webhook signature</b> — outbound webhooks are signed <code>X-VT-Signature: sha256=HMAC(body)</code>
|
||
with the tenant's webhook secret.</li>
|
||
</ul>
|
||
|
||
<h3>Rate Limiting</h3>
|
||
<p>Token-bucket per API key, enforced in Redis. Limits returned on every response:</p>
|
||
<pre>X-RateLimit-Limit: 60
|
||
X-RateLimit-Remaining: 57
|
||
X-RateLimit-Reset: 1754923200
|
||
Retry-After: 12 # only on 429</pre>
|
||
<table>
|
||
<tr><th>Plan</th><th>Reviews/min</th><th>Reviews/day</th><th>Burst</th></tr>
|
||
<tr><td>Free</td><td>2</td><td>10</td><td>3</td></tr>
|
||
<tr><td>Pro ($79/mo)</td><td>20</td><td>500 (fair-use)</td><td>30</td></tr>
|
||
<tr><td>Enterprise ($299/mo)</td><td>60</td><td>negotiated</td><td>100</td></tr>
|
||
</table>
|
||
|
||
<h3>Endpoints</h3>
|
||
<ul class="eplist">
|
||
<li><span class="tag t-post">POST</span><code>/auth/token</code> — exchange API key or credentials for a session JWT.</li>
|
||
<li><span class="tag t-post">POST</span><code>/reviews</code> — create a review (ingest artifact, start pipeline).</li>
|
||
<li><span class="tag t-get">GET</span><code>/reviews/{id}</code> — fetch a review session + verdict.</li>
|
||
<li><span class="tag t-get">GET</span><code>/reviews/{id}/verdict</code> — verdict + per-criterion detail only.</li>
|
||
<li><span class="tag t-get">GET</span><code>/reviews/{id}/fixes</code> — Fix-It ranked fix list.</li>
|
||
<li><span class="tag t-post">POST</span><code>/reviews/{id}/rescore</code> — submit revised artifact, get before/after delta.</li>
|
||
<li><span class="tag t-get">GET</span><code>/reviews</code> — list tenant reviews (paginated, filterable by verdict/state).</li>
|
||
<li><span class="tag t-get">GET</span><code>/rubrics</code> · <span class="tag t-post">POST</span><code>/rubrics</code> — manage scoring rubrics.</li>
|
||
<li><span class="tag t-get">GET</span><code>/webhooks</code> · <span class="tag t-post">POST</span><code>/webhooks</code> · <span class="tag t-del">DELETE</span><code>/webhooks/{id}</code> — manage webhook endpoints.</li>
|
||
<li><span class="tag t-get">GET</span><code>/whitelabel/status</code> — 5-condition gate ledger (see §8).</li>
|
||
<li><span class="tag t-get">GET</span><code>/healthz</code> · <span class="tag t-get">GET</span><code>/metrics</code> — liveness + Prometheus.</li>
|
||
</ul>
|
||
|
||
<h4>POST /reviews — request</h4>
|
||
<pre>POST /v4/reviews
|
||
Authorization: Bearer vt_live_XXXX
|
||
Content-Type: application/json
|
||
|
||
{ "rubric_id":"rub_default_v4",
|
||
"artifact":{ "filename":"proposal.pdf","mime":"application/pdf","bytes_b64":"JVBERi0x..." },
|
||
"options":{ "threshold":75, "webhook_url":"https://buyer.example/hooks/vt", "async":true } }</pre>
|
||
<h4>POST /reviews — response (202 async)</h4>
|
||
<pre>HTTP/1.1 202 Accepted
|
||
Location: /v4/reviews/rs_9f3a
|
||
{ "session_id":"rs_9f3a", "state":"RECEIVED", "poll_url":"/v4/reviews/rs_9f3a",
|
||
"estimated_ready_s":6 }</pre>
|
||
<p class="small">With <code>"async":false</code> the call blocks up to the 25 s wall-clock cap and
|
||
returns <code>200</code> with the full verdict inline.</p>
|
||
|
||
<h4>GET /reviews/{id} — response (200)</h4>
|
||
<pre>{ "session_id":"rs_9f3a", "state":"SCORED", "tenant_id":"t_8821",
|
||
"verdict":"BELOW", "overall":68.4, "threshold":75, "confidence":0.71, "degraded":false,
|
||
"criteria":[ {"id":"req_coverage","agg":55.0,"spread":18,"contested":false,
|
||
"rationales":{"judge-anthropic":"...","judge-openai":"...","judge-deepseek":"..."}} ],
|
||
"blocking_criteria":[], "report_url":"https://verdicttank.com/r/rs_9f3a",
|
||
"created_at":"2026-08-11T14:20:49Z", "computed_at":"2026-08-11T14:20:55Z" }</pre>
|
||
|
||
<h4>POST /reviews/{id}/rescore — request/response</h4>
|
||
<pre>POST /v4/reviews/rs_9f3a/rescore
|
||
{ "artifact":{ "filename":"proposal_v2.pdf","mime":"application/pdf","bytes_b64":"..." } }
|
||
|
||
→ 200 { ...rescore-delta/v4 object (see §3)... }
|
||
→ 409 { "error":"unchanged_artifact","message":"sha256 matches prior submission" }</pre>
|
||
|
||
<h3>Webhooks</h3>
|
||
<p>Registered endpoints receive signed POSTs on state transitions. Events: <code>review.scored</code>,
|
||
<code>review.completed</code>, <code>review.failed</code>, <code>review.rescored</code>.</p>
|
||
<pre>POST {webhook_url}
|
||
X-VT-Signature: sha256=9a3c...
|
||
X-VT-Event: review.scored
|
||
X-VT-Delivery: whd_44a1
|
||
|
||
{ "event":"review.scored", "session_id":"rs_9f3a", "verdict":"BELOW",
|
||
"overall":68.4, "occurred_at":"2026-08-11T14:20:55Z" }</pre>
|
||
<p class="small">Delivery: at-least-once with exponential backoff (max 6 attempts over ~1h). Consumers
|
||
must be idempotent on <code>X-VT-Delivery</code>. Verify by recomputing
|
||
<code>HMAC-SHA256(secret, raw_body)</code> and constant-time comparing to <code>X-VT-Signature</code>.</p>
|
||
|
||
<h3>Error Model</h3>
|
||
<p>All errors share one envelope. HTTP status conveys the class; <code>error</code> conveys the code.</p>
|
||
<pre>{ "error":"quorum_lost",
|
||
"message":"Only 1 of 3 judges returned a valid score.",
|
||
"session_id":"rs_9f3a", "request_id":"req_7c2e", "retryable":true }</pre>
|
||
<table>
|
||
<tr><th>HTTP</th><th>error code</th><th>Meaning</th><th>Retryable</th></tr>
|
||
<tr><td>400</td><td><code>invalid_request</code></td><td>Malformed body / unsupported mime / missing field.</td><td>no</td></tr>
|
||
<tr><td>401</td><td><code>unauthenticated</code></td><td>Missing/invalid API key or expired JWT.</td><td>no</td></tr>
|
||
<tr><td>403</td><td><code>forbidden</code></td><td>Role lacks permission (e.g. viewer creating a review).</td><td>no</td></tr>
|
||
<tr><td>404</td><td><code>not_found</code></td><td>Unknown session/rubric/webhook id (or cross-tenant access).</td><td>no</td></tr>
|
||
<tr><td>409</td><td><code>unchanged_artifact</code></td><td>Rescore artifact identical to prior (hash match).</td><td>no</td></tr>
|
||
<tr><td>413</td><td><code>artifact_too_large</code></td><td>Artifact exceeds size/token cap.</td><td>no</td></tr>
|
||
<tr><td>422</td><td><code>unprocessable_artifact</code></td><td>Parse failed (corrupt PDF, empty text).</td><td>no</td></tr>
|
||
<tr><td>423</td><td><code>whitelabel_locked</code></td><td>White-Label gate conditions unmet (see §8).</td><td>no</td></tr>
|
||
<tr><td>429</td><td><code>rate_limited</code></td><td>Bucket exhausted; see <code>Retry-After</code>.</td><td>yes</td></tr>
|
||
<tr><td>503</td><td><code>quorum_lost</code></td><td>Judge panel failed to reach quorum; no charge recorded.</td><td>yes</td></tr>
|
||
<tr><td>504</td><td><code>pipeline_timeout</code></td><td>Wall-clock cap hit with no partial verdict.</td><td>yes</td></tr>
|
||
</table>
|
||
</section>
|
||
|
||
<section id="datamodel">
|
||
<h2><span class="n">6</span>Data Model</h2>
|
||
<p class="lead">PostgreSQL 16. Multi-tenant with <code>tenant_id</code> on every business table and
|
||
row-level security enforced by the query layer. All money is integer cents; all scores are numeric(5,2).</p>
|
||
|
||
<h3>Entity Relationships</h3>
|
||
<pre class="ascii">
|
||
tenants ──1:N── api_keys
|
||
│
|
||
├──1:N── rubrics ──1:N── rubric_criteria
|
||
│
|
||
├──1:N── review_sessions ──1:N── judge_results ──1:N── judge_criterion_scores
|
||
│ │
|
||
│ ├──1:1── verdicts
|
||
│ └──1:N── fixes
|
||
│
|
||
├──1:N── webhooks
|
||
└──1:1── whitelabel_ledger
|
||
</pre>
|
||
|
||
<h3>DDL</h3>
|
||
<pre>CREATE TABLE tenants (
|
||
tenant_id TEXT PRIMARY KEY, -- 't_8821'
|
||
name TEXT NOT NULL,
|
||
plan TEXT NOT NULL DEFAULT 'free'
|
||
CHECK (plan IN ('free','pro','enterprise','whitelabel')),
|
||
data_role TEXT NOT NULL DEFAULT 'controller'
|
||
CHECK (data_role IN ('controller','joint_controller','processor')),
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||
);
|
||
|
||
CREATE TABLE api_keys (
|
||
key_id TEXT PRIMARY KEY, -- 'ak_...'
|
||
tenant_id TEXT NOT NULL REFERENCES tenants ON DELETE CASCADE,
|
||
key_hash TEXT NOT NULL, -- argon2id(secret)
|
||
role TEXT NOT NULL CHECK (role IN ('viewer','reviewer','admin')),
|
||
mode TEXT NOT NULL CHECK (mode IN ('live','test')),
|
||
last_used_at TIMESTAMPTZ,
|
||
revoked_at TIMESTAMPTZ,
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||
);
|
||
CREATE INDEX ix_api_keys_tenant ON api_keys(tenant_id);
|
||
|
||
CREATE TABLE rubrics (
|
||
rubric_id TEXT PRIMARY KEY, -- 'rub_default_v4'
|
||
tenant_id TEXT REFERENCES tenants ON DELETE CASCADE, -- NULL = system rubric
|
||
name TEXT NOT NULL,
|
||
threshold NUMERIC(5,2) NOT NULL DEFAULT 75.00,
|
||
version TEXT NOT NULL DEFAULT 'v4',
|
||
is_active BOOLEAN NOT NULL DEFAULT true,
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||
);
|
||
|
||
CREATE TABLE rubric_criteria (
|
||
criterion_id TEXT PRIMARY KEY, -- 'req_coverage'
|
||
rubric_id TEXT NOT NULL REFERENCES rubrics ON DELETE CASCADE,
|
||
label TEXT NOT NULL,
|
||
weight NUMERIC(4,3) NOT NULL CHECK (weight BETWEEN 0 AND 1),
|
||
is_mandatory BOOLEAN NOT NULL DEFAULT false,
|
||
hard_floor NUMERIC(5,2) NOT NULL DEFAULT 0.00, -- mandatory floor; below = forces BELOW
|
||
sort_order INT NOT NULL DEFAULT 0
|
||
);
|
||
CREATE INDEX ix_criteria_rubric ON rubric_criteria(rubric_id);
|
||
|
||
CREATE TABLE review_sessions (
|
||
session_id TEXT PRIMARY KEY, -- 'rs_9f3a'
|
||
tenant_id TEXT NOT NULL REFERENCES tenants ON DELETE CASCADE,
|
||
rubric_id TEXT NOT NULL REFERENCES rubrics,
|
||
state TEXT NOT NULL DEFAULT 'RECEIVED'
|
||
CHECK (state IN ('RECEIVED','SANITIZED','DISPATCHING','AGGREGATED',
|
||
'SCORED','FIXIT_READY','RESCORING','DEGRADED',
|
||
'COMPLETED','TIMEOUT_PARTIAL','FAILED')),
|
||
artifact_sha TEXT NOT NULL, -- sha256 of original artifact
|
||
sanitize_hash TEXT, -- idempotency key for judge cache
|
||
s3_key TEXT NOT NULL, -- vt-artifacts/{tenant}/{session}/original
|
||
threshold NUMERIC(5,2) NOT NULL,
|
||
webhook_url TEXT,
|
||
charged BOOLEAN NOT NULL DEFAULT false, -- billing guard (FAILED never charged)
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||
);
|
||
CREATE INDEX ix_sessions_tenant_state ON review_sessions(tenant_id, state);
|
||
|
||
CREATE TABLE judge_results (
|
||
id BIGSERIAL PRIMARY KEY,
|
||
session_id TEXT NOT NULL REFERENCES review_sessions ON DELETE CASCADE,
|
||
judge TEXT NOT NULL, -- 'judge-anthropic' | ...
|
||
status TEXT NOT NULL CHECK (status IN ('ok','substituted','unavailable')),
|
||
overall NUMERIC(5,2),
|
||
fallback_stage SMALLINT NOT NULL DEFAULT 1, -- which of the 6 stages produced this
|
||
latency_ms INT,
|
||
raw_output JSONB, -- persisted for replay/post-mortem
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||
UNIQUE (session_id, judge) -- idempotent per (session,judge)
|
||
);
|
||
|
||
CREATE TABLE judge_criterion_scores (
|
||
id BIGSERIAL PRIMARY KEY,
|
||
judge_result_id BIGINT NOT NULL REFERENCES judge_results ON DELETE CASCADE,
|
||
criterion_id TEXT NOT NULL,
|
||
score NUMERIC(5,2) NOT NULL CHECK (score BETWEEN 0 AND 100),
|
||
rationale TEXT,
|
||
evidence JSONB
|
||
);
|
||
CREATE INDEX ix_jcs_result ON judge_criterion_scores(judge_result_id);
|
||
|
||
CREATE TABLE verdicts (
|
||
session_id TEXT PRIMARY KEY REFERENCES review_sessions ON DELETE CASCADE,
|
||
verdict TEXT NOT NULL CHECK (verdict IN ('PASS','BELOW')),
|
||
overall NUMERIC(5,2) NOT NULL,
|
||
threshold NUMERIC(5,2) NOT NULL,
|
||
confidence NUMERIC(4,3) NOT NULL,
|
||
degraded BOOLEAN NOT NULL DEFAULT false,
|
||
criteria_agg JSONB NOT NULL, -- [{id,agg,spread,contested}]
|
||
blocking_criteria JSONB NOT NULL DEFAULT '[]',
|
||
computed_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||
);
|
||
|
||
CREATE TABLE fixes (
|
||
fix_id TEXT PRIMARY KEY, -- 'fix_01'
|
||
session_id TEXT NOT NULL REFERENCES review_sessions ON DELETE CASCADE,
|
||
criterion_id TEXT NOT NULL,
|
||
mode TEXT NOT NULL, -- MISSING_REQUIREMENT | WEAK_EVIDENCE | ...
|
||
priority NUMERIC(7,3) NOT NULL,
|
||
estimated_lift NUMERIC(5,2) NOT NULL,
|
||
body TEXT NOT NULL,
|
||
applied BOOLEAN NOT NULL DEFAULT false
|
||
);
|
||
CREATE INDEX ix_fixes_session ON fixes(session_id);
|
||
|
||
CREATE TABLE webhooks (
|
||
webhook_id TEXT PRIMARY KEY,
|
||
tenant_id TEXT NOT NULL REFERENCES tenants ON DELETE CASCADE,
|
||
url TEXT NOT NULL,
|
||
secret_hash TEXT NOT NULL,
|
||
events TEXT[] NOT NULL DEFAULT '{review.scored,review.completed}',
|
||
active BOOLEAN NOT NULL DEFAULT true,
|
||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||
);
|
||
|
||
CREATE TABLE whitelabel_ledger (
|
||
tenant_id TEXT PRIMARY KEY REFERENCES tenants ON DELETE CASCADE,
|
||
dpa_signed BOOLEAN NOT NULL DEFAULT false,
|
||
billing_active BOOLEAN NOT NULL DEFAULT false,
|
||
corpus_isolated BOOLEAN NOT NULL DEFAULT false,
|
||
branding_approved BOOLEAN NOT NULL DEFAULT false,
|
||
subprocessor_ack BOOLEAN NOT NULL DEFAULT false,
|
||
unlocked_at TIMESTAMPTZ,
|
||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||
);</pre>
|
||
<p class="small">Row-level security policies (omitted for brevity) restrict every business table to
|
||
<code>tenant_id = current_setting('app.tenant_id')</code>. The app sets this GUC per request from the
|
||
authenticated key, so a query can never span tenants even on a coding error.</p>
|
||
</section>
|
||
|
||
<section id="security">
|
||
<h2><span class="n">7</span>Security</h2>
|
||
<p class="lead">Customer proposals are confidential business documents. The security model assumes
|
||
every judge is an external sub-processor and every tenant is mutually distrustful.</p>
|
||
|
||
<h3>PII Sanitization</h3>
|
||
<p>Sanitization (data-flow stage ②) runs <b>before any bytes leave the trust boundary</b> for a model
|
||
call. It is deterministic and idempotent so a verdict is reproducible and the redaction is auditable.</p>
|
||
<ul>
|
||
<li><b>Detectors:</b> regex passes for EMAIL, PHONE, SSN/EIN, credit-card (Luhn-checked), IBAN,
|
||
street address; plus a NER pass for PERSON and ORG when the tenant enables aggressive mode.</li>
|
||
<li><b>Replacement:</b> each match becomes a typed token <code>[REDACTED:EMAIL]</code>. A per-session
|
||
salted map (never sent to judges) allows post-hoc rehydration in the customer's own report view only.</li>
|
||
<li><b>Idempotency:</b> <code>sanitize_hash = sha256(sanitized_text)</code> is the judge-cache key —
|
||
re-scoring identical sanitized text reuses cached judge results and never re-bills.</li>
|
||
<li><b>Audit:</b> the <code>redactions</code> count vector (no raw values) is stored on the session.</li>
|
||
</ul>
|
||
<div class="warnbox"><b>Fail-closed.</b> If the sanitizer errors or a detector library fails to load,
|
||
the session transitions to <code>FAILED</code> rather than dispatching unsanitized text. Redaction is
|
||
never skipped to "save the run."</div>
|
||
|
||
<h3>Corpus Isolation</h3>
|
||
<p>A tenant's reference corpus (past winning proposals, style guides) is never visible to another
|
||
tenant's scoring context. Enforced at three layers (AD-07):</p>
|
||
<ul>
|
||
<li><b>Storage:</b> S3 prefix <code>s3://vt-corpus/{tenant_id}/…</code> with a bucket policy denying
|
||
cross-prefix reads; each tenant's IAM-scoped credentials cannot list outside their prefix.</li>
|
||
<li><b>Database:</b> row-level security keyed on <code>tenant_id</code> GUC (see §6).</li>
|
||
<li><b>Retrieval:</b> corpus lookups always inject <code>WHERE tenant_id = :current</code>; the query
|
||
builder refuses to run a corpus query without a bound tenant.</li>
|
||
</ul>
|
||
|
||
<h3>Sub-Processor Training Guard</h3>
|
||
<p>Customer content must never train a third-party model (AD-08). Enforced at the LiteLLM boundary:</p>
|
||
<ul>
|
||
<li>Every judge call sets provider no-train / zero-data-retention flags where offered
|
||
(e.g. OpenAI <code>store:false</code>, Anthropic no-retention headers).</li>
|
||
<li>DeepSeek traffic is routed only through a contracted no-retention endpoint; if that route is
|
||
unavailable, the DeepSeek slot is marked <code>UNAVAILABLE</code> rather than falling back to a
|
||
retaining endpoint — the training guard overrides the fallback chain.</li>
|
||
<li>LiteLLM request logs store metadata + token counts only, never artifact text.</li>
|
||
<li>Contractual: each provider is a named sub-processor in the DPA with a no-training clause; the
|
||
tenant's <code>subprocessor_ack</code> ledger flag records acknowledgment (see §8).</li>
|
||
</ul>
|
||
|
||
<h3>Encryption</h3>
|
||
<table>
|
||
<tr><th>Layer</th><th>Control</th></tr>
|
||
<tr><td>In transit (edge)</td><td>TLS 1.3 via Caddy; HSTS; automatic cert renewal.</td></tr>
|
||
<tr><td>In transit (internal)</td><td>App↔Postgres, App↔Redis, App↔LiteLLM on the Docker private network; no plaintext service exposed to the host public interface.</td></tr>
|
||
<tr><td>At rest (DB)</td><td>Postgres volume on encrypted disk; secrets columns (key_hash, secret_hash) are one-way hashed (argon2id), never reversible.</td></tr>
|
||
<tr><td>At rest (object store)</td><td>Wasabi S3 server-side encryption (SSE) on all objects; artifact objects lifecycle-expired after retention window.</td></tr>
|
||
<tr><td>Secrets</td><td>Provider keys and JWT signing secret injected via environment from a root-only <code>.env</code> (chmod 600), never committed; rotated on a schedule.</td></tr>
|
||
</table>
|
||
|
||
<h3>RBAC</h3>
|
||
<table>
|
||
<tr><th>Role</th><th>Read verdicts</th><th>Create/rescore reviews</th><th>Manage keys/webhooks/rubrics</th><th>White-Label admin</th></tr>
|
||
<tr><td><code>viewer</code></td><td>✔</td><td>✗</td><td>✗</td><td>✗</td></tr>
|
||
<tr><td><code>reviewer</code></td><td>✔</td><td>✔</td><td>✗</td><td>✗</td></tr>
|
||
<tr><td><code>admin</code></td><td>✔</td><td>✔</td><td>✔</td><td>✔ (subject to §8 gate)</td></tr>
|
||
</table>
|
||
<p class="small">Roles are carried on the API key and in the session JWT. Every mutating endpoint
|
||
checks role before touching state; a role violation returns <code>403 forbidden</code>. Cross-tenant
|
||
access (valid key, wrong tenant's resource id) returns <code>404 not_found</code> — the platform does
|
||
not reveal that another tenant's resource exists.</p>
|
||
|
||
<h4>Error handling — security</h4>
|
||
<ul>
|
||
<li><b>Sanitizer failure:</b> fail-closed → session <code>FAILED</code>, no dispatch, alert fired.</li>
|
||
<li><b>Training-guard route down:</b> affected judge slot <code>UNAVAILABLE</code>; guard never bypassed.</li>
|
||
<li><b>RLS/GUC unset:</b> queries against business tables raise and abort the request (defense in depth against a missing tenant context).</li>
|
||
<li><b>Expired/rotated JWT secret:</b> old tokens fail verification → <code>401</code>; clients refresh via <code>/auth/refresh</code>.</li>
|
||
</ul>
|
||
</section>
|
||
|
||
<section id="wlgate">
|
||
<h2><span class="n">8</span>White-Label Gate</h2>
|
||
<p class="lead">White-Label makes VerdictTank a <b>processor</b> acting on a customer's behalf under their
|
||
brand. Processor-role features stay locked behind a 5-condition ledger until every prerequisite is
|
||
verifiably satisfied (AD-11). Any locked call returns <b>HTTP 423 Locked</b>.</p>
|
||
|
||
<h3>5-Condition Ledger</h3>
|
||
<p>Backed by the <code>whitelabel_ledger</code> table (§6). All five booleans must be <code>true</code>
|
||
before <code>unlocked_at</code> is stamped and White-Label endpoints activate.</p>
|
||
<table>
|
||
<tr><th>#</th><th>Ledger flag</th><th>Condition</th><th>Owner / evidence</th></tr>
|
||
<tr><td>1</td><td><code>dpa_signed</code></td><td>Data Processing Agreement executed (processor role, no-training sub-processor clause).</td><td>Legal — signed DPA on file.</td></tr>
|
||
<tr><td>2</td><td><code>billing_active</code></td><td>White-Label plan billing in good standing (no failed/overdue invoice).</td><td>Billing — active subscription record.</td></tr>
|
||
<tr><td>3</td><td><code>corpus_isolated</code></td><td>Dedicated S3 corpus prefix + RLS partition provisioned and verified for the tenant.</td><td>Platform — isolation probe passes.</td></tr>
|
||
<tr><td>4</td><td><code>branding_approved</code></td><td>Custom logo, domain, and report styling reviewed and approved (no trademark conflict).</td><td>Ops — branding checklist signed off.</td></tr>
|
||
<tr><td>5</td><td><code>subprocessor_ack</code></td><td>Tenant has acknowledged the current sub-processor list (Anthropic, OpenAI, DeepSeek) and no-training guarantee.</td><td>Legal/tenant — acknowledgment timestamp.</td></tr>
|
||
</table>
|
||
|
||
<div class="note"><b>Unlock rule.</b>
|
||
<code>unlocked = dpa_signed AND billing_active AND corpus_isolated AND branding_approved AND subprocessor_ack</code>.
|
||
On the transition to all-true the platform sets <code>unlocked_at = now()</code> in a single
|
||
transaction. If any flag later flips to <code>false</code> (e.g. billing lapses), White-Label
|
||
endpoints immediately re-lock and return 423 again — the gate is evaluated on every request, not cached.</div>
|
||
|
||
<h3>GET /whitelabel/status — response (200)</h3>
|
||
<pre>{ "tenant_id":"t_8821", "unlocked":false, "unlocked_at":null,
|
||
"conditions":{
|
||
"dpa_signed":true,
|
||
"billing_active":true,
|
||
"corpus_isolated":false, // ← blocking
|
||
"branding_approved":false, // ← blocking
|
||
"subprocessor_ack":true
|
||
},
|
||
"blocking":["corpus_isolated","branding_approved"] }</pre>
|
||
|
||
<h3>HTTP 423 Response (locked feature access)</h3>
|
||
<p>Any White-Label-scoped endpoint (custom-branded report render, processor-mode review, tenant-branded
|
||
webhook) invoked while the ledger is incomplete returns:</p>
|
||
<pre>HTTP/1.1 423 Locked
|
||
Content-Type: application/json
|
||
|
||
{ "error":"whitelabel_locked",
|
||
"message":"White-Label features are locked until all onboarding conditions are met.",
|
||
"tenant_id":"t_8821",
|
||
"blocking":["corpus_isolated","branding_approved"],
|
||
"status_url":"/v4/whitelabel/status",
|
||
"retryable":false }</pre>
|
||
<p class="small">423 is chosen deliberately over 403: the resource is not forbidden by role, it is
|
||
<i>temporarily locked pending a state change the tenant can resolve</i>. The <code>blocking</code>
|
||
array tells the caller exactly which conditions remain, and <code>status_url</code> points at the
|
||
live ledger. Once all conditions clear, the same call succeeds with no code change on the client side.</p>
|
||
</section>
|
||
|
||
<section id="deploy">
|
||
<h2><span class="n">9</span>Deployment Topology</h2>
|
||
<p class="lead">Single netcup RS 4000 host running Docker Compose. Caddy terminates TLS and reverse-proxies
|
||
to the app tier; Prometheus + Grafana provide observability; Wasabi S3 holds artifacts and corpus.</p>
|
||
|
||
<h3>Host</h3>
|
||
<table>
|
||
<tr><th>Attribute</th><th>Value</th></tr>
|
||
<tr><td>Provider / plan</td><td>netcup RS 4000 (root server)</td></tr>
|
||
<tr><td>Origin (app3)</td><td><code>152.53.241.111</code></td></tr>
|
||
<tr><td>Orchestration</td><td>Docker Compose (single-host, multi-container)</td></tr>
|
||
<tr><td>Edge</td><td>Caddy 2 — automatic TLS, HTTP/2, reverse proxy, per-route rate limit</td></tr>
|
||
<tr><td>Object store</td><td>Wasabi S3 (<code>vt-artifacts</code>, <code>vt-corpus</code>, <code>vt-backups</code>)</td></tr>
|
||
<tr><td>Observability</td><td>Prometheus (scrape) + Grafana (dashboards + alerts)</td></tr>
|
||
</table>
|
||
|
||
<h3>Topology Diagram</h3>
|
||
<pre class="ascii">
|
||
Internet
|
||
│ 443
|
||
┌──▼───────────────────────────────────────── netcup RS 4000 (app3) ──────────────┐
|
||
│ │
|
||
│ ┌────────────┐ verdicttank.com / api.verdicttank.com │
|
||
│ │ Caddy 2 │ TLS · HTTP/2 · rate-limit · reverse proxy │
|
||
│ └─────┬──────┘ │
|
||
│ │ (docker private net: vtnet) │
|
||
│ ┌────▼─────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────────────┐ │
|
||
│ │ api │ │ worker │ │ postgres │ │ redis │ │ litellm │ │
|
||
│ │ FastAPI │◀─▶│ arq │◀─▶│ 16 │ │ 7 │ │ gateway │──┼─▶ providers
|
||
│ │ (uvicorn)│ │ │ │ (vol) │ │ (vol) │ │ (3 judges) │ │ (Anthropic/
|
||
│ └────┬─────┘ └────┬─────┘ └──────────┘ └──────────┘ └────────────────┘ │ OpenAI/
|
||
│ │ │ │ DeepSeek)
|
||
│ ┌────▼──────────────▼────┐ ┌────────────┐ ┌───────────┐ │
|
||
│ │ /metrics │───▶│ Prometheus │──▶│ Grafana │ dashboards+alerts │
|
||
│ └────────────────────────┘ └────────────┘ └───────────┘ │
|
||
│ │
|
||
│ nightly: pg_dump + artifact sync ───────────────────────────────────────────────┼─▶ Wasabi S3
|
||
└─────────────────────────────────────────────────────────────────────────────────┘
|
||
</pre>
|
||
|
||
<h3>docker-compose.yml (reference)</h3>
|
||
<pre>services:
|
||
caddy:
|
||
image: caddy:2
|
||
ports: ["80:80","443:443"]
|
||
volumes: ["./Caddyfile:/etc/caddy/Caddyfile","caddy_data:/data"]
|
||
networks: [vtnet]
|
||
depends_on: [api]
|
||
|
||
api:
|
||
build: ./app
|
||
command: uvicorn vt.main:app --host 0.0.0.0 --port 8000 --workers 4
|
||
env_file: [.env]
|
||
networks: [vtnet]
|
||
depends_on: [postgres, redis, litellm]
|
||
healthcheck:
|
||
test: ["CMD","curl","-fsS","http://localhost:8000/healthz"]
|
||
interval: 15s
|
||
|
||
worker:
|
||
build: ./app
|
||
command: arq vt.worker.WorkerSettings
|
||
env_file: [.env]
|
||
networks: [vtnet]
|
||
depends_on: [postgres, redis, litellm]
|
||
|
||
postgres:
|
||
image: postgres:16
|
||
environment: [POSTGRES_DB=vt, POSTGRES_USER=vt]
|
||
env_file: [.env] # POSTGRES_PASSWORD
|
||
volumes: ["pg_data:/var/lib/postgresql/data"]
|
||
networks: [vtnet]
|
||
|
||
redis:
|
||
image: redis:7
|
||
command: ["redis-server","--appendonly","yes"]
|
||
volumes: ["redis_data:/data"]
|
||
networks: [vtnet]
|
||
|
||
litellm:
|
||
image: ghcr.io/berriai/litellm:main-stable
|
||
command: ["--config","/app/config.yaml"]
|
||
volumes: ["./litellm/config.yaml:/app/config.yaml"]
|
||
env_file: [.env] # provider keys, no-retention routes
|
||
networks: [vtnet]
|
||
|
||
prometheus:
|
||
image: prom/prometheus
|
||
volumes: ["./prometheus.yml:/etc/prometheus/prometheus.yml","prom_data:/prometheus"]
|
||
networks: [vtnet]
|
||
|
||
grafana:
|
||
image: grafana/grafana
|
||
volumes: ["grafana_data:/var/lib/grafana"]
|
||
networks: [vtnet]
|
||
depends_on: [prometheus]
|
||
|
||
networks: { vtnet: { driver: bridge } }
|
||
volumes: { caddy_data: {}, pg_data: {}, redis_data: {}, prom_data: {}, grafana_data: {} }</pre>
|
||
|
||
<h3>Caddyfile (reference)</h3>
|
||
<pre>verdicttank.com {
|
||
encode gzip zstd
|
||
reverse_proxy api:8000
|
||
header Strict-Transport-Security "max-age=31536000; includeSubDomains"
|
||
}
|
||
api.verdicttank.com {
|
||
encode gzip zstd
|
||
rate_limit { zone api { key {remote_host}; events 60; window 1m } }
|
||
reverse_proxy api:8000
|
||
}</pre>
|
||
|
||
<h3>Observability</h3>
|
||
<ul>
|
||
<li><b>Prometheus</b> scrapes <code>/metrics</code> from api, worker, and litellm every 15 s.</li>
|
||
<li><b>Key metrics:</b> <code>vt_review_latency_seconds</code> (histogram), <code>vt_judge_fallback_stage</code>
|
||
(counter by stage), <code>vt_quorum_lost_total</code>, <code>vt_verdict_total{verdict}</code>,
|
||
<code>vt_sanitize_failures_total</code>.</li>
|
||
<li><b>Grafana alerts:</b> p95 end-to-end > 18 s (5 min), quorum-loss rate > 1% (10 min),
|
||
any sanitizer failure (immediate), a judge slot stuck at fallback stage ≥5 (5 min).</li>
|
||
</ul>
|
||
|
||
<h3>Backups & DR</h3>
|
||
<ul>
|
||
<li>Nightly <code>pg_dump</code> → <code>s3://vt-backups/pg/</code> (Wasabi), 30-day retention.</li>
|
||
<li>Artifact + corpus objects already durable in Wasabi (SSE); versioning enabled on
|
||
<code>vt-artifacts</code>.</li>
|
||
<li>Config-as-code: compose file, Caddyfile, litellm <code>config.yaml</code>, and DDL migrations in
|
||
the infra repo — full host rebuild from repo + <code>.env</code> + latest pg dump.</li>
|
||
</ul>
|
||
|
||
<h4>Error handling — deployment</h4>
|
||
<ul>
|
||
<li><b>Container crash:</b> Compose <code>restart: unless-stopped</code>; api healthcheck failure
|
||
removes it from Caddy upstream until healthy.</li>
|
||
<li><b>Postgres unavailable:</b> api returns <code>503</code>; worker retries with backoff; no data loss (state is committed transactionally).</li>
|
||
<li><b>LiteLLM down:</b> all judge slots walk the fallback chain to stage 6; sessions go
|
||
<code>DEGRADED</code>/<code>FAILED</code> rather than hang; alert fires.</li>
|
||
<li><b>Wasabi unreachable:</b> ingest fails fast with <code>503</code>; nightly backup retries and alerts on repeated failure.</li>
|
||
</ul>
|
||
</section>
|
||
|
||
<section id="infra">
|
||
<h2><span class="n">10</span>Infrastructure Tracker</h2>
|
||
<p class="lead">Canonical record of the deployed footprint. The product domain is
|
||
<b>verdicttank.com</b>. <code>rfptank.com</code> is retained only as legacy/defensive and serves no
|
||
production traffic.</p>
|
||
|
||
<h3>Domains & DNS</h3>
|
||
<table>
|
||
<tr><th>Host</th><th>Type</th><th>Target</th><th>Purpose</th></tr>
|
||
<tr><td><code>verdicttank.com</code></td><td>A</td><td>152.53.241.111</td><td>Primary web app + report viewer</td></tr>
|
||
<tr><td><code>www.verdicttank.com</code></td><td>CNAME</td><td>verdicttank.com</td><td>Canonical redirect → apex</td></tr>
|
||
<tr><td><code>api.verdicttank.com</code></td><td>A</td><td>152.53.241.111</td><td>REST API (<code>/v4</code>)</td></tr>
|
||
<tr><td><code>verdicttank.com</code></td><td>CAA</td><td>letsencrypt.org</td><td>Restrict cert issuance</td></tr>
|
||
<tr><td><code>verdicttank.com</code></td><td>MX / TXT (SPF, DMARC)</td><td>provider MX; v=spf1 …; DMARC p=quarantine</td><td>Transactional email deliverability</td></tr>
|
||
</table>
|
||
|
||
<h3>TLS</h3>
|
||
<ul>
|
||
<li>Certificates issued and auto-renewed by Caddy (ACME / Let's Encrypt) for
|
||
<code>verdicttank.com</code>, <code>www.verdicttank.com</code>, <code>api.verdicttank.com</code>.</li>
|
||
<li>TLS 1.3, HSTS <code>max-age=31536000; includeSubDomains</code>. CAA locks issuance to the ACME CA.</li>
|
||
</ul>
|
||
|
||
<h3>Origin & Storage</h3>
|
||
<table>
|
||
<tr><th>Resource</th><th>Location</th></tr>
|
||
<tr><td>Origin host (app3)</td><td>netcup RS 4000 · <code>152.53.241.111</code></td></tr>
|
||
<tr><td>Deploy path (proposal/architecture pages)</td><td><code>/home/ippadmin/htdocs/proposals.itpropartner.com/verdicttank/</code></td></tr>
|
||
<tr><td>Artifact store</td><td><code>s3://vt-artifacts/</code> (Wasabi, SSE, versioned)</td></tr>
|
||
<tr><td>Corpus store</td><td><code>s3://vt-corpus/{tenant_id}/</code> (Wasabi, prefix-isolated)</td></tr>
|
||
<tr><td>Backup store</td><td><code>s3://vt-backups/pg/</code> (Wasabi, 30-day retention)</td></tr>
|
||
</table>
|
||
|
||
<h3>Backup Schedule</h3>
|
||
<table>
|
||
<tr><th>Asset</th><th>Method</th><th>Frequency</th><th>Retention</th></tr>
|
||
<tr><td>PostgreSQL</td><td><code>pg_dump</code> → Wasabi</td><td>Nightly</td><td>30 days</td></tr>
|
||
<tr><td>Artifacts / corpus</td><td>Wasabi versioning (durable by default)</td><td>Continuous</td><td>Per lifecycle policy</td></tr>
|
||
<tr><td>Config-as-code</td><td>Git (infra repo)</td><td>On change</td><td>Full history</td></tr>
|
||
</table>
|
||
|
||
<div class="legacy"><b>Legacy / defensive — rfptank.com.</b> <code>rfptank.com</code> was the product's
|
||
prior working name. It is held defensively to prevent squatting and to 301-redirect any inbound legacy
|
||
links to <code>verdicttank.com</code>. It runs <b>no</b> application, stores <b>no</b> data, and issues
|
||
<b>no</b> certificates for production services. Every production reference — API base URL, report URLs,
|
||
webhook origins, DNS, TLS, and object-store buckets — is namespaced to <code>verdicttank.com</code>.
|
||
If <code>rfptank.com</code> appears anywhere outside this note, treat it as a defect.</div>
|
||
|
||
<h3>Deployed Document Map</h3>
|
||
<table>
|
||
<tr><th>Document</th><th>URL</th></tr>
|
||
<tr><td>Proposal</td><td><code>https://proposals.itpropartner.com/verdicttank/index.html</code></td></tr>
|
||
<tr><td>Architecture (this doc)</td><td><code>https://proposals.itpropartner.com/verdicttank/architecture.html</code></td></tr>
|
||
<tr><td>Review (current)</td><td><code>https://proposals.itpropartner.com/verdicttank/review.html</code></td></tr>
|
||
<tr><td>Product app</td><td><code>https://verdicttank.com</code></td></tr>
|
||
<tr><td>API</td><td><code>https://api.verdicttank.com/v4</code></td></tr>
|
||
</table>
|
||
</section>
|
||
|
||
<!-- END SECTIONS -->
|
||
</main>
|
||
|
||
<footer>
|
||
<div class="wrap">
|
||
VerdictTank v4.0 Technical Architecture · verdicttank.com · Origin app3 (152.53.241.111) ·
|
||
Generated for implementation. All domain references are <code>verdicttank.com</code>;
|
||
<code>rfptank.com</code> appears only as documented legacy/defensive.
|
||
</div>
|
||
</footer>
|
||
</body>
|
||
</html>
|