docs: full v3 architecture doc — 728-line conductor rebuild, replaces partial recovery from rsync wipe
Publish Docs Site / build (push) Failing after 4s
Publish Docs Site / build (push) Failing after 4s
This commit is contained in:
Germaine Brown
@@ -1,329 +1,729 @@
|
||||
# VerdictTank — Technical Architecture
|
||||
|
||||
> **Version:** v1.0
|
||||
> **Status:** 3/3 Unanimous Conditional Go (Aug 10, 2026)
|
||||
> **Owner:** Germaine Brown
|
||||
> **Built by:** Sho'Nuff (Hermes Agent)
|
||||
> Standalone technical reference for the VerdictTank v3 platform architecture. This document covers component design, data models, API contracts, pipeline mechanics, and operational procedures for developers and technical evaluators.
|
||||
|
||||
---
|
||||
## 1. Overview
|
||||
|
||||
## Overview
|
||||
VerdictTank is an automated business proposal review platform. A user submits a proposal document; the system runs it through a multi-agent review pipeline and returns a scored, evidence-backed verdict as a branded PDF report, typically within 3 to 15 minutes.
|
||||
|
||||
VerdictTank is an AI-powered business proposal review platform. A user submits a business proposal, product spec, or architecture document. Five specialized agents process it across three phases, three independent judges deliver a majority-rules verdict, and a branded PDF report lands in the user's inbox.
|
||||
The core thesis is straightforward: a single reviewer opinion is a data point, not a verdict. VerdictTank instead runs a proposal through a sequence of independent review stages built on distinct model architectures, then aggregates their conclusions into a majority-rules decision. The goal is a brutally honest, evidence-grounded assessment that is hard to game and cheap enough to run at scale.
|
||||
|
||||
### Pipeline
|
||||
### 1.1 The Pipeline Concept
|
||||
|
||||
```
|
||||
User submits proposal
|
||||
│
|
||||
▼
|
||||
Phase 1: Research Agent
|
||||
Live web verification — competitors, trademarks, domain WHOIS,
|
||||
market data, pricing benchmarks. Citations attached to every claim.
|
||||
│
|
||||
▼
|
||||
Phase 2: Critic Agent
|
||||
10-dimension review (Name, Pricing, PMF, Competition, Financials,
|
||||
GTM, Risk, Missing Elements, Founder Fit, Overall Verdict).
|
||||
Scored 1-10 per dimension.
|
||||
│
|
||||
▼
|
||||
Phase 3: Three judges in parallel (majority rules)
|
||||
Judge A — Independent architecture
|
||||
Judge B — Independent architecture
|
||||
Judge C — Independent architecture
|
||||
│
|
||||
▼
|
||||
PDF verdict delivered by email
|
||||
Every review passes through three logical phases:
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[Proposal Submitted] --> B[Phase 1: Research]
|
||||
B --> C[Phase 2: Primary Review]
|
||||
C --> D[Phase 3: Parallel Judges]
|
||||
D --> E[Majority Verdict]
|
||||
E --> F[PDF Report]
|
||||
```
|
||||
|
||||
### Verdict System
|
||||
- **Phase 1 — Research**: A dedicated research agent independently verifies factual claims in the proposal (market size figures, competitor claims, pricing comparisons), checks domain and trademark availability, and compiles a citation-backed research brief.
|
||||
- **Phase 2 — Primary Review**: A primary reviewer performs a structured, ten-dimension critique of the proposal, informed by the research brief, and produces numeric scores plus qualitative findings.
|
||||
- **Phase 3 — Parallel Cross-Check**: Multiple independent judges, each built on a different underlying architecture, review the proposal and the primary reviewer's findings in parallel. Each judge issues its own verdict, confidence level, and list of agreements/disagreements with the primary review.
|
||||
- **Majority Verdict**: Verdicts from all judges are aggregated. A simple majority rule determines the final published verdict (Go / Conditional Go / No-Go), with dissent explicitly preserved rather than smoothed over.
|
||||
- **PDF Report**: All findings are compiled into an 11-section branded PDF via an automated document generation pipeline.
|
||||
|
||||
| Result | Meaning |
|
||||
|---|---|
|
||||
| 3/3 Unanimous | Maximum confidence — all judges agree |
|
||||
| 2/3 Majority | Majority verdict with dissenting opinion |
|
||||
| 1-1-1 Split | Human (Germaine) breaks the tie |
|
||||
| Degraded (2 judges) | One judge unavailable — runs with remaining two |
|
||||
### 1.2 v2 to v3 Evolution
|
||||
|
||||
---
|
||||
The live v2 platform runs five review stages across the three phases described above at roughly $0.47 per run, with a three-judge cross-check panel and a 3-9 minute turnaround. v3 is a proposed upgrade layered on top of that same core pipeline, organized into three enhancement tiers plus an infrastructure prerequisite phase:
|
||||
|
||||
## Component Architecture
|
||||
|
||||
### Server: Core (netcup RS 2000 — 152.53.192.33)
|
||||
|
||||
| Component | Technology | Port | Role |
|
||||
|---|---|---|---|
|
||||
| Caddy | Caddy 2 | 80/443 | Reverse proxy, TLS termination, static file serving |
|
||||
| VerdictTank API | FastAPI (Python) | 8201 | Review orchestration, corpus queries, email dispatch |
|
||||
| Corpus DB | SQLite | — | Proposal corpus, benchmark data, outcome tracking |
|
||||
| PDF Generator | WeasyPrint | — | HTML-to-PDF rendering from `pdf-template.html` |
|
||||
| Gitea Actions Runner | Gitea Act Runner | — | CI/CD for VerdictTank repo |
|
||||
|
||||
### AI Models (via admin-ai)
|
||||
|
||||
| Phase | Model | Role |
|
||||
| Tier | Theme | Additions |
|
||||
|---|---|---|
|
||||
| Research | Worker model | Web search, claim verification |
|
||||
| Critic | Reviewer model | 10-dimension critical review |
|
||||
| Judge A | Independent model | Architecture A — validation |
|
||||
| Judge B | Independent model | Architecture B — cross-check |
|
||||
| Judge C | Independent model | Architecture C — operational realism |
|
||||
| Phase 0 | Infrastructure prerequisites | Automated sanitization gate, automated PDF generation pipeline |
|
||||
| Tier 1 | Cumulative intelligence | Review corpus database, prediction-vs-outcome tracking, reviewer accuracy scoring |
|
||||
| Tier 2 | Deeper analysis | Vertical-specific templates, adversarial red-team per vertical, market simulation engine |
|
||||
| Tier 3 | Product and distribution | Public-facing shareable reports, review-as-a-service API, competitive corpus comparisons |
|
||||
|
||||
All model calls route through `admin-ai.itpropartner.com` with the `verdicttank-prod` API key. Multi-provider architecture ensures no single vendor lock-in and genuine cross-model disagreement.
|
||||
The net effect is a system that remembers its own history (corpus), learns whether its verdicts were correct (prediction tracking), specializes by industry vertical, and exposes itself as a product surface beyond a single web form.
|
||||
|
||||
### Caddy Routing
|
||||
!!! info "Pipeline verdict on the v3 proposal itself"
|
||||
The v3 proposal was run through the VerdictTank pipeline prior to build. Verdict: **3/3 Unanimous Conditional Go**. Conditions attached to that verdict are reflected throughout the Security Model and Failure Modes sections below (notably around corpus confidentiality and sanitization gate rigor).
|
||||
|
||||
---
|
||||
|
||||
## 2. Component Architecture
|
||||
|
||||
VerdictTank is deliberately a small number of well-defined components rather than a large microservice mesh. Each component has a single responsibility and a narrow interface to its neighbors.
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph Edge
|
||||
Caddy[Reverse Proxy — Caddy v2]
|
||||
end
|
||||
subgraph App
|
||||
API[Core API Server — FastAPI]
|
||||
Orchestrator[Review Pipeline Engine]
|
||||
Sim[Market Simulation Engine]
|
||||
Sanitize[Sanitization Gate]
|
||||
end
|
||||
subgraph Data
|
||||
DB[(Corpus Database — SQLite)]
|
||||
PDFGen[PDF Generation Engine — WeasyPrint]
|
||||
Reports[/reports/ static files/]
|
||||
end
|
||||
subgraph Scheduled
|
||||
Cron[Prediction Tracking Cron]
|
||||
end
|
||||
|
||||
Caddy --> API
|
||||
Caddy --> Reports
|
||||
API --> Orchestrator
|
||||
Orchestrator --> Sim
|
||||
Orchestrator --> DB
|
||||
Orchestrator --> PDFGen
|
||||
PDFGen --> Reports
|
||||
Sanitize -.blocks.-> Reports
|
||||
Cron --> DB
|
||||
```
|
||||
|
||||
### 2.1 Core API Server
|
||||
|
||||
- **Stack**: FastAPI on Python 3.11+, served via an ASGI worker process.
|
||||
- **Role**: Public-facing HTTP interface. Accepts review submissions, exposes polling and retrieval endpoints, enforces authentication and rate limits, and hands work off to the review pipeline engine.
|
||||
- **Responsibilities**: request validation, tier entitlement checks, API key scoping, response serialization. Contains no review logic itself; it is a thin contract layer over the orchestrator.
|
||||
|
||||
### 2.2 Reverse Proxy Layer
|
||||
|
||||
- **Stack**: Caddy v2.
|
||||
- **Role**: TLS termination, HTTP-to-HTTPS redirection, automatic certificate renewal, request routing to the API process, and direct static file serving for the `/reports/` path (completed PDFs and public share assets).
|
||||
- **Why Caddy**: automatic TLS, minimal configuration surface, and a config format simple enough to keep under version control alongside the rest of the deploy.
|
||||
- **Directory listing is explicitly disabled** on `/reports/` — see Security Model, 7.3.
|
||||
|
||||
### 2.3 Review Pipeline Engine
|
||||
|
||||
- **Role**: The orchestrator. Sequences the five-plus agent phases for a given review, manages timeouts per stage, handles degraded-mode fallback when a judge is unavailable, and persists intermediate state so a review can be resumed or inspected mid-flight.
|
||||
- **Design**: implemented as a state machine keyed by `review_id`. Each phase transition writes its output to the review record before advancing, so a crash mid-pipeline loses at most the in-flight stage rather than the whole review.
|
||||
- **Concurrency**: Phase 3 judges run concurrently (not sequentially) since they are independent of one another by design; this is what keeps total turnaround in the minutes range rather than compounding each judge's latency serially.
|
||||
|
||||
### 2.4 Corpus Database
|
||||
|
||||
- **Stack**: SQLite, single file, with the corpus schema described in Section 3.
|
||||
- **Role**: System of record for every review ever run, plus derived tables for predictions, outcomes, and per-judge accuracy scores.
|
||||
- **Why SQLite**: review volume at current and near-term projected scale does not warrant a networked database; SQLite gives transactional integrity, zero operational overhead, and trivial backup (file copy) for a workload that is read-heavy and write-light per unit time.
|
||||
- **Growth path**: schema is written with an eye toward a future migration to a networked engine if corpus size or concurrent-write volume outgrows SQLite's comfortable range; no component queries SQLite-specific SQL extensions that would block that migration.
|
||||
|
||||
### 2.5 PDF Generation Engine
|
||||
|
||||
- **Stack**: WeasyPrint, driven off an HTML template populated from the structured review JSON.
|
||||
- **Role**: Converts the completed review record into the branded 11-section PDF report. See Section 6 for the full pipeline.
|
||||
- **Validation**: every generated PDF passes through a post-generation validation script before being written to `/reports/` — see 6.3.
|
||||
|
||||
### 2.6 Automated Sanitization Gate
|
||||
|
||||
- **Role**: A pre-deploy content scanner that runs against all output destined for any public surface (shareable reports, OG images, public API responses, docs). It scans for architecture-identifying strings (model/vendor names) and internal infrastructure references, and **blocks the deploy** if a match is found rather than silently redacting.
|
||||
- **Rationale**: the platform's differentiator is cross-vendor architecture diversity; leaking which specific architectures are in the panel undermines both competitive position and the "independent judges" framing that gives the majority verdict its credibility.
|
||||
- **Placement in CI**: runs as a required check in the deploy pipeline, not as a runtime filter. A failure here is a build failure, not a logged warning. See 7.5 and 8.7.
|
||||
|
||||
### 2.7 Market Simulation Engine
|
||||
|
||||
- **Role**: v3 Tier 2 addition. Given a proposal's stated business model, generates a 12-month simulated trajectory: user acquisition curve, churn projection, and resulting revenue trajectory, rather than relying solely on the static financial figures the proposal itself provides.
|
||||
- **Output**: feeds into the Primary Reviewer's Financials dimension and into judge disagreement analysis (a judge may flag that the simulated trajectory materially diverges from the proposal's stated projections).
|
||||
- **Independence**: intentionally decoupled from the proposal's own numbers so it cannot simply echo back what was submitted; it is a sanity-check model, not a validator of the proposal's math.
|
||||
|
||||
### 2.8 Prediction-vs-Outcome Tracking Cron
|
||||
|
||||
- **Role**: v3 Tier 1 addition. Every review that contains a flagged prediction (e.g., "this pricing tier will suppress conversion," "this GTM channel will not reach target CAC") is scheduled for automated re-check at T+90, T+180, and T+365 days.
|
||||
- **Mechanics**: the cron job queries the corpus for predictions due for re-check, attempts to gather current public signal on the outcome (via the research agent's toolset), and records a best-effort actual-outcome assessment against the original prediction. See Data Model 3.6 and Failure Modes 8.3 for attribution caveats.
|
||||
|
||||
---
|
||||
|
||||
## 3. Data Model
|
||||
|
||||
All primary entities below are persisted in the corpus database. Fields marked `v3` are new relative to the v2 schema.
|
||||
|
||||
### 3.1 Review Record
|
||||
|
||||
The top-level entity representing one submitted proposal and its full review lifecycle.
|
||||
|
||||
```sql
|
||||
CREATE TABLE review (
|
||||
id TEXT PRIMARY KEY, -- UUID
|
||||
client_name TEXT,
|
||||
proposal_text TEXT NOT NULL,
|
||||
proposal_hash TEXT NOT NULL, -- dedup / integrity check
|
||||
pricing_tier TEXT NOT NULL, -- free | pro | enterprise | white_label
|
||||
vertical TEXT, -- v3: auto-classified (saas, consulting, marketplace, fintech, ...)
|
||||
status TEXT NOT NULL, -- queued | researching | reviewing | judging | aggregating | complete | failed
|
||||
verdict TEXT, -- go | conditional_go | no_go
|
||||
verdict_margin TEXT, -- e.g. "3/3 unanimous", "2/3 majority"
|
||||
corpus_opt_out BOOLEAN DEFAULT 0, -- v3: excludes from corpus search / comparisons
|
||||
created_at TIMESTAMP NOT NULL,
|
||||
updated_at TIMESTAMP NOT NULL,
|
||||
completed_at TIMESTAMP
|
||||
);
|
||||
```
|
||||
|
||||
### 3.2 ResearchBrief
|
||||
|
||||
Output of Phase 1. One-to-one with a review.
|
||||
|
||||
```sql
|
||||
CREATE TABLE research_brief (
|
||||
review_id TEXT PRIMARY KEY REFERENCES review(id),
|
||||
verified_claims TEXT, -- JSON array: {claim, source_url, verified: bool}
|
||||
discrepancies TEXT, -- JSON array: {claim, discrepancy_description}
|
||||
competitor_analysis TEXT, -- JSON array: {name, url, positioning_summary}
|
||||
domain_availability TEXT, -- JSON: {domain, available: bool, checked_at}
|
||||
trademark_flags TEXT, -- JSON array: {mark, jurisdiction, conflict_summary}
|
||||
market_data TEXT, -- JSON: sourced market size / growth figures with citations
|
||||
citation_count INTEGER,
|
||||
limited_citations_flag BOOLEAN DEFAULT 0, -- set when web verification degraded, see 8.3
|
||||
created_at TIMESTAMP NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
### 3.3 CriticReview
|
||||
|
||||
Output of Phase 2 (Primary Reviewer). One-to-one with a review.
|
||||
|
||||
```sql
|
||||
CREATE TABLE critic_review (
|
||||
review_id TEXT PRIMARY KEY REFERENCES review(id),
|
||||
score_name INTEGER, -- 1-10
|
||||
score_pricing INTEGER,
|
||||
score_pmf INTEGER,
|
||||
score_competition INTEGER,
|
||||
score_financials INTEGER,
|
||||
score_gtm INTEGER,
|
||||
score_risk INTEGER,
|
||||
score_missing_elements INTEGER,
|
||||
score_founder_fit INTEGER,
|
||||
score_overall_verdict INTEGER,
|
||||
fatal_flaws TEXT, -- JSON array of strings
|
||||
strengths TEXT, -- JSON array of strings
|
||||
blind_spots TEXT, -- JSON array of strings
|
||||
verdict TEXT, -- go | conditional_go | no_go
|
||||
conditions TEXT, -- JSON array: conditions attached to a conditional_go
|
||||
created_at TIMESTAMP NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
### 3.4 JudgeVerdict
|
||||
|
||||
Output of Phase 3. One row per judge per review (one-to-many with a review).
|
||||
|
||||
```sql
|
||||
CREATE TABLE judge_verdict (
|
||||
id TEXT PRIMARY KEY,
|
||||
review_id TEXT NOT NULL REFERENCES review(id),
|
||||
judge_id TEXT NOT NULL, -- e.g. "reasoning_verification", "execution_feasibility", "market_reality"
|
||||
architecture_family TEXT NOT NULL, -- opaque vendor/architecture tag, used only for diversity enforcement
|
||||
verdict TEXT NOT NULL, -- go | conditional_go | no_go
|
||||
confidence REAL, -- 0.0 - 1.0
|
||||
agreements TEXT, -- JSON array: points of agreement with critic_review
|
||||
disagreements TEXT, -- JSON array: points of disagreement
|
||||
novel_insights TEXT, -- JSON array: findings not raised by prior phases
|
||||
second_order_effects TEXT, -- JSON array: downstream consequences the judge flags
|
||||
latency_ms INTEGER,
|
||||
timed_out BOOLEAN DEFAULT 0,
|
||||
created_at TIMESTAMP NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
### 3.5 Corpus Record (v3)
|
||||
|
||||
Indexing/search layer over completed reviews. One-to-one with a review, populated on completion.
|
||||
|
||||
```sql
|
||||
CREATE TABLE corpus_record (
|
||||
review_id TEXT PRIMARY KEY REFERENCES review(id),
|
||||
embedding BLOB, -- vector embedding for semantic search
|
||||
vertical TEXT, -- indexed
|
||||
verdict TEXT, -- indexed
|
||||
overall_score_avg REAL, -- indexed, for percentile comparisons
|
||||
searchable BOOLEAN DEFAULT 1, -- respects corpus_opt_out
|
||||
indexed_at TIMESTAMP NOT NULL
|
||||
);
|
||||
CREATE INDEX idx_corpus_vertical ON corpus_record(vertical);
|
||||
CREATE INDEX idx_corpus_verdict ON corpus_record(verdict);
|
||||
```
|
||||
|
||||
### 3.6 Prediction / Outcome Record (v3)
|
||||
|
||||
```sql
|
||||
CREATE TABLE prediction (
|
||||
id TEXT PRIMARY KEY,
|
||||
review_id TEXT NOT NULL REFERENCES review(id),
|
||||
judge_id TEXT, -- nullable; may originate from critic_review instead
|
||||
prediction_text TEXT NOT NULL,
|
||||
expected_timeframe_days INTEGER NOT NULL, -- 90 | 180 | 365
|
||||
check_due_at TIMESTAMP NOT NULL,
|
||||
actual_outcome TEXT, -- filled by cron: materialized | not_materialized | inconclusive
|
||||
outcome_confidence REAL, -- 0.0 - 1.0, cron's confidence in its own assessment
|
||||
outcome_notes TEXT,
|
||||
checked_at TIMESTAMP,
|
||||
created_at TIMESTAMP NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
### 3.7 Accuracy Score (v3)
|
||||
|
||||
```sql
|
||||
CREATE TABLE accuracy_score (
|
||||
judge_id TEXT PRIMARY KEY,
|
||||
running_accuracy REAL, -- fraction of predictions that materialized as flagged
|
||||
agreement_rate REAL, -- fraction of reviews where this judge agreed with majority
|
||||
diversity_score REAL, -- inverse of agreement_rate, monitored for consensus drift, see 7.6 and 9.4
|
||||
total_predictions_scored INTEGER,
|
||||
last_updated_at TIMESTAMP NOT NULL
|
||||
);
|
||||
```
|
||||
|
||||
!!! warning "Accuracy scoring is read-only until Phase 4"
|
||||
Per the implementation roadmap, `accuracy_score` is computed and exposed on a dashboard starting in Phase 1, but it does **not** feed back into judge weighting until Phase 4. This separation is deliberate: it lets the team observe whether accuracy scoring itself is trustworthy (see Failure Modes 8.2, consensus-drift risk) before letting it influence live verdicts.
|
||||
|
||||
---
|
||||
|
||||
## 4. API Design
|
||||
|
||||
All endpoints are namespaced under `/api/verdicttank/`. Authentication is via scoped API keys (see Security Model 7.2) for Enterprise and White-Label tiers; Free and Pro tiers authenticate via session token from the web application.
|
||||
|
||||
### 4.1 Submit Review
|
||||
|
||||
```
|
||||
verdicttank.com, www.verdicttank.com {
|
||||
handle /api/verdicttank/* {
|
||||
reverse_proxy 127.0.0.1:8201
|
||||
}
|
||||
handle /reports/* {
|
||||
root * /var/www/verdicttank
|
||||
file_server
|
||||
}
|
||||
redir /reports /reports/ permanent
|
||||
POST /api/verdicttank/review
|
||||
```
|
||||
|
||||
**Request:**
|
||||
```json
|
||||
{
|
||||
"proposal_text": "string, required",
|
||||
"client_name": "string, optional",
|
||||
"pricing_tier": "free | pro | enterprise | white_label",
|
||||
"vertical_override": "string, optional"
|
||||
}
|
||||
```
|
||||
|
||||
Crucial detail: use `handle` (not `handle_path`) for `/reports/*` — `handle_path` strips the prefix, breaking file resolution for `/reports/Client-Report.pdf`.
|
||||
|
||||
---
|
||||
|
||||
## Data Model
|
||||
|
||||
### Review Record
|
||||
|
||||
```
|
||||
Review
|
||||
├── id: UUID
|
||||
├── client_name: str
|
||||
├── proposal_text: str (full submitted document)
|
||||
├── pricing_tier: enum[free, pro, enterprise, whitelabel]
|
||||
├── status: enum[pending, research, critique, judging, done, failed]
|
||||
├── created_at: datetime
|
||||
├── completed_at: datetime?
|
||||
│
|
||||
├── research_brief: ResearchBrief?
|
||||
├── critic_review: CriticReview?
|
||||
├── judge_verdicts: [JudgeVerdict]
|
||||
├── final_verdict: enum[go, no_go, conditional_go]?
|
||||
├── pdf_path: str?
|
||||
│
|
||||
├── cost_breakdown: CostBreakdown
|
||||
└── turnaround_seconds: int?
|
||||
**Response `202 Accepted`:**
|
||||
```json
|
||||
{
|
||||
"review_id": "uuid",
|
||||
"status": "queued",
|
||||
"estimated_completion_seconds": 540
|
||||
}
|
||||
```
|
||||
|
||||
### ResearchBrief
|
||||
**Errors:** `429` (tier quota exceeded), `422` (proposal text below minimum length or fails structural parse), `401` (invalid/missing key).
|
||||
|
||||
### 4.2 Poll Review Status
|
||||
|
||||
```
|
||||
ResearchBrief
|
||||
├── verified_claims: [{claim, source_url, confidence}]
|
||||
├── discrepancies: [{claimed, actual, severity, evidence}]
|
||||
├── competitor_analysis: [{name, url, pricing, features, notes}]
|
||||
├── domain_check: {domain, registered, expiry, registrar}
|
||||
├── trademark_risk: enum[none, low, medium, high, blocked]
|
||||
└── market_data: [{statistic, value, source, verified}]
|
||||
GET /api/verdicttank/review/{id}
|
||||
```
|
||||
|
||||
### CriticReview
|
||||
|
||||
```
|
||||
CriticReview
|
||||
├── dimension_scores: {name: str, score: int, notes: str}[10]
|
||||
├── fatal_flaws: [{description, severity, condition_to_fix}]
|
||||
├── strengths: [str]
|
||||
├── blind_spots: [str]
|
||||
├── overall_verdict: enum[go, no_go, conditional_go]
|
||||
└── conditions: [str]
|
||||
**Response `200 OK`:**
|
||||
```json
|
||||
{
|
||||
"review_id": "uuid",
|
||||
"status": "researching | reviewing | judging | aggregating | complete | failed",
|
||||
"verdict": "go | conditional_go | no_go | null",
|
||||
"verdict_margin": "string | null",
|
||||
"progress_phase": 2,
|
||||
"created_at": "iso8601",
|
||||
"completed_at": "iso8601 | null"
|
||||
}
|
||||
```
|
||||
|
||||
### JudgeVerdict
|
||||
### 4.3 Download PDF
|
||||
|
||||
```
|
||||
JudgeVerdict
|
||||
├── judge_id: str (Judge 1/2/3)
|
||||
├── verdict: enum[go, no_go, conditional_go]
|
||||
├── confidence: float (0.0-1.0)
|
||||
├── agreements_with_critic: [str]
|
||||
├── disagreements_with_critic: [str]
|
||||
├── novel_insights: [str] ← findings no other judge caught
|
||||
├── second_order_effects: [str]
|
||||
└── conditions: [str]
|
||||
GET /api/verdicttank/review/{id}/pdf
|
||||
```
|
||||
|
||||
Returns the generated PDF binary with `Content-Type: application/pdf` once `status == complete`. Returns `409 Conflict` if the review is not yet complete, `404` if the review id does not exist or belongs to another account.
|
||||
|
||||
### 4.4 Corpus Search (Enterprise tier)
|
||||
|
||||
```
|
||||
GET /api/verdicttank/corpus/search?vertical=saas&verdict=conditional_go&q=pricing+tier&limit=20
|
||||
```
|
||||
|
||||
**Response `200 OK`:**
|
||||
```json
|
||||
{
|
||||
"results": [
|
||||
{
|
||||
"review_id": "uuid",
|
||||
"vertical": "saas",
|
||||
"verdict": "conditional_go",
|
||||
"overall_score_avg": 6.4,
|
||||
"summary_snippet": "string",
|
||||
"created_at": "iso8601"
|
||||
}
|
||||
],
|
||||
"total_matches": 143
|
||||
}
|
||||
```
|
||||
|
||||
Only reviews with `corpus_opt_out = 0` are searchable, and results returned to Enterprise/White-Label customers other than the review's owner are aggregate-only (score and vertical, no proposal text) unless the searching account owns the underlying review. See Security Model 7.4.
|
||||
|
||||
### 4.5 Accuracy Dashboard
|
||||
|
||||
```
|
||||
GET /api/verdicttank/accuracy
|
||||
```
|
||||
|
||||
**Response `200 OK`:**
|
||||
```json
|
||||
{
|
||||
"judges": [
|
||||
{
|
||||
"judge_id": "reasoning_verification",
|
||||
"running_accuracy": 0.71,
|
||||
"agreement_rate": 0.58,
|
||||
"diversity_score": 0.42,
|
||||
"total_predictions_scored": 212
|
||||
}
|
||||
],
|
||||
"last_updated_at": "iso8601"
|
||||
}
|
||||
```
|
||||
|
||||
### 4.6 Prediction Tracking
|
||||
|
||||
```
|
||||
GET /api/verdicttank/predictions/{review_id}
|
||||
```
|
||||
|
||||
**Response `200 OK`:**
|
||||
```json
|
||||
{
|
||||
"review_id": "uuid",
|
||||
"predictions": [
|
||||
{
|
||||
"prediction_text": "string",
|
||||
"expected_timeframe_days": 180,
|
||||
"check_due_at": "iso8601",
|
||||
"actual_outcome": "materialized | not_materialized | inconclusive | pending",
|
||||
"outcome_confidence": 0.6,
|
||||
"checked_at": "iso8601 | null"
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## PDF Generation
|
||||
## 5. Pipeline Details
|
||||
|
||||
The PDF pipeline at `/root/projects/verdicttank/`:
|
||||
### 5.1 Phase 1: Research Agent
|
||||
|
||||
1. **HTML Template** (`pdf-template.html`) — 11-section branded report:
|
||||
- Cover page with verdict badge
|
||||
- Executive summary
|
||||
- Dimension score table
|
||||
- Research findings with citations
|
||||
- Fatal flaws ranked by severity
|
||||
- Priority-ranked action plan
|
||||
- Judge cards (anonymized — "Judge 1/2/3", no model names)
|
||||
- Judge notes section
|
||||
- **Disclaimer** (Not Legal Advice, Not Financial Advice, AI-Generated Content Disclosure, No Warranty, Limitation of Liability, Confidentiality Notice)
|
||||
- Citations
|
||||
The research agent's mandate is narrow and specific: verify, don't opine. It receives the raw proposal text and produces a `ResearchBrief`.
|
||||
|
||||
2. **Generator** (`generate-pdf.py`) — Reads structured JSON data, fills the template via WeasyPrint, outputs branded PDF
|
||||
**Steps:**
|
||||
|
||||
3. **Quality Gate** (post-generation):
|
||||
```python
|
||||
from pypdf import PdfReader
|
||||
r = PdfReader('report.pdf')
|
||||
text = ''.join(p.extract_text() or '' for p in r.pages)
|
||||
assert '—' not in text, 'EM DASH FOUND'
|
||||
assert 'dispatch' not in text.lower(), 'DISPATCH FOUND'
|
||||
assert 'Disclaimer' in text, 'DISCLAIMER SECTION MISSING'
|
||||
assert len(r.pages) >= 10, f'Expected 10+ pages, got {len(r.pages)}'
|
||||
```
|
||||
1. **Claim extraction** — parse the proposal into a list of checkable factual assertions (market size figures, named competitors, cited statistics, pricing comparisons to named alternatives).
|
||||
2. **Citation verification** — for each extracted claim, run live web search to find a corroborating or contradicting source. Every claim in the resulting brief carries a source URL and a verified/unverified flag.
|
||||
3. **Competitor identification** — independent of what the proposal itself names, search for adjacent products/services in the same space and summarize their positioning.
|
||||
4. **Domain and trademark checks** — verify availability of the proposed product's domain name and flag potential trademark conflicts in relevant jurisdictions.
|
||||
5. **Market data gathering** — pull sourced, current market sizing and growth figures for the proposal's category, to be used later by both the Primary Reviewer's Financials dimension and the Market Simulation Engine.
|
||||
|
||||
### Sanitization Rules (hard constraints)
|
||||
If live web verification is degraded or unavailable mid-run, the agent proceeds with whatever citations it could gather and sets `limited_citations_flag = 1` on the brief. This flag propagates visibly into the final PDF (see Failure Modes 8.3) rather than being silently absorbed.
|
||||
|
||||
| Rule | Enforcement |
|
||||
### 5.2 Phase 2: Primary Reviewer
|
||||
|
||||
The Primary Reviewer consumes the proposal and the `ResearchBrief` and produces a `CriticReview` scored across ten fixed dimensions, each 1-10:
|
||||
|
||||
| Dimension | Focus |
|
||||
|---|---|
|
||||
| No model names anywhere in PDF | `build_judge_cards()` uses data `name` field — expects "Judge 1/2/3" |
|
||||
| No internal methodology exposed | No "pipeline", "conductor", "Opus", "Sonnet", "Qwen", "Gemini" |
|
||||
| No em dashes | `clean()` function strips `\u2014` and `\u2013` |
|
||||
| 11-section report (was 10 before Aug 8) | Disclaimer is Section 10, Citations is Section 11 |
|
||||
| Filename: `<Client>-VerdictTank-Report.pdf` | Safe-filename generator strips non-alphanumeric |
|
||||
| Name | Brand clarity, memorability, domain/trademark conflict from research brief |
|
||||
| Pricing | Sanity of pricing tiers against comparable market pricing |
|
||||
| PMF | Evidence of product-market fit vs. assumption |
|
||||
| Competition | Competitive differentiation, informed by research brief's independent competitor list |
|
||||
| Financials | Plausibility of unit economics and projections |
|
||||
| GTM | Go-to-market channel selection and realism of CAC/timeline assumptions |
|
||||
| Risk | Identified operational, legal, and market risks |
|
||||
| Missing Elements | What the proposal fails to address at all |
|
||||
| Founder Fit | Alignment between stated founder background and proposal execution demands |
|
||||
| Overall Verdict | Synthesis dimension: go / conditional_go / no_go with attached conditions |
|
||||
|
||||
Output also includes free-text `fatal_flaws`, `strengths`, and `blind_spots` lists, which feed directly into the PDF's Fatal Flaws and Action Plan sections.
|
||||
|
||||
### 5.3 Phase 3: Parallel Cross-Check
|
||||
|
||||
**Multi-vendor architecture diversity requirement.** Judges in the panel are required to span distinct underlying architecture families. This is enforced structurally, not by convention: the orchestrator checks `architecture_family` diversity across the active judge roster before a review can be marked complete. The rationale is that judges built on the same underlying architecture tend to share correlated blind spots; independence of judgment requires independence of substrate, not just independent prompting.
|
||||
|
||||
**Minimum quorum.** A review requires a minimum of **2** judges to reach a valid verdict. Below that, no majority can meaningfully be computed.
|
||||
|
||||
**Degraded-mode fallback.** If one judge in an N-judge panel times out or crashes, the orchestrator proceeds with the remaining N-1 judges rather than failing the whole review, provided N-1 still meets the minimum quorum of 2. The review record and PDF explicitly note that the panel ran in degraded mode. If a crash brings the panel below quorum, see Failure Modes 8.2.
|
||||
|
||||
**v2 baseline panel** (three judges): a general critic, a devil's-advocate contrarian, and a domain-specialist judge, each on a distinct architecture.
|
||||
|
||||
**v3 panel expansion** (three to seven judges): adds three specialist roles:
|
||||
|
||||
- **Reasoning-Verification Judge** — focused specifically on catching numerical and logical inconsistencies across the proposal and the Primary Reviewer's own scoring (does the stated CAC math actually support the stated LTV claim, etc.).
|
||||
- **Execution-Feasibility Judge** — grades pure operational feasibility: given the team, timeline, and budget described, can this actually be built and shipped as scoped.
|
||||
- **Market-Reality Judge** — deliberately configured to be less agreeable; its mandate is a contrarian read specifically on market timing and demand claims, resisting the tendency of multi-judge panels to converge toward polite consensus.
|
||||
|
||||
### 5.4 v3 Additions to the Pipeline
|
||||
|
||||
- **Vertical auto-classification** runs early (immediately after claim extraction) and determines which domain-specific dimensions and red-team checks apply downstream. Classification output (`saas`, `consulting`, `marketplace`, `fintech`, etc.) is stored on the review record and drives template selection for both the Primary Reviewer and the PDF.
|
||||
- **Adversarial red-team per vertical** — an additional targeted check applied after classification, tailored to the vertical's characteristic failure mode: HIPAA/compliance exposure for healthcare, regulatory licensing exposure for fintech, churn-driver scrutiny for consumer SaaS. This runs as an additional structured pass, not a replacement for the general critique.
|
||||
- **Market simulation** (Section 2.7) feeds a projected trajectory into the Financials dimension and is available to judges as a data point distinct from the proposal's own stated projections.
|
||||
|
||||
### 5.5 The Modular Judge System
|
||||
|
||||
Judges are configured, not hardcoded. Each judge in the active roster is defined by: a role identifier, a prompt/instruction template, an architecture family tag (for diversity enforcement), and an active weight (currently always 1.0 pending Phase 4).
|
||||
|
||||
**Adding a judge**: define role identifier, instruction template, and architecture family; register in the active roster config; the orchestrator picks it up on the next review without a pipeline code change.
|
||||
|
||||
**Removing a judge**: deactivate in roster config; historical `judge_verdict` rows referencing that `judge_id` remain in the corpus for accuracy scoring continuity.
|
||||
|
||||
**Weighting** (Phase 4 and beyond): `accuracy_score.running_accuracy` becomes an input to a weighted majority calculation rather than a simple headcount majority. Until Phase 4 ships, weighting is display-only on the accuracy dashboard and has zero effect on verdict aggregation. This staged rollout exists specifically to observe accuracy scoring for the consensus-drift risk described in 7.6 before it can move a real verdict.
|
||||
|
||||
---
|
||||
|
||||
## Pricing Integration
|
||||
## 6. PDF Generation
|
||||
|
||||
| Tier | Price | Reviews/Month | Model Access |
|
||||
### 6.1 Report Structure (11 Sections)
|
||||
|
||||
| # | Section | Content |
|
||||
|---|---|---|
|
||||
| 1 | Cover | Client name, proposal title, verdict badge, date |
|
||||
| 2 | Executive Summary | One-paragraph synthesis of the majority verdict and its rationale |
|
||||
| 3 | Score Table | All ten `CriticReview` dimension scores, tabulated |
|
||||
| 4 | Research | Verified claims, discrepancies, competitor analysis, domain/trademark findings |
|
||||
| 5 | Fatal Flaws | Enumerated list from `CriticReview.fatal_flaws`, cross-referenced against judge disagreements |
|
||||
| 6 | Action Plan | Concrete, prioritized remediation steps derived from fatal flaws and conditions |
|
||||
| 7 | Judge Cards | One card per judge: verdict, confidence, headline finding |
|
||||
| 8 | Judge Notes | Full agreements/disagreements/novel insights/second-order effects per judge |
|
||||
| 9 | Vertical Analysis (v3) | Vertical-specific findings and red-team results, when applicable |
|
||||
| 10 | Disclaimer | Standard liability and methodology disclaimer language |
|
||||
| 11 | Citations | Full source list from the research brief |
|
||||
|
||||
### 6.2 Generation Pipeline
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[Completed Review JSON] --> B[Template Fill]
|
||||
B --> C[Rendered HTML]
|
||||
C --> D[WeasyPrint Render]
|
||||
D --> E[Draft PDF]
|
||||
E --> F[Validation Script]
|
||||
F -->|pass| G[Written to /reports/]
|
||||
F -->|fail| H[Retry Once]
|
||||
H -->|fail again| I[Plaintext Fallback]
|
||||
```
|
||||
|
||||
1. The completed review record (all tables joined) is serialized to a single JSON payload.
|
||||
2. That payload fills a Jinja-style HTML template implementing the 11-section structure, with conditional blocks for vertical-specific sections.
|
||||
3. The rendered HTML is passed to WeasyPrint, which produces the PDF binary directly from HTML/CSS, with no intermediate manual step.
|
||||
4. The draft PDF is passed through the post-generation validation script before being written to the public `/reports/` path.
|
||||
|
||||
### 6.3 Post-Generation Validation Script
|
||||
|
||||
Every generated PDF must pass all of the following checks before being served:
|
||||
|
||||
- **Em dash / double-hyphen scan** — the report's prose must not contain em dashes or double hyphens (house style rule enforced automatically, not just at prompt level).
|
||||
- **Section presence check** — all 11 (or 10, if vertical analysis does not apply) expected sections must be present and non-empty.
|
||||
- **Page count minimum** — the rendered PDF must meet a minimum page count threshold; a report that renders suspiciously short indicates a template fill failure upstream.
|
||||
- **Sanitization scan** — the same architecture-name and infrastructure-detail scan used by the deploy-time sanitization gate (Section 2.6) also runs here, per-report, before public write.
|
||||
|
||||
A validation failure triggers one automatic retry of the full generation pipeline. If the retry also fails, the system falls back to a plaintext summary (see Failure Modes 8.4) rather than serving a broken or non-compliant PDF.
|
||||
|
||||
### 6.4 Automated vs. Manual Generation
|
||||
|
||||
The v2 baseline generated PDFs through a partially manual per-review process. The Phase 0 infrastructure prerequisite scripts this end-to-end: submission to PDF delivery requires zero manual intervention under normal operation. Manual generation remains available as an operator-invoked fallback tool for support cases (e.g., regenerating a report after a template fix), but is not part of the default customer-facing path.
|
||||
|
||||
---
|
||||
|
||||
## 7. Security Model
|
||||
|
||||
### 7.1 Prompt Injection via Proposal Text
|
||||
|
||||
Proposal text is user-supplied free text and is treated as untrusted input throughout the pipeline. Mitigations:
|
||||
|
||||
- **Structured parsing before agent exposure** — the claim-extraction step in Phase 1 parses the proposal into discrete structured claims before any downstream agent reasons over it as a single blob, reducing the surface for injected instructions to be interpreted as system-level directives.
|
||||
- **Role-boundary reinforcement** — every agent's instruction template explicitly frames proposal content as data to be evaluated, not instructions to be followed, and this framing is tested as part of the sanitization gate's broader remit.
|
||||
- **Output re-validation** — the sanitization gate and PDF validation script both scan final output for signs that instructions embedded in a proposal leaked into the report's own voice or structure.
|
||||
|
||||
### 7.2 API Key Management
|
||||
|
||||
- API keys issued to Enterprise and White-Label tier customers are **scoped**: a key is bound to a specific account and tier, and cannot access corpus search or endpoints beyond what that tier entitles.
|
||||
- Each key carries a **usage cap** matching the account's plan (review count per billing period); the API server enforces the cap at the request layer, returning `429` once exceeded, independent of any downstream billing reconciliation.
|
||||
- Keys are revocable individually without affecting other keys on the same account, supporting key rotation without downtime.
|
||||
|
||||
### 7.3 Report Access Control
|
||||
|
||||
- Completed PDFs and public share assets live under `/reports/`, served directly by the reverse proxy as static files.
|
||||
- **Directory listing is disabled** on this path at the proxy level; a report is only retrievable by its specific, unguessable identifier-based URL.
|
||||
- Reports for non-Free tiers are not indexed or discoverable; only the customer with the corresponding `review_id` (and valid session/API key) can request the signed download link via `GET /api/verdicttank/review/{id}/pdf`.
|
||||
|
||||
### 7.4 Corpus Confidentiality
|
||||
|
||||
The corpus is the platform's most sensitive asset: it aggregates other companies' unreleased business proposals.
|
||||
|
||||
- **Encryption at rest** for the corpus database file.
|
||||
- **Per-user opt-out** — `review.corpus_opt_out` lets any customer exclude their review from search indexing and comparisons entirely; opt-out is respected at the `corpus_record.searchable` level, not just a UI-layer filter.
|
||||
- **Aggregate-only comparisons** — competitive comparison features (percentile vs. corpus median) expose only score distributions and vertical classification, never proposal text, to any account other than the review's owner.
|
||||
- This was flagged as a named risk in the v3 proposal review (corpus confidentiality liability) and the above controls are the direct mitigation; see Failure Modes for the case where a control fails.
|
||||
|
||||
### 7.5 Sanitization Gate
|
||||
|
||||
Covered in detail in 2.6 and 6.3. Security framing: this is the platform's primary defense against leaking architecture-identifying details or internal infrastructure references into any public-facing surface (shareable reports, OG images, docs, public API responses). It is deploy-blocking, not advisory, precisely because a leak here is a competitive and trust failure that is hard to walk back once a report has been shared publicly.
|
||||
|
||||
### 7.6 API Abuse Protections
|
||||
|
||||
- **Rate limiting** at the reverse proxy and API layers, tuned per tier.
|
||||
- **Anomaly detection** on submission patterns — e.g., a burst of near-identical proposal submissions from one account, which could indicate an attempt to probe the pipeline's judge behavior or extract architecture information through differential prompting.
|
||||
- Abuse detection findings feed into manual account review rather than automated suspension, to avoid false-positive lockouts on legitimate high-volume Enterprise/White-Label usage.
|
||||
|
||||
### 7.7 Training-Data Recursion Prevention
|
||||
|
||||
A structural risk unique to this kind of pipeline: if the panel's own review output were ever used, directly or indirectly, to further train or fine-tune models used by the panel itself, judge diversity would collapse over time as the panel converges on its own prior outputs.
|
||||
|
||||
- **Volume caps** on any data pipeline that could plausibly feed review output back toward model training.
|
||||
- **Agreement-rate monitoring** — `accuracy_score.agreement_rate` and `diversity_score` are tracked over time specifically to detect a drift toward artificial consensus (judges agreeing with each other more, and more often, than architectural independence would predict). A sustained upward drift in agreement rate across the panel is treated as an operational signal to investigate, not just a marketing metric.
|
||||
|
||||
---
|
||||
|
||||
## 8. Failure Modes & Recovery
|
||||
|
||||
| # | Failure Mode | Impact | Recovery Procedure |
|
||||
|---|---|---|---|
|
||||
| Free | $0 | 1 | Limited model set |
|
||||
| Pro | $79/mo | 10 ($8 overage) | Full model set |
|
||||
| Enterprise | $499/mo | 50 ($12 overage) | API access, prediction tracking |
|
||||
| White-Label | $1,999+/mo | Unlimited | Multi-tenant, custom branding |
|
||||
| 8.1 | Single judge timeout or crash | Panel drops to N-1 judges | Orchestrator proceeds automatically if N-1 still meets minimum quorum of 2; review and PDF are flagged as degraded-mode |
|
||||
| 8.2 | All judges fail, or surviving judges fall below quorum | No valid verdict can be computed | Review is marked `failed`; customer is offered a free re-run at no charge against their quota |
|
||||
| 8.3 | Research agent web verification failure | Citations incomplete or absent | Pipeline proceeds with `limited_citations_flag = 1`; flag is surfaced visibly in the final PDF's Research section rather than silently omitted |
|
||||
| 8.4 | PDF generation failure (post-validation) | No compliant PDF produced | One automatic retry of the full generation pipeline; if retry also fails, deliver a plaintext summary fallback and flag the review for manual PDF regeneration |
|
||||
| 8.5 | Corpus database corruption | Loss of search, prediction tracking, and accuracy scoring continuity | Restore from nightly backup (Section 10.4); any reviews written between last backup and corruption event are re-derived from review-record source data where still available |
|
||||
| 8.6 | Model/architecture provider outage | One or more judges or the primary reviewer unavailable | Failover chain routes affected role to an alternate configured provider/architecture within the same role; if no failover is configured for that role, treat as case 8.1 or 8.2 depending on scope |
|
||||
| 8.7 | Sanitization gate failure (a scan match is found) | Deploy or report publication is blocked | This is by design: the gate blocks rather than warns. Operator must resolve the flagged content (redact/rephrase) before the deploy or report can proceed. Failure is never silently bypassed |
|
||||
|
||||
Cost per review: $0.70-0.75 (v3 corpus-backed). Turnaround: 5-15 minutes.
|
||||
!!! danger "Sanitization gate failures are not incidents to route around"
|
||||
If the sanitization gate blocks a deploy or a report, the correct response is to fix the flagged content, not to disable or bypass the gate to unblock a release. A bypass here directly reintroduces the leak risk the gate exists to prevent.
|
||||
|
||||
### 8.1 Attribution Difficulty in Prediction Tracking
|
||||
|
||||
Not a system failure in the crash sense, but a known limitation worth documenting alongside the other failure modes: the prediction-vs-outcome cron's `actual_outcome` assessment is inherently a correlation judgment, not a causal one. A flagged prediction ("this pricing tier will suppress conversion") that appears to materialize by T+180 may have done so for unrelated reasons. `outcome_confidence` on the `prediction` record exists specifically to carry this uncertainty forward rather than presenting the cron's assessment as ground truth; accuracy scoring calculations weight predictions by this confidence rather than treating every checked prediction as a binary hit/miss.
|
||||
|
||||
---
|
||||
|
||||
## Cost Model
|
||||
## 9. Cost Model
|
||||
|
||||
### Per-Review Breakdown
|
||||
### 9.1 v2 Baseline
|
||||
|
||||
| Phase | Tokens (est.) | Cost |
|
||||
|---|---|---|
|
||||
| Research Agent | ~5K in, ~3K out | ~$0.10 |
|
||||
| Critic Agent | ~15K in, ~5K out | ~$0.25 |
|
||||
| Judge A | ~20K in, ~3K out | ~$0.12 |
|
||||
| Judge B | ~20K in, ~2K out | ~$0.05 |
|
||||
| Judge C | ~20K in, ~2K out | ~$0.05 |
|
||||
| PDF Generation | N/A | ~$0.01 |
|
||||
| **Total** | | **~$0.58-0.75** |
|
||||
| Component | Cost per run |
|
||||
|---|---|
|
||||
| Model inference (research + primary review + 3 judges) | $0.38 |
|
||||
| Fully loaded (inference + infra amortization) | $0.47 |
|
||||
|
||||
At 10 reviews/month (Pro tier): ~$7.50 cost, $79 revenue = ~90% gross margin.
|
||||
At scale (100+ reviews/month): corpus caching reduces research costs significantly.
|
||||
### 9.2 v3 Full Pipeline
|
||||
|
||||
| Component | Contribution |
|
||||
|---|---|
|
||||
| Base pipeline (research + primary review) | carried forward from v2 baseline |
|
||||
| Specialist judges (up to 7-judge panel) | incremental per additional judge |
|
||||
| Market simulation engine | fixed per-run compute cost |
|
||||
| Corpus write and indexing | marginal, per-review |
|
||||
| Infra amortization | Caddy/API/WeasyPrint/cron overhead spread across run volume |
|
||||
| **Total v3 full pipeline** | **$0.86/run** |
|
||||
|
||||
### 9.3 Per-Tier Economics
|
||||
|
||||
| Tier | Price | Cost per run | Margin |
|
||||
|---|---|---|---|
|
||||
| Free | $0/mo (1 review/mo) | $0.07 (single-reviewer, no full panel) | N/A (loss-leader / funnel) |
|
||||
| Pro | $79/mo (20 reviews) | $0.86/run | 78% |
|
||||
| Enterprise | $499/mo (100 reviews) | $1.05/run | 79% |
|
||||
| White-Label | from $1,999/mo (unlimited cohort) | $1.05-$1.20/run | 75-82% |
|
||||
|
||||
Free tier deliberately runs a reduced single-reviewer pipeline (no full judge panel, score summary only, corpus percentile teaser) rather than the full pipeline at a loss per unit; this keeps the funnel economically sane while still giving prospective customers a real taste of the product.
|
||||
|
||||
### 9.4 Cost Assumptions and Optimization Levers
|
||||
|
||||
- **AI cost deflation assumption**: underlying inference costs are modeled to decline 15-20% annually based on historical trend, which is factored into margin projections for Enterprise and White-Label tiers over a multi-year horizon. This is an assumption, not a guarantee, and margin models should be re-validated against actual provider pricing at each planning cycle.
|
||||
- **Judge panel cost optimization**: judge count is a tunable parameter, not a fixed constant. The system supports running fewer judges for cost-sensitive contexts (e.g., Free tier) and more for Enterprise/White-Label, and the accuracy-vs-cost ratio (accuracy dashboard metrics against per-run cost) is the intended basis for deciding whether panel size should grow further, hold, or shrink for a given tier. The three-judge minimum-viable panel and the seven-judge maximal panel bound this tradeoff space; see 7.6 and 8.2 for the operational floor (minimum quorum of 2).
|
||||
|
||||
---
|
||||
|
||||
## Security Model
|
||||
## 10. Deployment Topology
|
||||
|
||||
### Attack Surface
|
||||
### 10.1 Single-Server Architecture
|
||||
|
||||
| Surface | Risk | Mitigation |
|
||||
|---|---|---|
|
||||
| Prompt injection via proposal text | HIGH | Input sanitization, structured parsing, no raw tool execution from user input |
|
||||
| Research agent web access | MEDIUM | URL allowlist, rate limiting, response size caps |
|
||||
| PDF injection via WeasyPrint | LOW | No user-controlled HTML in template, all content sanitized |
|
||||
| API key exposure | LOW | `verdicttank-prod` key scoped to admin-ai, usage caps |
|
||||
| Report access | LOW | `/reports/` directory served by Caddy, no directory listing |
|
||||
| Corpus poisoning | LOW | SQLite read-only for judge queries, write path gated |
|
||||
The platform runs as a single-server deployment: one host running the API server process, the Caddy reverse proxy, the SQLite corpus database file, and the WeasyPrint PDF generation process. This is an intentional simplicity choice given current scale; there is no distributed consensus, no service mesh, and no multi-region failover at this stage.
|
||||
|
||||
### Input Sanitization Pipeline
|
||||
|
||||
```
|
||||
Raw proposal submission
|
||||
→ Strip control characters
|
||||
→ Validate UTF-8
|
||||
→ Size cap (100KB)
|
||||
→ MIME type validation
|
||||
→ Pass to Research Agent as structured data, not raw prompt
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph Host[Single Application Host]
|
||||
Caddy[Caddy v2 — TLS + routing + static /reports/]
|
||||
API[API Server Process]
|
||||
DB[(SQLite Corpus DB)]
|
||||
PDF[WeasyPrint Process]
|
||||
Cron[Prediction Tracking Cron]
|
||||
end
|
||||
Internet -->|HTTPS| Caddy
|
||||
Caddy --> API
|
||||
API --> DB
|
||||
API --> PDF
|
||||
Cron --> DB
|
||||
```
|
||||
|
||||
### 10.2 DNS and TLS
|
||||
|
||||
- Public entry point is a standard DNS A/AAAA record pointed at the application host.
|
||||
- TLS termination and certificate issuance/renewal are handled automatically by Caddy v2; no manual certificate management step exists in the deploy path.
|
||||
|
||||
### 10.3 Build and Deploy Pipeline
|
||||
|
||||
- **Documentation** (this site): built with MkDocs (Material theme) and deployed as static output.
|
||||
- **API and pipeline code**: deployed via a standard git-push-triggered pipeline. The sanitization gate (Section 2.6) runs as a required check in this pipeline for any change touching public-facing output paths; a gate failure blocks the deploy outright.
|
||||
- Deploys to the API layer and deploys to the docs site are independent pipelines and can ship on separate cadences.
|
||||
|
||||
### 10.4 Backup Strategy
|
||||
|
||||
- **Nightly backups** to off-host object storage (S3-compatible), covering the full application state including the corpus database file.
|
||||
- The corpus is treated as first-class backup content, not an afterthought: it is the platform's accumulated institutional knowledge (Tier 1 cumulative intelligence) and its loss would silently degrade corpus search, prediction tracking, and accuracy scoring without necessarily causing an immediately visible outage.
|
||||
- Restore procedure for corpus corruption is covered in Failure Modes 8.5.
|
||||
|
||||
---
|
||||
|
||||
## Failure Modes & Recovery
|
||||
## 11. Operations
|
||||
|
||||
| Failure | Impact | Recovery |
|
||||
### 11.1 Key Metrics
|
||||
|
||||
| Metric | Description |
|
||||
|---|---|
|
||||
| Review success rate | Fraction of submitted reviews reaching `status = complete` vs. `failed` |
|
||||
| API latency (p95) | 95th percentile response time for synchronous API endpoints (submission, polling, dashboard) |
|
||||
| PDF generation failure rate | Fraction of reviews requiring PDF retry or falling back to plaintext (Section 8.4) |
|
||||
| Model/architecture latency | Per-role latency for research, primary review, and each judge, used to catch a specific provider degrading before it causes a full timeout |
|
||||
| Panel diversity health | `agreement_rate` / `diversity_score` trend across the judge panel over time (Section 7.7) |
|
||||
| Corpus growth rate | Reviews indexed per period, used to sanity-check corpus and backup sizing assumptions |
|
||||
|
||||
### 11.2 Alerting Thresholds
|
||||
|
||||
- Review success rate dropping below an agreed operational floor over a rolling window triggers investigation (distinguish between a systemic pipeline issue and a single provider outage per Failure Mode 8.6).
|
||||
- Any sanitization gate block on a production deploy or report generation attempt (8.7) should notify the operator immediately; this is a security-relevant event even though it is functioning as designed.
|
||||
- Sustained upward drift in panel `agreement_rate` beyond a defined band triggers a manual review of judge configuration, per the training-data recursion and consensus-drift concerns in 7.6 and 7.7.
|
||||
- PDF fallback-to-plaintext events (8.4) should alert on any occurrence, not just above a threshold, since they represent a customer-visible degradation of the deliverable.
|
||||
|
||||
### 11.3 Backup Schedule
|
||||
|
||||
- Nightly full backup of application state (Section 10.4).
|
||||
- Backup integrity should be spot-verified on a periodic cadence (e.g., a scheduled restore-to-scratch test), rather than assumed functional purely because the backup job reports success.
|
||||
|
||||
### 11.4 Corpus Health Checks
|
||||
|
||||
- Periodic verification that `corpus_record` entries stay in sync with their source `review` rows (no orphaned or stale index entries), particularly after any manual corpus maintenance operation.
|
||||
- Periodic audit that `corpus_opt_out` reviews are in fact excluded from search and comparison results, as a direct verification of the confidentiality control in 7.4 rather than trusting the flag's existence alone.
|
||||
- Prediction tracking cron runs (Section 2.8) should be monitored for completion; a silently-failing cron would cause `prediction.actual_outcome` to remain perpetually `pending`, quietly degrading the value of Tier 1 cumulative intelligence without any loud failure signal.
|
||||
|
||||
---
|
||||
|
||||
## Appendix A: Implementation Roadmap Reference
|
||||
|
||||
For planning and evaluation purposes, the v3 build is staged as follows:
|
||||
|
||||
| Phase | Weeks | Scope |
|
||||
|---|---|---|
|
||||
| Single judge timeout/crash | Degraded to 2 judges | Majority of 2 rules; note missing judge in addendum |
|
||||
| All judges fail | No verdict produced | Free re-run; investigate admin-ai |
|
||||
| Research Agent web failure | Incomplete citations | Flag as "limited research" in brief; proceed with available data |
|
||||
| PDF generation failure | Report not delivered | Retry once; email plain-text summary on persistent failure |
|
||||
| Core server outage | Platform unavailable | Warm standby (app1-bu) can take over Caddy + API |
|
||||
| admin-ai outage | All phases blocked | Model failover chain (Flash → Gemini → Grok → Sonnet) |
|
||||
| SQLite corruption | Corpus lost | Nightly backup to S3; rebuild from backup |
|
||||
| Phase 0 | 1-3 | Sanitization gate, automated PDF generation, corpus schema and backfill |
|
||||
| Phase 1 | 4-7 | Corpus search, prediction tracking cron, accuracy scoring (read-only, dashboard only) |
|
||||
| Phase 2 | 8-12 | Vertical-specific templates, adversarial red-team per vertical, market simulation engine |
|
||||
| Phase 3 | 13-18 | New specialist judges (reasoning-verification, execution-feasibility, market-reality), public shareable reports, review-as-a-service API beta |
|
||||
| Phase 4 | 19-22 | Live accuracy-based judge weighting, general availability, White-Label pilots |
|
||||
|
||||
---
|
||||
|
||||
## Operations
|
||||
|
||||
### Monitoring
|
||||
|
||||
| Metric | Source | Alert Threshold |
|
||||
|---|---|---|
|
||||
| API response time | Prometheus (Core:9090) | >30s p95 |
|
||||
| Review success rate | App-level counter | <90% over 1h |
|
||||
| PDF generation failures | App-level counter | Any failure |
|
||||
| admin-ai latency | LiteLLM metrics | >10s p95 |
|
||||
| Disk usage (reports) | node_exporter | >80% |
|
||||
|
||||
### Backup
|
||||
|
||||
- **Corpus DB:** Nightly to S3 via `hermes-backup.sh`
|
||||
- **PDF reports:** `/var/www/verdicttank/reports/` included in full backup
|
||||
- **API config:** In git repo, pushed to Gitea
|
||||
|
||||
---
|
||||
|
||||
## Known Limitations
|
||||
|
||||
1. **Single-box architecture** — Core is a single VPS. If Core goes down, VerdictTank is down until app1-bu failover completes (~5-10 min). A dedicated VerdictTank instance would eliminate this dependency.
|
||||
2. **No payment integration** — Currently manual billing. Stripe integration planned for Phase 2.
|
||||
3. **No user accounts** — Each review is standalone. User dashboard and review history planned for Phase 2.
|
||||
4. **Corpus freshness** — Benchmark data is static until manually refreshed. Automated crawl planned.
|
||||
5. **PDF email delivery** — Via MXroute SMTP, no delivery tracking. Postmark/SendGrid integration planned.
|
||||
|
||||
---
|
||||
|
||||
## Roadmap
|
||||
|
||||
| Phase | Deliverable | Status |
|
||||
|---|---|---|
|
||||
| Phase 0 | PDF pipeline, corpus DB, basic API | **DONE** |
|
||||
| Phase 1 | Outcome tracking, accuracy engine, corpus refresh | Proposed |
|
||||
| Phase 2 | Stripe integration, user dashboard, API access | Proposed |
|
||||
| Phase 3 | Multi-tenant, white-label, SSO | Proposed |
|
||||
|
||||
---
|
||||
|
||||
## Deployment
|
||||
|
||||
```
|
||||
# Deploy site
|
||||
scp -i /root/.ssh/itpp-infra /tmp/verdicttank-architecture.md \
|
||||
root@152.53.192.33:/root/itpp-docs/docs-source/verdicttank/architecture.md
|
||||
|
||||
# Build and deploy
|
||||
ssh core "cd /root/itpp-docs && mkdocs build --clean && \
|
||||
tar czf - site/" | ssh app3 "tar xzf - --strip-components=1 \
|
||||
-C /home/ippadmin/htdocs/docs.itpropartner.com/"
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
*Last updated: August 10, 2026*
|
||||
*Pipeline verdict: 3/3 Unanimous Conditional Go*
|
||||
*Commit: see git.itpropartner.com/ippadmin/itpp-docs*
|
||||
This staging is deliberate about sequencing risk: infrastructure hardening (sanitization, automated PDF, corpus foundation) ships before any feature that depends on it; accuracy scoring ships read-only well before it is trusted to influence a live verdict; and the highest-blast-radius change (letting historical accuracy actually move a verdict) is the very last thing to go live.
|
||||
Reference in New Issue
Block a user