ippadmin/itpp-docs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
1ecaeac35007ba9ed431d5f38ebcfb854f7cdf2f
itpp-docs/docs-source/verdicttank/architecture.md
T

44 KiB
Raw Blame History

VerdictTank — Technical Architecture

Standalone technical reference for the VerdictTank v3 platform architecture. This document covers component design, data models, API contracts, pipeline mechanics, and operational procedures for developers and technical evaluators.

1. Overview

VerdictTank is an automated business proposal review platform. A user submits a proposal document; the system runs it through a multi-agent review pipeline and returns a scored, evidence-backed verdict as a branded PDF report, typically within 3 to 15 minutes.

The core thesis is straightforward: a single reviewer opinion is a data point, not a verdict. VerdictTank instead runs a proposal through a sequence of independent review stages built on distinct model architectures, then aggregates their conclusions into a majority-rules decision. The goal is a brutally honest, evidence-grounded assessment that is hard to game and cheap enough to run at scale.

1.1 The Pipeline Concept

Every review passes through three logical phases:

flowchart LR
    A[Proposal Submitted] --> B[Phase 1: Research]
    B --> C[Phase 2: Primary Review]
    C --> D[Phase 3: Parallel Judges]
    D --> E[Majority Verdict]
    E --> F[PDF Report]
  • Phase 1 — Research: A dedicated research agent independently verifies factual claims in the proposal (market size figures, competitor claims, pricing comparisons), checks domain and trademark availability, and compiles a citation-backed research brief.
  • Phase 2 — Primary Review: A primary reviewer performs a structured, ten-dimension critique of the proposal, informed by the research brief, and produces numeric scores plus qualitative findings.
  • Phase 3 — Parallel Cross-Check: Multiple independent judges, each built on a different underlying architecture, review the proposal and the primary reviewer's findings in parallel. Each judge issues its own verdict, confidence level, and list of agreements/disagreements with the primary review.
  • Majority Verdict: Verdicts from all judges are aggregated. A simple majority rule determines the final published verdict (Go / Conditional Go / No-Go), with dissent explicitly preserved rather than smoothed over.
  • PDF Report: All findings are compiled into an 11-section branded PDF via an automated document generation pipeline.

1.2 v2 to v3 Evolution

The live v2 platform runs five review stages across the three phases described above at roughly $0.47 per run, with a three-judge cross-check panel and a 3-9 minute turnaround. v3 is a proposed upgrade layered on top of that same core pipeline, organized into three enhancement tiers plus an infrastructure prerequisite phase:

Tier Theme Additions
Phase 0 Infrastructure prerequisites Automated sanitization gate, automated PDF generation pipeline
Tier 1 Cumulative intelligence Review corpus database, prediction-vs-outcome tracking, reviewer accuracy scoring
Tier 2 Deeper analysis Vertical-specific templates, adversarial red-team per vertical, market simulation engine
Tier 3 Product and distribution Public-facing shareable reports, review-as-a-service API, competitive corpus comparisons

The net effect is a system that remembers its own history (corpus), learns whether its verdicts were correct (prediction tracking), specializes by industry vertical, and exposes itself as a product surface beyond a single web form.

!!! info "Pipeline verdict on the v3 proposal itself" The v3 proposal was run through the VerdictTank pipeline prior to build. Verdict: 3/3 Unanimous Conditional Go. Conditions attached to that verdict are reflected throughout the Security Model and Failure Modes sections below (notably around corpus confidentiality and sanitization gate rigor).

2. Component Architecture

VerdictTank is deliberately a small number of well-defined components rather than a large microservice mesh. Each component has a single responsibility and a narrow interface to its neighbors.

flowchart TB
    subgraph Edge
        Caddy[Reverse Proxy — Caddy v2]
    end
    subgraph App
        API[Core API Server — FastAPI]
        Orchestrator[Review Pipeline Engine]
        Sim[Market Simulation Engine]
        Sanitize[Sanitization Gate]
    end
    subgraph Data
        DB[(Corpus Database — SQLite)]
        PDFGen[PDF Generation Engine — WeasyPrint]
        Reports[/reports/ static files/]
    end
    subgraph Scheduled
        Cron[Prediction Tracking Cron]
    end

    Caddy --> API
    Caddy --> Reports
    API --> Orchestrator
    Orchestrator --> Sim
    Orchestrator --> DB
    Orchestrator --> PDFGen
    PDFGen --> Reports
    Sanitize -.blocks.-> Reports
    Cron --> DB

2.1 Core API Server

  • Stack: FastAPI on Python 3.11+, served via an ASGI worker process.
  • Role: Public-facing HTTP interface. Accepts review submissions, exposes polling and retrieval endpoints, enforces authentication and rate limits, and hands work off to the review pipeline engine.
  • Responsibilities: request validation, tier entitlement checks, API key scoping, response serialization. Contains no review logic itself; it is a thin contract layer over the orchestrator.

2.2 Reverse Proxy Layer

  • Stack: Caddy v2.
  • Role: TLS termination, HTTP-to-HTTPS redirection, automatic certificate renewal, request routing to the API process, and direct static file serving for the /reports/ path (completed PDFs and public share assets).
  • Why Caddy: automatic TLS, minimal configuration surface, and a config format simple enough to keep under version control alongside the rest of the deploy.
  • Directory listing is explicitly disabled on /reports/ — see Security Model, 7.3.

2.3 Review Pipeline Engine

  • Role: The orchestrator. Sequences the five-plus agent phases for a given review, manages timeouts per stage, handles degraded-mode fallback when a judge is unavailable, and persists intermediate state so a review can be resumed or inspected mid-flight.
  • Design: implemented as a state machine keyed by review_id. Each phase transition writes its output to the review record before advancing, so a crash mid-pipeline loses at most the in-flight stage rather than the whole review.
  • Concurrency: Phase 3 judges run concurrently (not sequentially) since they are independent of one another by design; this is what keeps total turnaround in the minutes range rather than compounding each judge's latency serially.

2.4 Corpus Database

  • Stack: SQLite, single file, with the corpus schema described in Section 3.
  • Role: System of record for every review ever run, plus derived tables for predictions, outcomes, and per-judge accuracy scores.
  • Why SQLite: review volume at current and near-term projected scale does not warrant a networked database; SQLite gives transactional integrity, zero operational overhead, and trivial backup (file copy) for a workload that is read-heavy and write-light per unit time.
  • Growth path: schema is written with an eye toward a future migration to a networked engine if corpus size or concurrent-write volume outgrows SQLite's comfortable range; no component queries SQLite-specific SQL extensions that would block that migration.

2.5 PDF Generation Engine

  • Stack: WeasyPrint, driven off an HTML template populated from the structured review JSON.
  • Role: Converts the completed review record into the branded 11-section PDF report. See Section 6 for the full pipeline.
  • Validation: every generated PDF passes through a post-generation validation script before being written to /reports/ — see 6.3.

2.6 Automated Sanitization Gate

  • Role: A pre-deploy content scanner that runs against all output destined for any public surface (shareable reports, OG images, public API responses, docs). It scans for architecture-identifying strings (model/vendor names) and internal infrastructure references, and blocks the deploy if a match is found rather than silently redacting.
  • Rationale: the platform's differentiator is cross-vendor architecture diversity; leaking which specific architectures are in the panel undermines both competitive position and the "independent judges" framing that gives the majority verdict its credibility.
  • Placement in CI: runs as a required check in the deploy pipeline, not as a runtime filter. A failure here is a build failure, not a logged warning. See 7.5 and 8.7.

2.7 Market Simulation Engine

  • Role: v3 Tier 2 addition. Given a proposal's stated business model, generates a 12-month simulated trajectory: user acquisition curve, churn projection, and resulting revenue trajectory, rather than relying solely on the static financial figures the proposal itself provides.
  • Output: feeds into the Primary Reviewer's Financials dimension and into judge disagreement analysis (a judge may flag that the simulated trajectory materially diverges from the proposal's stated projections).
  • Independence: intentionally decoupled from the proposal's own numbers so it cannot simply echo back what was submitted; it is a sanity-check model, not a validator of the proposal's math.

2.8 Prediction-vs-Outcome Tracking Cron

  • Role: v3 Tier 1 addition. Every review that contains a flagged prediction (e.g., "this pricing tier will suppress conversion," "this GTM channel will not reach target CAC") is scheduled for automated re-check at T+90, T+180, and T+365 days.
  • Mechanics: the cron job queries the corpus for predictions due for re-check, attempts to gather current public signal on the outcome (via the research agent's toolset), and records a best-effort actual-outcome assessment against the original prediction. See Data Model 3.6 and Failure Modes 8.3 for attribution caveats.

3. Data Model

All primary entities below are persisted in the corpus database. Fields marked v3 are new relative to the v2 schema.

3.1 Review Record

The top-level entity representing one submitted proposal and its full review lifecycle.

CREATE TABLE review (
    id                  TEXT PRIMARY KEY,        -- UUID
    client_name         TEXT,
    proposal_text       TEXT NOT NULL,
    proposal_hash       TEXT NOT NULL,            -- dedup / integrity check
    pricing_tier        TEXT NOT NULL,            -- free | pro | enterprise | white_label
    vertical            TEXT,                     -- v3: auto-classified (saas, consulting, marketplace, fintech, ...)
    status              TEXT NOT NULL,            -- queued | researching | reviewing | judging | aggregating | complete | failed
    verdict             TEXT,                      -- go | conditional_go | no_go
    verdict_margin      TEXT,                      -- e.g. "3/3 unanimous", "2/3 majority"
    corpus_opt_out      BOOLEAN DEFAULT 0,        -- v3: excludes from corpus search / comparisons
    created_at          TIMESTAMP NOT NULL,
    updated_at          TIMESTAMP NOT NULL,
    completed_at        TIMESTAMP
);

3.2 ResearchBrief

Output of Phase 1. One-to-one with a review.

CREATE TABLE research_brief (
    review_id           TEXT PRIMARY KEY REFERENCES review(id),
    verified_claims      TEXT,   -- JSON array: {claim, source_url, verified: bool}
    discrepancies        TEXT,   -- JSON array: {claim, discrepancy_description}
    competitor_analysis  TEXT,   -- JSON array: {name, url, positioning_summary}
    domain_availability   TEXT,  -- JSON: {domain, available: bool, checked_at}
    trademark_flags      TEXT,   -- JSON array: {mark, jurisdiction, conflict_summary}
    market_data          TEXT,   -- JSON: sourced market size / growth figures with citations
    citation_count       INTEGER,
    limited_citations_flag BOOLEAN DEFAULT 0,     -- set when web verification degraded, see 8.3
    created_at           TIMESTAMP NOT NULL
);

3.3 CriticReview

Output of Phase 2 (Primary Reviewer). One-to-one with a review.

CREATE TABLE critic_review (
    review_id           TEXT PRIMARY KEY REFERENCES review(id),
    score_name           INTEGER,   -- 1-10
    score_pricing         INTEGER,
    score_pmf             INTEGER,
    score_competition      INTEGER,
    score_financials       INTEGER,
    score_gtm              INTEGER,
    score_risk             INTEGER,
    score_missing_elements  INTEGER,
    score_founder_fit       INTEGER,
    score_overall_verdict   INTEGER,
    fatal_flaws          TEXT,   -- JSON array of strings
    strengths            TEXT,   -- JSON array of strings
    blind_spots           TEXT,  -- JSON array of strings
    verdict               TEXT,  -- go | conditional_go | no_go
    conditions             TEXT, -- JSON array: conditions attached to a conditional_go
    created_at             TIMESTAMP NOT NULL
);

3.4 JudgeVerdict

Output of Phase 3. One row per judge per review (one-to-many with a review).

CREATE TABLE judge_verdict (
    id                   TEXT PRIMARY KEY,
    review_id            TEXT NOT NULL REFERENCES review(id),
    judge_id             TEXT NOT NULL,        -- e.g. "reasoning_verification", "execution_feasibility", "market_reality"
    architecture_family   TEXT NOT NULL,        -- opaque vendor/architecture tag, used only for diversity enforcement
    verdict               TEXT NOT NULL,         -- go | conditional_go | no_go
    confidence             REAL,                -- 0.0 - 1.0
    agreements             TEXT,                -- JSON array: points of agreement with critic_review
    disagreements           TEXT,               -- JSON array: points of disagreement
    novel_insights          TEXT,               -- JSON array: findings not raised by prior phases
    second_order_effects     TEXT,              -- JSON array: downstream consequences the judge flags
    latency_ms              INTEGER,
    timed_out               BOOLEAN DEFAULT 0,
    created_at               TIMESTAMP NOT NULL
);

3.5 Corpus Record (v3)

Indexing/search layer over completed reviews. One-to-one with a review, populated on completion.

CREATE TABLE corpus_record (
    review_id             TEXT PRIMARY KEY REFERENCES review(id),
    embedding              BLOB,        -- vector embedding for semantic search
    vertical                TEXT,       -- indexed
    verdict                 TEXT,       -- indexed
    overall_score_avg        REAL,      -- indexed, for percentile comparisons
    searchable                BOOLEAN DEFAULT 1,  -- respects corpus_opt_out
    indexed_at                TIMESTAMP NOT NULL
);
CREATE INDEX idx_corpus_vertical ON corpus_record(vertical);
CREATE INDEX idx_corpus_verdict ON corpus_record(verdict);

3.6 Prediction / Outcome Record (v3)

CREATE TABLE prediction (
    id                    TEXT PRIMARY KEY,
    review_id              TEXT NOT NULL REFERENCES review(id),
    judge_id                TEXT,             -- nullable; may originate from critic_review instead
    prediction_text          TEXT NOT NULL,
    expected_timeframe_days   INTEGER NOT NULL,  -- 90 | 180 | 365
    check_due_at              TIMESTAMP NOT NULL,
    actual_outcome            TEXT,            -- filled by cron: materialized | not_materialized | inconclusive
    outcome_confidence         REAL,           -- 0.0 - 1.0, cron's confidence in its own assessment
    outcome_notes              TEXT,
    checked_at                 TIMESTAMP,
    created_at                 TIMESTAMP NOT NULL
);

3.7 Accuracy Score (v3)

CREATE TABLE accuracy_score (
    judge_id                TEXT PRIMARY KEY,
    running_accuracy          REAL,     -- fraction of predictions that materialized as flagged
    agreement_rate             REAL,    -- fraction of reviews where this judge agreed with majority
    diversity_score             REAL,   -- inverse of agreement_rate, monitored for consensus drift, see 7.6 and 9.4
    total_predictions_scored     INTEGER,
    last_updated_at               TIMESTAMP NOT NULL
);

!!! warning "Accuracy scoring is read-only until Phase 4" Per the implementation roadmap, accuracy_score is computed and exposed on a dashboard starting in Phase 1, but it does not feed back into judge weighting until Phase 4. This separation is deliberate: it lets the team observe whether accuracy scoring itself is trustworthy (see Failure Modes 8.2, consensus-drift risk) before letting it influence live verdicts.

4. API Design

All endpoints are namespaced under /api/verdicttank/. Authentication is via scoped API keys (see Security Model 7.2) for Enterprise and White-Label tiers; Free and Pro tiers authenticate via session token from the web application.

4.1 Submit Review

POST /api/verdicttank/review

Request:

{
  "proposal_text": "string, required",
  "client_name": "string, optional",
  "pricing_tier": "free | pro | enterprise | white_label",
  "vertical_override": "string, optional"
}

Response 202 Accepted:

{
  "review_id": "uuid",
  "status": "queued",
  "estimated_completion_seconds": 540
}

Errors: 429 (tier quota exceeded), 422 (proposal text below minimum length or fails structural parse), 401 (invalid/missing key).

4.2 Poll Review Status

GET /api/verdicttank/review/{id}

Response 200 OK:

{
  "review_id": "uuid",
  "status": "researching | reviewing | judging | aggregating | complete | failed",
  "verdict": "go | conditional_go | no_go | null",
  "verdict_margin": "string | null",
  "progress_phase": 2,
  "created_at": "iso8601",
  "completed_at": "iso8601 | null"
}

4.3 Download PDF

GET /api/verdicttank/review/{id}/pdf

Returns the generated PDF binary with Content-Type: application/pdf once status == complete. Returns 409 Conflict if the review is not yet complete, 404 if the review id does not exist or belongs to another account.

4.4 Corpus Search (Enterprise tier)

GET /api/verdicttank/corpus/search?vertical=saas&verdict=conditional_go&q=pricing+tier&limit=20

Response 200 OK:

{
  "results": [
    {
      "review_id": "uuid",
      "vertical": "saas",
      "verdict": "conditional_go",
      "overall_score_avg": 6.4,
      "summary_snippet": "string",
      "created_at": "iso8601"
    }
  ],
  "total_matches": 143
}

Only reviews with corpus_opt_out = 0 are searchable, and results returned to Enterprise/White-Label customers other than the review's owner are aggregate-only (score and vertical, no proposal text) unless the searching account owns the underlying review. See Security Model 7.4.

4.5 Accuracy Dashboard

GET /api/verdicttank/accuracy

Response 200 OK:

{
  "judges": [
    {
      "judge_id": "reasoning_verification",
      "running_accuracy": 0.71,
      "agreement_rate": 0.58,
      "diversity_score": 0.42,
      "total_predictions_scored": 212
    }
  ],
  "last_updated_at": "iso8601"
}

4.6 Prediction Tracking

GET /api/verdicttank/predictions/{review_id}

Response 200 OK:

{
  "review_id": "uuid",
  "predictions": [
    {
      "prediction_text": "string",
      "expected_timeframe_days": 180,
      "check_due_at": "iso8601",
      "actual_outcome": "materialized | not_materialized | inconclusive | pending",
      "outcome_confidence": 0.6,
      "checked_at": "iso8601 | null"
    }
  ]
}

5. Pipeline Details

5.1 Phase 1: Research Agent

The research agent's mandate is narrow and specific: verify, don't opine. It receives the raw proposal text and produces a ResearchBrief.

Steps:

  1. Claim extraction — parse the proposal into a list of checkable factual assertions (market size figures, named competitors, cited statistics, pricing comparisons to named alternatives).
  2. Citation verification — for each extracted claim, run live web search to find a corroborating or contradicting source. Every claim in the resulting brief carries a source URL and a verified/unverified flag.
  3. Competitor identification — independent of what the proposal itself names, search for adjacent products/services in the same space and summarize their positioning.
  4. Domain and trademark checks — verify availability of the proposed product's domain name and flag potential trademark conflicts in relevant jurisdictions.
  5. Market data gathering — pull sourced, current market sizing and growth figures for the proposal's category, to be used later by both the Primary Reviewer's Financials dimension and the Market Simulation Engine.

If live web verification is degraded or unavailable mid-run, the agent proceeds with whatever citations it could gather and sets limited_citations_flag = 1 on the brief. This flag propagates visibly into the final PDF (see Failure Modes 8.3) rather than being silently absorbed.

5.2 Phase 2: Primary Reviewer

The Primary Reviewer consumes the proposal and the ResearchBrief and produces a CriticReview scored across ten fixed dimensions, each 1-10:

Dimension Focus
Name Brand clarity, memorability, domain/trademark conflict from research brief
Pricing Sanity of pricing tiers against comparable market pricing
PMF Evidence of product-market fit vs. assumption
Competition Competitive differentiation, informed by research brief's independent competitor list
Financials Plausibility of unit economics and projections
GTM Go-to-market channel selection and realism of CAC/timeline assumptions
Risk Identified operational, legal, and market risks
Missing Elements What the proposal fails to address at all
Founder Fit Alignment between stated founder background and proposal execution demands
Overall Verdict Synthesis dimension: go / conditional_go / no_go with attached conditions

Output also includes free-text fatal_flaws, strengths, and blind_spots lists, which feed directly into the PDF's Fatal Flaws and Action Plan sections.

5.3 Phase 3: Parallel Cross-Check

Multi-vendor architecture diversity requirement. Judges in the panel are required to span distinct underlying architecture families. This is enforced structurally, not by convention: the orchestrator checks architecture_family diversity across the active judge roster before a review can be marked complete. The rationale is that judges built on the same underlying architecture tend to share correlated blind spots; independence of judgment requires independence of substrate, not just independent prompting.

Minimum quorum. A review requires a minimum of 2 judges to reach a valid verdict. Below that, no majority can meaningfully be computed.

Degraded-mode fallback. If one judge in an N-judge panel times out or crashes, the orchestrator proceeds with the remaining N-1 judges rather than failing the whole review, provided N-1 still meets the minimum quorum of 2. The review record and PDF explicitly note that the panel ran in degraded mode. If a crash brings the panel below quorum, see Failure Modes 8.2.

v2 baseline panel (three judges): a general critic, a devil's-advocate contrarian, and a domain-specialist judge, each on a distinct architecture.

v3 panel expansion (three to seven judges): adds three specialist roles:

  • Reasoning-Verification Judge — focused specifically on catching numerical and logical inconsistencies across the proposal and the Primary Reviewer's own scoring (does the stated CAC math actually support the stated LTV claim, etc.).
  • Execution-Feasibility Judge — grades pure operational feasibility: given the team, timeline, and budget described, can this actually be built and shipped as scoped.
  • Market-Reality Judge — deliberately configured to be less agreeable; its mandate is a contrarian read specifically on market timing and demand claims, resisting the tendency of multi-judge panels to converge toward polite consensus.

5.4 v3 Additions to the Pipeline

  • Vertical auto-classification runs early (immediately after claim extraction) and determines which domain-specific dimensions and red-team checks apply downstream. Classification output (saas, consulting, marketplace, fintech, etc.) is stored on the review record and drives template selection for both the Primary Reviewer and the PDF.
  • Adversarial red-team per vertical — an additional targeted check applied after classification, tailored to the vertical's characteristic failure mode: HIPAA/compliance exposure for healthcare, regulatory licensing exposure for fintech, churn-driver scrutiny for consumer SaaS. This runs as an additional structured pass, not a replacement for the general critique.
  • Market simulation (Section 2.7) feeds a projected trajectory into the Financials dimension and is available to judges as a data point distinct from the proposal's own stated projections.

5.5 The Modular Judge System

Judges are configured, not hardcoded. Each judge in the active roster is defined by: a role identifier, a prompt/instruction template, an architecture family tag (for diversity enforcement), and an active weight (currently always 1.0 pending Phase 4).

Adding a judge: define role identifier, instruction template, and architecture family; register in the active roster config; the orchestrator picks it up on the next review without a pipeline code change.

Removing a judge: deactivate in roster config; historical judge_verdict rows referencing that judge_id remain in the corpus for accuracy scoring continuity.

Weighting (Phase 4 and beyond): accuracy_score.running_accuracy becomes an input to a weighted majority calculation rather than a simple headcount majority. Until Phase 4 ships, weighting is display-only on the accuracy dashboard and has zero effect on verdict aggregation. This staged rollout exists specifically to observe accuracy scoring for the consensus-drift risk described in 7.6 before it can move a real verdict.

6. PDF Generation

6.1 Report Structure (11 Sections)

# Section Content
1 Cover Client name, proposal title, verdict badge, date
2 Executive Summary One-paragraph synthesis of the majority verdict and its rationale
3 Score Table All ten CriticReview dimension scores, tabulated
4 Research Verified claims, discrepancies, competitor analysis, domain/trademark findings
5 Fatal Flaws Enumerated list from CriticReview.fatal_flaws, cross-referenced against judge disagreements
6 Action Plan Concrete, prioritized remediation steps derived from fatal flaws and conditions
7 Judge Cards One card per judge: verdict, confidence, headline finding
8 Judge Notes Full agreements/disagreements/novel insights/second-order effects per judge
9 Vertical Analysis (v3) Vertical-specific findings and red-team results, when applicable
10 Disclaimer Standard liability and methodology disclaimer language
11 Citations Full source list from the research brief

6.2 Generation Pipeline

flowchart LR
    A[Completed Review JSON] --> B[Template Fill]
    B --> C[Rendered HTML]
    C --> D[WeasyPrint Render]
    D --> E[Draft PDF]
    E --> F[Validation Script]
    F -->|pass| G[Written to /reports/]
    F -->|fail| H[Retry Once]
    H -->|fail again| I[Plaintext Fallback]
  1. The completed review record (all tables joined) is serialized to a single JSON payload.
  2. That payload fills a Jinja-style HTML template implementing the 11-section structure, with conditional blocks for vertical-specific sections.
  3. The rendered HTML is passed to WeasyPrint, which produces the PDF binary directly from HTML/CSS, with no intermediate manual step.
  4. The draft PDF is passed through the post-generation validation script before being written to the public /reports/ path.

6.3 Post-Generation Validation Script

Every generated PDF must pass all of the following checks before being served:

  • Em dash / double-hyphen scan — the report's prose must not contain em dashes or double hyphens (house style rule enforced automatically, not just at prompt level).
  • Section presence check — all 11 (or 10, if vertical analysis does not apply) expected sections must be present and non-empty.
  • Page count minimum — the rendered PDF must meet a minimum page count threshold; a report that renders suspiciously short indicates a template fill failure upstream.
  • Sanitization scan — the same architecture-name and infrastructure-detail scan used by the deploy-time sanitization gate (Section 2.6) also runs here, per-report, before public write.

A validation failure triggers one automatic retry of the full generation pipeline. If the retry also fails, the system falls back to a plaintext summary (see Failure Modes 8.4) rather than serving a broken or non-compliant PDF.

6.4 Automated vs. Manual Generation

The v2 baseline generated PDFs through a partially manual per-review process. The Phase 0 infrastructure prerequisite scripts this end-to-end: submission to PDF delivery requires zero manual intervention under normal operation. Manual generation remains available as an operator-invoked fallback tool for support cases (e.g., regenerating a report after a template fix), but is not part of the default customer-facing path.

7. Security Model

7.1 Prompt Injection via Proposal Text

Proposal text is user-supplied free text and is treated as untrusted input throughout the pipeline. Mitigations:

  • Structured parsing before agent exposure — the claim-extraction step in Phase 1 parses the proposal into discrete structured claims before any downstream agent reasons over it as a single blob, reducing the surface for injected instructions to be interpreted as system-level directives.
  • Role-boundary reinforcement — every agent's instruction template explicitly frames proposal content as data to be evaluated, not instructions to be followed, and this framing is tested as part of the sanitization gate's broader remit.
  • Output re-validation — the sanitization gate and PDF validation script both scan final output for signs that instructions embedded in a proposal leaked into the report's own voice or structure.

7.2 API Key Management

  • API keys issued to Enterprise and White-Label tier customers are scoped: a key is bound to a specific account and tier, and cannot access corpus search or endpoints beyond what that tier entitles.
  • Each key carries a usage cap matching the account's plan (review count per billing period); the API server enforces the cap at the request layer, returning 429 once exceeded, independent of any downstream billing reconciliation.
  • Keys are revocable individually without affecting other keys on the same account, supporting key rotation without downtime.

7.3 Report Access Control

  • Completed PDFs and public share assets live under /reports/, served directly by the reverse proxy as static files.
  • Directory listing is disabled on this path at the proxy level; a report is only retrievable by its specific, unguessable identifier-based URL.
  • Reports for non-Free tiers are not indexed or discoverable; only the customer with the corresponding review_id (and valid session/API key) can request the signed download link via GET /api/verdicttank/review/{id}/pdf.

7.4 Corpus Confidentiality

The corpus is the platform's most sensitive asset: it aggregates other companies' unreleased business proposals.

  • Encryption at rest for the corpus database file.
  • Per-user opt-out — review.corpus_opt_out lets any customer exclude their review from search indexing and comparisons entirely; opt-out is respected at the corpus_record.searchable level, not just a UI-layer filter.
  • Aggregate-only comparisons — competitive comparison features (percentile vs. corpus median) expose only score distributions and vertical classification, never proposal text, to any account other than the review's owner.
  • This was flagged as a named risk in the v3 proposal review (corpus confidentiality liability) and the above controls are the direct mitigation; see Failure Modes for the case where a control fails.

7.5 Sanitization Gate

Covered in detail in 2.6 and 6.3. Security framing: this is the platform's primary defense against leaking architecture-identifying details or internal infrastructure references into any public-facing surface (shareable reports, OG images, docs, public API responses). It is deploy-blocking, not advisory, precisely because a leak here is a competitive and trust failure that is hard to walk back once a report has been shared publicly.

7.6 API Abuse Protections

  • Rate limiting at the reverse proxy and API layers, tuned per tier.
  • Anomaly detection on submission patterns — e.g., a burst of near-identical proposal submissions from one account, which could indicate an attempt to probe the pipeline's judge behavior or extract architecture information through differential prompting.
  • Abuse detection findings feed into manual account review rather than automated suspension, to avoid false-positive lockouts on legitimate high-volume Enterprise/White-Label usage.

7.7 Training-Data Recursion Prevention

A structural risk unique to this kind of pipeline: if the panel's own review output were ever used, directly or indirectly, to further train or fine-tune models used by the panel itself, judge diversity would collapse over time as the panel converges on its own prior outputs.

  • Volume caps on any data pipeline that could plausibly feed review output back toward model training.
  • Agreement-rate monitoring — accuracy_score.agreement_rate and diversity_score are tracked over time specifically to detect a drift toward artificial consensus (judges agreeing with each other more, and more often, than architectural independence would predict). A sustained upward drift in agreement rate across the panel is treated as an operational signal to investigate, not just a marketing metric.

8. Failure Modes & Recovery

# Failure Mode Impact Recovery Procedure
8.1 Single judge timeout or crash Panel drops to N-1 judges Orchestrator proceeds automatically if N-1 still meets minimum quorum of 2; review and PDF are flagged as degraded-mode
8.2 All judges fail, or surviving judges fall below quorum No valid verdict can be computed Review is marked failed; customer is offered a free re-run at no charge against their quota
8.3 Research agent web verification failure Citations incomplete or absent Pipeline proceeds with limited_citations_flag = 1; flag is surfaced visibly in the final PDF's Research section rather than silently omitted
8.4 PDF generation failure (post-validation) No compliant PDF produced One automatic retry of the full generation pipeline; if retry also fails, deliver a plaintext summary fallback and flag the review for manual PDF regeneration
8.5 Corpus database corruption Loss of search, prediction tracking, and accuracy scoring continuity Restore from nightly backup (Section 10.4); any reviews written between last backup and corruption event are re-derived from review-record source data where still available
8.6 Model/architecture provider outage One or more judges or the primary reviewer unavailable Failover chain routes affected role to an alternate configured provider/architecture within the same role; if no failover is configured for that role, treat as case 8.1 or 8.2 depending on scope
8.7 Sanitization gate failure (a scan match is found) Deploy or report publication is blocked This is by design: the gate blocks rather than warns. Operator must resolve the flagged content (redact/rephrase) before the deploy or report can proceed. Failure is never silently bypassed

!!! danger "Sanitization gate failures are not incidents to route around" If the sanitization gate blocks a deploy or a report, the correct response is to fix the flagged content, not to disable or bypass the gate to unblock a release. A bypass here directly reintroduces the leak risk the gate exists to prevent.

8.1 Attribution Difficulty in Prediction Tracking

Not a system failure in the crash sense, but a known limitation worth documenting alongside the other failure modes: the prediction-vs-outcome cron's actual_outcome assessment is inherently a correlation judgment, not a causal one. A flagged prediction ("this pricing tier will suppress conversion") that appears to materialize by T+180 may have done so for unrelated reasons. outcome_confidence on the prediction record exists specifically to carry this uncertainty forward rather than presenting the cron's assessment as ground truth; accuracy scoring calculations weight predictions by this confidence rather than treating every checked prediction as a binary hit/miss.

9. Cost Model

9.1 v2 Baseline

Component Cost per run
Model inference (research + primary review + 3 judges) $0.38
Fully loaded (inference + infra amortization) $0.47

9.2 v3 Full Pipeline

Component Contribution
Base pipeline (research + primary review) carried forward from v2 baseline
Specialist judges (up to 7-judge panel) incremental per additional judge
Market simulation engine fixed per-run compute cost
Corpus write and indexing marginal, per-review
Infra amortization Caddy/API/WeasyPrint/cron overhead spread across run volume
Total v3 full pipeline $0.86/run

9.3 Per-Tier Economics

Tier Price Cost per run Margin
Free $0/mo (1 review/mo) $0.07 (single-reviewer, no full panel) N/A (loss-leader / funnel)
Pro $79/mo (20 reviews) $0.86/run 78%
Enterprise $499/mo (100 reviews) $1.05/run 79%
White-Label from $1,999/mo (unlimited cohort) $1.05-$1.20/run 75-82%

Free tier deliberately runs a reduced single-reviewer pipeline (no full judge panel, score summary only, corpus percentile teaser) rather than the full pipeline at a loss per unit; this keeps the funnel economically sane while still giving prospective customers a real taste of the product.

9.4 Cost Assumptions and Optimization Levers

  • AI cost deflation assumption: underlying inference costs are modeled to decline 15-20% annually based on historical trend, which is factored into margin projections for Enterprise and White-Label tiers over a multi-year horizon. This is an assumption, not a guarantee, and margin models should be re-validated against actual provider pricing at each planning cycle.
  • Judge panel cost optimization: judge count is a tunable parameter, not a fixed constant. The system supports running fewer judges for cost-sensitive contexts (e.g., Free tier) and more for Enterprise/White-Label, and the accuracy-vs-cost ratio (accuracy dashboard metrics against per-run cost) is the intended basis for deciding whether panel size should grow further, hold, or shrink for a given tier. The three-judge minimum-viable panel and the seven-judge maximal panel bound this tradeoff space; see 7.6 and 8.2 for the operational floor (minimum quorum of 2).

10. Deployment Topology

10.1 Single-Server Architecture

The platform runs as a single-server deployment: one host running the API server process, the Caddy reverse proxy, the SQLite corpus database file, and the WeasyPrint PDF generation process. This is an intentional simplicity choice given current scale; there is no distributed consensus, no service mesh, and no multi-region failover at this stage.

flowchart TB
    subgraph Host[Single Application Host]
        Caddy[Caddy v2 — TLS + routing + static /reports/]
        API[API Server Process]
        DB[(SQLite Corpus DB)]
        PDF[WeasyPrint Process]
        Cron[Prediction Tracking Cron]
    end
    Internet -->|HTTPS| Caddy
    Caddy --> API
    API --> DB
    API --> PDF
    Cron --> DB

10.2 DNS and TLS

  • Public entry point is a standard DNS A/AAAA record pointed at the application host.
  • TLS termination and certificate issuance/renewal are handled automatically by Caddy v2; no manual certificate management step exists in the deploy path.

10.3 Build and Deploy Pipeline

  • Documentation (this site): built with MkDocs (Material theme) and deployed as static output.
  • API and pipeline code: deployed via a standard git-push-triggered pipeline. The sanitization gate (Section 2.6) runs as a required check in this pipeline for any change touching public-facing output paths; a gate failure blocks the deploy outright.
  • Deploys to the API layer and deploys to the docs site are independent pipelines and can ship on separate cadences.

10.4 Backup Strategy

  • Nightly backups to off-host object storage (S3-compatible), covering the full application state including the corpus database file.
  • The corpus is treated as first-class backup content, not an afterthought: it is the platform's accumulated institutional knowledge (Tier 1 cumulative intelligence) and its loss would silently degrade corpus search, prediction tracking, and accuracy scoring without necessarily causing an immediately visible outage.
  • Restore procedure for corpus corruption is covered in Failure Modes 8.5.

11. Operations

11.1 Key Metrics

Metric Description
Review success rate Fraction of submitted reviews reaching status = complete vs. failed
API latency (p95) 95th percentile response time for synchronous API endpoints (submission, polling, dashboard)
PDF generation failure rate Fraction of reviews requiring PDF retry or falling back to plaintext (Section 8.4)
Model/architecture latency Per-role latency for research, primary review, and each judge, used to catch a specific provider degrading before it causes a full timeout
Panel diversity health agreement_rate / diversity_score trend across the judge panel over time (Section 7.7)
Corpus growth rate Reviews indexed per period, used to sanity-check corpus and backup sizing assumptions

11.2 Alerting Thresholds

  • Review success rate dropping below an agreed operational floor over a rolling window triggers investigation (distinguish between a systemic pipeline issue and a single provider outage per Failure Mode 8.6).
  • Any sanitization gate block on a production deploy or report generation attempt (8.7) should notify the operator immediately; this is a security-relevant event even though it is functioning as designed.
  • Sustained upward drift in panel agreement_rate beyond a defined band triggers a manual review of judge configuration, per the training-data recursion and consensus-drift concerns in 7.6 and 7.7.
  • PDF fallback-to-plaintext events (8.4) should alert on any occurrence, not just above a threshold, since they represent a customer-visible degradation of the deliverable.

11.3 Backup Schedule

  • Nightly full backup of application state (Section 10.4).
  • Backup integrity should be spot-verified on a periodic cadence (e.g., a scheduled restore-to-scratch test), rather than assumed functional purely because the backup job reports success.

11.4 Corpus Health Checks

  • Periodic verification that corpus_record entries stay in sync with their source review rows (no orphaned or stale index entries), particularly after any manual corpus maintenance operation.
  • Periodic audit that corpus_opt_out reviews are in fact excluded from search and comparison results, as a direct verification of the confidentiality control in 7.4 rather than trusting the flag's existence alone.
  • Prediction tracking cron runs (Section 2.8) should be monitored for completion; a silently-failing cron would cause prediction.actual_outcome to remain perpetually pending, quietly degrading the value of Tier 1 cumulative intelligence without any loud failure signal.

Appendix A: Implementation Roadmap Reference

For planning and evaluation purposes, the v3 build is staged as follows:

Phase Weeks Scope
Phase 0 1-3 Sanitization gate, automated PDF generation, corpus schema and backfill
Phase 1 4-7 Corpus search, prediction tracking cron, accuracy scoring (read-only, dashboard only)
Phase 2 8-12 Vertical-specific templates, adversarial red-team per vertical, market simulation engine
Phase 3 13-18 New specialist judges (reasoning-verification, execution-feasibility, market-reality), public shareable reports, review-as-a-service API beta
Phase 4 19-22 Live accuracy-based judge weighting, general availability, White-Label pilots

This staging is deliberate about sequencing risk: infrastructure hardening (sanitization, automated PDF, corpus foundation) ships before any feature that depends on it; accuracy scoring ships read-only well before it is trusted to influence a live verdict; and the highest-blast-radius change (letting historical accuracy actually move a verdict) is the very last thing to go live.

Reference in New Issue View Git Blame Copy Permalink