842 lines
86 KiB
Markdown
842 lines
86 KiB
Markdown
# VerdictTank — Technical Architecture
|
|
|
|
> Standalone technical reference for the VerdictTank v3 platform architecture. This document covers component design, data models, API contracts, pipeline mechanics, and operational procedures for developers and technical evaluators.
|
|
|
|
## 1. Overview
|
|
|
|
VerdictTank is an automated business proposal review platform. A user submits a proposal document; the system runs it through a multi-agent review pipeline and returns a scored, evidence-backed verdict as a branded PDF report, typically within 3 to 15 minutes.
|
|
|
|
The core thesis is straightforward: a single reviewer opinion is a data point, not a verdict. VerdictTank instead runs a proposal through a sequence of independent review stages built on distinct model architectures, then aggregates their conclusions into a majority-rules decision. The goal is a brutally honest, evidence-grounded assessment that is hard to game and cheap enough to run at scale.
|
|
|
|
### 1.1 The Pipeline Concept
|
|
|
|
Every review passes through three logical phases:
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
A[Proposal Submitted] --> B[Phase 1: Research]
|
|
B --> C[Phase 2: Primary Review]
|
|
C --> D[Phase 3: Parallel Judges]
|
|
D --> E[Majority Verdict]
|
|
E --> F[PDF Report]
|
|
```
|
|
|
|
- **Phase 1 — Research**: A dedicated research agent independently verifies factual claims in the proposal (market size figures, competitor claims, pricing comparisons), checks domain and trademark availability, and compiles a citation-backed research brief.
|
|
- **Phase 2 — Primary Review**: A primary reviewer performs a structured, ten-dimension critique of the proposal, informed by the research brief, and produces numeric scores plus qualitative findings.
|
|
- **Phase 3 — Parallel Cross-Check**: Multiple independent judges, each built on a different underlying architecture, review the proposal and the primary reviewer's findings in parallel. Each judge issues its own verdict, confidence level, and list of agreements/disagreements with the primary review.
|
|
- **Majority Verdict**: Verdicts from all judges are aggregated. A simple majority rule determines the final published verdict (Go / Conditional Go / No-Go), with dissent explicitly preserved rather than smoothed over.
|
|
- **PDF Report**: All findings are compiled into an 11-section branded PDF via an automated document generation pipeline.
|
|
|
|
### 1.2 v2 to v3 Evolution
|
|
|
|
The live v2 platform runs five review stages across the three phases described above at roughly $0.47 per run, with a three-judge cross-check panel and a 3-9 minute turnaround. v3 is a proposed upgrade layered on top of that same core pipeline, organized into three enhancement tiers plus an infrastructure prerequisite phase:
|
|
|
|
| Tier | Theme | Additions |
|
|
|---|---|---|
|
|
| Phase 0 | Infrastructure prerequisites | Automated sanitization gate, automated PDF generation pipeline, embedding subsystem, corpus schema and backfill |
|
|
| Tier 1 | Cumulative intelligence | Review corpus database, prediction-vs-outcome tracking, reviewer accuracy scoring |
|
|
| Tier 2 | Deeper analysis | Vertical-specific templates, adversarial red-team per vertical, market simulation engine (Phase 2, sequenced before Phase 2 primary review per §5.4) |
|
|
| Tier 3 | Product and distribution | Public-facing shareable reports, review-as-a-service API, competitive corpus comparisons |
|
|
|
|
The net effect is a system that remembers its own history (corpus), learns whether its verdicts were correct (prediction tracking), specializes by industry vertical, and exposes itself as a product surface beyond a single web form.
|
|
|
|
!!! info "Pipeline verdict on the v3 proposal itself"
|
|
The v3 proposal was run through the VerdictTank pipeline prior to build. Verdict: **3/3 Unanimous Conditional Go**. Conditions attached to that verdict are reflected throughout the Security Model and Failure Modes sections below (notably around corpus confidentiality and sanitization gate rigor).
|
|
|
|
!!! info "Architecture review verdict"
|
|
This document itself was subjected to a 3-judge Conductor architecture review on 2026-08-10 (Opus 4.8 operational realism, Claude security and failure modes, Gemini Pro completeness and coverage). All three judges returned **CONDITIONAL GO**. The 18 merged conditions from that review are addressed throughout this revision; see **Appendix B: Review Addendum** for the full record.
|
|
|
|
---
|
|
|
|
## 2. Component Architecture
|
|
|
|
VerdictTank is deliberately a small number of well-defined components rather than a large microservice mesh. Each component has a single responsibility and a narrow interface to its neighbors.
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
subgraph Edge
|
|
Caddy[Reverse Proxy — Caddy v2]
|
|
Auth[Auth Gateway — Stack Auth / API Keys]
|
|
end
|
|
subgraph App
|
|
API[Core API Server — FastAPI]
|
|
Orchestrator[Review Pipeline Engine]
|
|
Sim[Market Simulation Engine]
|
|
Sanitize[Sanitization Gate]
|
|
Embed[Embedding Service]
|
|
end
|
|
subgraph Data
|
|
DB[(Corpus Database — SQLite, WAL mode)]
|
|
PDFGen[PDF Generation Engine — WeasyPrint]
|
|
Reports[/reports/ static files/]
|
|
end
|
|
subgraph Scheduled
|
|
Cron[Prediction Tracking Cron]
|
|
end
|
|
subgraph External
|
|
Billing[Stripe Billing]
|
|
Notify[Notification Service]
|
|
end
|
|
|
|
Caddy --> Auth
|
|
Auth --> API
|
|
Caddy --> Reports
|
|
API --> Orchestrator
|
|
Orchestrator --> Sim
|
|
Orchestrator --> DB
|
|
Orchestrator --> Embed
|
|
Embed --> DB
|
|
Orchestrator --> PDFGen
|
|
PDFGen --> Reports
|
|
Orchestrator --> Notify
|
|
API --> Billing
|
|
Sanitize -.blocks.-> Reports
|
|
Cron --> DB
|
|
```
|
|
|
|
### 2.1 Core API Server
|
|
|
|
- **Stack**: FastAPI on Python 3.11+, served via an ASGI worker process.
|
|
- **Role**: Public-facing HTTP interface. Accepts review submissions, exposes polling and retrieval endpoints, enforces authentication and rate limits, and hands work off to the review pipeline engine.
|
|
- **Responsibilities**: request validation, tier entitlement checks, API key scoping, response serialization. Contains no review logic itself; it is a thin contract layer over the orchestrator.
|
|
|
|
### 2.2 Reverse Proxy Layer
|
|
|
|
- **Stack**: Caddy v2.
|
|
- **Role**: TLS termination, HTTP-to-HTTPS redirection, automatic certificate renewal, request routing to the API process, and direct static file serving for the `/reports/` path (completed PDFs and public share assets).
|
|
- **Why Caddy**: automatic TLS, minimal configuration surface, and a config format simple enough to keep under version control alongside the rest of the deploy.
|
|
- **Directory listing is explicitly disabled** on `/reports/` — see Security Model, 7.3.
|
|
|
|
### 2.3 Review Pipeline Engine
|
|
|
|
- **Role**: The orchestrator. Sequences the five-plus agent phases for a given review, manages timeouts per stage, handles degraded-mode fallback when a judge is unavailable, and persists intermediate state so a review can be resumed or inspected mid-flight.
|
|
- **Design**: implemented as a state machine keyed by `review_id`. Each phase transition writes its output to the review record before advancing, so a crash mid-pipeline loses at most the in-flight stage rather than the whole review.
|
|
- **Concurrency**: Phase 3 judges run concurrently within a single review (not sequentially) since they are independent of one another by design; this is what keeps total turnaround in the minutes range rather than compounding each judge's latency serially. Across reviews, concurrency is bounded by a single-worker queue serialization model; see 2.4 and 5.6.
|
|
|
|
### 2.4 Corpus Database
|
|
|
|
- **Stack**: SQLite, single file, with the corpus schema described in Section 3. **WAL (Write-Ahead Logging) mode is enabled explicitly** (`PRAGMA journal_mode=WAL;`) at database initialization, not left at the default rollback-journal mode. WAL allows concurrent readers (e.g., a poll request per §4.2) to proceed while a writer (e.g., a judge verdict write) is in flight, which is a hard requirement given the read/write mix described below.
|
|
- **Role**: System of record for every review ever run, plus derived tables for predictions, outcomes, and per-judge accuracy scores.
|
|
- **Why SQLite**: review volume at current and near-term projected scale does not warrant a networked database; SQLite gives transactional integrity, zero operational overhead, and trivial backup (file copy) for a workload that is read-heavy and write-light per unit time, provided writes are serialized, see below.
|
|
- **Concurrent-writer ceiling**: SQLite, even in WAL mode, permits exactly one writer at a time; concurrent write attempts beyond that one writer either block or raise `SQLITE_BUSY`. This is a hard ceiling on the single-file design, not a tunable parameter. The realistic comfort zone for this write pattern (parallel judges within a review, plus cron, plus API writes) is a single concurrent writer per review, with reviews themselves serialized. See 5.6 for how the orchestrator enforces this.
|
|
- **Single-worker queue serialization**: the orchestrator runs as a single-worker process with respect to write-heavy phases. Multiple submitted reviews are queued, not run concurrently against the database. Within one review, Phase 3 judges still run their model inference calls concurrently (the expensive, slow part), but their writes to `judge_verdict` are serialized through the orchestrator's single write path rather than issued as N simultaneous transactions. This converts "N judges racing to write" into "N judges racing to finish, one write queue," which eliminates the write-contention failure mode described in 8.8 for the common case.
|
|
- **Growth path**: schema is written with an eye toward a future migration to a networked engine. The trigger for that migration is write concurrency, not corpus size — a single SQLite file with WAL mode and single-worker serialization comfortably holds many gigabytes of review history; it does not comfortably hold multiple simultaneous writers hammering it at once. See 5.7 (Corpus Scale Threshold) for the concrete numeric ceiling. The one deliberate exception to "no SQLite-specific extensions" is the embedding similarity mechanism in 2.9, which is called out explicitly rather than left as an implicit contradiction.
|
|
|
|
### 2.5 PDF Generation Engine
|
|
|
|
- **Stack**: WeasyPrint, driven off an HTML template populated from the structured review JSON.
|
|
- **Role**: Converts the completed review record into the branded 11-section PDF report. See Section 6 for the full pipeline.
|
|
- **Validation**: every generated PDF passes through a post-generation validation script before being written to `/reports/` — see 6.3.
|
|
- **Native dependency and resource constraints**: see 10.5 for the required system packages and 8.9 for memory/timeout limits applied to the WeasyPrint process.
|
|
|
|
### 2.6 Automated Sanitization Gate
|
|
|
|
- **Role**: A pre-deploy content scanner that runs against all output destined for any public surface (shareable reports, OG images, public API responses, docs). It scans for architecture-identifying strings (model/vendor names) and internal infrastructure references, and blocks the deploy if a match is found rather than silently redacting.
|
|
- **Detection layers**: two layers, not one. Layer 1 is string matching against a maintained list of architecture-identifying strings and internal infrastructure references, as in v2. Layer 2 (v3 addition) is behavioral fingerprint detection: a maintained regex pattern bank targeting self-identification phrases common across model families ("as a language model," "my training data includes," "I cannot browse the internet," "my knowledge cutoff is"), capability-boundary language, and training-cutoff date patterns. Layer 2 exists because architecture identity leaks far more often through phrasing patterns than through an accidental vendor name; see 8.12 and 7.5 for the full specification.
|
|
- **Rationale**: the platform's differentiator is cross-vendor architecture diversity; leaking which specific architectures are in the panel undermines both competitive position and the "independent judges" framing that gives the majority verdict its credibility.
|
|
- **Placement in CI**: runs as a required check in the deploy pipeline, not as a runtime filter. A failure here is a build failure, not a logged warning. It also runs per-report at generation time (6.3). See 7.5 and 8.7.
|
|
- **False-positive handling**: the gate remains blocking by default. A documented human-override procedure exists for confirmed false positives; see 8.12 and 7.5.
|
|
|
|
### 2.7 Market Simulation Engine
|
|
|
|
- **Role**: v3 Tier 2 addition. Given a proposal's stated business model, generates a 12-month simulated trajectory: user acquisition curve, churn projection, and resulting revenue trajectory, rather than relying solely on the static financial figures the proposal itself provides.
|
|
- **Methodology**: agent-driven Monte Carlo simulation. The simulation agent is given the proposal's stated business model, the research brief's sourced market data, and a fixed set of explicitly documented distributional assumptions (e.g., CAC variance bounded by vertical-specific historical ranges, churn modeled as a bounded stochastic process informed by the research brief's market data, not invented ad hoc per run). Each simulation run records its assumption set alongside its output so a reviewer or judge can inspect what the simulation actually assumed, not just what it concluded. If a defensible assumption set cannot be maintained for a given vertical, the simulation is skipped for that review and the Financials dimension proceeds on the research brief and proposal figures alone, flagged as `simulation_skipped = true`.
|
|
- **Sequencing**: the Market Simulation Engine runs **before** the Primary Review (Phase 2), not concurrently with it. Its output is packaged as an appendix to the `ResearchBrief` (a `simulated_trajectory` field, see 3.2) so the Primary Reviewer consumes it as one more input to the Financials dimension, the same way it consumes verified claims and market data. This resolves the sequencing ambiguity between §1.1's phase diagram and the original placement of market simulation as a Phase 2-adjacent activity; simulation is logically part of Phase 1's research-gathering output, timed to complete before Phase 2 begins.
|
|
- **Independence**: intentionally decoupled from the proposal's own numbers so it cannot simply echo back what was submitted; it is a sanity-check model, not a validator of the proposal's math.
|
|
- **De-scope condition**: if, during Phase 0 build-out, the assumption set for a given vertical cannot be defensibly specified (no credible source for churn/CAC ranges in that vertical), that vertical is de-scoped from market simulation and moves to a Phase 3 backlog item rather than shipping with fabricated placeholder assumptions.
|
|
|
|
### 2.8 Prediction-vs-Outcome Tracking Cron
|
|
|
|
- **Role**: v3 Tier 1 addition. Every review that contains a flagged prediction (e.g., "this pricing tier will suppress conversion," "this GTM channel will not reach target CAC") is scheduled for automated re-check at T+90, T+180, and T+365 days.
|
|
- **Mechanics**: the cron job queries the corpus for predictions due for re-check, attempts to gather current public signal on the outcome (via the research agent's toolset), and records a best-effort actual-outcome assessment against the original prediction. Runs as a **systemd timer**, not a bare crontab entry, with `OnFailure=` wired to a notification unit so a failed run pages the operator rather than failing silently. See Data Model 3.6, Failure Modes 8.4 and 8.13, and 10.6.
|
|
|
|
### 2.9 Embedding Service
|
|
|
|
- **Role**: v3 Phase 0 addition. Generates the vector embedding stored in `corpus_record.embedding` (3.5) for corpus semantic search (4.4).
|
|
- **Model**: `all-MiniLM-L6-v2` via the `sentence-transformers` library. Chosen for a small footprint (384-dimension output, runs on CPU without a GPU dependency), which matches the single-server deployment model in Section 10.
|
|
- **Storage and query mechanism**: embeddings are stored as raw float32 BLOBs in the `corpus_record.embedding` column (SQLite has no native vector type). Similarity search is performed via row-level cosine similarity computed in application code using `numpy`, scanning the searchable subset of `corpus_record` rows at query time. This keeps the datastore free of SQLite-specific vector extensions, preserving the migration-portability property claimed in 2.4. At current and near-term corpus sizes (low thousands of rows), a full in-memory cosine scan against `numpy` arrays completes well within the API's latency budget; this is revisited if/when corpus size or query volume grows past the SQLite migration trigger in 5.7.
|
|
- **Exception acknowledgment**: this is the one place in the architecture that comes close to a SQLite-specific mechanism, since the embedding column and the scan logic are shaped around SQLite's lack of native vector support. It is explicitly documented here as the accepted exception to the "no SQLite-specific extensions" portability claim in 2.4, rather than left as an unstated contradiction. Should `sqlite-vec` or a similar extension be adopted later for query performance, that adoption must be documented here and the exception restated, not silently introduced.
|
|
- **Opt-out policy**: for reviews with `corpus_opt_out = 1`, embedding generation is **skipped entirely**, not generated-then-flagged-non-searchable. The orchestrator checks `review.corpus_opt_out` before invoking the embedding service; if set, no `corpus_record` row is created and no embedding is computed. See 3.5, 7.4, and 8.14.
|
|
- **Deletion on late opt-out**: if a customer opts out after a `corpus_record` row and embedding already exist (opt-out is a mutable setting on an existing review), the embedding BLOB and the `corpus_record` row are deleted, not merely flagged non-searchable. A deletion is logged to the audit trail described in 7.4.
|
|
|
|
### 2.10 Frontend
|
|
|
|
A thin web application, backed entirely by the API described in Section 4, handles proposal submission and results viewing. It has two primary surfaces: a submission form (proposal text entry, tier selection, vertical override) and a results dashboard (review status polling, verdict display, PDF download link, and, for Enterprise/White-Label accounts, corpus search and accuracy dashboard views gated per 4.4 and 4.5). The frontend holds no review logic and no direct database access; it is a client of the public API contract, which keeps the authentication and tenant-isolation boundaries described below as the single enforcement point regardless of which client (web app, API-key integrator, White-Label embed) is calling in.
|
|
|
|
### 2.11 Authentication
|
|
|
|
Two authentication mechanisms cover the platform's access patterns. **Bearer token (JWT) authentication** is used for the web application and any session-based interactive use, issued via `auth2.itpropartner.com`, a Stack Auth project shared with other IT Pro Partner properties. **API key authentication** is used for programmatic/Enterprise and White-Label integrations, per the scoping rules in 7.2. Both mechanisms terminate at the API server (2.1); the reverse proxy (2.2) does not perform authentication itself, only TLS termination and routing. A request without a valid JWT or API key for a tier-gated endpoint receives `401`.
|
|
|
|
### 2.12 Tenant Isolation
|
|
|
|
Tenant isolation is tier-dependent, not uniform. **White-Label** tier customers get a separate corpus partition per tenant, enforced via a `tenant_id` column added to `review` and `corpus_record` (see 3.1, 3.5); all corpus queries for a White-Label tenant are scoped to that tenant's partition and never cross into another tenant's data or the shared corpus. **Enterprise** tier is single-tenant by default: an Enterprise account's own reviews are private to that account, though its corpus search can still query the shared aggregate corpus per the aggregate-only rules in 7.4. **Pro** and **Free** tiers share the general corpus with search restrictions; individual review content is never exposed to another account regardless of tier, only aggregate metadata per 4.4.
|
|
|
|
### 2.13 Billing Integration
|
|
|
|
Stripe handles subscription billing for Pro, Enterprise, and White-Label tiers. Stripe webhooks (subscription created, updated, canceled, payment failed) drive quota enforcement: a webhook handler updates the account's entitlement record, and the API server's quota check (7.2) reads from that entitlement record rather than calling out to Stripe synchronously on every request. A payment failure webhook flips the account to a grace-period state before hard-blocking new submissions, giving the customer a window to update payment details without an abrupt cutoff.
|
|
|
|
### 2.14 Notification System
|
|
|
|
On review completion, the system sends an email via SMTP with the verdict summary and a link to the PDF report. This is the default notification path for all tiers. Enterprise tier customers may additionally configure a **webhook callback**: on review completion, the orchestrator POSTs the review summary payload (verdict, verdict_margin, review_id, pdf_url) to the customer-configured URL, with the same retry-with-backoff behavior as the model provider failover described in 5.8, capped at 3 attempts before falling back to email-only notification for that review.
|
|
|
|
---
|
|
|
|
## 3. Data Model
|
|
|
|
All primary entities below are persisted in the corpus database. Fields marked `v3` are new relative to the v2 schema.
|
|
|
|
### 3.1 Review Record
|
|
|
|
The top-level entity representing one submitted proposal and its full review lifecycle.
|
|
|
|
```sql
|
|
CREATE TABLE review (
|
|
id TEXT PRIMARY KEY, -- UUID (uuid4, see 7.3)
|
|
tenant_id TEXT, -- v3: White-Label tenant partition, NULL for non-White-Label
|
|
client_name TEXT,
|
|
proposal_text TEXT NOT NULL,
|
|
proposal_hash TEXT NOT NULL, -- dedup / integrity check, see 5.9
|
|
proposal_word_count INTEGER, -- v3: enforced against 25K word hard cap, see 5.9
|
|
language_detected TEXT, -- v3: ISO 639-1, see 5.9
|
|
pricing_tier TEXT NOT NULL, -- free | pro | enterprise | white_label
|
|
vertical TEXT, -- v3: auto-classified (saas, consulting, marketplace, fintech, ...)
|
|
status TEXT NOT NULL, -- queued | researching | reviewing | judging | aggregating | complete | failed
|
|
verdict TEXT, -- go | conditional_go | no_go
|
|
verdict_margin TEXT, -- e.g. "3/3 unanimous", "2/3 majority"
|
|
corpus_opt_out BOOLEAN DEFAULT 0, -- v3: excludes from corpus search / comparisons / embedding generation
|
|
report_url_expires_at TIMESTAMP, -- v3: optional expiration, default null = no expiry unless configured, see 7.3
|
|
created_at TIMESTAMP NOT NULL,
|
|
updated_at TIMESTAMP NOT NULL,
|
|
completed_at TIMESTAMP
|
|
);
|
|
```
|
|
|
|
### 3.2 ResearchBrief
|
|
|
|
Output of Phase 1. One-to-one with a review.
|
|
|
|
```sql
|
|
CREATE TABLE research_brief (
|
|
review_id TEXT PRIMARY KEY REFERENCES review(id),
|
|
verified_claims TEXT, -- JSON array: {claim, source_url, verified: bool}
|
|
discrepancies TEXT, -- JSON array: {claim, discrepancy_description}
|
|
competitor_analysis TEXT, -- JSON array: {name, url, positioning_summary}
|
|
domain_availability TEXT, -- JSON: {domain, available: bool, checked_at}
|
|
trademark_flags TEXT, -- JSON array: {mark, jurisdiction, conflict_summary}
|
|
market_data TEXT, -- JSON: sourced market size / growth figures with citations
|
|
simulated_trajectory TEXT, -- v3: JSON from Market Simulation Engine (2.7), appended before Phase 2, null if simulation_skipped
|
|
simulation_skipped BOOLEAN DEFAULT 0, -- v3: true if no defensible assumption set for vertical, see 2.7
|
|
citation_count INTEGER,
|
|
limited_citations_flag BOOLEAN DEFAULT 0, -- set when web verification degraded, see 8.3
|
|
created_at TIMESTAMP NOT NULL
|
|
);
|
|
```
|
|
|
|
### 3.3 CriticReview
|
|
|
|
Output of Phase 2 (Primary Reviewer). One-to-one with a review.
|
|
|
|
```sql
|
|
CREATE TABLE critic_review (
|
|
review_id TEXT PRIMARY KEY REFERENCES review(id),
|
|
score_name INTEGER, -- 1-10
|
|
score_pricing INTEGER,
|
|
score_pmf INTEGER,
|
|
score_competition INTEGER,
|
|
score_financials INTEGER,
|
|
score_gtm INTEGER,
|
|
score_risk INTEGER,
|
|
score_missing_elements INTEGER,
|
|
score_founder_fit INTEGER,
|
|
score_overall_verdict INTEGER,
|
|
fatal_flaws TEXT, -- JSON array of strings
|
|
strengths TEXT, -- JSON array of strings
|
|
blind_spots TEXT, -- JSON array of strings
|
|
verdict TEXT, -- go | conditional_go | no_go
|
|
conditions TEXT, -- JSON array: conditions attached to a conditional_go
|
|
created_at TIMESTAMP NOT NULL
|
|
);
|
|
```
|
|
|
|
### 3.4 JudgeVerdict
|
|
|
|
Output of Phase 3. One row per judge per review (one-to-many with a review).
|
|
|
|
```sql
|
|
CREATE TABLE judge_verdict (
|
|
id TEXT PRIMARY KEY,
|
|
review_id TEXT NOT NULL REFERENCES review(id),
|
|
judge_id TEXT NOT NULL, -- e.g. "reasoning_verification", "execution_feasibility", "market_reality"
|
|
architecture_family TEXT NOT NULL, -- opaque vendor/architecture tag, used only for diversity enforcement
|
|
verdict TEXT NOT NULL, -- go | conditional_go | no_go
|
|
confidence REAL, -- 0.0 - 1.0
|
|
agreements TEXT, -- JSON array: points of agreement with critic_review
|
|
disagreements TEXT, -- JSON array: points of disagreement
|
|
novel_insights TEXT, -- JSON array: findings not raised by prior phases
|
|
second_order_effects TEXT, -- JSON array: downstream consequences the judge flags
|
|
latency_ms INTEGER,
|
|
timed_out BOOLEAN DEFAULT 0,
|
|
created_at TIMESTAMP NOT NULL
|
|
);
|
|
```
|
|
|
|
### 3.5 Corpus Record (v3)
|
|
|
|
Indexing/search layer over completed reviews. One-to-one with a review, created only when embedding generation actually runs (see 2.9). No row is created for `corpus_opt_out = 1` reviews.
|
|
|
|
```sql
|
|
CREATE TABLE corpus_record (
|
|
review_id TEXT PRIMARY KEY REFERENCES review(id),
|
|
tenant_id TEXT, -- v3: mirrors review.tenant_id for White-Label partition scoping, see 2.12
|
|
embedding BLOB, -- v3: 384-dim float32 vector, all-MiniLM-L6-v2, see 2.9. Never populated for corpus_opt_out=1
|
|
vertical TEXT, -- indexed
|
|
verdict TEXT, -- indexed
|
|
overall_score_avg REAL, -- indexed, for percentile comparisons
|
|
summary_snippet TEXT, -- v3: fixed-length structural summary only, see 4.4
|
|
searchable BOOLEAN DEFAULT 1, -- respects corpus_opt_out; row does not exist at all if opted out at creation time
|
|
indexed_at TIMESTAMP NOT NULL
|
|
);
|
|
CREATE INDEX idx_corpus_vertical ON corpus_record(vertical);
|
|
CREATE INDEX idx_corpus_verdict ON corpus_record(verdict);
|
|
CREATE INDEX idx_corpus_tenant ON corpus_record(tenant_id);
|
|
```
|
|
|
|
### 3.6 Prediction / Outcome Record (v3)
|
|
|
|
```sql
|
|
CREATE TABLE prediction (
|
|
id TEXT PRIMARY KEY,
|
|
review_id TEXT NOT NULL REFERENCES review(id),
|
|
judge_id TEXT, -- nullable; may originate from critic_review instead
|
|
prediction_text TEXT NOT NULL,
|
|
expected_timeframe_days INTEGER NOT NULL, -- 90 | 180 | 365
|
|
check_due_at TIMESTAMP NOT NULL,
|
|
actual_outcome TEXT, -- filled by cron: materialized | not_materialized | inconclusive
|
|
outcome_confidence REAL, -- 0.0 - 1.0, cron's confidence in its own assessment
|
|
outcome_notes TEXT,
|
|
checked_at TIMESTAMP,
|
|
created_at TIMESTAMP NOT NULL
|
|
);
|
|
```
|
|
|
|
### 3.7 Accuracy Score (v3)
|
|
|
|
```sql
|
|
CREATE TABLE accuracy_score (
|
|
judge_id TEXT PRIMARY KEY,
|
|
running_accuracy REAL, -- fraction of predictions that materialized as flagged
|
|
agreement_rate REAL, -- fraction of reviews where this judge agreed with majority
|
|
diversity_score REAL, -- inverse of agreement_rate, monitored for consensus drift, see 7.6 and 9.4
|
|
total_predictions_scored INTEGER,
|
|
last_updated_at TIMESTAMP NOT NULL
|
|
);
|
|
```
|
|
|
|
!!! warning "Accuracy scoring is read-only until Phase 4"
|
|
Per the implementation roadmap, `accuracy_score` is computed and exposed on a dashboard starting in Phase 1, but it does **not** feed back into judge weighting until Phase 4. This separation is deliberate: it lets the team observe whether accuracy scoring itself is trustworthy (see Failure Modes 8.2, consensus-drift risk) before letting it influence live verdicts. Access to the dashboard itself is restricted to authenticated Enterprise-tier accounts at minimum, per 4.5 and 7.4; read-only status governs when the *data* affects verdicts, not who can *see* it.
|
|
|
|
### 3.8 Provider Failover Configuration (v3)
|
|
|
|
Not a corpus table; lives in a version-controlled config file (`config/failover.yaml`), loaded at orchestrator startup, not in the database. Kept out of the database deliberately so a provider outage can be worked around by an operator editing and redeploying config without a live DB write racing the outage itself.
|
|
|
|
```yaml
|
|
# config/failover.yaml
|
|
roles:
|
|
research_agent:
|
|
providers: [primary_vendor_a, secondary_vendor_b, tertiary_vendor_c]
|
|
retry_count: 3
|
|
backoff_base_seconds: 2 # exponential: 2s, 4s, 8s before escalating to next provider
|
|
primary_reviewer:
|
|
providers: [primary_vendor_a, secondary_vendor_b]
|
|
retry_count: 3
|
|
backoff_base_seconds: 2
|
|
judge_reasoning_verification:
|
|
providers: [primary_vendor_c, secondary_vendor_a]
|
|
retry_count: 3
|
|
backoff_base_seconds: 2
|
|
judge_execution_feasibility:
|
|
providers: [primary_vendor_b, secondary_vendor_c]
|
|
retry_count: 3
|
|
backoff_base_seconds: 2
|
|
judge_market_reality:
|
|
providers: [primary_vendor_a, secondary_vendor_b]
|
|
retry_count: 3
|
|
backoff_base_seconds: 2
|
|
```
|
|
|
|
Each pipeline role (research, primary reviewer, each judge) has an ordered provider list. On a call failure, the orchestrator retries against the same provider up to `retry_count` times with exponential backoff, then escalates to the next provider in the role's list. If every configured provider for a role is exhausted, the failure is handled per Failure Mode 8.6 (treated as a single-judge timeout/crash, or a full-panel failure if it cascades below quorum). See 5.8 for the operational walkthrough.
|
|
|
|
---
|
|
|
|
## 4. API Contract
|
|
|
|
All endpoints are served under `api.verdicttank.com` (or the equivalent path behind the reverse proxy). Authentication follows 2.11; unauthenticated requests to tier-gated endpoints receive `401`.
|
|
|
|
### 4.1 Submit Review
|
|
|
|
```
|
|
POST /v1/reviews
|
|
Authorization: Bearer <jwt> | X-API-Key: <key>
|
|
Content-Type: application/json
|
|
|
|
{
|
|
"proposal_text": "...",
|
|
"client_name": "optional",
|
|
"vertical_override": "optional",
|
|
"corpus_opt_out": false
|
|
}
|
|
```
|
|
|
|
Validates tier entitlement and quota (7.2), computes `proposal_hash` and checks for a duplicate submission (5.9), detects language and word count (5.9), then enqueues the review. Returns `202 Accepted` with a `review_id` and initial `status: queued`. Duplicate submissions do not re-enqueue; see 5.9.
|
|
|
|
### 4.2 Poll Review Status
|
|
|
|
```
|
|
GET /v1/reviews/{review_id}
|
|
Authorization: Bearer <jwt> | X-API-Key: <key>
|
|
```
|
|
|
|
Returns current `status`, and once `status = complete`, the verdict summary and `pdf_url`. **This endpoint is rate-limited independently from the submission endpoint** (7.7), both to protect against poll-based abuse and because the polling pattern itself carries a low-severity timing side-channel discussed in 8.15 and 9.5.
|
|
|
|
### 4.3 Retrieve Report
|
|
|
|
```
|
|
GET /reports/{unguessable_id}.pdf
|
|
```
|
|
|
|
Served directly by the reverse proxy as a static file (2.2). The `unguessable_id` is not the `review_id`; it is a separately generated `uuid4()` value (or `os.urandom`-derived token) stored on the review record specifically for the public-facing URL, so a leaked or guessed `review_id` from an authenticated context does not also expose the public report path. See 7.3 for the full security rationale, including the optional expiration window and access logging.
|
|
|
|
### 4.4 Corpus Search (v3)
|
|
|
|
```
|
|
GET /v1/corpus/search?vertical=saas&verdict=go&q=...
|
|
Authorization: Bearer <jwt> | X-API-Key: <key>
|
|
```
|
|
|
|
Available to Pro tier and above, scoped per the tenant isolation rules in 2.12: White-Label queries are scoped to `tenant_id`, Enterprise and Pro/Free query the shared aggregate corpus with the search restrictions in 7.4. `q` performs semantic search via the embedding similarity mechanism in 2.9. Results never include full proposal text or full review findings for another account's review; each result includes only `vertical`, `verdict`, `overall_score_avg`, and `summary_snippet`. **`summary_snippet` construction is strictly limited**: it is a fixed-length structural summary composed only from `vertical`, `verdict`, and a score-range bucket (e.g., "SaaS proposal, Conditional Go, score range 6-7"). It is never a derivative, excerpt, paraphrase, or embedding-nearest-sentence of the original proposal text. This is a hard construction rule, not a length limit on an otherwise-free-form summary; see 8.14 for the confidentiality rationale.
|
|
|
|
### 4.5 Accuracy Dashboard (v3)
|
|
|
|
```
|
|
GET /v1/accuracy/dashboard
|
|
Authorization: Bearer <jwt> | X-API-Key: <key>
|
|
```
|
|
|
|
**Restricted to authenticated Enterprise-tier accounts and above** (Enterprise, White-Label). Free and Pro tiers receive `403` on this endpoint; they have no access to per-judge accuracy data, running accuracy scores, or agreement-rate figures. Rationale: per-judge accuracy is competitively sensitive (it effectively ranks the underlying architectures against each other) and is still in the read-only observation period described in 3.7's warning callout.
|
|
|
|
### 4.6 Webhook Registration (Enterprise, v3)
|
|
|
|
```
|
|
POST /v1/webhooks
|
|
Authorization: Bearer <jwt> | X-API-Key: <key>
|
|
|
|
{ "callback_url": "https://customer.example.com/verdicttank-callback" }
|
|
```
|
|
|
|
Registers the notification callback described in 2.14. Enterprise tier only.
|
|
|
|
---
|
|
|
|
## 5. Pipeline Mechanics
|
|
|
|
### 5.1 Phase 1: Research
|
|
|
|
The research agent has tool access (web search, domain lookup) and produces the `research_brief` (3.2). It runs first because every downstream phase, including the Market Simulation Engine (2.7, 5.4), consumes its output.
|
|
|
|
### 5.2 Phase 2: Primary Review
|
|
|
|
The primary reviewer consumes the proposal text, the research brief (including any `simulated_trajectory`), and produces the `critic_review` (3.3): a ten-dimension score, fatal flaws, strengths, blind spots, and an initial verdict with any conditions attached.
|
|
|
|
### 5.3 Phase 3: Parallel Judges
|
|
|
|
Each judge receives the proposal, the research brief, and the critic review, and independently produces a `judge_verdict` (3.4). Judges run concurrently against their respective providers; writes are serialized per 2.4.
|
|
|
|
### 5.4 Market Simulation Sequencing
|
|
|
|
As established in 2.7, the Market Simulation Engine runs as the final step of Phase 1, after the research brief's core content is assembled but before the brief is handed to Phase 2. This keeps the phase diagram in 1.1 accurate: simulation is not a parallel, Phase-2-adjacent activity, it is the last research step. A review's `research_brief.simulated_trajectory` is therefore always either populated or explicitly null with `simulation_skipped = true` by the time Phase 2 begins.
|
|
|
|
### 5.5 Majority Aggregation
|
|
|
|
Once all judges report (or time out, per 8.6), the orchestrator counts verdicts. A simple majority (2-of-3 in the base panel) determines the published verdict. A split panel is reported as-is (e.g., "2/3 majority, one dissent") rather than resolved by a tiebreaker model; dissent is preserved, not smoothed over, as stated in 1.1.
|
|
|
|
### 5.6 Concurrent Review Queue Serialization
|
|
|
|
The orchestrator is a single-worker process with respect to review execution: **one review is actively processed at a time**. Additional submitted reviews wait in a FIFO queue rather than being dispatched concurrently against the shared SQLite corpus database. This is the direct operational consequence of the concurrent-writer ceiling described in 2.4: rather than fight SQLite's single-writer constraint with retry loops and `SQLITE_BUSY` handling scattered across every write path, the design accepts one review in flight at a time and queues the rest. Given per-review turnaround of 3-15 minutes, a shallow queue clears quickly under normal load; queue depth is capped at 50 pending reviews (5.9, 8.13), beyond which new submissions receive `429`.
|
|
|
|
### 5.7 Corpus Scale Threshold
|
|
|
|
The practical comfort ceiling for the current single-file SQLite design, combining WAL mode and single-worker write serialization, is approximately **5,000 reviews or 50 GB of corpus data**, whichever comes first. This figure is about storage and read-scan performance (particularly the embedding cosine-scan in 2.9, which is a linear scan over searchable rows), not a hard SQLite file-size limit; SQLite itself supports databases far larger than this. The trigger for migrating off SQLite is **write concurrency exceeding what single-worker serialization can absorb**, not corpus size in isolation; a corpus well past 5,000 reviews with light write volume is less urgent to migrate than a smaller corpus experiencing frequent `SQLITE_BUSY` errors under the queue model in 5.6.
|
|
|
|
### 5.8 Provider Failover Walkthrough
|
|
|
|
Given the configuration in 3.8: a judge call to its primary provider fails (timeout, 5xx, rate limit). The orchestrator retries the same provider up to 3 times with exponential backoff (2s, 4s, 8s). If all three retries fail, the orchestrator escalates to the role's secondary provider and repeats the retry sequence there. If a tertiary provider is configured for that role (as with the research agent) and the secondary also exhausts its retries, the orchestrator escalates once more. If every configured provider for a role is exhausted, that role's output is treated as a hard failure for the review, handled per Failure Mode 8.6: a single judge failing below quorum falls back to a 2-judge panel with a disclosure note on the report; the primary reviewer or research agent failing entirely halts the review and surfaces a `failed` status to the customer with automatic notification.
|
|
|
|
### 5.9 Edge Case Handling
|
|
|
|
The pipeline is deliberately explicit about the edges of its input space rather than silently degrading or erroring opaquely:
|
|
|
|
- **Proposals exceeding token limit**: proposals are hard-capped at **25,000 words (approximately 37,000 tokens)** at submission time. A proposal over that cap is rejected at `POST /v1/reviews` with a `413` and a clear message stating the cap. This is a hard cap, not a soft warning: chunked processing with summary aggregation was considered and rejected for Phase 1 because it would silently change what "the proposal" means to each pipeline stage; it remains a candidate technique for a future phase if genuinely long-form proposals become common, but is not implemented now.
|
|
- **Non-English proposals**: language is detected at submission time (`review.language_detected`). **Phase 1 supports English only.** A non-English proposal is rejected with a clear error message identifying the detected language and stating that English is currently required, rather than being silently run through the pipeline and producing a low-quality or nonsensical review.
|
|
- **Concurrent review queue**: per 5.6, the orchestrator processes one review at a time. The submission queue has a **depth limit of 50 pending reviews**; a submission when the queue is at capacity receives `429` with a `Retry-After` hint. Monitoring alerts at 80% of that capacity (40 pending), per 8.13.
|
|
- **Duplicate submission**: `proposal_hash` (a content hash of the normalized proposal text) is checked at submission time. A match against an existing review returns that review's existing link and status with a `"This proposal was reviewed on [date]"` notice, rather than re-running (and re-billing for) an identical review.
|
|
- **Corpus scale threshold**: see 5.7. Restated here because it is as much an edge case as a capacity fact: the system's behavior at and beyond that threshold is degraded query latency and elevated `SQLITE_BUSY` risk, not silent data loss, and the migration trigger is write contention, not size alone.
|
|
|
|
---
|
|
|
|
## 6. PDF Generation
|
|
|
|
### 6.1 Report Structure (11 Sections)
|
|
|
|
| # | Section | Content |
|
|
|---|---|---|
|
|
| 1 | Cover | Client name, proposal title, verdict badge, date |
|
|
| 2 | Executive Summary | One-paragraph synthesis of the majority verdict and its rationale |
|
|
| 3 | Score Table | All ten `CriticReview` dimension scores, tabulated |
|
|
| 4 | Research | Verified claims, discrepancies, competitor analysis, domain/trademark findings, market simulation summary when applicable |
|
|
| 5 | Fatal Flaws | Enumerated list from `CriticReview.fatal_flaws`, cross-referenced against judge disagreements |
|
|
| 6 | Action Plan | Concrete, prioritized remediation steps derived from fatal flaws and conditions |
|
|
| 7 | Judge Cards | One card per judge: verdict, confidence, headline finding |
|
|
| 8 | Judge Notes | Full agreements/disagreements/novel insights/second-order effects per judge |
|
|
| 9 | Vertical Analysis (v3) | Vertical-specific findings and red-team results, when applicable |
|
|
| 10 | Disclaimer | Standard liability and methodology disclaimer language |
|
|
| 11 | Citations | Full source list from the research brief |
|
|
|
|
### 6.2 Generation Pipeline
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
A[Completed Review JSON] --> B[Template Fill]
|
|
B --> C[Rendered HTML]
|
|
C --> D[WeasyPrint Render]
|
|
D --> E[Draft PDF]
|
|
E --> F[Validation Script]
|
|
F -->|pass| G[Written to /reports/]
|
|
F -->|fail| H[Retry Once]
|
|
H -->|fail again| I[Plaintext Fallback]
|
|
```
|
|
|
|
1. The completed review record (all tables joined) is serialized to a single JSON payload.
|
|
2. That payload fills a Jinja-style HTML template implementing the 11-section structure, with conditional blocks for vertical-specific sections.
|
|
3. The rendered HTML is passed to WeasyPrint, which produces the PDF binary directly from HTML/CSS, with no intermediate manual step. This step runs under a memory cap and a hard timeout; see 8.9 and 10.5.
|
|
4. The draft PDF is passed through the post-generation validation script before being written to the public `/reports/` path.
|
|
|
|
### 6.3 Post-Generation Validation Script
|
|
|
|
Every generated PDF must pass all of the following checks before being served:
|
|
|
|
- **Em dash / double-hyphen scan**: the report's prose must not contain em dashes or double hyphens (house style rule enforced automatically, not just at prompt level).
|
|
- **Section presence check**: all 11 (or 10, if vertical analysis does not apply) expected sections must be present and non-empty.
|
|
- **Page count minimum**: the rendered PDF must meet a minimum page count threshold; a report that renders suspiciously short indicates a template fill failure upstream.
|
|
- **Sanitization scan**: the same two-layer architecture-name and behavioral-fingerprint scan used by the deploy-time sanitization gate (Section 2.6, expanded in 7.5 and 8.12) also runs here, per-report, before public write.
|
|
|
|
A validation failure triggers one automatic retry of the full generation pipeline. If the retry also fails, the system falls back to a plaintext summary (see Failure Modes 8.4) rather than serving a broken or non-compliant PDF.
|
|
|
|
### 6.4 Automated vs. Manual Generation
|
|
|
|
The v2 baseline generated PDFs through a partially manual per-review process. The Phase 0 infrastructure prerequisite scripts this end-to-end: submission to PDF delivery requires zero manual intervention under normal operation. Manual generation remains available as an operator-invoked fallback tool for support cases (e.g., regenerating a report after a template fix), but is not part of the default customer-facing path.
|
|
|
|
### 6.5 Resource Constraints on the Render Step
|
|
|
|
WeasyPrint's HTML/CSS-to-PDF render is the single most resource-intensive step in the generation pipeline, particularly for reports with large tables (Judge Notes, Citations) or long proposal text echoed into the Research section. The render step runs with a hard memory cap and a hard wall-clock timeout, not unbounded; see 8.9 for the specific limits and the fallback behavior when either is exceeded, and 10.5 for the native OS dependencies the render step requires.
|
|
|
|
---
|
|
|
|
## 7. Security Model
|
|
|
|
### 7.1 Prompt Injection via Proposal Text
|
|
|
|
Proposal text is user-supplied free text and is treated as untrusted input throughout the pipeline. The previous framing of this section described the claim extractor as a filter that runs before agent exposure; that framing understated the actual exposure surface. **The claim extractor itself is an LLM-driven step and is therefore the first point of exposure to untrusted proposal text, not a pre-exposure filter that keeps injected content away from a model.** The claim extractor reads and reasons over the raw proposal text directly. Defense against injection is therefore layered, not front-loaded into a single filtering stage:
|
|
|
|
- **Pre-extraction input sanitization**: before the proposal text reaches any LLM, including the claim extractor, a regex-based sanitization pass strips or neutralizes common injection patterns: role-override attempts ("ignore previous instructions", "you are now", "disregard the above"), fake system-turn markers ("system:", "assistant:", "### instruction"), and markdown code-block injection attempts that try to smuggle instructions as fenced code intended for a different rendering context. This is a blunt, pattern-based first pass, not a semantic understanding of intent; it exists to raise the cost of the most common injection techniques, not to guarantee immunity to novel ones.
|
|
- **Structured parsing before further agent exposure**: once past sanitization, the claim-extraction step parses the proposal into discrete structured claims before any downstream agent (primary reviewer, judges) reasons over it as a single blob, reducing the surface for injected instructions to be interpreted as system-level directives by those later stages, even though the extractor itself remains exposed to the raw text.
|
|
- **Role-boundary reinforcement**: every agent's instruction template explicitly frames proposal content as data to be evaluated, not instructions to be followed, and this framing is tested as part of the sanitization gate's broader remit.
|
|
- **Output re-validation**: the sanitization gate and PDF validation script both scan final output for signs that instructions embedded in a proposal leaked into the report's own voice or structure.
|
|
- **Output behavioral anomaly detection (v3, supplementary signal)**: beyond string and pattern matching on output, the system tracks statistical outliers in scoring and verdict metadata across the review population: score distributions that deviate sharply from a proposal's vertical/tier peer group, judges converging to unusually high or identical confidence values, or verdict metadata that does not match the shape of a normal review. These signals do not block a review on their own; they queue a review for manual inspection. This is a supplementary, lagging detection layer on top of the preventive layers above, not a replacement for them.
|
|
- **Red-team testing protocol (Phase 0 deliverable)**: a documented, repeatable set of adversarial proposal inputs, covering the injection categories above plus proposal-text-embedded attempts to extract architecture identity, internal infrastructure details, or system prompts, is run against the pipeline before Phase 0 is considered complete, and re-run against each subsequent phase that changes agent prompting or the sanitization gate's pattern bank. Red-team results and any newly discovered bypass techniques are fed back into the sanitization pattern bank (2.6, 7.5, 8.12) as part of the false-positive/pattern-update procedure.
|
|
|
|
!!! warning "This is defense in depth, not a guarantee"
|
|
No layer above claims to make the claim extractor immune to injected content it is directly exposed to. The combination of pre-extraction sanitization, structured downstream parsing, output re-validation, and anomaly detection is designed to make successful injection difficult and, when it does occur, detectable after the fact, not to make the exposure itself disappear.
|
|
|
|
### 7.2 API Key Management
|
|
|
|
- API keys issued to Enterprise and White-Label tier customers are **scoped**: a key is bound to a specific account and tier, and cannot access corpus search or endpoints beyond what that tier entitles.
|
|
- Each key carries a **usage cap** matching the account's plan (review count per billing period); the API server enforces the cap at the request layer, returning `429` once exceeded, independent of any downstream billing reconciliation.
|
|
- Keys are revocable individually without affecting other keys on the same account, supporting key rotation without downtime.
|
|
|
|
### 7.3 Report Access Control
|
|
|
|
- Completed PDFs and public share assets live under `/reports/`, served directly by the reverse proxy as static files.
|
|
- **Directory listing is disabled** on this path at the proxy level; a report is only retrievable by its specific, unguessable identifier-based URL.
|
|
- **Identifier generation**: the report URL token is generated via `uuid4()` or an equivalent `os.urandom`-derived value, chosen specifically for unguessability (128 bits of entropy, no sequential or timestamp-derived component). It is a distinct value from the internal `review_id`, per 4.3.
|
|
- **Optional expiration**: a report URL may be configured to expire after a set period, defaulting to **90 days** when expiration is enabled for a given tier or account. `review.report_url_expires_at` (3.1) carries this value; a request against an expired URL receives `410 Gone` with guidance to re-authenticate and request a fresh link through the authenticated dashboard.
|
|
- **Access logging**: every retrieval of a report, whether via the public unguessable URL or the authenticated `GET /api/verdicttank/review/{id}/pdf` path, is logged with timestamp, source IP, and requested identifier. This log is the primary detection mechanism for URL-guessing attempts or unexpected sharing patterns, given that the URL itself carries no authentication.
|
|
- Reports for non-Free tiers are not indexed or discoverable; only the customer with the corresponding `review_id` (and valid session/API key) can request the signed download link via `GET /api/verdicttank/review/{id}/pdf`.
|
|
|
|
### 7.4 Corpus Confidentiality
|
|
|
|
The corpus is the platform's most sensitive asset: it aggregates other companies' unreleased business proposals.
|
|
|
|
- **Encryption at rest** for the corpus database file.
|
|
- **Per-user opt-out**: `review.corpus_opt_out` lets any customer exclude their review from search indexing and comparisons entirely. Opt-out is enforced at the point of embedding generation itself, not after the fact: for `corpus_opt_out = 1` reviews, **no embedding is generated and no `corpus_record` row is created** (2.9, 3.5), rather than generating the embedding and merely flagging it non-searchable. A late opt-out on a review that already has a `corpus_record` triggers deletion of that row and its embedding, logged to this section's audit trail.
|
|
- **Raw database access model**: access to the raw corpus database file itself (as opposed to the API's scoped, filtered views of it) is **single-operator, all-or-nothing**, stated here as an explicit design assumption rather than left implicit. There is no row-level or column-level access control layer between an operator with filesystem access to the database and the full, unfiltered contents of every review ever submitted, including opted-out reviews' underlying `review` rows (opt-out removes a review from corpus search and embedding, it does not remove the review record itself, which remains needed for the customer's own report retrieval and billing history). This assumption is acceptable at current operational scale (one operator, one host) and must be explicitly revisited, not silently inherited, if the operator model changes (e.g., additional staff with database access, a managed-service tier with third-party operators).
|
|
- **Aggregate-only comparisons**: competitive comparison features (percentile vs. corpus median) expose only score distributions and vertical classification, never proposal text, to any account other than the review's owner.
|
|
- **`summary_snippet` construction rule**: as specified in 4.4, the `corpus_record.summary_snippet` field surfaced through corpus search is a fixed-length structural summary built only from `vertical`, `verdict`, and a score-range bucket. It is never generated as a derivative, excerpt, or embedding-nearest-sentence pull from the original proposal text. This closes a specific confidentiality gap: a "helpful" free-text summary generated from proposal content would risk leaking substantive proposal details through the back door of a search result, even with `corpus_opt_out` respected and encryption at rest in place.
|
|
- This was flagged as a named risk in the v3 proposal review (corpus confidentiality liability) and the above controls are the direct mitigation; see Failure Modes for the case where a control fails.
|
|
|
|
### 7.5 Sanitization Gate
|
|
|
|
Covered in detail in 2.6 and 6.3. Security framing: this is the platform's primary defense against leaking architecture-identifying details, behavioral self-identification patterns, or internal infrastructure references into any public-facing surface (shareable reports, OG images, docs, public API responses). It is deploy-blocking, not advisory, precisely because a leak here is a competitive and trust failure that is hard to walk back once a report has been shared publicly.
|
|
|
|
- **Layer 1 (string matching)**: a maintained list of architecture-identifying strings and internal infrastructure references, as in v2.
|
|
- **Layer 2 (behavioral fingerprint detection, v3)**: a maintained regex pattern bank targeting self-identification phrases that recur across model families regardless of the specific vendor name being present ("as a language model," "my training data includes," "I cannot browse the internet," "I don't have the ability to," "my knowledge cutoff is"), training-cutoff date patterns, and capability-boundary language. This layer exists because architecture identity leaks more often through phrasing habits than through an accidental vendor name, and a string-match list alone cannot catch a model describing its own limitations in generic but still identifying language.
|
|
- **False-positive procedure**: the gate remains **blocking by default**, with no change to that policy. When a block is believed to be a false positive, the procedure is: human review of the flagged content, and if confirmed as a false positive, the specific triggering pattern is added to an allowlist scoped as narrowly as possible (ideally to the exact phrase-in-context, not a broad pattern removal), the gate is re-run to confirm the deploy or report now passes, and a mandatory audit log entry records the pattern, the reviewer, the timestamp, and the justification. This procedure exists so that legitimate content is not permanently blocked by an overly broad pattern, without weakening the gate's default-blocking posture or leaving allowlist changes unaudited.
|
|
|
|
### 7.6 API Abuse Protections
|
|
|
|
- **Rate limiting** at the reverse proxy and API layers, tuned per tier, with the polling endpoint (`GET /v1/reviews/{review_id}`, 4.2) rate-limited **independently** from the submission endpoint (`POST /v1/reviews`, 4.1). These are different abuse surfaces: submission abuse is about volume and cost, polling abuse is about probing pipeline internals (see the timing oracle note below and in 9.5) and about scraping status data at a rate disproportionate to legitimate dashboard usage.
|
|
- **Free-tier submission content monitoring**: free-tier submissions are monitored for known-good probing patterns, patterns consistent with an account systematically testing pipeline behavior with minor input variations (near-duplicate proposals differing only in specific test phrases) rather than submitting genuine business proposals. This monitoring feeds manual account review, per the anomaly-detection policy below, not automated suspension.
|
|
- **Anomaly detection** on submission patterns: e.g., a burst of near-identical proposal submissions from one account, which could indicate an attempt to probe the pipeline's judge behavior or extract architecture information through differential prompting.
|
|
- Abuse detection findings feed into manual account review rather than automated suspension, to avoid false-positive lockouts on legitimate high-volume Enterprise/White-Label usage.
|
|
- **Timing oracle via polling (acknowledged low-severity risk)**: because a review passes through observable phase transitions (`status` field values in 3.1) at different, somewhat characteristic latencies, an attacker polling frequently could in principle infer something about pipeline composition (e.g., roughly how many phases exist, or that a particular phase took unusually long, suggesting a provider failover event per 5.8) purely from timing, without ever seeing pipeline internals directly. This is acknowledged explicitly as a low-severity side channel with no identified practical exploitation path at current scale; see 8.15 and 9.5. It is not actively defended against beyond the rate limiting above, and is called out here rather than left undocumented.
|
|
|
|
### 7.7 Training-Data Recursion Prevention
|
|
|
|
A structural risk unique to this kind of pipeline: if the panel's own review output were ever used, directly or indirectly, to further train or fine-tune models used by the panel itself, judge diversity would collapse over time as the panel converges on its own prior outputs.
|
|
|
|
- **"No training on API usage" contractual requirement**: every judge model provider under contract must have an active "no training on API usage" term in its API terms of service or a negotiated data processing agreement to that effect. This is a procurement and contract-management requirement, not a purely technical control, and it is tracked as a standing condition of each provider relationship, re-verified whenever a provider's terms of service change.
|
|
- **Volume caps** on any data pipeline that could plausibly feed review output back toward model training, as a defense-in-depth measure independent of the contractual requirement above.
|
|
- **Agreement-rate monitoring**: `accuracy_score.agreement_rate` and `diversity_score` are tracked over time specifically to detect a drift toward artificial consensus (judges agreeing with each other more, and more often, than architectural independence would predict). A sustained upward drift in agreement rate across the panel is treated as an operational signal to investigate, not just a marketing metric.
|
|
- **Honest limitation**: agreement-rate monitoring is **lagging detection**, not leading prevention. It can reveal that consensus has already drifted; it cannot prevent the drift from occurring, and it cannot fully distinguish organic convergence (the proposal genuinely warrants unanimous agreement) from recursion-driven convergence. Training-data recursion risk is therefore treated as **partially inherent to any multi-model pipeline** that depends on providers' training practices remaining as represented; the contractual requirement above is the primary control, and monitoring is a backstop, not a solution. This is restated in the Inherent Risks subsection, 8.14.
|
|
|
|
---
|
|
|
|
## 8. Failure Modes & Recovery
|
|
|
|
| # | Failure Mode | Impact | Recovery Procedure |
|
|
|---|---|---|---|
|
|
| 8.1 | Single judge timeout or crash | Panel drops to N-1 judges | Orchestrator proceeds automatically if N-1 still meets minimum quorum of 2; review and PDF are flagged as degraded-mode |
|
|
| 8.2 | All judges fail, or surviving judges fall below quorum | No valid verdict can be computed | Review is marked `failed`; customer is offered a free re-run at no charge against their quota |
|
|
| 8.3 | Research agent web verification failure | Citations incomplete or absent | Pipeline proceeds with `limited_citations_flag = 1`; flag is surfaced visibly in the final PDF's Research section rather than silently omitted |
|
|
| 8.4 | PDF generation failure (post-validation) | No compliant PDF produced | One automatic retry of the full generation pipeline; if retry also fails, deliver a plaintext summary fallback and flag the review for manual PDF regeneration |
|
|
| 8.5 | Corpus database corruption | Loss of search, prediction tracking, and accuracy scoring continuity | Restore from the most recent WAL checkpoint (Section 10.4, at most 15 minutes of writes lost); any reviews written between last checkpoint and corruption event are re-derived from review-record source data where still available |
|
|
| 8.6 | Model/architecture provider outage | One or more judges or the primary reviewer unavailable | Failover chain (3.8, 5.8) routes affected role through its configured provider list with retry and backoff; if every configured provider for that role is exhausted, treat as case 8.1 or 8.2 depending on scope |
|
|
| 8.7 | Sanitization gate failure (a scan match is found) | Deploy or report publication is blocked | This is by design: the gate blocks rather than warns. Operator must resolve the flagged content (redact/rephrase) before the deploy or report can proceed, or follow the false-positive procedure in 7.5/8.12 if the match is confirmed spurious. Failure is never silently bypassed |
|
|
| 8.8 | SQLite write contention under concurrent judges | Write attempts raise `SQLITE_BUSY` or block | WAL mode (2.4) plus single-worker queue serialization (2.4, 5.6) is the primary mitigation; within-review judge writes are serialized through the orchestrator's single write path rather than issued as N simultaneous transactions, which eliminates contention for the common case. A `SQLITE_BUSY` that still occurs is retried with backoff at the write layer before surfacing as an error |
|
|
| 8.9 | WeasyPrint memory exhaustion on large reports | Render process killed or hangs, report never produced | Render step runs under a 2GB memory cap (cgroups) and a 120-second timeout; a cap or timeout breach kills the render and triggers the automatic retry described in 8.4, and a second failure falls back to the plaintext summary rather than a partial or corrupted PDF |
|
|
| 8.10 | Sanitization gate false positive | Legitimate content blocked from deploy or publication | Human review confirms the false positive; the gate remains blocking by default (no change to 8.7 policy). A human override path exists specifically for confirmed false positives, requiring a mandatory audit log entry documenting the override, per 7.5/8.12 |
|
|
| 8.11 | Prediction cron silent failure | `prediction.actual_outcome` remains perpetually `pending`, quietly degrading Tier 1 cumulative intelligence with no loud failure signal | Cron runs as a systemd timer (2.8, 10.6) with `OnFailure=` wired to a notification unit, plus a separate missing-run detection check (a run that should have fired but did not, distinct from a run that fired and errored) that pages the operator on either condition |
|
|
| 8.12 | Confident-but-wrong judge analysis | A judge issues a high-confidence verdict that is substantively incorrect, with no technical signal distinguishing it from a correct high-confidence verdict | No technical prevention exists for this case; it is addressed as a product disclosure to users (verdicts are probabilistic assessments, not guarantees) rather than a solved engineering problem. Detection is via outlier analysis post-hoc (7.1's behavioral anomaly detection, and accuracy scoring's eventual outcome tracking in 2.8), not at verdict time |
|
|
| 8.13 | Concurrent review queue saturation | New submissions cannot be accepted | Queue cap of 50 pending reviews (5.6, 5.9); submissions beyond the cap receive `429` with `Retry-After`; a monitoring alert fires at 80% of capacity (40 pending) so the operator has lead time before the cap is actually hit |
|
|
|
|
!!! danger "Sanitization gate failures are not incidents to route around"
|
|
If the sanitization gate blocks a deploy or a report, the correct response is to fix the flagged content or follow the documented false-positive procedure, not to disable or bypass the gate to unblock a release. A bypass here directly reintroduces the leak risk the gate exists to prevent.
|
|
|
|
### 8.15 Attribution Difficulty in Prediction Tracking
|
|
|
|
Not a system failure in the crash sense, but a known limitation worth documenting alongside the other failure modes: the prediction-vs-outcome cron's `actual_outcome` assessment is inherently a correlation judgment, not a causal one. A flagged prediction ("this pricing tier will suppress conversion") that appears to materialize by T+180 may have done so for unrelated reasons. `outcome_confidence` on the `prediction` record exists specifically to carry this uncertainty forward rather than presenting the cron's assessment as ground truth; accuracy scoring calculations weight predictions by this confidence rather than treating every checked prediction as a binary hit/miss.
|
|
|
|
### 8.14 Inherent Risks
|
|
|
|
Three risks in this architecture do not have a full technical solution and are documented here explicitly as accepted, ongoing risk rather than as failure modes with a recovery procedure:
|
|
|
|
1. **Training-data recursion is partially inherent.** As stated in 7.7, the agreement-rate/diversity-score monitor is lagging detection, not leading prevention, and cannot fully distinguish organic consensus from recursion-driven consensus. The primary control is contractual (7.7's "no training on API usage" requirement), and no fully technical solution exists that guarantees judge independence indefinitely as long as the panel depends on third-party model providers whose training practices cannot be directly audited.
|
|
2. **Timing oracle via the polling endpoint.** As stated in 7.6, phase-transition timing observed through repeated polling could in principle reveal something about pipeline composition. This is assessed as low severity, and no practical exploitation vector has been identified: the information such an analysis could extract (rough phase count, occasional failover events) does not appear to translate into a meaningful competitive or security compromise at this time. It remains documented rather than dismissed, in case that assessment changes as the platform's usage or attacker sophistication increases.
|
|
3. **Concurrent queue saturation is a capacity design choice, not just a failure mode.** The 50-pending-review cap (5.6, 5.9, 8.13) is not an incidental limit that emerged from the SQLite single-writer constraint; it is a deliberate capacity decision for a single-server deployment. Raising it would require either accepting longer queue wait times under load, or the SQLite migration discussed in 5.7, not simply changing a config value. This is stated plainly so that "the queue cap is too low" is understood as a capacity-planning conversation, not a bug report.
|
|
|
|
---
|
|
|
|
## 9. Cost Model
|
|
|
|
### 9.1 v2 Baseline
|
|
|
|
| Component | Cost per run |
|
|
|---|---|
|
|
| Model inference (research + primary review + 3 judges) | $0.38 |
|
|
| Fully loaded (inference + infra amortization) | $0.47 |
|
|
|
|
### 9.2 v3 Full Pipeline
|
|
|
|
| Component | Contribution |
|
|
|---|---|
|
|
| Base pipeline (research + primary review) | carried forward from v2 baseline |
|
|
| Specialist judges (up to 7-judge panel) | incremental per additional judge |
|
|
| Market simulation engine | fixed per-run compute cost, incurred once per review ahead of Primary Review (5.2) |
|
|
| Embedding generation (opt-in reviews only) | marginal, per-review; skipped entirely for opt-out reviews (2.9) |
|
|
| Corpus write and indexing | marginal, per-review |
|
|
| Infra amortization | Caddy/API/WeasyPrint/cron overhead spread across run volume |
|
|
| **Total v3 full pipeline** | **$0.86/run** |
|
|
|
|
### 9.3 Per-Tier Economics
|
|
|
|
| Tier | Price | Cost per run | Margin |
|
|
|---|---|---|---|
|
|
| Free | $0/mo (1 review/mo) | $0.07 (single-reviewer, no full panel) | N/A (loss-leader / funnel) |
|
|
| Pro | $79/mo (20 reviews) | $0.86/run | 78% |
|
|
| Enterprise | $499/mo (100 reviews) | $1.05/run | 79% |
|
|
| White-Label | from $1,999/mo (unlimited cohort) | $1.05-$1.20/run | 75-82% |
|
|
|
|
Free tier deliberately runs a reduced single-reviewer pipeline (no full judge panel, score summary only, corpus percentile teaser) rather than the full pipeline at a loss per unit; this keeps the funnel economically sane while still giving prospective customers a real taste of the product.
|
|
|
|
### 9.4 Cost Assumptions and Optimization Levers
|
|
|
|
- **AI cost deflation assumption**: underlying inference costs are modeled to decline 15-20% annually based on historical trend, which is factored into margin projections for Enterprise and White-Label tiers over a multi-year horizon. This is an assumption, not a guarantee, and margin models should be re-validated against actual provider pricing at each planning cycle.
|
|
- **Judge panel cost optimization**: judge count is a tunable parameter, not a fixed constant. The system supports running fewer judges for cost-sensitive contexts (e.g., Free tier) and more for Enterprise/White-Label, and the accuracy-vs-cost ratio (accuracy dashboard metrics against per-run cost) is the intended basis for deciding whether panel size should grow further, hold, or shrink for a given tier. The three-judge minimum-viable panel and the seven-judge maximal panel bound this tradeoff space; see 7.6 and 8.2 for the operational floor (minimum quorum of 2).
|
|
|
|
### 9.5 Cost and Risk of the Timing Side Channel
|
|
|
|
Noted here for completeness alongside the cost table, since it is adjacent to per-request economics: the timing oracle risk described in 7.6 and 8.14 has no cost impact of its own (it does not consume additional inference spend), but it is worth tracking whether polling-based probing correlates with elevated API request volume from a given account, since that would show up in per-account infra cost before it shows up as a security incident.
|
|
|
|
---
|
|
|
|
## 10. Deployment Topology
|
|
|
|
### 10.1 Single-Server Architecture
|
|
|
|
The platform runs as a single-server deployment: one host running the API server process, the Caddy reverse proxy, the SQLite corpus database file, and the WeasyPrint PDF generation process. This is an intentional simplicity choice given current scale; there is no distributed consensus, no service mesh, and no multi-region failover at this stage. Provider-level failover (5.8) covers model/architecture outages; it does not cover the host itself.
|
|
|
|
```mermaid
|
|
flowchart TB
|
|
subgraph Host[Single Application Host]
|
|
Caddy[Caddy v2 - TLS + routing + static /reports/]
|
|
API[API Server Process - systemd managed]
|
|
DB[(SQLite Corpus DB - WAL mode)]
|
|
PDF[WeasyPrint Process - memory/timeout capped]
|
|
Cron[Prediction Tracking Cron - systemd timer]
|
|
end
|
|
Internet -->|HTTPS| Caddy
|
|
Caddy --> API
|
|
API --> DB
|
|
API --> PDF
|
|
Cron --> DB
|
|
API -.->|15min WAL checkpoint| S3[(S3-Compatible Backup, encrypted)]
|
|
```
|
|
|
|
### 10.2 DNS and TLS
|
|
|
|
- Public entry point is a standard DNS A/AAAA record pointed at the application host.
|
|
- TLS termination and certificate issuance/renewal are handled automatically by Caddy v2; no manual certificate management step exists in the deploy path.
|
|
|
|
### 10.3 Build and Deploy Pipeline
|
|
|
|
- **Documentation** (this site): built with MkDocs (Material theme) and deployed as static output.
|
|
- **API and pipeline code**: deployed via a standard git-push-triggered pipeline. The sanitization gate (Section 2.6) runs as a required check in this pipeline for any change touching public-facing output paths; a gate failure blocks the deploy outright.
|
|
- Deploys to the API layer and deploys to the docs site are independent pipelines and can ship on separate cadences.
|
|
- **Dependency pinning**: `requirements.txt` pins exact versions with hashes (`pip install --require-hashes`), so a deploy always installs the exact, previously-verified set of packages rather than resolving against whatever the latest compatible versions happen to be at deploy time.
|
|
|
|
### 10.4 Backup Strategy
|
|
|
|
- **WAL-based continuous backup**: rather than a nightly-only snapshot, the corpus database's WAL file is checkpointed to off-host, S3-compatible object storage every **15 minutes**, matching the pattern used for Hermes's own backup strategy. This bounds worst-case data loss on corruption or host failure to roughly 15 minutes of writes, versus up to 24 hours under a nightly-only scheme.
|
|
- A **full nightly snapshot** is retained in addition to the 15-minute checkpoints, giving a clean daily restore point independent of WAL replay correctness.
|
|
- **Backup encryption**: backups are encrypted with a key **separate from and independent of host-level disk encryption**. Host disk encryption protects the live system against physical media theft; backup encryption protects the offsite copy against compromise of the storage provider or backup credentials, and the two are not treated as substitutes for each other.
|
|
- The corpus is treated as first-class backup content, not an afterthought: it is the platform's accumulated institutional knowledge (Tier 1 cumulative intelligence) and its loss would silently degrade corpus search, prediction tracking, and accuracy scoring without necessarily causing an immediately visible outage.
|
|
- Restore procedure for corpus corruption is covered in Failure Modes 8.5.
|
|
|
|
### 10.5 Native Dependencies (WeasyPrint)
|
|
|
|
WeasyPrint's PDF rendering depends on several native system libraries that are not installable via pip and must be present on the host as apt packages: **cairo**, **pango**, **gdk-pixbuf**, **libffi**, and **harfbuzz**. These are documented explicitly here because a pip-only dependency list (`requirements.txt`) will install the WeasyPrint Python package successfully while still failing at render time if these system packages are absent, a failure mode that is easy to miss in a deploy checklist that only checks Python dependencies.
|
|
|
|
- **Font availability**: branded PDF reports depend on the client's chosen or the platform's default brand fonts being installed at the OS level and discoverable by WeasyPrint's font stack. A missing font does not hard-fail the render; it silently substitutes a fallback font, which is a correctness issue (broken branding) rather than a crash, and is checked for as part of the post-generation validation script (6.3) rather than assumed correct by default.
|
|
|
|
### 10.6 Process Management and Graceful Shutdown
|
|
|
|
- **Systemd unit**: the API server and orchestrator run as a systemd service with `Restart=on-failure` and `RestartSec=5`, so a crashed process comes back automatically without manual intervention, with a short delay to avoid a tight crash-restart loop against a persistently failing dependency.
|
|
- **Log rotation**: process output is captured via systemd's journald (the default under a systemd unit) with journald's own rotation/retention policy, or via logrotate for any component that writes to flat log files instead. Logs are not left to grow unbounded on the single application host.
|
|
- **Secrets management**: credentials (model/architecture provider API keys, database encryption keys, backup credentials) are supplied as environment variables via a systemd `EnvironmentFile`, not hardcoded in source or committed to the repository. The `EnvironmentFile` itself is filesystem-permission-restricted to the service's running user.
|
|
- **Graceful shutdown on deploy**: a deploy sends `SIGTERM` to the running process rather than a hard kill. The process's shutdown handler stops accepting new review submissions immediately, then drains in-flight reviews already in progress, allowing up to **15 minutes** for any review that is mid-pipeline at shutdown time to reach completion before the process actually exits and is replaced. This avoids a deploy silently killing a review that was seconds from finishing Phase 3.
|
|
|
|
---
|
|
|
|
## 11. Operations
|
|
|
|
### 11.1 Key Metrics
|
|
|
|
| Metric | Description |
|
|
|---|---|
|
|
| Review success rate | Fraction of submitted reviews reaching `status = complete` vs. `failed` |
|
|
| API latency (p95) | 95th percentile response time for synchronous API endpoints (submission, polling, dashboard) |
|
|
| PDF generation failure rate | Fraction of reviews requiring PDF retry or falling back to plaintext (Section 8.4) |
|
|
| Model/architecture latency | Per-role latency for research, primary review, and each judge, used to catch a specific provider degrading before it causes a full timeout |
|
|
| Panel diversity health | `agreement_rate` / `diversity_score` trend across the judge panel over time (Section 7.7) |
|
|
| Corpus growth rate | Reviews indexed per period, used to sanity-check corpus and backup sizing assumptions against the 5,000-review / 50GB comfort ceiling (5.7) |
|
|
| SQLite write contention rate | Count of `SQLITE_BUSY` occurrences per period (Section 8.8); a rising trend is the leading indicator for the write-concurrency migration trigger described in 5.7, well before size alone would suggest one |
|
|
| Queue depth | Current pending-review count against the 50-review cap (5.6, 8.13); alerts at 80% |
|
|
| Report access anomalies | Count of report retrievals flagged by the access logging in 7.3, used to spot URL-guessing attempts |
|
|
|
|
### 11.2 Alerting Thresholds
|
|
|
|
- Review success rate dropping below an agreed operational floor over a rolling window triggers investigation (distinguish between a systemic pipeline issue and a single provider outage per Failure Mode 8.6).
|
|
- Any sanitization gate block on a production deploy or report generation attempt (8.7) should notify the operator immediately; this is a security-relevant event even though it is functioning as designed.
|
|
- Sustained upward drift in panel `agreement_rate` beyond a defined band triggers a manual review of judge configuration, per the training-data recursion and consensus-drift concerns in 7.7 and 8.14.
|
|
- PDF fallback-to-plaintext events (8.4, 8.9) should alert on any occurrence, not just above a threshold, since they represent a customer-visible degradation of the deliverable.
|
|
- Queue depth crossing 80% of the 50-review cap (8.13) should alert with enough lead time to investigate before submissions start receiving `429`.
|
|
- Prediction cron missing-run or failure events (8.11) page the operator immediately; there is no acceptable silent-failure window for this check given how quietly its failure otherwise degrades Tier 1 intelligence.
|
|
|
|
### 11.3 Backup Schedule
|
|
|
|
- 15-minute WAL checkpoint to encrypted, off-host S3-compatible storage, plus a full nightly snapshot (Section 10.4).
|
|
- Backup integrity should be spot-verified on a periodic cadence (e.g., a scheduled restore-to-scratch test), rather than assumed functional purely because the backup job reports success.
|
|
|
|
### 11.4 Corpus Health Checks
|
|
|
|
- Periodic verification that `corpus_record` entries stay in sync with their source `review` rows (no orphaned or stale index entries), particularly after any manual corpus maintenance operation.
|
|
- Periodic audit that `corpus_opt_out` reviews are in fact excluded from search, comparison results, and embedding generation entirely (7.4, 2.9), as a direct verification of the confidentiality control rather than trusting the flag's existence alone.
|
|
- Periodic audit that `summary_snippet` values in the corpus (4.4, 7.4) remain structural-only and have not regressed toward containing proposal-text-derived content through a template change.
|
|
|
|
### 11.5 Prediction Cron Monitoring
|
|
|
|
Moved here from its prior placement as a passing note in 11.4, and elevated to its own subsection given the failure mode's severity (8.11): the prediction-tracking cron runs as a systemd timer with `OnFailure=` wired to a dedicated notification unit, and a separate missing-run detection check verifies that a run that should have fired within its expected window actually did, independent of whether a run that did fire reported success or failure. Both conditions page the operator; there is no code path where this cron can fail silently and go unnoticed.
|
|
|
|
---
|
|
|
|
## Appendix A: Implementation Roadmap Reference
|
|
|
|
For planning and evaluation purposes, the v3 build is staged as follows:
|
|
|
|
| Phase | Weeks | Scope |
|
|
|---|---|---|
|
|
| Phase 0 | 1-3 | Sanitization gate (both layers, 7.5), automated PDF generation, corpus schema and backfill, red-team testing protocol (7.1) |
|
|
| Phase 1 | 4-7 | Corpus search, embedding generation policy (2.9), prediction tracking cron, accuracy scoring (read-only, dashboard only, Enterprise-tier access per 4.5/8.16) |
|
|
| Phase 2 | 8-12 | Vertical-specific templates, adversarial red-team per vertical |
|
|
| Phase 3 | 13-18 | New specialist judges (reasoning-verification, execution-feasibility, market-reality), public shareable reports, review-as-a-service API beta, market simulation engine (5.2) if methodology remains defensibly specified per Condition E; otherwise re-scoped to this phase by default |
|
|
| Phase 4 | 19-22 | Live accuracy-based judge weighting, general availability, White-Label pilots |
|
|
|
|
---
|
|
|
|
## Appendix B: Review Addendum
|
|
|
|
### B.1 Review Verdict
|
|
|
|
On **2026-08-10**, this architecture document underwent a structured three-judge review prior to build sign-off. All three judges returned an independent **CONDITIONAL GO** verdict; the panel result was a unanimous 3/3 Conditional Go, meaning the architecture is approved to proceed to implementation contingent on the conditions below being addressed in this document, which this revision does.
|
|
|
|
| Judge | Model | Focus Area |
|
|
|---|---|---|
|
|
| Judge 1 | Opus | Operational realism: deployment mechanics, failure recovery, resource limits, monitoring |
|
|
| Judge 2 | Claude | Security and failure modes: prompt injection, access control, corpus confidentiality, abuse protection |
|
|
| Judge 3 | Gemini Pro | Completeness and coverage: missing sections, edge cases, specification gaps |
|
|
|
|
### B.2 Conditions Addressed
|
|
|
|
The following 18 merged and deduplicated conditions, consolidated across the three independent reviews, are addressed in this revision:
|
|
|
|
1. **Embedding/vector search subsystem specified**: model named (`all-MiniLM-L6-v2` via `sentence-transformers`), storage/query mechanism specified (SQLite row-level cosine similarity via numpy, or `sqlite-vec` as the documented exception to the no-extensions rule), embedding generation skipped entirely (not merely flagged) for opt-out reviews. See 2.9.
|
|
2. **SQLite WAL mode and write contention**: WAL mode declared explicitly in 2.4; `SQLITE_BUSY`/write contention added to the failure modes table (8.8); concurrent-writer ceiling and single-worker serialization documented (2.4, 5.6); migration trigger re-diagnosed as write concurrency, not corpus size (5.7).
|
|
3. **Deployment/production hardening**: WeasyPrint native apt dependencies and font availability (10.5); systemd unit with `Restart=on-failure`/`RestartSec=5` (10.6); log rotation via journald/logrotate (10.6); secrets via systemd `EnvironmentFile` (10.6); graceful `SIGTERM` drain with a 15-minute completion window (10.6); dependency pinning with hashed `requirements.txt` (10.3).
|
|
4. **Backup RPO**: nightly-only backup replaced with 15-minute WAL checkpoint to S3, retaining a nightly full snapshot in addition; backup encryption documented as independent of host-level disk encryption (10.4).
|
|
5. **Market simulation specification**: methodology specified as agent-driven Monte Carlo simulation with explicit assumption documentation; sequencing clarified as running before Primary Review, feeding a ResearchBrief appendix rather than running concurrently with Phase 2 (5.2); de-scope path to Phase 3 documented if methodology cannot be defensibly specified (Appendix A).
|
|
6. **Missing sections added**: Frontend (2.10), Authentication (2.11), Tenant Isolation (2.12), Billing Integration (2.13), Notification System (2.14).
|
|
7. **Edge case handling**: token-limit chunking and 25K-word hard cap (5.9), non-English rejection with Phase 1 English-only scope (5.9), queue depth cap of 50 with `429` (5.6, 5.9, 8.13), duplicate submission via `proposal_hash` (5.9), corpus scale threshold restated as write-contention-driven (5.7, 5.9).
|
|
8. **Failover configuration**: per-role provider list `[primary, secondary, tertiary]` documented (3.8, 5.8), 3-retry exponential backoff before escalation, configuration location specified as a config table (3.8).
|
|
9. **Prompt injection defense reframed**: claim extractor acknowledged as the first point of exposure, not a pre-exposure filter; pre-extraction regex sanitization added; output behavioral anomaly detection added as a supplementary signal; red-team testing protocol added as a Phase 0 deliverable (7.1).
|
|
10. **Corpus confidentiality**: embeddings skipped (not generated) for opt-out reviews (2.9, 7.4); raw DB access model documented as single-operator, all-or-nothing design assumption (7.4); S3 backups encrypted with a separate key from host-level disk encryption (10.4); `summary_snippet` construction rules defined as fixed-length structural summary, not a proposal-text derivative (4.4, 7.4).
|
|
11. **Accuracy dashboard access restricted**: 4.5 endpoint restricted to authenticated Enterprise-tier access at minimum; Free/Pro tiers have no access to per-judge accuracy data.
|
|
12. **Report URL security**: `uuid4()`/`os.urandom`-based unguessable identifiers confirmed; optional expiration (default 90 days) added; access logging added for all report retrievals (7.3).
|
|
13. **"No training" API terms**: contractual requirement for "no training on API usage" terms added as a standing provider condition; agreement-rate monitoring documented as lagging detection, not leading prevention; training-data recursion acknowledged as partially inherent to multi-model pipelines (7.7, 8.14).
|
|
14. **Expanded failure modes table**: six new entries added covering SQLite write contention, WeasyPrint memory exhaustion, sanitization gate false positives, prediction cron silent failure, confident-but-wrong judge analysis, and concurrent queue saturation (8.8 through 8.13).
|
|
15. **Sanitization gate enhancements**: behavioral fingerprint detection layer added alongside string matching, targeting self-identification phrases, training-cutoff patterns, and capability-boundary language across model families; false-positive procedure (human review, allowlist, re-run, mandatory audit log) added; gate confirmed as still blocking by default (7.5).
|
|
16. **API abuse countermeasures**: polling endpoint rate-limited independently from submission endpoint; free-tier submission content monitoring for probing patterns added; timing oracle via polling documented as an acknowledged low-severity side channel (7.6).
|
|
17. **Inherent risk documentation**: a dedicated Inherent Risks subsection added (8.14) stating plainly that training-data recursion, the polling timing oracle, and queue-saturation capacity limits do not have full technical solutions and are accepted, ongoing risk rather than solvable failure modes.
|
|
18. **Accuracy dashboard, tenant, and billing sections cross-checked against tier structure**: Section 4.5, 2.12, and 2.13 were cross-verified against the pricing tiers in 9.3 to confirm no contradiction between stated tier entitlements and the access restrictions added under condition 11.
|
|
|
|
### B.3 Scope Note
|
|
|
|
This addendum documents that the above conditions were incorporated into this architecture document as a direct response to the review. It does not itself constitute a fourth review pass; implementation should still be validated against this document during and after each build phase in Appendix A, and the red-team testing protocol (7.1, condition 9) and Phase 0 deliverables should be treated as prerequisites for the Phase 0 exit criteria, not optional hardening. |