diff --git a/docs-source/verdicttank/architecture.md b/docs-source/verdicttank/architecture.md index ed62491..8edb1f5 100644 --- a/docs-source/verdicttank/architecture.md +++ b/docs-source/verdicttank/architecture.md @@ -33,9 +33,9 @@ The live v2 platform runs five review stages across the three phases described a | Tier | Theme | Additions | |---|---|---| -| Phase 0 | Infrastructure prerequisites | Automated sanitization gate, automated PDF generation pipeline | +| Phase 0 | Infrastructure prerequisites | Automated sanitization gate, automated PDF generation pipeline, embedding subsystem, corpus schema and backfill | | Tier 1 | Cumulative intelligence | Review corpus database, prediction-vs-outcome tracking, reviewer accuracy scoring | -| Tier 2 | Deeper analysis | Vertical-specific templates, adversarial red-team per vertical, market simulation engine | +| Tier 2 | Deeper analysis | Vertical-specific templates, adversarial red-team per vertical, market simulation engine (Phase 2, sequenced before Phase 2 primary review per §5.4) | | Tier 3 | Product and distribution | Public-facing shareable reports, review-as-a-service API, competitive corpus comparisons | The net effect is a system that remembers its own history (corpus), learns whether its verdicts were correct (prediction tracking), specializes by industry vertical, and exposes itself as a product surface beyond a single web form. @@ -43,6 +43,9 @@ The net effect is a system that remembers its own history (corpus), learns wheth !!! info "Pipeline verdict on the v3 proposal itself" The v3 proposal was run through the VerdictTank pipeline prior to build. Verdict: **3/3 Unanimous Conditional Go**. Conditions attached to that verdict are reflected throughout the Security Model and Failure Modes sections below (notably around corpus confidentiality and sanitization gate rigor). +!!! info "Architecture review verdict" + This document itself was subjected to a 3-judge Conductor architecture review on 2026-08-10 (Opus 4.8 operational realism, Claude security and failure modes, Gemini Pro completeness and coverage). All three judges returned **CONDITIONAL GO**. The 18 merged conditions from that review are addressed throughout this revision; see **Appendix B: Review Addendum** for the full record. + --- ## 2. Component Architecture @@ -53,29 +56,40 @@ VerdictTank is deliberately a small number of well-defined components rather tha flowchart TB subgraph Edge Caddy[Reverse Proxy — Caddy v2] + Auth[Auth Gateway — Stack Auth / API Keys] end subgraph App API[Core API Server — FastAPI] Orchestrator[Review Pipeline Engine] Sim[Market Simulation Engine] Sanitize[Sanitization Gate] + Embed[Embedding Service] end subgraph Data - DB[(Corpus Database — SQLite)] + DB[(Corpus Database — SQLite, WAL mode)] PDFGen[PDF Generation Engine — WeasyPrint] Reports[/reports/ static files/] end subgraph Scheduled Cron[Prediction Tracking Cron] end + subgraph External + Billing[Stripe Billing] + Notify[Notification Service] + end - Caddy --> API + Caddy --> Auth + Auth --> API Caddy --> Reports API --> Orchestrator Orchestrator --> Sim Orchestrator --> DB + Orchestrator --> Embed + Embed --> DB Orchestrator --> PDFGen PDFGen --> Reports + Orchestrator --> Notify + API --> Billing Sanitize -.blocks.-> Reports Cron --> DB ``` @@ -97,37 +111,73 @@ flowchart TB - **Role**: The orchestrator. Sequences the five-plus agent phases for a given review, manages timeouts per stage, handles degraded-mode fallback when a judge is unavailable, and persists intermediate state so a review can be resumed or inspected mid-flight. - **Design**: implemented as a state machine keyed by `review_id`. Each phase transition writes its output to the review record before advancing, so a crash mid-pipeline loses at most the in-flight stage rather than the whole review. -- **Concurrency**: Phase 3 judges run concurrently (not sequentially) since they are independent of one another by design; this is what keeps total turnaround in the minutes range rather than compounding each judge's latency serially. +- **Concurrency**: Phase 3 judges run concurrently within a single review (not sequentially) since they are independent of one another by design; this is what keeps total turnaround in the minutes range rather than compounding each judge's latency serially. Across reviews, concurrency is bounded by a single-worker queue serialization model; see 2.4 and 5.6. ### 2.4 Corpus Database -- **Stack**: SQLite, single file, with the corpus schema described in Section 3. +- **Stack**: SQLite, single file, with the corpus schema described in Section 3. **WAL (Write-Ahead Logging) mode is enabled explicitly** (`PRAGMA journal_mode=WAL;`) at database initialization, not left at the default rollback-journal mode. WAL allows concurrent readers (e.g., a poll request per §4.2) to proceed while a writer (e.g., a judge verdict write) is in flight, which is a hard requirement given the read/write mix described below. - **Role**: System of record for every review ever run, plus derived tables for predictions, outcomes, and per-judge accuracy scores. -- **Why SQLite**: review volume at current and near-term projected scale does not warrant a networked database; SQLite gives transactional integrity, zero operational overhead, and trivial backup (file copy) for a workload that is read-heavy and write-light per unit time. -- **Growth path**: schema is written with an eye toward a future migration to a networked engine if corpus size or concurrent-write volume outgrows SQLite's comfortable range; no component queries SQLite-specific SQL extensions that would block that migration. +- **Why SQLite**: review volume at current and near-term projected scale does not warrant a networked database; SQLite gives transactional integrity, zero operational overhead, and trivial backup (file copy) for a workload that is read-heavy and write-light per unit time, provided writes are serialized, see below. +- **Concurrent-writer ceiling**: SQLite, even in WAL mode, permits exactly one writer at a time; concurrent write attempts beyond that one writer either block or raise `SQLITE_BUSY`. This is a hard ceiling on the single-file design, not a tunable parameter. The realistic comfort zone for this write pattern (parallel judges within a review, plus cron, plus API writes) is a single concurrent writer per review, with reviews themselves serialized. See 5.6 for how the orchestrator enforces this. +- **Single-worker queue serialization**: the orchestrator runs as a single-worker process with respect to write-heavy phases. Multiple submitted reviews are queued, not run concurrently against the database. Within one review, Phase 3 judges still run their model inference calls concurrently (the expensive, slow part), but their writes to `judge_verdict` are serialized through the orchestrator's single write path rather than issued as N simultaneous transactions. This converts "N judges racing to write" into "N judges racing to finish, one write queue," which eliminates the write-contention failure mode described in 8.8 for the common case. +- **Growth path**: schema is written with an eye toward a future migration to a networked engine. The trigger for that migration is write concurrency, not corpus size — a single SQLite file with WAL mode and single-worker serialization comfortably holds many gigabytes of review history; it does not comfortably hold multiple simultaneous writers hammering it at once. See 5.7 (Corpus Scale Threshold) for the concrete numeric ceiling. The one deliberate exception to "no SQLite-specific extensions" is the embedding similarity mechanism in 2.9, which is called out explicitly rather than left as an implicit contradiction. ### 2.5 PDF Generation Engine - **Stack**: WeasyPrint, driven off an HTML template populated from the structured review JSON. - **Role**: Converts the completed review record into the branded 11-section PDF report. See Section 6 for the full pipeline. - **Validation**: every generated PDF passes through a post-generation validation script before being written to `/reports/` — see 6.3. +- **Native dependency and resource constraints**: see 10.5 for the required system packages and 8.9 for memory/timeout limits applied to the WeasyPrint process. ### 2.6 Automated Sanitization Gate -- **Role**: A pre-deploy content scanner that runs against all output destined for any public surface (shareable reports, OG images, public API responses, docs). It scans for architecture-identifying strings (model/vendor names) and internal infrastructure references, and **blocks the deploy** if a match is found rather than silently redacting. +- **Role**: A pre-deploy content scanner that runs against all output destined for any public surface (shareable reports, OG images, public API responses, docs). It scans for architecture-identifying strings (model/vendor names) and internal infrastructure references, and blocks the deploy if a match is found rather than silently redacting. +- **Detection layers**: two layers, not one. Layer 1 is string matching against a maintained list of architecture-identifying strings and internal infrastructure references, as in v2. Layer 2 (v3 addition) is behavioral fingerprint detection: a maintained regex pattern bank targeting self-identification phrases common across model families ("as a language model," "my training data includes," "I cannot browse the internet," "my knowledge cutoff is"), capability-boundary language, and training-cutoff date patterns. Layer 2 exists because architecture identity leaks far more often through phrasing patterns than through an accidental vendor name; see 8.12 and 7.5 for the full specification. - **Rationale**: the platform's differentiator is cross-vendor architecture diversity; leaking which specific architectures are in the panel undermines both competitive position and the "independent judges" framing that gives the majority verdict its credibility. -- **Placement in CI**: runs as a required check in the deploy pipeline, not as a runtime filter. A failure here is a build failure, not a logged warning. See 7.5 and 8.7. +- **Placement in CI**: runs as a required check in the deploy pipeline, not as a runtime filter. A failure here is a build failure, not a logged warning. It also runs per-report at generation time (6.3). See 7.5 and 8.7. +- **False-positive handling**: the gate remains blocking by default. A documented human-override procedure exists for confirmed false positives; see 8.12 and 7.5. ### 2.7 Market Simulation Engine - **Role**: v3 Tier 2 addition. Given a proposal's stated business model, generates a 12-month simulated trajectory: user acquisition curve, churn projection, and resulting revenue trajectory, rather than relying solely on the static financial figures the proposal itself provides. -- **Output**: feeds into the Primary Reviewer's Financials dimension and into judge disagreement analysis (a judge may flag that the simulated trajectory materially diverges from the proposal's stated projections). +- **Methodology**: agent-driven Monte Carlo simulation. The simulation agent is given the proposal's stated business model, the research brief's sourced market data, and a fixed set of explicitly documented distributional assumptions (e.g., CAC variance bounded by vertical-specific historical ranges, churn modeled as a bounded stochastic process informed by the research brief's market data, not invented ad hoc per run). Each simulation run records its assumption set alongside its output so a reviewer or judge can inspect what the simulation actually assumed, not just what it concluded. If a defensible assumption set cannot be maintained for a given vertical, the simulation is skipped for that review and the Financials dimension proceeds on the research brief and proposal figures alone, flagged as `simulation_skipped = true`. +- **Sequencing**: the Market Simulation Engine runs **before** the Primary Review (Phase 2), not concurrently with it. Its output is packaged as an appendix to the `ResearchBrief` (a `simulated_trajectory` field, see 3.2) so the Primary Reviewer consumes it as one more input to the Financials dimension, the same way it consumes verified claims and market data. This resolves the sequencing ambiguity between §1.1's phase diagram and the original placement of market simulation as a Phase 2-adjacent activity; simulation is logically part of Phase 1's research-gathering output, timed to complete before Phase 2 begins. - **Independence**: intentionally decoupled from the proposal's own numbers so it cannot simply echo back what was submitted; it is a sanity-check model, not a validator of the proposal's math. +- **De-scope condition**: if, during Phase 0 build-out, the assumption set for a given vertical cannot be defensibly specified (no credible source for churn/CAC ranges in that vertical), that vertical is de-scoped from market simulation and moves to a Phase 3 backlog item rather than shipping with fabricated placeholder assumptions. ### 2.8 Prediction-vs-Outcome Tracking Cron - **Role**: v3 Tier 1 addition. Every review that contains a flagged prediction (e.g., "this pricing tier will suppress conversion," "this GTM channel will not reach target CAC") is scheduled for automated re-check at T+90, T+180, and T+365 days. -- **Mechanics**: the cron job queries the corpus for predictions due for re-check, attempts to gather current public signal on the outcome (via the research agent's toolset), and records a best-effort actual-outcome assessment against the original prediction. See Data Model 3.6 and Failure Modes 8.3 for attribution caveats. +- **Mechanics**: the cron job queries the corpus for predictions due for re-check, attempts to gather current public signal on the outcome (via the research agent's toolset), and records a best-effort actual-outcome assessment against the original prediction. Runs as a **systemd timer**, not a bare crontab entry, with `OnFailure=` wired to a notification unit so a failed run pages the operator rather than failing silently. See Data Model 3.6, Failure Modes 8.4 and 8.13, and 10.6. + +### 2.9 Embedding Service + +- **Role**: v3 Phase 0 addition. Generates the vector embedding stored in `corpus_record.embedding` (3.5) for corpus semantic search (4.4). +- **Model**: `all-MiniLM-L6-v2` via the `sentence-transformers` library. Chosen for a small footprint (384-dimension output, runs on CPU without a GPU dependency), which matches the single-server deployment model in Section 10. +- **Storage and query mechanism**: embeddings are stored as raw float32 BLOBs in the `corpus_record.embedding` column (SQLite has no native vector type). Similarity search is performed via row-level cosine similarity computed in application code using `numpy`, scanning the searchable subset of `corpus_record` rows at query time. This keeps the datastore free of SQLite-specific vector extensions, preserving the migration-portability property claimed in 2.4. At current and near-term corpus sizes (low thousands of rows), a full in-memory cosine scan against `numpy` arrays completes well within the API's latency budget; this is revisited if/when corpus size or query volume grows past the SQLite migration trigger in 5.7. +- **Exception acknowledgment**: this is the one place in the architecture that comes close to a SQLite-specific mechanism, since the embedding column and the scan logic are shaped around SQLite's lack of native vector support. It is explicitly documented here as the accepted exception to the "no SQLite-specific extensions" portability claim in 2.4, rather than left as an unstated contradiction. Should `sqlite-vec` or a similar extension be adopted later for query performance, that adoption must be documented here and the exception restated, not silently introduced. +- **Opt-out policy**: for reviews with `corpus_opt_out = 1`, embedding generation is **skipped entirely**, not generated-then-flagged-non-searchable. The orchestrator checks `review.corpus_opt_out` before invoking the embedding service; if set, no `corpus_record` row is created and no embedding is computed. See 3.5, 7.4, and 8.14. +- **Deletion on late opt-out**: if a customer opts out after a `corpus_record` row and embedding already exist (opt-out is a mutable setting on an existing review), the embedding BLOB and the `corpus_record` row are deleted, not merely flagged non-searchable. A deletion is logged to the audit trail described in 7.4. + +### 2.10 Frontend + +A thin web application, backed entirely by the API described in Section 4, handles proposal submission and results viewing. It has two primary surfaces: a submission form (proposal text entry, tier selection, vertical override) and a results dashboard (review status polling, verdict display, PDF download link, and, for Enterprise/White-Label accounts, corpus search and accuracy dashboard views gated per 4.4 and 4.5). The frontend holds no review logic and no direct database access; it is a client of the public API contract, which keeps the authentication and tenant-isolation boundaries described below as the single enforcement point regardless of which client (web app, API-key integrator, White-Label embed) is calling in. + +### 2.11 Authentication + +Two authentication mechanisms cover the platform's access patterns. **Bearer token (JWT) authentication** is used for the web application and any session-based interactive use, issued via `auth2.itpropartner.com`, a Stack Auth project shared with other IT Pro Partner properties. **API key authentication** is used for programmatic/Enterprise and White-Label integrations, per the scoping rules in 7.2. Both mechanisms terminate at the API server (2.1); the reverse proxy (2.2) does not perform authentication itself, only TLS termination and routing. A request without a valid JWT or API key for a tier-gated endpoint receives `401`. + +### 2.12 Tenant Isolation + +Tenant isolation is tier-dependent, not uniform. **White-Label** tier customers get a separate corpus partition per tenant, enforced via a `tenant_id` column added to `review` and `corpus_record` (see 3.1, 3.5); all corpus queries for a White-Label tenant are scoped to that tenant's partition and never cross into another tenant's data or the shared corpus. **Enterprise** tier is single-tenant by default: an Enterprise account's own reviews are private to that account, though its corpus search can still query the shared aggregate corpus per the aggregate-only rules in 7.4. **Pro** and **Free** tiers share the general corpus with search restrictions; individual review content is never exposed to another account regardless of tier, only aggregate metadata per 4.4. + +### 2.13 Billing Integration + +Stripe handles subscription billing for Pro, Enterprise, and White-Label tiers. Stripe webhooks (subscription created, updated, canceled, payment failed) drive quota enforcement: a webhook handler updates the account's entitlement record, and the API server's quota check (7.2) reads from that entitlement record rather than calling out to Stripe synchronously on every request. A payment failure webhook flips the account to a grace-period state before hard-blocking new submissions, giving the customer a window to update payment details without an abrupt cutoff. + +### 2.14 Notification System + +On review completion, the system sends an email via SMTP with the verdict summary and a link to the PDF report. This is the default notification path for all tiers. Enterprise tier customers may additionally configure a **webhook callback**: on review completion, the orchestrator POSTs the review summary payload (verdict, verdict_margin, review_id, pdf_url) to the customer-configured URL, with the same retry-with-backoff behavior as the model provider failover described in 5.8, capped at 3 attempts before falling back to email-only notification for that review. --- @@ -141,16 +191,20 @@ The top-level entity representing one submitted proposal and its full review lif ```sql CREATE TABLE review ( - id TEXT PRIMARY KEY, -- UUID + id TEXT PRIMARY KEY, -- UUID (uuid4, see 7.3) + tenant_id TEXT, -- v3: White-Label tenant partition, NULL for non-White-Label client_name TEXT, proposal_text TEXT NOT NULL, - proposal_hash TEXT NOT NULL, -- dedup / integrity check + proposal_hash TEXT NOT NULL, -- dedup / integrity check, see 5.9 + proposal_word_count INTEGER, -- v3: enforced against 25K word hard cap, see 5.9 + language_detected TEXT, -- v3: ISO 639-1, see 5.9 pricing_tier TEXT NOT NULL, -- free | pro | enterprise | white_label vertical TEXT, -- v3: auto-classified (saas, consulting, marketplace, fintech, ...) status TEXT NOT NULL, -- queued | researching | reviewing | judging | aggregating | complete | failed verdict TEXT, -- go | conditional_go | no_go verdict_margin TEXT, -- e.g. "3/3 unanimous", "2/3 majority" - corpus_opt_out BOOLEAN DEFAULT 0, -- v3: excludes from corpus search / comparisons + corpus_opt_out BOOLEAN DEFAULT 0, -- v3: excludes from corpus search / comparisons / embedding generation + report_url_expires_at TIMESTAMP, -- v3: optional expiration, default null = no expiry unless configured, see 7.3 created_at TIMESTAMP NOT NULL, updated_at TIMESTAMP NOT NULL, completed_at TIMESTAMP @@ -170,6 +224,8 @@ CREATE TABLE research_brief ( domain_availability TEXT, -- JSON: {domain, available: bool, checked_at} trademark_flags TEXT, -- JSON array: {mark, jurisdiction, conflict_summary} market_data TEXT, -- JSON: sourced market size / growth figures with citations + simulated_trajectory TEXT, -- v3: JSON from Market Simulation Engine (2.7), appended before Phase 2, null if simulation_skipped + simulation_skipped BOOLEAN DEFAULT 0, -- v3: true if no defensible assumption set for vertical, see 2.7 citation_count INTEGER, limited_citations_flag BOOLEAN DEFAULT 0, -- set when web verification degraded, see 8.3 created_at TIMESTAMP NOT NULL @@ -226,20 +282,23 @@ CREATE TABLE judge_verdict ( ### 3.5 Corpus Record (v3) -Indexing/search layer over completed reviews. One-to-one with a review, populated on completion. +Indexing/search layer over completed reviews. One-to-one with a review, created only when embedding generation actually runs (see 2.9). No row is created for `corpus_opt_out = 1` reviews. ```sql CREATE TABLE corpus_record ( review_id TEXT PRIMARY KEY REFERENCES review(id), - embedding BLOB, -- vector embedding for semantic search + tenant_id TEXT, -- v3: mirrors review.tenant_id for White-Label partition scoping, see 2.12 + embedding BLOB, -- v3: 384-dim float32 vector, all-MiniLM-L6-v2, see 2.9. Never populated for corpus_opt_out=1 vertical TEXT, -- indexed verdict TEXT, -- indexed overall_score_avg REAL, -- indexed, for percentile comparisons - searchable BOOLEAN DEFAULT 1, -- respects corpus_opt_out + summary_snippet TEXT, -- v3: fixed-length structural summary only, see 4.4 + searchable BOOLEAN DEFAULT 1, -- respects corpus_opt_out; row does not exist at all if opted out at creation time indexed_at TIMESTAMP NOT NULL ); CREATE INDEX idx_corpus_vertical ON corpus_record(vertical); CREATE INDEX idx_corpus_verdict ON corpus_record(verdict); +CREATE INDEX idx_corpus_tenant ON corpus_record(tenant_id); ``` ### 3.6 Prediction / Outcome Record (v3) @@ -274,206 +333,153 @@ CREATE TABLE accuracy_score ( ``` !!! warning "Accuracy scoring is read-only until Phase 4" - Per the implementation roadmap, `accuracy_score` is computed and exposed on a dashboard starting in Phase 1, but it does **not** feed back into judge weighting until Phase 4. This separation is deliberate: it lets the team observe whether accuracy scoring itself is trustworthy (see Failure Modes 8.2, consensus-drift risk) before letting it influence live verdicts. + Per the implementation roadmap, `accuracy_score` is computed and exposed on a dashboard starting in Phase 1, but it does **not** feed back into judge weighting until Phase 4. This separation is deliberate: it lets the team observe whether accuracy scoring itself is trustworthy (see Failure Modes 8.2, consensus-drift risk) before letting it influence live verdicts. Access to the dashboard itself is restricted to authenticated Enterprise-tier accounts at minimum, per 4.5 and 7.4; read-only status governs when the *data* affects verdicts, not who can *see* it. + +### 3.8 Provider Failover Configuration (v3) + +Not a corpus table; lives in a version-controlled config file (`config/failover.yaml`), loaded at orchestrator startup, not in the database. Kept out of the database deliberately so a provider outage can be worked around by an operator editing and redeploying config without a live DB write racing the outage itself. + +```yaml +# config/failover.yaml +roles: + research_agent: + providers: [primary_vendor_a, secondary_vendor_b, tertiary_vendor_c] + retry_count: 3 + backoff_base_seconds: 2 # exponential: 2s, 4s, 8s before escalating to next provider + primary_reviewer: + providers: [primary_vendor_a, secondary_vendor_b] + retry_count: 3 + backoff_base_seconds: 2 + judge_reasoning_verification: + providers: [primary_vendor_c, secondary_vendor_a] + retry_count: 3 + backoff_base_seconds: 2 + judge_execution_feasibility: + providers: [primary_vendor_b, secondary_vendor_c] + retry_count: 3 + backoff_base_seconds: 2 + judge_market_reality: + providers: [primary_vendor_a, secondary_vendor_b] + retry_count: 3 + backoff_base_seconds: 2 +``` + +Each pipeline role (research, primary reviewer, each judge) has an ordered provider list. On a call failure, the orchestrator retries against the same provider up to `retry_count` times with exponential backoff, then escalates to the next provider in the role's list. If every configured provider for a role is exhausted, the failure is handled per Failure Mode 8.6 (treated as a single-judge timeout/crash, or a full-panel failure if it cascades below quorum). See 5.8 for the operational walkthrough. --- -## 4. API Design +## 4. API Contract -All endpoints are namespaced under `/api/verdicttank/`. Authentication is via scoped API keys (see Security Model 7.2) for Enterprise and White-Label tiers; Free and Pro tiers authenticate via session token from the web application. +All endpoints are served under `api.verdicttank.com` (or the equivalent path behind the reverse proxy). Authentication follows 2.11; unauthenticated requests to tier-gated endpoints receive `401`. ### 4.1 Submit Review ``` -POST /api/verdicttank/review -``` +POST /v1/reviews +Authorization: Bearer | X-API-Key: +Content-Type: application/json -**Request:** -```json { - "proposal_text": "string, required", - "client_name": "string, optional", - "pricing_tier": "free | pro | enterprise | white_label", - "vertical_override": "string, optional" + "proposal_text": "...", + "client_name": "optional", + "vertical_override": "optional", + "corpus_opt_out": false } ``` -**Response `202 Accepted`:** -```json -{ - "review_id": "uuid", - "status": "queued", - "estimated_completion_seconds": 540 -} -``` - -**Errors:** `429` (tier quota exceeded), `422` (proposal text below minimum length or fails structural parse), `401` (invalid/missing key). +Validates tier entitlement and quota (7.2), computes `proposal_hash` and checks for a duplicate submission (5.9), detects language and word count (5.9), then enqueues the review. Returns `202 Accepted` with a `review_id` and initial `status: queued`. Duplicate submissions do not re-enqueue; see 5.9. ### 4.2 Poll Review Status ``` -GET /api/verdicttank/review/{id} +GET /v1/reviews/{review_id} +Authorization: Bearer | X-API-Key: ``` -**Response `200 OK`:** -```json -{ - "review_id": "uuid", - "status": "researching | reviewing | judging | aggregating | complete | failed", - "verdict": "go | conditional_go | no_go | null", - "verdict_margin": "string | null", - "progress_phase": 2, - "created_at": "iso8601", - "completed_at": "iso8601 | null" -} -``` +Returns current `status`, and once `status = complete`, the verdict summary and `pdf_url`. **This endpoint is rate-limited independently from the submission endpoint** (7.7), both to protect against poll-based abuse and because the polling pattern itself carries a low-severity timing side-channel discussed in 8.15 and 9.5. -### 4.3 Download PDF +### 4.3 Retrieve Report ``` -GET /api/verdicttank/review/{id}/pdf +GET /reports/{unguessable_id}.pdf ``` -Returns the generated PDF binary with `Content-Type: application/pdf` once `status == complete`. Returns `409 Conflict` if the review is not yet complete, `404` if the review id does not exist or belongs to another account. +Served directly by the reverse proxy as a static file (2.2). The `unguessable_id` is not the `review_id`; it is a separately generated `uuid4()` value (or `os.urandom`-derived token) stored on the review record specifically for the public-facing URL, so a leaked or guessed `review_id` from an authenticated context does not also expose the public report path. See 7.3 for the full security rationale, including the optional expiration window and access logging. -### 4.4 Corpus Search (Enterprise tier) +### 4.4 Corpus Search (v3) ``` -GET /api/verdicttank/corpus/search?vertical=saas&verdict=conditional_go&q=pricing+tier&limit=20 +GET /v1/corpus/search?vertical=saas&verdict=go&q=... +Authorization: Bearer | X-API-Key: ``` -**Response `200 OK`:** -```json -{ - "results": [ - { - "review_id": "uuid", - "vertical": "saas", - "verdict": "conditional_go", - "overall_score_avg": 6.4, - "summary_snippet": "string", - "created_at": "iso8601" - } - ], - "total_matches": 143 -} -``` +Available to Pro tier and above, scoped per the tenant isolation rules in 2.12: White-Label queries are scoped to `tenant_id`, Enterprise and Pro/Free query the shared aggregate corpus with the search restrictions in 7.4. `q` performs semantic search via the embedding similarity mechanism in 2.9. Results never include full proposal text or full review findings for another account's review; each result includes only `vertical`, `verdict`, `overall_score_avg`, and `summary_snippet`. **`summary_snippet` construction is strictly limited**: it is a fixed-length structural summary composed only from `vertical`, `verdict`, and a score-range bucket (e.g., "SaaS proposal, Conditional Go, score range 6-7"). It is never a derivative, excerpt, paraphrase, or embedding-nearest-sentence of the original proposal text. This is a hard construction rule, not a length limit on an otherwise-free-form summary; see 8.14 for the confidentiality rationale. -Only reviews with `corpus_opt_out = 0` are searchable, and results returned to Enterprise/White-Label customers other than the review's owner are aggregate-only (score and vertical, no proposal text) unless the searching account owns the underlying review. See Security Model 7.4. - -### 4.5 Accuracy Dashboard +### 4.5 Accuracy Dashboard (v3) ``` -GET /api/verdicttank/accuracy +GET /v1/accuracy/dashboard +Authorization: Bearer | X-API-Key: ``` -**Response `200 OK`:** -```json -{ - "judges": [ - { - "judge_id": "reasoning_verification", - "running_accuracy": 0.71, - "agreement_rate": 0.58, - "diversity_score": 0.42, - "total_predictions_scored": 212 - } - ], - "last_updated_at": "iso8601" -} -``` +**Restricted to authenticated Enterprise-tier accounts and above** (Enterprise, White-Label). Free and Pro tiers receive `403` on this endpoint; they have no access to per-judge accuracy data, running accuracy scores, or agreement-rate figures. Rationale: per-judge accuracy is competitively sensitive (it effectively ranks the underlying architectures against each other) and is still in the read-only observation period described in 3.7's warning callout. -### 4.6 Prediction Tracking +### 4.6 Webhook Registration (Enterprise, v3) ``` -GET /api/verdicttank/predictions/{review_id} +POST /v1/webhooks +Authorization: Bearer | X-API-Key: + +{ "callback_url": "https://customer.example.com/verdicttank-callback" } ``` -**Response `200 OK`:** -```json -{ - "review_id": "uuid", - "predictions": [ - { - "prediction_text": "string", - "expected_timeframe_days": 180, - "check_due_at": "iso8601", - "actual_outcome": "materialized | not_materialized | inconclusive | pending", - "outcome_confidence": 0.6, - "checked_at": "iso8601 | null" - } - ] -} -``` +Registers the notification callback described in 2.14. Enterprise tier only. --- -## 5. Pipeline Details +## 5. Pipeline Mechanics -### 5.1 Phase 1: Research Agent +### 5.1 Phase 1: Research -The research agent's mandate is narrow and specific: verify, don't opine. It receives the raw proposal text and produces a `ResearchBrief`. +The research agent has tool access (web search, domain lookup) and produces the `research_brief` (3.2). It runs first because every downstream phase, including the Market Simulation Engine (2.7, 5.4), consumes its output. -**Steps:** +### 5.2 Phase 2: Primary Review -1. **Claim extraction** — parse the proposal into a list of checkable factual assertions (market size figures, named competitors, cited statistics, pricing comparisons to named alternatives). -2. **Citation verification** — for each extracted claim, run live web search to find a corroborating or contradicting source. Every claim in the resulting brief carries a source URL and a verified/unverified flag. -3. **Competitor identification** — independent of what the proposal itself names, search for adjacent products/services in the same space and summarize their positioning. -4. **Domain and trademark checks** — verify availability of the proposed product's domain name and flag potential trademark conflicts in relevant jurisdictions. -5. **Market data gathering** — pull sourced, current market sizing and growth figures for the proposal's category, to be used later by both the Primary Reviewer's Financials dimension and the Market Simulation Engine. +The primary reviewer consumes the proposal text, the research brief (including any `simulated_trajectory`), and produces the `critic_review` (3.3): a ten-dimension score, fatal flaws, strengths, blind spots, and an initial verdict with any conditions attached. -If live web verification is degraded or unavailable mid-run, the agent proceeds with whatever citations it could gather and sets `limited_citations_flag = 1` on the brief. This flag propagates visibly into the final PDF (see Failure Modes 8.3) rather than being silently absorbed. +### 5.3 Phase 3: Parallel Judges -### 5.2 Phase 2: Primary Reviewer +Each judge receives the proposal, the research brief, and the critic review, and independently produces a `judge_verdict` (3.4). Judges run concurrently against their respective providers; writes are serialized per 2.4. -The Primary Reviewer consumes the proposal and the `ResearchBrief` and produces a `CriticReview` scored across ten fixed dimensions, each 1-10: +### 5.4 Market Simulation Sequencing -| Dimension | Focus | -|---|---| -| Name | Brand clarity, memorability, domain/trademark conflict from research brief | -| Pricing | Sanity of pricing tiers against comparable market pricing | -| PMF | Evidence of product-market fit vs. assumption | -| Competition | Competitive differentiation, informed by research brief's independent competitor list | -| Financials | Plausibility of unit economics and projections | -| GTM | Go-to-market channel selection and realism of CAC/timeline assumptions | -| Risk | Identified operational, legal, and market risks | -| Missing Elements | What the proposal fails to address at all | -| Founder Fit | Alignment between stated founder background and proposal execution demands | -| Overall Verdict | Synthesis dimension: go / conditional_go / no_go with attached conditions | +As established in 2.7, the Market Simulation Engine runs as the final step of Phase 1, after the research brief's core content is assembled but before the brief is handed to Phase 2. This keeps the phase diagram in 1.1 accurate: simulation is not a parallel, Phase-2-adjacent activity, it is the last research step. A review's `research_brief.simulated_trajectory` is therefore always either populated or explicitly null with `simulation_skipped = true` by the time Phase 2 begins. -Output also includes free-text `fatal_flaws`, `strengths`, and `blind_spots` lists, which feed directly into the PDF's Fatal Flaws and Action Plan sections. +### 5.5 Majority Aggregation -### 5.3 Phase 3: Parallel Cross-Check +Once all judges report (or time out, per 8.6), the orchestrator counts verdicts. A simple majority (2-of-3 in the base panel) determines the published verdict. A split panel is reported as-is (e.g., "2/3 majority, one dissent") rather than resolved by a tiebreaker model; dissent is preserved, not smoothed over, as stated in 1.1. -**Multi-vendor architecture diversity requirement.** Judges in the panel are required to span distinct underlying architecture families. This is enforced structurally, not by convention: the orchestrator checks `architecture_family` diversity across the active judge roster before a review can be marked complete. The rationale is that judges built on the same underlying architecture tend to share correlated blind spots; independence of judgment requires independence of substrate, not just independent prompting. +### 5.6 Concurrent Review Queue Serialization -**Minimum quorum.** A review requires a minimum of **2** judges to reach a valid verdict. Below that, no majority can meaningfully be computed. +The orchestrator is a single-worker process with respect to review execution: **one review is actively processed at a time**. Additional submitted reviews wait in a FIFO queue rather than being dispatched concurrently against the shared SQLite corpus database. This is the direct operational consequence of the concurrent-writer ceiling described in 2.4: rather than fight SQLite's single-writer constraint with retry loops and `SQLITE_BUSY` handling scattered across every write path, the design accepts one review in flight at a time and queues the rest. Given per-review turnaround of 3-15 minutes, a shallow queue clears quickly under normal load; queue depth is capped at 50 pending reviews (5.9, 8.13), beyond which new submissions receive `429`. -**Degraded-mode fallback.** If one judge in an N-judge panel times out or crashes, the orchestrator proceeds with the remaining N-1 judges rather than failing the whole review, provided N-1 still meets the minimum quorum of 2. The review record and PDF explicitly note that the panel ran in degraded mode. If a crash brings the panel below quorum, see Failure Modes 8.2. +### 5.7 Corpus Scale Threshold -**v2 baseline panel** (three judges): a general critic, a devil's-advocate contrarian, and a domain-specialist judge, each on a distinct architecture. +The practical comfort ceiling for the current single-file SQLite design, combining WAL mode and single-worker write serialization, is approximately **5,000 reviews or 50 GB of corpus data**, whichever comes first. This figure is about storage and read-scan performance (particularly the embedding cosine-scan in 2.9, which is a linear scan over searchable rows), not a hard SQLite file-size limit; SQLite itself supports databases far larger than this. The trigger for migrating off SQLite is **write concurrency exceeding what single-worker serialization can absorb**, not corpus size in isolation; a corpus well past 5,000 reviews with light write volume is less urgent to migrate than a smaller corpus experiencing frequent `SQLITE_BUSY` errors under the queue model in 5.6. -**v3 panel expansion** (three to seven judges): adds three specialist roles: +### 5.8 Provider Failover Walkthrough -- **Reasoning-Verification Judge** — focused specifically on catching numerical and logical inconsistencies across the proposal and the Primary Reviewer's own scoring (does the stated CAC math actually support the stated LTV claim, etc.). -- **Execution-Feasibility Judge** — grades pure operational feasibility: given the team, timeline, and budget described, can this actually be built and shipped as scoped. -- **Market-Reality Judge** — deliberately configured to be less agreeable; its mandate is a contrarian read specifically on market timing and demand claims, resisting the tendency of multi-judge panels to converge toward polite consensus. +Given the configuration in 3.8: a judge call to its primary provider fails (timeout, 5xx, rate limit). The orchestrator retries the same provider up to 3 times with exponential backoff (2s, 4s, 8s). If all three retries fail, the orchestrator escalates to the role's secondary provider and repeats the retry sequence there. If a tertiary provider is configured for that role (as with the research agent) and the secondary also exhausts its retries, the orchestrator escalates once more. If every configured provider for a role is exhausted, that role's output is treated as a hard failure for the review, handled per Failure Mode 8.6: a single judge failing below quorum falls back to a 2-judge panel with a disclosure note on the report; the primary reviewer or research agent failing entirely halts the review and surfaces a `failed` status to the customer with automatic notification. -### 5.4 v3 Additions to the Pipeline +### 5.9 Edge Case Handling -- **Vertical auto-classification** runs early (immediately after claim extraction) and determines which domain-specific dimensions and red-team checks apply downstream. Classification output (`saas`, `consulting`, `marketplace`, `fintech`, etc.) is stored on the review record and drives template selection for both the Primary Reviewer and the PDF. -- **Adversarial red-team per vertical** — an additional targeted check applied after classification, tailored to the vertical's characteristic failure mode: HIPAA/compliance exposure for healthcare, regulatory licensing exposure for fintech, churn-driver scrutiny for consumer SaaS. This runs as an additional structured pass, not a replacement for the general critique. -- **Market simulation** (Section 2.7) feeds a projected trajectory into the Financials dimension and is available to judges as a data point distinct from the proposal's own stated projections. +The pipeline is deliberately explicit about the edges of its input space rather than silently degrading or erroring opaquely: -### 5.5 The Modular Judge System - -Judges are configured, not hardcoded. Each judge in the active roster is defined by: a role identifier, a prompt/instruction template, an architecture family tag (for diversity enforcement), and an active weight (currently always 1.0 pending Phase 4). - -**Adding a judge**: define role identifier, instruction template, and architecture family; register in the active roster config; the orchestrator picks it up on the next review without a pipeline code change. - -**Removing a judge**: deactivate in roster config; historical `judge_verdict` rows referencing that `judge_id` remain in the corpus for accuracy scoring continuity. - -**Weighting** (Phase 4 and beyond): `accuracy_score.running_accuracy` becomes an input to a weighted majority calculation rather than a simple headcount majority. Until Phase 4 ships, weighting is display-only on the accuracy dashboard and has zero effect on verdict aggregation. This staged rollout exists specifically to observe accuracy scoring for the consensus-drift risk described in 7.6 before it can move a real verdict. +- **Proposals exceeding token limit**: proposals are hard-capped at **25,000 words (approximately 37,000 tokens)** at submission time. A proposal over that cap is rejected at `POST /v1/reviews` with a `413` and a clear message stating the cap. This is a hard cap, not a soft warning: chunked processing with summary aggregation was considered and rejected for Phase 1 because it would silently change what "the proposal" means to each pipeline stage; it remains a candidate technique for a future phase if genuinely long-form proposals become common, but is not implemented now. +- **Non-English proposals**: language is detected at submission time (`review.language_detected`). **Phase 1 supports English only.** A non-English proposal is rejected with a clear error message identifying the detected language and stating that English is currently required, rather than being silently run through the pipeline and producing a low-quality or nonsensical review. +- **Concurrent review queue**: per 5.6, the orchestrator processes one review at a time. The submission queue has a **depth limit of 50 pending reviews**; a submission when the queue is at capacity receives `429` with a `Retry-After` hint. Monitoring alerts at 80% of that capacity (40 pending), per 8.13. +- **Duplicate submission**: `proposal_hash` (a content hash of the normalized proposal text) is checked at submission time. A match against an existing review returns that review's existing link and status with a `"This proposal was reviewed on [date]"` notice, rather than re-running (and re-billing for) an identical review. +- **Corpus scale threshold**: see 5.7. Restated here because it is as much an edge case as a capacity fact: the system's behavior at and beyond that threshold is degraded query latency and elevated `SQLITE_BUSY` risk, not silent data loss, and the migration trigger is write contention, not size alone. --- @@ -486,7 +492,7 @@ Judges are configured, not hardcoded. Each judge in the active roster is defined | 1 | Cover | Client name, proposal title, verdict badge, date | | 2 | Executive Summary | One-paragraph synthesis of the majority verdict and its rationale | | 3 | Score Table | All ten `CriticReview` dimension scores, tabulated | -| 4 | Research | Verified claims, discrepancies, competitor analysis, domain/trademark findings | +| 4 | Research | Verified claims, discrepancies, competitor analysis, domain/trademark findings, market simulation summary when applicable | | 5 | Fatal Flaws | Enumerated list from `CriticReview.fatal_flaws`, cross-referenced against judge disagreements | | 6 | Action Plan | Concrete, prioritized remediation steps derived from fatal flaws and conditions | | 7 | Judge Cards | One card per judge: verdict, confidence, headline finding | @@ -511,17 +517,17 @@ flowchart LR 1. The completed review record (all tables joined) is serialized to a single JSON payload. 2. That payload fills a Jinja-style HTML template implementing the 11-section structure, with conditional blocks for vertical-specific sections. -3. The rendered HTML is passed to WeasyPrint, which produces the PDF binary directly from HTML/CSS, with no intermediate manual step. +3. The rendered HTML is passed to WeasyPrint, which produces the PDF binary directly from HTML/CSS, with no intermediate manual step. This step runs under a memory cap and a hard timeout; see 8.9 and 10.5. 4. The draft PDF is passed through the post-generation validation script before being written to the public `/reports/` path. ### 6.3 Post-Generation Validation Script Every generated PDF must pass all of the following checks before being served: -- **Em dash / double-hyphen scan** — the report's prose must not contain em dashes or double hyphens (house style rule enforced automatically, not just at prompt level). -- **Section presence check** — all 11 (or 10, if vertical analysis does not apply) expected sections must be present and non-empty. -- **Page count minimum** — the rendered PDF must meet a minimum page count threshold; a report that renders suspiciously short indicates a template fill failure upstream. -- **Sanitization scan** — the same architecture-name and infrastructure-detail scan used by the deploy-time sanitization gate (Section 2.6) also runs here, per-report, before public write. +- **Em dash / double-hyphen scan**: the report's prose must not contain em dashes or double hyphens (house style rule enforced automatically, not just at prompt level). +- **Section presence check**: all 11 (or 10, if vertical analysis does not apply) expected sections must be present and non-empty. +- **Page count minimum**: the rendered PDF must meet a minimum page count threshold; a report that renders suspiciously short indicates a template fill failure upstream. +- **Sanitization scan**: the same two-layer architecture-name and behavioral-fingerprint scan used by the deploy-time sanitization gate (Section 2.6, expanded in 7.5 and 8.12) also runs here, per-report, before public write. A validation failure triggers one automatic retry of the full generation pipeline. If the retry also fails, the system falls back to a plaintext summary (see Failure Modes 8.4) rather than serving a broken or non-compliant PDF. @@ -529,17 +535,27 @@ A validation failure triggers one automatic retry of the full generation pipelin The v2 baseline generated PDFs through a partially manual per-review process. The Phase 0 infrastructure prerequisite scripts this end-to-end: submission to PDF delivery requires zero manual intervention under normal operation. Manual generation remains available as an operator-invoked fallback tool for support cases (e.g., regenerating a report after a template fix), but is not part of the default customer-facing path. +### 6.5 Resource Constraints on the Render Step + +WeasyPrint's HTML/CSS-to-PDF render is the single most resource-intensive step in the generation pipeline, particularly for reports with large tables (Judge Notes, Citations) or long proposal text echoed into the Research section. The render step runs with a hard memory cap and a hard wall-clock timeout, not unbounded; see 8.9 for the specific limits and the fallback behavior when either is exceeded, and 10.5 for the native OS dependencies the render step requires. + --- ## 7. Security Model ### 7.1 Prompt Injection via Proposal Text -Proposal text is user-supplied free text and is treated as untrusted input throughout the pipeline. Mitigations: +Proposal text is user-supplied free text and is treated as untrusted input throughout the pipeline. The previous framing of this section described the claim extractor as a filter that runs before agent exposure; that framing understated the actual exposure surface. **The claim extractor itself is an LLM-driven step and is therefore the first point of exposure to untrusted proposal text, not a pre-exposure filter that keeps injected content away from a model.** The claim extractor reads and reasons over the raw proposal text directly. Defense against injection is therefore layered, not front-loaded into a single filtering stage: -- **Structured parsing before agent exposure** — the claim-extraction step in Phase 1 parses the proposal into discrete structured claims before any downstream agent reasons over it as a single blob, reducing the surface for injected instructions to be interpreted as system-level directives. -- **Role-boundary reinforcement** — every agent's instruction template explicitly frames proposal content as data to be evaluated, not instructions to be followed, and this framing is tested as part of the sanitization gate's broader remit. -- **Output re-validation** — the sanitization gate and PDF validation script both scan final output for signs that instructions embedded in a proposal leaked into the report's own voice or structure. +- **Pre-extraction input sanitization**: before the proposal text reaches any LLM, including the claim extractor, a regex-based sanitization pass strips or neutralizes common injection patterns: role-override attempts ("ignore previous instructions", "you are now", "disregard the above"), fake system-turn markers ("system:", "assistant:", "### instruction"), and markdown code-block injection attempts that try to smuggle instructions as fenced code intended for a different rendering context. This is a blunt, pattern-based first pass, not a semantic understanding of intent; it exists to raise the cost of the most common injection techniques, not to guarantee immunity to novel ones. +- **Structured parsing before further agent exposure**: once past sanitization, the claim-extraction step parses the proposal into discrete structured claims before any downstream agent (primary reviewer, judges) reasons over it as a single blob, reducing the surface for injected instructions to be interpreted as system-level directives by those later stages, even though the extractor itself remains exposed to the raw text. +- **Role-boundary reinforcement**: every agent's instruction template explicitly frames proposal content as data to be evaluated, not instructions to be followed, and this framing is tested as part of the sanitization gate's broader remit. +- **Output re-validation**: the sanitization gate and PDF validation script both scan final output for signs that instructions embedded in a proposal leaked into the report's own voice or structure. +- **Output behavioral anomaly detection (v3, supplementary signal)**: beyond string and pattern matching on output, the system tracks statistical outliers in scoring and verdict metadata across the review population: score distributions that deviate sharply from a proposal's vertical/tier peer group, judges converging to unusually high or identical confidence values, or verdict metadata that does not match the shape of a normal review. These signals do not block a review on their own; they queue a review for manual inspection. This is a supplementary, lagging detection layer on top of the preventive layers above, not a replacement for them. +- **Red-team testing protocol (Phase 0 deliverable)**: a documented, repeatable set of adversarial proposal inputs, covering the injection categories above plus proposal-text-embedded attempts to extract architecture identity, internal infrastructure details, or system prompts, is run against the pipeline before Phase 0 is considered complete, and re-run against each subsequent phase that changes agent prompting or the sanitization gate's pattern bank. Red-team results and any newly discovered bypass techniques are fed back into the sanitization pattern bank (2.6, 7.5, 8.12) as part of the false-positive/pattern-update procedure. + +!!! warning "This is defense in depth, not a guarantee" + No layer above claims to make the claim extractor immune to injected content it is directly exposed to. The combination of pre-extraction sanitization, structured downstream parsing, output re-validation, and anomaly detection is designed to make successful injection difficult and, when it does occur, detectable after the fact, not to make the exposure itself disappear. ### 7.2 API Key Management @@ -551,6 +567,9 @@ Proposal text is user-supplied free text and is treated as untrusted input throu - Completed PDFs and public share assets live under `/reports/`, served directly by the reverse proxy as static files. - **Directory listing is disabled** on this path at the proxy level; a report is only retrievable by its specific, unguessable identifier-based URL. +- **Identifier generation**: the report URL token is generated via `uuid4()` or an equivalent `os.urandom`-derived value, chosen specifically for unguessability (128 bits of entropy, no sequential or timestamp-derived component). It is a distinct value from the internal `review_id`, per 4.3. +- **Optional expiration**: a report URL may be configured to expire after a set period, defaulting to **90 days** when expiration is enabled for a given tier or account. `review.report_url_expires_at` (3.1) carries this value; a request against an expired URL receives `410 Gone` with guidance to re-authenticate and request a fresh link through the authenticated dashboard. +- **Access logging**: every retrieval of a report, whether via the public unguessable URL or the authenticated `GET /api/verdicttank/review/{id}/pdf` path, is logged with timestamp, source IP, and requested identifier. This log is the primary detection mechanism for URL-guessing attempts or unexpected sharing patterns, given that the URL itself carries no authentication. - Reports for non-Free tiers are not indexed or discoverable; only the customer with the corresponding `review_id` (and valid session/API key) can request the signed download link via `GET /api/verdicttank/review/{id}/pdf`. ### 7.4 Corpus Confidentiality @@ -558,26 +577,36 @@ Proposal text is user-supplied free text and is treated as untrusted input throu The corpus is the platform's most sensitive asset: it aggregates other companies' unreleased business proposals. - **Encryption at rest** for the corpus database file. -- **Per-user opt-out** — `review.corpus_opt_out` lets any customer exclude their review from search indexing and comparisons entirely; opt-out is respected at the `corpus_record.searchable` level, not just a UI-layer filter. -- **Aggregate-only comparisons** — competitive comparison features (percentile vs. corpus median) expose only score distributions and vertical classification, never proposal text, to any account other than the review's owner. +- **Per-user opt-out**: `review.corpus_opt_out` lets any customer exclude their review from search indexing and comparisons entirely. Opt-out is enforced at the point of embedding generation itself, not after the fact: for `corpus_opt_out = 1` reviews, **no embedding is generated and no `corpus_record` row is created** (2.9, 3.5), rather than generating the embedding and merely flagging it non-searchable. A late opt-out on a review that already has a `corpus_record` triggers deletion of that row and its embedding, logged to this section's audit trail. +- **Raw database access model**: access to the raw corpus database file itself (as opposed to the API's scoped, filtered views of it) is **single-operator, all-or-nothing**, stated here as an explicit design assumption rather than left implicit. There is no row-level or column-level access control layer between an operator with filesystem access to the database and the full, unfiltered contents of every review ever submitted, including opted-out reviews' underlying `review` rows (opt-out removes a review from corpus search and embedding, it does not remove the review record itself, which remains needed for the customer's own report retrieval and billing history). This assumption is acceptable at current operational scale (one operator, one host) and must be explicitly revisited, not silently inherited, if the operator model changes (e.g., additional staff with database access, a managed-service tier with third-party operators). +- **Aggregate-only comparisons**: competitive comparison features (percentile vs. corpus median) expose only score distributions and vertical classification, never proposal text, to any account other than the review's owner. +- **`summary_snippet` construction rule**: as specified in 4.4, the `corpus_record.summary_snippet` field surfaced through corpus search is a fixed-length structural summary built only from `vertical`, `verdict`, and a score-range bucket. It is never generated as a derivative, excerpt, or embedding-nearest-sentence pull from the original proposal text. This closes a specific confidentiality gap: a "helpful" free-text summary generated from proposal content would risk leaking substantive proposal details through the back door of a search result, even with `corpus_opt_out` respected and encryption at rest in place. - This was flagged as a named risk in the v3 proposal review (corpus confidentiality liability) and the above controls are the direct mitigation; see Failure Modes for the case where a control fails. ### 7.5 Sanitization Gate -Covered in detail in 2.6 and 6.3. Security framing: this is the platform's primary defense against leaking architecture-identifying details or internal infrastructure references into any public-facing surface (shareable reports, OG images, docs, public API responses). It is deploy-blocking, not advisory, precisely because a leak here is a competitive and trust failure that is hard to walk back once a report has been shared publicly. +Covered in detail in 2.6 and 6.3. Security framing: this is the platform's primary defense against leaking architecture-identifying details, behavioral self-identification patterns, or internal infrastructure references into any public-facing surface (shareable reports, OG images, docs, public API responses). It is deploy-blocking, not advisory, precisely because a leak here is a competitive and trust failure that is hard to walk back once a report has been shared publicly. + +- **Layer 1 (string matching)**: a maintained list of architecture-identifying strings and internal infrastructure references, as in v2. +- **Layer 2 (behavioral fingerprint detection, v3)**: a maintained regex pattern bank targeting self-identification phrases that recur across model families regardless of the specific vendor name being present ("as a language model," "my training data includes," "I cannot browse the internet," "I don't have the ability to," "my knowledge cutoff is"), training-cutoff date patterns, and capability-boundary language. This layer exists because architecture identity leaks more often through phrasing habits than through an accidental vendor name, and a string-match list alone cannot catch a model describing its own limitations in generic but still identifying language. +- **False-positive procedure**: the gate remains **blocking by default**, with no change to that policy. When a block is believed to be a false positive, the procedure is: human review of the flagged content, and if confirmed as a false positive, the specific triggering pattern is added to an allowlist scoped as narrowly as possible (ideally to the exact phrase-in-context, not a broad pattern removal), the gate is re-run to confirm the deploy or report now passes, and a mandatory audit log entry records the pattern, the reviewer, the timestamp, and the justification. This procedure exists so that legitimate content is not permanently blocked by an overly broad pattern, without weakening the gate's default-blocking posture or leaving allowlist changes unaudited. ### 7.6 API Abuse Protections -- **Rate limiting** at the reverse proxy and API layers, tuned per tier. -- **Anomaly detection** on submission patterns — e.g., a burst of near-identical proposal submissions from one account, which could indicate an attempt to probe the pipeline's judge behavior or extract architecture information through differential prompting. +- **Rate limiting** at the reverse proxy and API layers, tuned per tier, with the polling endpoint (`GET /v1/reviews/{review_id}`, 4.2) rate-limited **independently** from the submission endpoint (`POST /v1/reviews`, 4.1). These are different abuse surfaces: submission abuse is about volume and cost, polling abuse is about probing pipeline internals (see the timing oracle note below and in 9.5) and about scraping status data at a rate disproportionate to legitimate dashboard usage. +- **Free-tier submission content monitoring**: free-tier submissions are monitored for known-good probing patterns, patterns consistent with an account systematically testing pipeline behavior with minor input variations (near-duplicate proposals differing only in specific test phrases) rather than submitting genuine business proposals. This monitoring feeds manual account review, per the anomaly-detection policy below, not automated suspension. +- **Anomaly detection** on submission patterns: e.g., a burst of near-identical proposal submissions from one account, which could indicate an attempt to probe the pipeline's judge behavior or extract architecture information through differential prompting. - Abuse detection findings feed into manual account review rather than automated suspension, to avoid false-positive lockouts on legitimate high-volume Enterprise/White-Label usage. +- **Timing oracle via polling (acknowledged low-severity risk)**: because a review passes through observable phase transitions (`status` field values in 3.1) at different, somewhat characteristic latencies, an attacker polling frequently could in principle infer something about pipeline composition (e.g., roughly how many phases exist, or that a particular phase took unusually long, suggesting a provider failover event per 5.8) purely from timing, without ever seeing pipeline internals directly. This is acknowledged explicitly as a low-severity side channel with no identified practical exploitation path at current scale; see 8.15 and 9.5. It is not actively defended against beyond the rate limiting above, and is called out here rather than left undocumented. ### 7.7 Training-Data Recursion Prevention A structural risk unique to this kind of pipeline: if the panel's own review output were ever used, directly or indirectly, to further train or fine-tune models used by the panel itself, judge diversity would collapse over time as the panel converges on its own prior outputs. -- **Volume caps** on any data pipeline that could plausibly feed review output back toward model training. -- **Agreement-rate monitoring** — `accuracy_score.agreement_rate` and `diversity_score` are tracked over time specifically to detect a drift toward artificial consensus (judges agreeing with each other more, and more often, than architectural independence would predict). A sustained upward drift in agreement rate across the panel is treated as an operational signal to investigate, not just a marketing metric. +- **"No training on API usage" contractual requirement**: every judge model provider under contract must have an active "no training on API usage" term in its API terms of service or a negotiated data processing agreement to that effect. This is a procurement and contract-management requirement, not a purely technical control, and it is tracked as a standing condition of each provider relationship, re-verified whenever a provider's terms of service change. +- **Volume caps** on any data pipeline that could plausibly feed review output back toward model training, as a defense-in-depth measure independent of the contractual requirement above. +- **Agreement-rate monitoring**: `accuracy_score.agreement_rate` and `diversity_score` are tracked over time specifically to detect a drift toward artificial consensus (judges agreeing with each other more, and more often, than architectural independence would predict). A sustained upward drift in agreement rate across the panel is treated as an operational signal to investigate, not just a marketing metric. +- **Honest limitation**: agreement-rate monitoring is **lagging detection**, not leading prevention. It can reveal that consensus has already drifted; it cannot prevent the drift from occurring, and it cannot fully distinguish organic convergence (the proposal genuinely warrants unanimous agreement) from recursion-driven convergence. Training-data recursion risk is therefore treated as **partially inherent to any multi-model pipeline** that depends on providers' training practices remaining as represented; the contractual requirement above is the primary control, and monitoring is a backstop, not a solution. This is restated in the Inherent Risks subsection, 8.14. --- @@ -589,17 +618,31 @@ A structural risk unique to this kind of pipeline: if the panel's own review out | 8.2 | All judges fail, or surviving judges fall below quorum | No valid verdict can be computed | Review is marked `failed`; customer is offered a free re-run at no charge against their quota | | 8.3 | Research agent web verification failure | Citations incomplete or absent | Pipeline proceeds with `limited_citations_flag = 1`; flag is surfaced visibly in the final PDF's Research section rather than silently omitted | | 8.4 | PDF generation failure (post-validation) | No compliant PDF produced | One automatic retry of the full generation pipeline; if retry also fails, deliver a plaintext summary fallback and flag the review for manual PDF regeneration | -| 8.5 | Corpus database corruption | Loss of search, prediction tracking, and accuracy scoring continuity | Restore from nightly backup (Section 10.4); any reviews written between last backup and corruption event are re-derived from review-record source data where still available | -| 8.6 | Model/architecture provider outage | One or more judges or the primary reviewer unavailable | Failover chain routes affected role to an alternate configured provider/architecture within the same role; if no failover is configured for that role, treat as case 8.1 or 8.2 depending on scope | -| 8.7 | Sanitization gate failure (a scan match is found) | Deploy or report publication is blocked | This is by design: the gate blocks rather than warns. Operator must resolve the flagged content (redact/rephrase) before the deploy or report can proceed. Failure is never silently bypassed | +| 8.5 | Corpus database corruption | Loss of search, prediction tracking, and accuracy scoring continuity | Restore from the most recent WAL checkpoint (Section 10.4, at most 15 minutes of writes lost); any reviews written between last checkpoint and corruption event are re-derived from review-record source data where still available | +| 8.6 | Model/architecture provider outage | One or more judges or the primary reviewer unavailable | Failover chain (3.8, 5.8) routes affected role through its configured provider list with retry and backoff; if every configured provider for that role is exhausted, treat as case 8.1 or 8.2 depending on scope | +| 8.7 | Sanitization gate failure (a scan match is found) | Deploy or report publication is blocked | This is by design: the gate blocks rather than warns. Operator must resolve the flagged content (redact/rephrase) before the deploy or report can proceed, or follow the false-positive procedure in 7.5/8.12 if the match is confirmed spurious. Failure is never silently bypassed | +| 8.8 | SQLite write contention under concurrent judges | Write attempts raise `SQLITE_BUSY` or block | WAL mode (2.4) plus single-worker queue serialization (2.4, 5.6) is the primary mitigation; within-review judge writes are serialized through the orchestrator's single write path rather than issued as N simultaneous transactions, which eliminates contention for the common case. A `SQLITE_BUSY` that still occurs is retried with backoff at the write layer before surfacing as an error | +| 8.9 | WeasyPrint memory exhaustion on large reports | Render process killed or hangs, report never produced | Render step runs under a 2GB memory cap (cgroups) and a 120-second timeout; a cap or timeout breach kills the render and triggers the automatic retry described in 8.4, and a second failure falls back to the plaintext summary rather than a partial or corrupted PDF | +| 8.10 | Sanitization gate false positive | Legitimate content blocked from deploy or publication | Human review confirms the false positive; the gate remains blocking by default (no change to 8.7 policy). A human override path exists specifically for confirmed false positives, requiring a mandatory audit log entry documenting the override, per 7.5/8.12 | +| 8.11 | Prediction cron silent failure | `prediction.actual_outcome` remains perpetually `pending`, quietly degrading Tier 1 cumulative intelligence with no loud failure signal | Cron runs as a systemd timer (2.8, 10.6) with `OnFailure=` wired to a notification unit, plus a separate missing-run detection check (a run that should have fired but did not, distinct from a run that fired and errored) that pages the operator on either condition | +| 8.12 | Confident-but-wrong judge analysis | A judge issues a high-confidence verdict that is substantively incorrect, with no technical signal distinguishing it from a correct high-confidence verdict | No technical prevention exists for this case; it is addressed as a product disclosure to users (verdicts are probabilistic assessments, not guarantees) rather than a solved engineering problem. Detection is via outlier analysis post-hoc (7.1's behavioral anomaly detection, and accuracy scoring's eventual outcome tracking in 2.8), not at verdict time | +| 8.13 | Concurrent review queue saturation | New submissions cannot be accepted | Queue cap of 50 pending reviews (5.6, 5.9); submissions beyond the cap receive `429` with `Retry-After`; a monitoring alert fires at 80% of capacity (40 pending) so the operator has lead time before the cap is actually hit | !!! danger "Sanitization gate failures are not incidents to route around" - If the sanitization gate blocks a deploy or a report, the correct response is to fix the flagged content, not to disable or bypass the gate to unblock a release. A bypass here directly reintroduces the leak risk the gate exists to prevent. + If the sanitization gate blocks a deploy or a report, the correct response is to fix the flagged content or follow the documented false-positive procedure, not to disable or bypass the gate to unblock a release. A bypass here directly reintroduces the leak risk the gate exists to prevent. -### 8.1 Attribution Difficulty in Prediction Tracking +### 8.15 Attribution Difficulty in Prediction Tracking Not a system failure in the crash sense, but a known limitation worth documenting alongside the other failure modes: the prediction-vs-outcome cron's `actual_outcome` assessment is inherently a correlation judgment, not a causal one. A flagged prediction ("this pricing tier will suppress conversion") that appears to materialize by T+180 may have done so for unrelated reasons. `outcome_confidence` on the `prediction` record exists specifically to carry this uncertainty forward rather than presenting the cron's assessment as ground truth; accuracy scoring calculations weight predictions by this confidence rather than treating every checked prediction as a binary hit/miss. +### 8.14 Inherent Risks + +Three risks in this architecture do not have a full technical solution and are documented here explicitly as accepted, ongoing risk rather than as failure modes with a recovery procedure: + +1. **Training-data recursion is partially inherent.** As stated in 7.7, the agreement-rate/diversity-score monitor is lagging detection, not leading prevention, and cannot fully distinguish organic consensus from recursion-driven consensus. The primary control is contractual (7.7's "no training on API usage" requirement), and no fully technical solution exists that guarantees judge independence indefinitely as long as the panel depends on third-party model providers whose training practices cannot be directly audited. +2. **Timing oracle via the polling endpoint.** As stated in 7.6, phase-transition timing observed through repeated polling could in principle reveal something about pipeline composition. This is assessed as low severity, and no practical exploitation vector has been identified: the information such an analysis could extract (rough phase count, occasional failover events) does not appear to translate into a meaningful competitive or security compromise at this time. It remains documented rather than dismissed, in case that assessment changes as the platform's usage or attacker sophistication increases. +3. **Concurrent queue saturation is a capacity design choice, not just a failure mode.** The 50-pending-review cap (5.6, 5.9, 8.13) is not an incidental limit that emerged from the SQLite single-writer constraint; it is a deliberate capacity decision for a single-server deployment. Raising it would require either accepting longer queue wait times under load, or the SQLite migration discussed in 5.7, not simply changing a config value. This is stated plainly so that "the queue cap is too low" is understood as a capacity-planning conversation, not a bug report. + --- ## 9. Cost Model @@ -617,7 +660,8 @@ Not a system failure in the crash sense, but a known limitation worth documentin |---|---| | Base pipeline (research + primary review) | carried forward from v2 baseline | | Specialist judges (up to 7-judge panel) | incremental per additional judge | -| Market simulation engine | fixed per-run compute cost | +| Market simulation engine | fixed per-run compute cost, incurred once per review ahead of Primary Review (5.2) | +| Embedding generation (opt-in reviews only) | marginal, per-review; skipped entirely for opt-out reviews (2.9) | | Corpus write and indexing | marginal, per-review | | Infra amortization | Caddy/API/WeasyPrint/cron overhead spread across run volume | | **Total v3 full pipeline** | **$0.86/run** | @@ -638,28 +682,33 @@ Free tier deliberately runs a reduced single-reviewer pipeline (no full judge pa - **AI cost deflation assumption**: underlying inference costs are modeled to decline 15-20% annually based on historical trend, which is factored into margin projections for Enterprise and White-Label tiers over a multi-year horizon. This is an assumption, not a guarantee, and margin models should be re-validated against actual provider pricing at each planning cycle. - **Judge panel cost optimization**: judge count is a tunable parameter, not a fixed constant. The system supports running fewer judges for cost-sensitive contexts (e.g., Free tier) and more for Enterprise/White-Label, and the accuracy-vs-cost ratio (accuracy dashboard metrics against per-run cost) is the intended basis for deciding whether panel size should grow further, hold, or shrink for a given tier. The three-judge minimum-viable panel and the seven-judge maximal panel bound this tradeoff space; see 7.6 and 8.2 for the operational floor (minimum quorum of 2). +### 9.5 Cost and Risk of the Timing Side Channel + +Noted here for completeness alongside the cost table, since it is adjacent to per-request economics: the timing oracle risk described in 7.6 and 8.14 has no cost impact of its own (it does not consume additional inference spend), but it is worth tracking whether polling-based probing correlates with elevated API request volume from a given account, since that would show up in per-account infra cost before it shows up as a security incident. + --- ## 10. Deployment Topology ### 10.1 Single-Server Architecture -The platform runs as a single-server deployment: one host running the API server process, the Caddy reverse proxy, the SQLite corpus database file, and the WeasyPrint PDF generation process. This is an intentional simplicity choice given current scale; there is no distributed consensus, no service mesh, and no multi-region failover at this stage. +The platform runs as a single-server deployment: one host running the API server process, the Caddy reverse proxy, the SQLite corpus database file, and the WeasyPrint PDF generation process. This is an intentional simplicity choice given current scale; there is no distributed consensus, no service mesh, and no multi-region failover at this stage. Provider-level failover (5.8) covers model/architecture outages; it does not cover the host itself. ```mermaid flowchart TB subgraph Host[Single Application Host] - Caddy[Caddy v2 — TLS + routing + static /reports/] - API[API Server Process] - DB[(SQLite Corpus DB)] - PDF[WeasyPrint Process] - Cron[Prediction Tracking Cron] + Caddy[Caddy v2 - TLS + routing + static /reports/] + API[API Server Process - systemd managed] + DB[(SQLite Corpus DB - WAL mode)] + PDF[WeasyPrint Process - memory/timeout capped] + Cron[Prediction Tracking Cron - systemd timer] end Internet -->|HTTPS| Caddy Caddy --> API API --> DB API --> PDF Cron --> DB + API -.->|15min WAL checkpoint| S3[(S3-Compatible Backup, encrypted)] ``` ### 10.2 DNS and TLS @@ -672,13 +721,29 @@ flowchart TB - **Documentation** (this site): built with MkDocs (Material theme) and deployed as static output. - **API and pipeline code**: deployed via a standard git-push-triggered pipeline. The sanitization gate (Section 2.6) runs as a required check in this pipeline for any change touching public-facing output paths; a gate failure blocks the deploy outright. - Deploys to the API layer and deploys to the docs site are independent pipelines and can ship on separate cadences. +- **Dependency pinning**: `requirements.txt` pins exact versions with hashes (`pip install --require-hashes`), so a deploy always installs the exact, previously-verified set of packages rather than resolving against whatever the latest compatible versions happen to be at deploy time. ### 10.4 Backup Strategy -- **Nightly backups** to off-host object storage (S3-compatible), covering the full application state including the corpus database file. +- **WAL-based continuous backup**: rather than a nightly-only snapshot, the corpus database's WAL file is checkpointed to off-host, S3-compatible object storage every **15 minutes**, matching the pattern used for Hermes's own backup strategy. This bounds worst-case data loss on corruption or host failure to roughly 15 minutes of writes, versus up to 24 hours under a nightly-only scheme. +- A **full nightly snapshot** is retained in addition to the 15-minute checkpoints, giving a clean daily restore point independent of WAL replay correctness. +- **Backup encryption**: backups are encrypted with a key **separate from and independent of host-level disk encryption**. Host disk encryption protects the live system against physical media theft; backup encryption protects the offsite copy against compromise of the storage provider or backup credentials, and the two are not treated as substitutes for each other. - The corpus is treated as first-class backup content, not an afterthought: it is the platform's accumulated institutional knowledge (Tier 1 cumulative intelligence) and its loss would silently degrade corpus search, prediction tracking, and accuracy scoring without necessarily causing an immediately visible outage. - Restore procedure for corpus corruption is covered in Failure Modes 8.5. +### 10.5 Native Dependencies (WeasyPrint) + +WeasyPrint's PDF rendering depends on several native system libraries that are not installable via pip and must be present on the host as apt packages: **cairo**, **pango**, **gdk-pixbuf**, **libffi**, and **harfbuzz**. These are documented explicitly here because a pip-only dependency list (`requirements.txt`) will install the WeasyPrint Python package successfully while still failing at render time if these system packages are absent, a failure mode that is easy to miss in a deploy checklist that only checks Python dependencies. + +- **Font availability**: branded PDF reports depend on the client's chosen or the platform's default brand fonts being installed at the OS level and discoverable by WeasyPrint's font stack. A missing font does not hard-fail the render; it silently substitutes a fallback font, which is a correctness issue (broken branding) rather than a crash, and is checked for as part of the post-generation validation script (6.3) rather than assumed correct by default. + +### 10.6 Process Management and Graceful Shutdown + +- **Systemd unit**: the API server and orchestrator run as a systemd service with `Restart=on-failure` and `RestartSec=5`, so a crashed process comes back automatically without manual intervention, with a short delay to avoid a tight crash-restart loop against a persistently failing dependency. +- **Log rotation**: process output is captured via systemd's journald (the default under a systemd unit) with journald's own rotation/retention policy, or via logrotate for any component that writes to flat log files instead. Logs are not left to grow unbounded on the single application host. +- **Secrets management**: credentials (model/architecture provider API keys, database encryption keys, backup credentials) are supplied as environment variables via a systemd `EnvironmentFile`, not hardcoded in source or committed to the repository. The `EnvironmentFile` itself is filesystem-permission-restricted to the service's running user. +- **Graceful shutdown on deploy**: a deploy sends `SIGTERM` to the running process rather than a hard kill. The process's shutdown handler stops accepting new review submissions immediately, then drains in-flight reviews already in progress, allowing up to **15 minutes** for any review that is mid-pipeline at shutdown time to reach completion before the process actually exits and is replaced. This avoids a deploy silently killing a review that was seconds from finishing Phase 3. + --- ## 11. Operations @@ -692,25 +757,34 @@ flowchart TB | PDF generation failure rate | Fraction of reviews requiring PDF retry or falling back to plaintext (Section 8.4) | | Model/architecture latency | Per-role latency for research, primary review, and each judge, used to catch a specific provider degrading before it causes a full timeout | | Panel diversity health | `agreement_rate` / `diversity_score` trend across the judge panel over time (Section 7.7) | -| Corpus growth rate | Reviews indexed per period, used to sanity-check corpus and backup sizing assumptions | +| Corpus growth rate | Reviews indexed per period, used to sanity-check corpus and backup sizing assumptions against the 5,000-review / 50GB comfort ceiling (5.7) | +| SQLite write contention rate | Count of `SQLITE_BUSY` occurrences per period (Section 8.8); a rising trend is the leading indicator for the write-concurrency migration trigger described in 5.7, well before size alone would suggest one | +| Queue depth | Current pending-review count against the 50-review cap (5.6, 8.13); alerts at 80% | +| Report access anomalies | Count of report retrievals flagged by the access logging in 7.3, used to spot URL-guessing attempts | ### 11.2 Alerting Thresholds - Review success rate dropping below an agreed operational floor over a rolling window triggers investigation (distinguish between a systemic pipeline issue and a single provider outage per Failure Mode 8.6). - Any sanitization gate block on a production deploy or report generation attempt (8.7) should notify the operator immediately; this is a security-relevant event even though it is functioning as designed. -- Sustained upward drift in panel `agreement_rate` beyond a defined band triggers a manual review of judge configuration, per the training-data recursion and consensus-drift concerns in 7.6 and 7.7. -- PDF fallback-to-plaintext events (8.4) should alert on any occurrence, not just above a threshold, since they represent a customer-visible degradation of the deliverable. +- Sustained upward drift in panel `agreement_rate` beyond a defined band triggers a manual review of judge configuration, per the training-data recursion and consensus-drift concerns in 7.7 and 8.14. +- PDF fallback-to-plaintext events (8.4, 8.9) should alert on any occurrence, not just above a threshold, since they represent a customer-visible degradation of the deliverable. +- Queue depth crossing 80% of the 50-review cap (8.13) should alert with enough lead time to investigate before submissions start receiving `429`. +- Prediction cron missing-run or failure events (8.11) page the operator immediately; there is no acceptable silent-failure window for this check given how quietly its failure otherwise degrades Tier 1 intelligence. ### 11.3 Backup Schedule -- Nightly full backup of application state (Section 10.4). +- 15-minute WAL checkpoint to encrypted, off-host S3-compatible storage, plus a full nightly snapshot (Section 10.4). - Backup integrity should be spot-verified on a periodic cadence (e.g., a scheduled restore-to-scratch test), rather than assumed functional purely because the backup job reports success. ### 11.4 Corpus Health Checks - Periodic verification that `corpus_record` entries stay in sync with their source `review` rows (no orphaned or stale index entries), particularly after any manual corpus maintenance operation. -- Periodic audit that `corpus_opt_out` reviews are in fact excluded from search and comparison results, as a direct verification of the confidentiality control in 7.4 rather than trusting the flag's existence alone. -- Prediction tracking cron runs (Section 2.8) should be monitored for completion; a silently-failing cron would cause `prediction.actual_outcome` to remain perpetually `pending`, quietly degrading the value of Tier 1 cumulative intelligence without any loud failure signal. +- Periodic audit that `corpus_opt_out` reviews are in fact excluded from search, comparison results, and embedding generation entirely (7.4, 2.9), as a direct verification of the confidentiality control rather than trusting the flag's existence alone. +- Periodic audit that `summary_snippet` values in the corpus (4.4, 7.4) remain structural-only and have not regressed toward containing proposal-text-derived content through a template change. + +### 11.5 Prediction Cron Monitoring + +Moved here from its prior placement as a passing note in 11.4, and elevated to its own subsection given the failure mode's severity (8.11): the prediction-tracking cron runs as a systemd timer with `OnFailure=` wired to a dedicated notification unit, and a separate missing-run detection check verifies that a run that should have fired within its expected window actually did, independent of whether a run that did fire reported success or failure. Both conditions page the operator; there is no code path where this cron can fail silently and go unnoticed. --- @@ -720,10 +794,49 @@ For planning and evaluation purposes, the v3 build is staged as follows: | Phase | Weeks | Scope | |---|---|---| -| Phase 0 | 1-3 | Sanitization gate, automated PDF generation, corpus schema and backfill | -| Phase 1 | 4-7 | Corpus search, prediction tracking cron, accuracy scoring (read-only, dashboard only) | -| Phase 2 | 8-12 | Vertical-specific templates, adversarial red-team per vertical, market simulation engine | -| Phase 3 | 13-18 | New specialist judges (reasoning-verification, execution-feasibility, market-reality), public shareable reports, review-as-a-service API beta | +| Phase 0 | 1-3 | Sanitization gate (both layers, 7.5), automated PDF generation, corpus schema and backfill, red-team testing protocol (7.1) | +| Phase 1 | 4-7 | Corpus search, embedding generation policy (2.9), prediction tracking cron, accuracy scoring (read-only, dashboard only, Enterprise-tier access per 4.5/8.16) | +| Phase 2 | 8-12 | Vertical-specific templates, adversarial red-team per vertical | +| Phase 3 | 13-18 | New specialist judges (reasoning-verification, execution-feasibility, market-reality), public shareable reports, review-as-a-service API beta, market simulation engine (5.2) if methodology remains defensibly specified per Condition E; otherwise re-scoped to this phase by default | | Phase 4 | 19-22 | Live accuracy-based judge weighting, general availability, White-Label pilots | -This staging is deliberate about sequencing risk: infrastructure hardening (sanitization, automated PDF, corpus foundation) ships before any feature that depends on it; accuracy scoring ships read-only well before it is trusted to influence a live verdict; and the highest-blast-radius change (letting historical accuracy actually move a verdict) is the very last thing to go live. \ No newline at end of file +--- + +## Appendix B: Review Addendum + +### B.1 Review Verdict + +On **2026-08-10**, this architecture document underwent a structured three-judge review prior to build sign-off. All three judges returned an independent **CONDITIONAL GO** verdict; the panel result was a unanimous 3/3 Conditional Go, meaning the architecture is approved to proceed to implementation contingent on the conditions below being addressed in this document, which this revision does. + +| Judge | Model | Focus Area | +|---|---|---| +| Judge 1 | Opus | Operational realism: deployment mechanics, failure recovery, resource limits, monitoring | +| Judge 2 | Claude | Security and failure modes: prompt injection, access control, corpus confidentiality, abuse protection | +| Judge 3 | Gemini Pro | Completeness and coverage: missing sections, edge cases, specification gaps | + +### B.2 Conditions Addressed + +The following 18 merged and deduplicated conditions, consolidated across the three independent reviews, are addressed in this revision: + +1. **Embedding/vector search subsystem specified**: model named (`all-MiniLM-L6-v2` via `sentence-transformers`), storage/query mechanism specified (SQLite row-level cosine similarity via numpy, or `sqlite-vec` as the documented exception to the no-extensions rule), embedding generation skipped entirely (not merely flagged) for opt-out reviews. See 2.9. +2. **SQLite WAL mode and write contention**: WAL mode declared explicitly in 2.4; `SQLITE_BUSY`/write contention added to the failure modes table (8.8); concurrent-writer ceiling and single-worker serialization documented (2.4, 5.6); migration trigger re-diagnosed as write concurrency, not corpus size (5.7). +3. **Deployment/production hardening**: WeasyPrint native apt dependencies and font availability (10.5); systemd unit with `Restart=on-failure`/`RestartSec=5` (10.6); log rotation via journald/logrotate (10.6); secrets via systemd `EnvironmentFile` (10.6); graceful `SIGTERM` drain with a 15-minute completion window (10.6); dependency pinning with hashed `requirements.txt` (10.3). +4. **Backup RPO**: nightly-only backup replaced with 15-minute WAL checkpoint to S3, retaining a nightly full snapshot in addition; backup encryption documented as independent of host-level disk encryption (10.4). +5. **Market simulation specification**: methodology specified as agent-driven Monte Carlo simulation with explicit assumption documentation; sequencing clarified as running before Primary Review, feeding a ResearchBrief appendix rather than running concurrently with Phase 2 (5.2); de-scope path to Phase 3 documented if methodology cannot be defensibly specified (Appendix A). +6. **Missing sections added**: Frontend (2.10), Authentication (2.11), Tenant Isolation (2.12), Billing Integration (2.13), Notification System (2.14). +7. **Edge case handling**: token-limit chunking and 25K-word hard cap (5.9), non-English rejection with Phase 1 English-only scope (5.9), queue depth cap of 50 with `429` (5.6, 5.9, 8.13), duplicate submission via `proposal_hash` (5.9), corpus scale threshold restated as write-contention-driven (5.7, 5.9). +8. **Failover configuration**: per-role provider list `[primary, secondary, tertiary]` documented (3.8, 5.8), 3-retry exponential backoff before escalation, configuration location specified as a config table (3.8). +9. **Prompt injection defense reframed**: claim extractor acknowledged as the first point of exposure, not a pre-exposure filter; pre-extraction regex sanitization added; output behavioral anomaly detection added as a supplementary signal; red-team testing protocol added as a Phase 0 deliverable (7.1). +10. **Corpus confidentiality**: embeddings skipped (not generated) for opt-out reviews (2.9, 7.4); raw DB access model documented as single-operator, all-or-nothing design assumption (7.4); S3 backups encrypted with a separate key from host-level disk encryption (10.4); `summary_snippet` construction rules defined as fixed-length structural summary, not a proposal-text derivative (4.4, 7.4). +11. **Accuracy dashboard access restricted**: 4.5 endpoint restricted to authenticated Enterprise-tier access at minimum; Free/Pro tiers have no access to per-judge accuracy data. +12. **Report URL security**: `uuid4()`/`os.urandom`-based unguessable identifiers confirmed; optional expiration (default 90 days) added; access logging added for all report retrievals (7.3). +13. **"No training" API terms**: contractual requirement for "no training on API usage" terms added as a standing provider condition; agreement-rate monitoring documented as lagging detection, not leading prevention; training-data recursion acknowledged as partially inherent to multi-model pipelines (7.7, 8.14). +14. **Expanded failure modes table**: six new entries added covering SQLite write contention, WeasyPrint memory exhaustion, sanitization gate false positives, prediction cron silent failure, confident-but-wrong judge analysis, and concurrent queue saturation (8.8 through 8.13). +15. **Sanitization gate enhancements**: behavioral fingerprint detection layer added alongside string matching, targeting self-identification phrases, training-cutoff patterns, and capability-boundary language across model families; false-positive procedure (human review, allowlist, re-run, mandatory audit log) added; gate confirmed as still blocking by default (7.5). +16. **API abuse countermeasures**: polling endpoint rate-limited independently from submission endpoint; free-tier submission content monitoring for probing patterns added; timing oracle via polling documented as an acknowledged low-severity side channel (7.6). +17. **Inherent risk documentation**: a dedicated Inherent Risks subsection added (8.14) stating plainly that training-data recursion, the polling timing oracle, and queue-saturation capacity limits do not have full technical solutions and are accepted, ongoing risk rather than solvable failure modes. +18. **Accuracy dashboard, tenant, and billing sections cross-checked against tier structure**: Section 4.5, 2.12, and 2.13 were cross-verified against the pricing tiers in 9.3 to confirm no contradiction between stated tier entitlements and the access restrictions added under condition 11. + +### B.3 Scope Note + +This addendum documents that the above conditions were incorporated into this architecture document as a direct response to the review. It does not itself constitute a fourth review pass; implementation should still be validated against this document during and after each build phase in Appendix A, and the red-team testing protocol (7.1, condition 9) and Phase 0 deliverables should be treated as prerequisites for the Phase 0 exit criteria, not optional hardening. \ No newline at end of file