itpp-infrastructure/projects/wall-o/01-product-model.md

Wall-O: Product Model and Data Architecture

Status: PLANNED First client: Wall Orthodontics Scope: STRICTLY internal staff knowledge (employee handbook, policies, procedures, billing questions). NOT patient records, NOT PHI, no HIPAA scope.

1. Core Primitive

Every channel is exactly three things bound together:

1. One knowledge domain (for example "Employee Resources", "Billing", "IT Help").
2. One attached AI agent (a domain-tuned persona).
3. One scoped knowledge source (a set of documents/indices that maps to one vector namespace).

This is a hard invariant. A channel has exactly one agent and exactly one knowledge scope. A channel cannot span two domains, and an agent cannot serve two channels. If a practice needs a second knowledge domain, it creates a second channel, a second agent, and a second scope.

2. Canonical Architecture Split

Do not redesign this split. It is the foundation of the tenancy model.

Layer	Responsibility	Owns
Rocket.Chat	Chat transport only	One workspace per tenant (MIT core, EE stripped). Rooms, users, messages, DMs. Zero intelligence.
Orchestrator	All intelligence	Multi-tenant Python FastAPI service. Tenancy, agents, kb_scope, M365 connector, retrieval, LLM, posting.
Postgres + pgvector	State and vectors	Tenants, channels, agents, scopes, documents, chunks, messages.
admin-ai	LLM	DeepSeek V4 Pro primary, configured fallback chain.
Wasabi S3	Object storage	M365 sync staging, backups, agent assets (avatars), audit exports.

Deployment: Rocket.Chat in Docker on netcup Core/app servers; orchestrator is FastAPI behind Caddy; Postgres + pgvector on the app data tier; M365 connector runs as a worker inside the orchestrator.

Entity relationship summary

tenants 1:N channels 1:1 agents 1:1 kb_scopes
                          |               |
                          |               + 1:N documents 1:N chunks
                          |
                          + 1:N messages (per channel)
tenants 1:N users

3. Relational Data Model

All identifiers are UUIDv4. Every table that holds tenant data carries tenant_id and is filtered by it on every query. See section 9 for isolation.

3.1 tenants

One row per practice. Root of the tenancy tree.

Column	Type	Constraints	Notes
tenant_id	UUID	PK	Canonical identifier.
slug	TEXT	UNIQUE, NOT NULL	URL-safe slug, used in the webhook URL path.
name	TEXT	NOT NULL	Practice display name.
logo_url	TEXT	NULL	Branding asset, stored on Wasabi S3.
accent_color	TEXT	NULL	Branding hex color.
m365_tenant_id	TEXT	NULL	Microsoft Entra directory (tenant) id.
m365_client_id	TEXT	NULL	Entra app registration client id.
m365_credential_ref	TEXT	NULL	Vaultwarden secret reference. Never inline the client secret.
rocket_chat_url	TEXT	NULL	Workspace root URL for this tenant.
rocket_chat_admin_token_ref	TEXT	NULL	Vaultwarden reference for the admin REST token used to provision bots/rooms.
webhook_secret_ref	TEXT	NULL	Vaultwarden reference for the HMAC secret that signs webhook POSTs.
status	TEXT	NOT NULL DEFAULT 'provisioning'	provisioning, active, suspended.
created_at	TIMESTAMPTZ	NOT NULL DEFAULT now()
updated_at	TIMESTAMPTZ	NOT NULL DEFAULT now()

Indexes: UNIQUE (slug).

3.2 channels

One row per knowledge domain. Belongs to exactly one tenant.

Column	Type	Constraints	Notes
channel_id	UUID	PK	Canonical identifier.
tenant_id	UUID	FK to tenants.tenant_id, NOT NULL	Multi-tenant isolation.
name	TEXT	NOT NULL	Knowledge domain name, for example "Billing".
slug	TEXT	NOT NULL	URL-safe, unique per tenant.
description	TEXT	NULL	What this channel answers.
rocket_chat_room_id	TEXT	NULL	Rocket.Chat room/team id this channel maps to.
rocket_chat_room_name	TEXT	NULL	Human-readable room name.
agent_id	UUID	FK to agents.agent_id, UNIQUE, NULL	Exactly one agent per channel. Null until the agent is attached.
kb_scope_id	UUID	FK to kb_scopes.kb_scope_id, UNIQUE, NULL	Exactly one scope per channel. Null until bound.
status	TEXT	NOT NULL DEFAULT 'draft'	draft, active, archived.
created_at	TIMESTAMPTZ	NOT NULL DEFAULT now()
updated_at	TIMESTAMPTZ	NOT NULL DEFAULT now()

Indexes: UNIQUE (tenant_id, slug), INDEX (tenant_id), UNIQUE (agent_id), UNIQUE (kb_scope_id), INDEX (rocket_chat_room_id).

The dual agent_id/kb_scope_id unique columns enforce the one-to-one invariant from both directions.

3.3 agents

One row per AI persona. Bound to exactly one channel.

Column	Type	Constraints	Notes
agent_id	UUID	PK	Canonical identifier.
tenant_id	UUID	FK to tenants.tenant_id, NOT NULL	Multi-tenant isolation.
channel_id	UUID	FK to channels.channel_id, UNIQUE, NOT NULL	One agent per channel.
name	TEXT	NOT NULL	Agent display name, for example "Wallo Billing".
system_prompt	TEXT	NOT NULL	Domain-tuned persona and instructions.
kb_scope_id	UUID	FK to kb_scopes.kb_scope_id, NULL	What this agent may retrieve.
model_provider	TEXT	NOT NULL DEFAULT 'admin-ai'	LLM gateway.
model_name	TEXT	NOT NULL DEFAULT 'deepseek-v4-pro'	Primary model.
model_params	JSONB	NOT NULL DEFAULT '{}'	temperature, max_tokens, top_p.
fallback_model	TEXT	NULL	Next model in the failover chain.
rocket_chat_bot_username	TEXT	UNIQUE, NOT NULL	Bot username in Rocket.Chat.
rocket_chat_bot_user_id	TEXT	NULL	Rocket.Chat internal _id, filled after provisioning.
rocket_chat_bot_token_ref	TEXT	NULL	Vaultwarden reference for the bot personal access token.
status	TEXT	NOT NULL DEFAULT 'draft'	draft, provisioning, active, disabled.
created_at	TIMESTAMPTZ	NOT NULL DEFAULT now()
updated_at	TIMESTAMPTZ	NOT NULL DEFAULT now()

Indexes: UNIQUE (channel_id), UNIQUE (rocket_chat_bot_username), INDEX (tenant_id).

3.4 kb_scopes

One row per knowledge scope. Maps to exactly one vector namespace and one channel.

Column	Type	Constraints	Notes
kb_scope_id	UUID	PK
tenant_id	UUID	FK to tenants.tenant_id, NOT NULL	Multi-tenant isolation.
channel_id	UUID	FK to channels.channel_id, UNIQUE, NOT NULL	One scope per channel.
vector_namespace	TEXT	UNIQUE, NOT NULL	Logical pgvector namespace, for example tenant_{tenant_id}__scope_{kb_scope_id} (see Part 2, section 5).
document_libraries	JSONB	NOT NULL DEFAULT '[]'	List of M365 document libraries this scope may search. Entries carry site_id, drive_id, list_id, and display name.
index_refs	JSONB	NOT NULL DEFAULT '[]'	Search index names the scope may query (Graph fallback).
embedding_model	TEXT	NOT NULL	Model used to embed chunks and queries.
embedding_dim	INT	NOT NULL	Vector dimension. Must match the platform-wide column dimension.
chunk_size	INT	NOT NULL DEFAULT 512	Tokens per chunk (approx 2000 chars). See Part 2, section 2.3.
chunk_overlap	INT	NOT NULL DEFAULT 64	Overlap tokens between chunks (approx 250 chars). See Part 2, section 2.3.
sync_policy	JSONB	NULL	Delta sync schedule and file-type filters.
last_synced_at	TIMESTAMPTZ	NULL	Last successful M365 sync.
created_at	TIMESTAMPTZ	NOT NULL DEFAULT now()
updated_at	TIMESTAMPTZ	NOT NULL DEFAULT now()

Indexes: UNIQUE (channel_id), UNIQUE (vector_namespace), INDEX (tenant_id).

Constraint: pgvector stores a fixed dimension per column. The platform therefore standardizes on one embedding model and dimension across all scopes so chunks.embedding is a single vector(N) column. Changing the model requires a full re-embed and reindex.

3.5 documents

One row per synced source document. Always scoped to a tenant and a channel.

Column	Type	Constraints	Notes
document_id	UUID	PK
tenant_id	UUID	FK to tenants.tenant_id, NOT NULL	Multi-tenant isolation.
channel_id	UUID	FK to channels.channel_id, NOT NULL
kb_scope_id	UUID	FK to kb_scopes.kb_scope_id, NOT NULL
source_type	TEXT	NOT NULL	sharepoint, onedrive, manual_upload.
m365_drive_id	TEXT	NULL	For Graph delta sync.
m365_item_id	TEXT	NULL	For Graph delta sync.
source_path	TEXT	NULL	Full source path, for example the SharePoint URL.
title	TEXT	NULL
file_name	TEXT	NULL
mime_type	TEXT	NULL
content_hash	TEXT	NOT NULL	SHA256 of raw content, for change detection.
size_bytes	BIGINT	NULL
metadata	JSONB	NOT NULL DEFAULT '{}'	author, last modified time, page count.
status	TEXT	NOT NULL DEFAULT 'pending'	pending, extracting, indexing, indexed, failed, deleted.
last_synced_at	TIMESTAMPTZ	NULL
indexed_at	TIMESTAMPTZ	NULL
error	TEXT	NULL	Last sync or index error.
created_at	TIMESTAMPTZ	NOT NULL DEFAULT now()
updated_at	TIMESTAMPTZ	NOT NULL DEFAULT now()

Indexes: INDEX (tenant_id, channel_id), INDEX (kb_scope_id), INDEX (status), partial UNIQUE (tenant_id, m365_drive_id, m365_item_id) WHERE m365_item_id IS NOT NULL.

3.6 chunks

One row per embedded chunk. Carries the vector and the denormalized tenant/channel/scope keys for fast, isolated retrieval.

Column	Type	Constraints	Notes
chunk_id	UUID	PK
document_id	UUID	FK to documents.document_id ON DELETE CASCADE, NOT NULL
tenant_id	UUID	FK to tenants.tenant_id, NOT NULL	Denormalized for retrieval filtering.
channel_id	UUID	FK to channels.channel_id, NOT NULL
kb_scope_id	UUID	FK to kb_scopes.kb_scope_id, NOT NULL
chunk_index	INT	NOT NULL	Position within the document.
content	TEXT	NOT NULL	The chunk text.
token_count	INT	NOT NULL
embedding	vector(N)	NOT NULL	pgvector column. N is the platform-wide dimension.
metadata	JSONB	NOT NULL DEFAULT '{}'	page number, section heading, anchor text.

Indexes: INDEX (document_id), INDEX (tenant_id, kb_scope_id), and a vector index:

CREATE INDEX chunks_embedding_idx ON chunks
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

For large tenants, partition chunks by tenant_id so each partition is its own physical namespace and the HNSW index is built per partition.

3.7 messages

One row per inbound question and per outbound answer. Doubles as the audit log and the idempotency ledger.

Column	Type	Constraints	Notes
message_id	UUID	PK
tenant_id	UUID	FK to tenants.tenant_id, NOT NULL	Multi-tenant isolation.
channel_id	UUID	FK to channels.channel_id, NOT NULL
agent_id	UUID	FK to agents.agent_id, NULL	Null for inbound user messages.
rocket_chat_message_id	TEXT	UNIQUE, NOT NULL	Rocket.Chat message _id, used as the idempotency key.
rocket_chat_room_id	TEXT	NOT NULL
rocket_chat_user_id	TEXT	NULL	Author _id on Rocket.Chat.
direction	TEXT	NOT NULL	inbound, outbound.
role	TEXT	NOT NULL	user, assistant.
content	TEXT	NOT NULL
citations	JSONB	NULL	Provenance array, see section 7.
prompt_tokens	INT	NULL
completion_tokens	INT	NULL
total_tokens	INT	NULL
model_name	TEXT	NULL	Model that produced the answer.
latency_ms	INT	NULL	End to end latency.
status	TEXT	NOT NULL	received, resolving, retrieving, prompting, posted, failed.
error	TEXT	NULL	Failure detail.
created_at	TIMESTAMPTZ	NOT NULL DEFAULT now()

Indexes: UNIQUE (rocket_chat_message_id), INDEX (tenant_id, channel_id, created_at), INDEX (created_at).

3.8 users

One row per human staff member. Mirrors the Rocket.Chat user for identity and role mapping.

Column	Type	Constraints	Notes
user_id	UUID	PK
tenant_id	UUID	FK to tenants.tenant_id, NOT NULL	Multi-tenant isolation.
rocket_chat_user_id	TEXT	NOT NULL	Rocket.Chat user _id.
email	TEXT	NULL
display_name	TEXT	NULL
role	TEXT	NOT NULL DEFAULT 'staff'	admin, staff. Controls orchestrator admin surface only.
last_seen_at	TIMESTAMPTZ	NULL
created_at	TIMESTAMPTZ	NOT NULL DEFAULT now()
updated_at	TIMESTAMPTZ	NOT NULL DEFAULT now()

Indexes: UNIQUE (tenant_id, rocket_chat_user_id), INDEX (tenant_id).

4. Expanded Message Flow

The canonical 6-step flow, with exact mechanics.

Step 1: User posts a question

A staff member either @mentions the agent bot in a channel, or DMs the bot directly (section 8).

Step 2: Rocket.Chat fires the webhook

Channel mentions are captured by an outgoing webhook integration configured on the channel with trigger word equal to the bot username. Rocket.Chat POSTs to:

POST https://orchestrator.wall-o.<domain>/api/v1/webhook/rocketchat/{tenant_slug}

Headers:

Header	Value
X-WallO-Signature	Hex HMAC-SHA256 of the raw request body, keyed by the tenant webhook secret.
X-WallO-Timestamp	Unix seconds. Reject if skew is greater than 300 seconds.
Content-Type	application/json

Payload (Rocket.Chat outgoing webhook shape):

{
  "token": "<outgoing webhook token>",
  "channel_id": "<rocketchat room id>",
  "channel_name": "employee-resources",
  "timestamp": "2026-08-15T12:00:00.000Z",
  "user_id": "<rocketchat user id>",
  "user_name": "jane.doe",
  "text": "@wallo-billing How do I submit a PTO request?",
  "trigger_word": "@wallo-billing",
  "bot": false
}

The orchestrator strips the trigger word and mention prefix from text before treating the remainder as the question.

DM capture does not use this webhook. Rocket.Chat outgoing webhooks do not fire inside DMs, so the orchestrator's bot-listener sidecar (section 5) receives DM events over the Realtime API and normalizes them into the identical inbound message object.

Step 3: Orchestrator resolves tenant, channel, agent, scope

Resolution order, all in one request context:

1. Authenticate: verify X-WallO-Signature against the tenant webhook secret looked up by tenant_slug.
2. Idempotency: derive rocket_chat_message_id from a hash of (channel_id, user_id, timestamp, text). If a messages row already exists with that id, return HTTP 200 immediately and do nothing. This deduplicates webhook retries.
3. Resolve channel: SELECT * FROM channels WHERE tenant_id = ? AND rocket_chat_room_id = ?, with a fallback match on rocket_chat_room_name/slug.
4. Resolve agent: read channel.agent_id, then SELECT * FROM agents WHERE agent_id = ?.
5. Resolve scope: read agent.kb_scope_id, then SELECT vector_namespace, embedding_model FROM kb_scopes WHERE kb_scope_id = ?.
6. If the channel has no agent or no scope, reply with a configuration-error message (or a silent no-op, configurable per tenant) and stop.

Step 4: Retrieval

Primary path: pgvector semantic search over the channel's vector namespace.

SELECT chunk_id, document_id, content, metadata, 1 - (embedding <=> $query_vec) AS score
FROM chunks
WHERE tenant_id = $tenant_id AND kb_scope_id = $kb_scope_id
ORDER BY embedding <=> $query_vec
LIMIT 20;  -- candidate set; rerank to 5 per Part 2, section 3.2

The query is embedded with the scope's embedding_model. Both tenant_id and kb_scope_id filters are mandatory in the same query, which is what makes the namespace isolated.

Fallback path: if the top score is below a configured threshold (for example 0.70) or the query returns zero rows, the orchestrator calls Microsoft Graph /search/query over the scope's document_libraries, with the search entity type set to driveItem and the site/drive scoped to the scope's libraries. Results are optionally re-ranked before prompt assembly.

Step 5: Prompt build and LLM call

The prompt is assembled from four parts:

1. agents.system_prompt (the domain-tuned persona).
2. A retrieved-context block where each chunk is labeled with a source marker [1], [2], and so on, carrying its document title and section heading.
3. A citation instruction: answer using only the provided context, cite sources inline as [n], and if the context does not contain an answer, say so instead of guessing.
4. The user question.

The call goes to admin-ai with model_name (DeepSeek V4 Pro primary) and model_params. A per-call timeout (for example 30 seconds) is enforced. On timeout or model error, the orchestrator retries once with fallback_model, then returns a canned "I could not reach the model" reply. It never fabricates an answer.

Step 6: Post the answer back to Rocket.Chat

POST https://<rocket_chat_url>/api/v1/chat.postMessage
Headers: X-Auth-Token: <bot token>, X-User-Id: <bot user id>
Body: { "roomId": "<room id>", "text": "<answer with [n] citations + sources footer>" }

The answer is posted as the agent bot (section 7). On post failure the orchestrator retries with exponential backoff (up to 3 attempts), then marks the message failed and records the error.

Error and timeout handling

• The webhook acknowledges immediately (HTTP 200/202) after persisting the inbound message and enqueuing async processing, so the Rocket.Chat webhook never blocks or times out on the LLM call. The answer is posted out of band via REST.
• Retrieval empty and below threshold: reply "I could not find an answer in the knowledge base for this question" with no sources. Never hallucinate.
• LLM timeout/error: failover model, then a canned error reply.
• Rocket.Chat post failure: retry with backoff, then mark failed and surface to a tenant alert.
• Poison messages go to a dead-letter status with the error retained; alerting fires on elevated failure rate.

5. Rocket.Chat Bot Integration

Recommended mechanism

Primary path: (a) bot user type + REST API, with the outgoing webhook (b) as the canonical channel-mention trigger and a websocket listener for DMs. The Apps Engine (c) is not used.

Justification: the bot user gives each agent a stable, named identity (username, avatar, alias) inside Rocket.Chat and the REST API is versioned, stateless, and fully scriptable from the orchestrator, while the outgoing webhook delivers the canonical inbound POST with no code running inside the chat layer. The Apps Engine is rejected because it would place logic inside the transport layer, violating the architecture split, and would require maintaining a JS app per tenant. Incoming webhooks alone (b by itself) cannot provide a dynamic per-agent identity, so they are used only as an optional posting convenience, not as the identity layer.

Register an agent bot

For each agent, the orchestrator (using the tenant admin token):

1. Create the bot user: POST /api/v1/users.create with:

   {
     "name": "<agent name>",
     "username": "<rocket_chat_bot_username>",
     "email": "<username>@<tenant-domain>",
     "password": "<random 32-char>",
     "roles": ["bot"],
     "joinDefaultChannels": false,
     "requirePasswordChange": false,
     "sendWelcomeEmail": false,
     "verified": true
   }
   

2. Create a personal access token: POST /api/v1/users.createToken with { "userId": "<bot _id>" }. The returned authToken and userId are stored in Vaultwarden; the reference goes in agents.rocket_chat_bot_token_ref and the _id in agents.rocket_chat_bot_user_id.
3. Optionally set the avatar: POST /api/v1/users.setAvatar (uploaded image) or users.setAvatarFromUrl.

Join a channel

The orchestrator (via the tenant admin token) invites the bot to the channel's room:

POST /api/v1/channels.invite with { "roomId": "<rocket_chat_room_id>", "userId": "<bot _id>" }.

Bots auto-accept invitations. After this the bot is a room member and can be @mentioned and post as itself.

Route @mentions and commands

• @mention/trigger word: configure an outgoing webhook integration on the channel with trigger word set to the bot username. A user mentioning the bot (or typing the trigger word) causes Rocket.Chat to POST to the orchestrator webhook URL (section 4, step 2). The orchestrator strips the mention/trigger prefix.
• Slash command (optional): POST /api/v1/commands.create to register a command such as /ask in the channel that routes to the same webhook. Useful when a channel hosts more than one purpose and explicit scoping is wanted.

Route DMs

Because outgoing webhooks do not fire in DMs, the orchestrator runs a bot-listener sidecar that maintains a websocket (Realtime API) session per bot: connect, call method: "login" with the bot token, then subscribe to stream-room-messages for the bot's own user id. Both DMs and room mentions arrive as the same message event type. The sidecar normalizes them into the identical inbound message object as the webhook path and enqueues them through the same pipeline. The sidecar is a pure transport shim; it holds no intelligence.

6. Channel Lifecycle

Step	Action	Performed by
1. Create channel	Insert a channels row (tenant, name, slug, description, status = draft). Optionally create the Rocket.Chat room via POST /api/v1/channels.create and store rocket_chat_room_id.	Orchestrator API, tenant admin role
2. Attach agent	Insert an agents row (system_prompt, model config, bot username) with channel_id set. Provision the Rocket.Chat bot user and token (section 5). Invite the bot to the room. Set channels.agent_id.	Orchestrator API + Rocket.Chat REST
3. Bind kb_scope	Insert a kb_scopes row (vector_namespace, document_libraries, embedding model/dim, chunk params). Set channels.kb_scope_id.	Orchestrator API
4. Initial sync and index	The M365 connector pulls the named document libraries, extracts text, chunks, embeds, and writes documents + chunks rows under the namespace.	M365 connector worker
5. Go live	Set channels.status = active. Configure the outgoing webhook and optional slash command. Run an end-to-end smoke test question and confirm a cited answer posts back as the bot.	Orchestrator API + operator

Steps 1 through 3 are idempotent API calls driven by an onboarding form in the orchestrator admin surface. Step 4 is the only long-running step; the channel stays in draft until indexed document count is nonzero. Step 5 flips status and does not require a redeploy.

7. Agent Identity and Response Citations

Identity: the bot posts with its own username and avatar (set at provisioning). Every answer is attributed to the bot user, never to a human, so staff can distinguish agent answers from colleague messages. The system_prompt instructs the agent to introduce itself as the channel's assistant (for example "I am the Wall-O assistant for Employee Resources") and to stay inside the domain.

Citations: retrieved chunks carry provenance in metadata (document title, section heading, page). During prompt assembly each chunk is labeled [1], [2], and so on. The orchestrator renders the final answer with inline [n] markers and appends a "Sources" footer listing each cited document title and a link (Microsoft Graph sharing link, or a Rocket.Chat file link). The full provenance (document_id, chunk_id, title, snippet, url) is stored in messages.citations as JSONB for audit and re-render. If the answer cites nothing, no footer is emitted.

8. User Interaction Model

Mode	When to use	Trigger path
Channel @mention	Shared or discoverable questions, team-wide answers, anything others should see.	Outgoing webhook (section 4, step 2).
DM the bot	Private follow-up, iterative clarification, personal-but-not-PHI questions, or a 1:1 thread the user does not want in the channel.	Bot-listener websocket (section 5).

Guidance surfaced to staff: use the channel for Q&A that benefits the team (answers stay searchable in the room), and DM the bot for private or back-and-forth questions. Both paths produce the same cited answer format; only the destination differs.

9. Multi-tenant Isolation and Security

• Tenant_id on every table: tenants, channels, agents, kb_scopes, documents, chunks, messages, and users all carry tenant_id. The application derives the tenant from the webhook path and HMAC signature, never from a client-supplied body field alone, and scopes every query with it.
• Postgres row-level security: enable RLS on all tenant tables with a policy tenant_id = current_setting('app.tenant_id'), set per request, as defense in depth under the application-level filter.
• Vector namespace isolation: retrieval always filters tenant_id AND kb_scope_id in the same query, so a chunk can never leak across channels or tenants. Partitioning by tenant_id makes each tenant's vectors physically separate.
• Secrets: M365 credentials, bot tokens, and webhook secrets live in Vaultwarden. The database stores references only. Never commit secrets; .env.example placeholders only.
• Scope guard: the product never ingests patient records or PHI. The M365 connector's document_libraries are allowlisted per scope, and a content filter flags documents outside the allowlisted libraries before indexing.