itpp-infrastructure/projects/wall-o.md

Wall-O: System Architecture

Status: DESIGN (internal technical architecture) First client: Wall Orthodontics Scope: internal staff knowledge-base chat. Employee handbook, policies, procedures, billing questions. NOT patient records, NOT PHI, no HIPAA scope.

This is the consolidated Wall-O architecture. It aggregates three detailed sections (kept as source files under wall-o/):

Part	Section	Source
1	Product model and data architecture	wall-o/01-product-model.md
2	M365 connector and retrieval	wall-o/02-m365-connector.md
3	Deployment, white-label, and Phase 0	wall-o/03-deployment-white-label.md

The three parts were produced in parallel by specialized subagents against one shared canonical model, then reconciled by the parent. The canonical entities are:

• tenant_id = one practice (the Microsoft 365 tenant boundary)
• channel_id = one knowledge domain with exactly one attached agent and one kb_scope
• agent_id = one AI persona bound to exactly one channel
• kb_scope = per-tenant plus per-channel document scope, mapped 1:1 to a vector namespace

Architecture split (do not redesign): Rocket.Chat is chat transport only (one workspace per tenant, MIT core, EE stripped). All intelligence lives in a multi-tenant Python FastAPI orchestrator that owns tenancy, agents, kb_scope, the M365 connector, retrieval, and the LLM. State and vectors in Postgres + pgvector; secrets in Vaultwarden; raw-file archive and backups in Wasabi S3; LLM and embeddings via admin-ai (LiteLLM).

Cross-team discrepancies and their authoritative resolutions are listed in the appendix at the end.

---

Wall-O: Product Model and Data Architecture

Status: PLANNED First client: Wall Orthodontics Scope: STRICTLY internal staff knowledge (employee handbook, policies, procedures, billing questions). NOT patient records, NOT PHI, no HIPAA scope.

1. Core Primitive

Every channel is exactly three things bound together:

1. One knowledge domain (for example "Employee Resources", "Billing", "IT Help").
2. One attached AI agent (a domain-tuned persona).
3. One scoped knowledge source (a set of documents/indices that maps to one vector namespace).

This is a hard invariant. A channel has exactly one agent and exactly one knowledge scope. A channel cannot span two domains, and an agent cannot serve two channels. If a practice needs a second knowledge domain, it creates a second channel, a second agent, and a second scope.

2. Canonical Architecture Split

Do not redesign this split. It is the foundation of the tenancy model.

Layer	Responsibility	Owns
Rocket.Chat	Chat transport only	One workspace per tenant (MIT core, EE stripped). Rooms, users, messages, DMs. Zero intelligence.
Orchestrator	All intelligence	Multi-tenant Python FastAPI service. Tenancy, agents, kb_scope, M365 connector, retrieval, LLM, posting.
Postgres + pgvector	State and vectors	Tenants, channels, agents, scopes, documents, chunks, messages.
admin-ai	LLM	DeepSeek V4 Pro primary, configured fallback chain.
Wasabi S3	Object storage	M365 sync staging, backups, agent assets (avatars), audit exports.

Deployment: Rocket.Chat in Docker on netcup Core/app servers; orchestrator is FastAPI behind Caddy; Postgres + pgvector on the app data tier; M365 connector runs as a worker inside the orchestrator.

Entity relationship summary

tenants 1:N channels 1:1 agents 1:1 kb_scopes
                          |               |
                          |               + 1:N documents 1:N chunks
                          |
                          + 1:N messages (per channel)
tenants 1:N users

3. Relational Data Model

All identifiers are UUIDv4. Every table that holds tenant data carries tenant_id and is filtered by it on every query. See section 9 for isolation.

3.1 tenants

One row per practice. Root of the tenancy tree.

Column	Type	Constraints	Notes
tenant_id	UUID	PK	Canonical identifier.
slug	TEXT	UNIQUE, NOT NULL	URL-safe slug, used in the webhook URL path.
name	TEXT	NOT NULL	Practice display name.
logo_url	TEXT	NULL	Branding asset, stored on Wasabi S3.
accent_color	TEXT	NULL	Branding hex color.
m365_tenant_id	TEXT	NULL	Microsoft Entra directory (tenant) id.
m365_client_id	TEXT	NULL	Entra app registration client id.
m365_credential_ref	TEXT	NULL	Vaultwarden secret reference. Never inline the client secret.
rocket_chat_url	TEXT	NULL	Workspace root URL for this tenant.
rocket_chat_admin_token_ref	TEXT	NULL	Vaultwarden reference for the admin REST token used to provision bots/rooms.
webhook_secret_ref	TEXT	NULL	Vaultwarden reference for the HMAC secret that signs webhook POSTs.
status	TEXT	NOT NULL DEFAULT 'provisioning'	provisioning, active, suspended.
created_at	TIMESTAMPTZ	NOT NULL DEFAULT now()
updated_at	TIMESTAMPTZ	NOT NULL DEFAULT now()

Indexes: UNIQUE (slug).

3.2 channels

One row per knowledge domain. Belongs to exactly one tenant.

Column	Type	Constraints	Notes
channel_id	UUID	PK	Canonical identifier.
tenant_id	UUID	FK to tenants.tenant_id, NOT NULL	Multi-tenant isolation.
name	TEXT	NOT NULL	Knowledge domain name, for example "Billing".
slug	TEXT	NOT NULL	URL-safe, unique per tenant.
description	TEXT	NULL	What this channel answers.
rocket_chat_room_id	TEXT	NULL	Rocket.Chat room/team id this channel maps to.
rocket_chat_room_name	TEXT	NULL	Human-readable room name.
agent_id	UUID	FK to agents.agent_id, UNIQUE, NULL	Exactly one agent per channel. Null until the agent is attached.
kb_scope_id	UUID	FK to kb_scopes.kb_scope_id, UNIQUE, NULL	Exactly one scope per channel. Null until bound.
status	TEXT	NOT NULL DEFAULT 'draft'	draft, active, archived.
created_at	TIMESTAMPTZ	NOT NULL DEFAULT now()
updated_at	TIMESTAMPTZ	NOT NULL DEFAULT now()

Indexes: UNIQUE (tenant_id, slug), INDEX (tenant_id), UNIQUE (agent_id), UNIQUE (kb_scope_id), INDEX (rocket_chat_room_id).

The dual agent_id/kb_scope_id unique columns enforce the one-to-one invariant from both directions.

3.3 agents

One row per AI persona. Bound to exactly one channel.

Column	Type	Constraints	Notes
agent_id	UUID	PK	Canonical identifier.
tenant_id	UUID	FK to tenants.tenant_id, NOT NULL	Multi-tenant isolation.
channel_id	UUID	FK to channels.channel_id, UNIQUE, NOT NULL	One agent per channel.
name	TEXT	NOT NULL	Agent display name, for example "Wallo Billing".
system_prompt	TEXT	NOT NULL	Domain-tuned persona and instructions.
kb_scope_id	UUID	FK to kb_scopes.kb_scope_id, NULL	What this agent may retrieve.
model_provider	TEXT	NOT NULL DEFAULT 'admin-ai'	LLM gateway.
model_name	TEXT	NOT NULL DEFAULT 'deepseek-v4-pro'	Primary model.
model_params	JSONB	NOT NULL DEFAULT '{}'	temperature, max_tokens, top_p.
fallback_model	TEXT	NULL	Next model in the failover chain.
rocket_chat_bot_username	TEXT	UNIQUE, NOT NULL	Bot username in Rocket.Chat.
rocket_chat_bot_user_id	TEXT	NULL	Rocket.Chat internal _id, filled after provisioning.
rocket_chat_bot_token_ref	TEXT	NULL	Vaultwarden reference for the bot personal access token.
status	TEXT	NOT NULL DEFAULT 'draft'	draft, provisioning, active, disabled.
created_at	TIMESTAMPTZ	NOT NULL DEFAULT now()
updated_at	TIMESTAMPTZ	NOT NULL DEFAULT now()

Indexes: UNIQUE (channel_id), UNIQUE (rocket_chat_bot_username), INDEX (tenant_id).

3.4 kb_scopes

One row per knowledge scope. Maps to exactly one vector namespace and one channel.

Column	Type	Constraints	Notes
kb_scope_id	UUID	PK
tenant_id	UUID	FK to tenants.tenant_id, NOT NULL	Multi-tenant isolation.
channel_id	UUID	FK to channels.channel_id, UNIQUE, NOT NULL	One scope per channel.
vector_namespace	TEXT	UNIQUE, NOT NULL	Logical pgvector namespace, for example tenant_{tenant_id}__scope_{kb_scope_id} (see Part 2, section 5).
document_libraries	JSONB	NOT NULL DEFAULT '[]'	List of M365 document libraries this scope may search. Entries carry site_id, drive_id, list_id, and display name.
index_refs	JSONB	NOT NULL DEFAULT '[]'	Search index names the scope may query (Graph fallback).
embedding_model	TEXT	NOT NULL	Model used to embed chunks and queries.
embedding_dim	INT	NOT NULL	Vector dimension. Must match the platform-wide column dimension.
chunk_size	INT	NOT NULL DEFAULT 512	Tokens per chunk (approx 2000 chars). See Part 2, section 2.3.
chunk_overlap	INT	NOT NULL DEFAULT 64	Overlap tokens between chunks (approx 250 chars). See Part 2, section 2.3.
sync_policy	JSONB	NULL	Delta sync schedule and file-type filters.
last_synced_at	TIMESTAMPTZ	NULL	Last successful M365 sync.
created_at	TIMESTAMPTZ	NOT NULL DEFAULT now()
updated_at	TIMESTAMPTZ	NOT NULL DEFAULT now()

Indexes: UNIQUE (channel_id), UNIQUE (vector_namespace), INDEX (tenant_id).

Constraint: pgvector stores a fixed dimension per column. The platform therefore standardizes on one embedding model and dimension across all scopes so chunks.embedding is a single vector(N) column. Changing the model requires a full re-embed and reindex.

3.5 documents

One row per synced source document. Always scoped to a tenant and a channel.

Column	Type	Constraints	Notes
document_id	UUID	PK
tenant_id	UUID	FK to tenants.tenant_id, NOT NULL	Multi-tenant isolation.
channel_id	UUID	FK to channels.channel_id, NOT NULL
kb_scope_id	UUID	FK to kb_scopes.kb_scope_id, NOT NULL
source_type	TEXT	NOT NULL	sharepoint, onedrive, manual_upload.
m365_drive_id	TEXT	NULL	For Graph delta sync.
m365_item_id	TEXT	NULL	For Graph delta sync.
source_path	TEXT	NULL	Full source path, for example the SharePoint URL.
title	TEXT	NULL
file_name	TEXT	NULL
mime_type	TEXT	NULL
content_hash	TEXT	NOT NULL	SHA256 of raw content, for change detection.
size_bytes	BIGINT	NULL
metadata	JSONB	NOT NULL DEFAULT '{}'	author, last modified time, page count.
status	TEXT	NOT NULL DEFAULT 'pending'	pending, extracting, indexing, indexed, failed, deleted.
last_synced_at	TIMESTAMPTZ	NULL
indexed_at	TIMESTAMPTZ	NULL
error	TEXT	NULL	Last sync or index error.
created_at	TIMESTAMPTZ	NOT NULL DEFAULT now()
updated_at	TIMESTAMPTZ	NOT NULL DEFAULT now()

Indexes: INDEX (tenant_id, channel_id), INDEX (kb_scope_id), INDEX (status), partial UNIQUE (tenant_id, m365_drive_id, m365_item_id) WHERE m365_item_id IS NOT NULL.

3.6 chunks

One row per embedded chunk. Carries the vector and the denormalized tenant/channel/scope keys for fast, isolated retrieval.

Column	Type	Constraints	Notes
chunk_id	UUID	PK
document_id	UUID	FK to documents.document_id ON DELETE CASCADE, NOT NULL
tenant_id	UUID	FK to tenants.tenant_id, NOT NULL	Denormalized for retrieval filtering.
channel_id	UUID	FK to channels.channel_id, NOT NULL
kb_scope_id	UUID	FK to kb_scopes.kb_scope_id, NOT NULL
chunk_index	INT	NOT NULL	Position within the document.
content	TEXT	NOT NULL	The chunk text.
token_count	INT	NOT NULL
embedding	vector(N)	NOT NULL	pgvector column. N is the platform-wide dimension.
metadata	JSONB	NOT NULL DEFAULT '{}'	page number, section heading, anchor text.

Indexes: INDEX (document_id), INDEX (tenant_id, kb_scope_id), and a vector index:

CREATE INDEX chunks_embedding_idx ON chunks
USING hnsw (embedding vector_cosine_ops)
WITH (m = 16, ef_construction = 64);

For large tenants, partition chunks by tenant_id so each partition is its own physical namespace and the HNSW index is built per partition.

3.7 messages

One row per inbound question and per outbound answer. Doubles as the audit log and the idempotency ledger.

Column	Type	Constraints	Notes
message_id	UUID	PK
tenant_id	UUID	FK to tenants.tenant_id, NOT NULL	Multi-tenant isolation.
channel_id	UUID	FK to channels.channel_id, NOT NULL
agent_id	UUID	FK to agents.agent_id, NULL	Null for inbound user messages.
rocket_chat_message_id	TEXT	UNIQUE, NOT NULL	Rocket.Chat message _id, used as the idempotency key.
rocket_chat_room_id	TEXT	NOT NULL
rocket_chat_user_id	TEXT	NULL	Author _id on Rocket.Chat.
direction	TEXT	NOT NULL	inbound, outbound.
role	TEXT	NOT NULL	user, assistant.
content	TEXT	NOT NULL
citations	JSONB	NULL	Provenance array, see section 7.
prompt_tokens	INT	NULL
completion_tokens	INT	NULL
total_tokens	INT	NULL
model_name	TEXT	NULL	Model that produced the answer.
latency_ms	INT	NULL	End to end latency.
status	TEXT	NOT NULL	received, resolving, retrieving, prompting, posted, failed.
error	TEXT	NULL	Failure detail.
created_at	TIMESTAMPTZ	NOT NULL DEFAULT now()

Indexes: UNIQUE (rocket_chat_message_id), INDEX (tenant_id, channel_id, created_at), INDEX (created_at).

3.8 users

One row per human staff member. Mirrors the Rocket.Chat user for identity and role mapping.

Column	Type	Constraints	Notes
user_id	UUID	PK
tenant_id	UUID	FK to tenants.tenant_id, NOT NULL	Multi-tenant isolation.
rocket_chat_user_id	TEXT	NOT NULL	Rocket.Chat user _id.
email	TEXT	NULL
display_name	TEXT	NULL
role	TEXT	NOT NULL DEFAULT 'staff'	admin, staff. Controls orchestrator admin surface only.
last_seen_at	TIMESTAMPTZ	NULL
created_at	TIMESTAMPTZ	NOT NULL DEFAULT now()
updated_at	TIMESTAMPTZ	NOT NULL DEFAULT now()

Indexes: UNIQUE (tenant_id, rocket_chat_user_id), INDEX (tenant_id).

4. Expanded Message Flow

The canonical 6-step flow, with exact mechanics.

Step 1: User posts a question

A staff member either @mentions the agent bot in a channel, or DMs the bot directly (section 8).

Step 2: Rocket.Chat fires the webhook

Channel mentions are captured by an outgoing webhook integration configured on the channel with trigger word equal to the bot username. Rocket.Chat POSTs to:

POST https://orchestrator.wall-o.<domain>/api/v1/webhook/rocketchat/{tenant_slug}

Headers:

Header	Value
X-WallO-Signature	Hex HMAC-SHA256 of the raw request body, keyed by the tenant webhook secret.
X-WallO-Timestamp	Unix seconds. Reject if skew is greater than 300 seconds.
Content-Type	application/json

Payload (Rocket.Chat outgoing webhook shape):

{
  "token": "<outgoing webhook token>",
  "channel_id": "<rocketchat room id>",
  "channel_name": "employee-resources",
  "timestamp": "2026-08-15T12:00:00.000Z",
  "user_id": "<rocketchat user id>",
  "user_name": "jane.doe",
  "text": "@wallo-billing How do I submit a PTO request?",
  "trigger_word": "@wallo-billing",
  "bot": false
}

The orchestrator strips the trigger word and mention prefix from text before treating the remainder as the question.

DM capture does not use this webhook. Rocket.Chat outgoing webhooks do not fire inside DMs, so the orchestrator's bot-listener sidecar (section 5) receives DM events over the Realtime API and normalizes them into the identical inbound message object.

Step 3: Orchestrator resolves tenant, channel, agent, scope

Resolution order, all in one request context:

1. Authenticate: verify X-WallO-Signature against the tenant webhook secret looked up by tenant_slug.
2. Idempotency: derive rocket_chat_message_id from a hash of (channel_id, user_id, timestamp, text). If a messages row already exists with that id, return HTTP 200 immediately and do nothing. This deduplicates webhook retries.
3. Resolve channel: SELECT * FROM channels WHERE tenant_id = ? AND rocket_chat_room_id = ?, with a fallback match on rocket_chat_room_name/slug.
4. Resolve agent: read channel.agent_id, then SELECT * FROM agents WHERE agent_id = ?.
5. Resolve scope: read agent.kb_scope_id, then SELECT vector_namespace, embedding_model FROM kb_scopes WHERE kb_scope_id = ?.
6. If the channel has no agent or no scope, reply with a configuration-error message (or a silent no-op, configurable per tenant) and stop.

Step 4: Retrieval

Primary path: pgvector semantic search over the channel's vector namespace.

SELECT chunk_id, document_id, content, metadata, 1 - (embedding <=> $query_vec) AS score
FROM chunks
WHERE tenant_id = $tenant_id AND kb_scope_id = $kb_scope_id
ORDER BY embedding <=> $query_vec
LIMIT 20;  -- candidate set; rerank to 5 per Part 2, section 3.2

The query is embedded with the scope's embedding_model. Both tenant_id and kb_scope_id filters are mandatory in the same query, which is what makes the namespace isolated.

Fallback path: if the top score is below a configured threshold (for example 0.70) or the query returns zero rows, the orchestrator calls Microsoft Graph /search/query over the scope's document_libraries, with the search entity type set to driveItem and the site/drive scoped to the scope's libraries. Results are optionally re-ranked before prompt assembly.

Step 5: Prompt build and LLM call

The prompt is assembled from four parts:

1. agents.system_prompt (the domain-tuned persona).
2. A retrieved-context block where each chunk is labeled with a source marker [1], [2], and so on, carrying its document title and section heading.
3. A citation instruction: answer using only the provided context, cite sources inline as [n], and if the context does not contain an answer, say so instead of guessing.
4. The user question.

The call goes to admin-ai with model_name (DeepSeek V4 Pro primary) and model_params. A per-call timeout (for example 30 seconds) is enforced. On timeout or model error, the orchestrator retries once with fallback_model, then returns a canned "I could not reach the model" reply. It never fabricates an answer.

Step 6: Post the answer back to Rocket.Chat

POST https://<rocket_chat_url>/api/v1/chat.postMessage
Headers: X-Auth-Token: <bot token>, X-User-Id: <bot user id>
Body: { "roomId": "<room id>", "text": "<answer with [n] citations + sources footer>" }

The answer is posted as the agent bot (section 7). On post failure the orchestrator retries with exponential backoff (up to 3 attempts), then marks the message failed and records the error.

Error and timeout handling

• The webhook acknowledges immediately (HTTP 200/202) after persisting the inbound message and enqueuing async processing, so the Rocket.Chat webhook never blocks or times out on the LLM call. The answer is posted out of band via REST.
• Retrieval empty and below threshold: reply "I could not find an answer in the knowledge base for this question" with no sources. Never hallucinate.
• LLM timeout/error: failover model, then a canned error reply.
• Rocket.Chat post failure: retry with backoff, then mark failed and surface to a tenant alert.
• Poison messages go to a dead-letter status with the error retained; alerting fires on elevated failure rate.

5. Rocket.Chat Bot Integration

Recommended mechanism

Primary path: (a) bot user type + REST API, with the outgoing webhook (b) as the canonical channel-mention trigger and a websocket listener for DMs. The Apps Engine (c) is not used.

Justification: the bot user gives each agent a stable, named identity (username, avatar, alias) inside Rocket.Chat and the REST API is versioned, stateless, and fully scriptable from the orchestrator, while the outgoing webhook delivers the canonical inbound POST with no code running inside the chat layer. The Apps Engine is rejected because it would place logic inside the transport layer, violating the architecture split, and would require maintaining a JS app per tenant. Incoming webhooks alone (b by itself) cannot provide a dynamic per-agent identity, so they are used only as an optional posting convenience, not as the identity layer.

Register an agent bot

For each agent, the orchestrator (using the tenant admin token):

1. Create the bot user: POST /api/v1/users.create with:

   {
     "name": "<agent name>",
     "username": "<rocket_chat_bot_username>",
     "email": "<username>@<tenant-domain>",
     "password": "<random 32-char>",
     "roles": ["bot"],
     "joinDefaultChannels": false,
     "requirePasswordChange": false,
     "sendWelcomeEmail": false,
     "verified": true
   }
   

2. Create a personal access token: POST /api/v1/users.createToken with { "userId": "<bot _id>" }. The returned authToken and userId are stored in Vaultwarden; the reference goes in agents.rocket_chat_bot_token_ref and the _id in agents.rocket_chat_bot_user_id.
3. Optionally set the avatar: POST /api/v1/users.setAvatar (uploaded image) or users.setAvatarFromUrl.

Join a channel

The orchestrator (via the tenant admin token) invites the bot to the channel's room:

POST /api/v1/channels.invite with { "roomId": "<rocket_chat_room_id>", "userId": "<bot _id>" }.

Bots auto-accept invitations. After this the bot is a room member and can be @mentioned and post as itself.

Route @mentions and commands

• @mention/trigger word: configure an outgoing webhook integration on the channel with trigger word set to the bot username. A user mentioning the bot (or typing the trigger word) causes Rocket.Chat to POST to the orchestrator webhook URL (section 4, step 2). The orchestrator strips the mention/trigger prefix.
• Slash command (optional): POST /api/v1/commands.create to register a command such as /ask in the channel that routes to the same webhook. Useful when a channel hosts more than one purpose and explicit scoping is wanted.

Route DMs

Because outgoing webhooks do not fire in DMs, the orchestrator runs a bot-listener sidecar that maintains a websocket (Realtime API) session per bot: connect, call method: "login" with the bot token, then subscribe to stream-room-messages for the bot's own user id. Both DMs and room mentions arrive as the same message event type. The sidecar normalizes them into the identical inbound message object as the webhook path and enqueues them through the same pipeline. The sidecar is a pure transport shim; it holds no intelligence.

6. Channel Lifecycle

Step	Action	Performed by
1. Create channel	Insert a channels row (tenant, name, slug, description, status = draft). Optionally create the Rocket.Chat room via POST /api/v1/channels.create and store rocket_chat_room_id.	Orchestrator API, tenant admin role
2. Attach agent	Insert an agents row (system_prompt, model config, bot username) with channel_id set. Provision the Rocket.Chat bot user and token (section 5). Invite the bot to the room. Set channels.agent_id.	Orchestrator API + Rocket.Chat REST
3. Bind kb_scope	Insert a kb_scopes row (vector_namespace, document_libraries, embedding model/dim, chunk params). Set channels.kb_scope_id.	Orchestrator API
4. Initial sync and index	The M365 connector pulls the named document libraries, extracts text, chunks, embeds, and writes documents + chunks rows under the namespace.	M365 connector worker
5. Go live	Set channels.status = active. Configure the outgoing webhook and optional slash command. Run an end-to-end smoke test question and confirm a cited answer posts back as the bot.	Orchestrator API + operator

Steps 1 through 3 are idempotent API calls driven by an onboarding form in the orchestrator admin surface. Step 4 is the only long-running step; the channel stays in draft until indexed document count is nonzero. Step 5 flips status and does not require a redeploy.

7. Agent Identity and Response Citations

Identity: the bot posts with its own username and avatar (set at provisioning). Every answer is attributed to the bot user, never to a human, so staff can distinguish agent answers from colleague messages. The system_prompt instructs the agent to introduce itself as the channel's assistant (for example "I am the Wall-O assistant for Employee Resources") and to stay inside the domain.

Citations: retrieved chunks carry provenance in metadata (document title, section heading, page). During prompt assembly each chunk is labeled [1], [2], and so on. The orchestrator renders the final answer with inline [n] markers and appends a "Sources" footer listing each cited document title and a link (Microsoft Graph sharing link, or a Rocket.Chat file link). The full provenance (document_id, chunk_id, title, snippet, url) is stored in messages.citations as JSONB for audit and re-render. If the answer cites nothing, no footer is emitted.

8. User Interaction Model

Mode	When to use	Trigger path
Channel @mention	Shared or discoverable questions, team-wide answers, anything others should see.	Outgoing webhook (section 4, step 2).
DM the bot	Private follow-up, iterative clarification, personal-but-not-PHI questions, or a 1:1 thread the user does not want in the channel.	Bot-listener websocket (section 5).

Guidance surfaced to staff: use the channel for Q&A that benefits the team (answers stay searchable in the room), and DM the bot for private or back-and-forth questions. Both paths produce the same cited answer format; only the destination differs.

9. Multi-tenant Isolation and Security

• Tenant_id on every table: tenants, channels, agents, kb_scopes, documents, chunks, messages, and users all carry tenant_id. The application derives the tenant from the webhook path and HMAC signature, never from a client-supplied body field alone, and scopes every query with it.
• Postgres row-level security: enable RLS on all tenant tables with a policy tenant_id = current_setting('app.tenant_id'), set per request, as defense in depth under the application-level filter.
• Vector namespace isolation: retrieval always filters tenant_id AND kb_scope_id in the same query, so a chunk can never leak across channels or tenants. Partitioning by tenant_id makes each tenant's vectors physically separate.
• Secrets: M365 credentials, bot tokens, and webhook secrets live in Vaultwarden. The database stores references only. Never commit secrets; .env.example placeholders only.
• Scope guard: the product never ingests patient records or PHI. The M365 connector's document_libraries are allowlisted per scope, and a content filter flags documents outside the allowlisted libraries before indexing.

Wall-O: M365 Connector + Retrieval Architecture

Status: DESIGN (internal technical architecture) Owner: Wall-O build team Scope: Internal staff knowledge only. Employee handbook, policies, procedures, billing questions. Excluded: Patient records, PHI, any HIPAA-regulated data. This connector MUST NOT be pointed at clinical or patient data sources.

Canonical entities used in this document

Entity	Definition	Cardinality
tenant_id	A practice (e.g. Wall Orthodontics). The Microsoft 365 tenant boundary.	1 M365 tenant per tenant_id
channel_id	A knowledge domain within a tenant (e.g. "Employee Resources").	many per tenant
agent_id	An AI persona bound to a channel. One agent serves one channel.	1:1 with channel_id
kb_scope	The set of documents/indices an agent may search. Scoped per tenant AND per channel. Maps 1:1 to a vector namespace.	1:1 with vector namespace
documents/chunks	Files synced from M365, chunked and indexed into the tenant+channel vector namespace.	many per kb_scope

The vector namespace is a logical concept. In pgvector it is implemented as a namespace column on the chunk table and filtered in the WHERE clause of every similarity query. In Qdrant it would be a native collection. The logical name format is fixed regardless of backend:

tenant_{tenant_id}__scope_{kb_scope_id}

---

1. Authentication model (Microsoft Graph, app-only)

1.1 Entra app registration

Single multi-tenant app registration in the IT Pro Partner (IPP) home tenant. Supported account type:

Accounts in any organizational directory (Any Microsoft Entra ID tenant - Multitenant)

This lets one app identity serve every practice tenant. Each practice admin consents the app into their own tenant; no app is registered inside the client's tenant.

1.2 Permission strategy: Sites.Selected over Sites.Read.All

Wall-O reads documents, never writes. Least privilege is achieved with the Sites.Selected application permission rather than the broad Sites.Read.All.

Permission	Type	Scope	Why accepted / rejected
Sites.Selected	Application	Only sites explicitly granted via the site permissions API	ACCEPTED. Primary. Grants per-site read; the app sees nothing else in the tenant.
Sites.Read.All	Application	Every site in the tenant	REJECTED for production. Violates least privilege; exposes all site collections including any we are not meant to index.
Files.Read.All	Application	Every file in every drive	REJECTED. Superseded by Sites.Selected for site-scoped read.
User.Read.All	Application	Read directory user profiles	OPTIONAL. Needed only to resolve author display names from OneDrive drive owner IDs. Not required for retrieval.

Sites.Selected supports site-level roles: read, write, fullcontrol, manage. Wall-O requests read only. The grant is issued per site collection via:

POST https://graph.microsoft.com/v1.0/sites/{site-id}/permissions
Content-Type: application/json

{
  "roles": ["read"],
  "grantedToIdentities": [
    { "application": { "id": "{wall-o-app-client-id}", "displayName": "Wall-O" } }
  ]
}

This request is made by the orchestrator using the app's own token for the tenant. Because the app already holds Sites.Selected, it can grant itself read on a specific site once a practice admin has approved the site in onboarding. A stricter variant has a practice global admin run the grant via Graph Explorer so the app never self-grants. Wall-O uses the admin-driven variant: the grant is issued during onboarding by the practice admin (or by the orchestrator on a one-time admin-approved site list), not by the app unprompted.

1.3 Client credentials grant

App-only OAuth 2.0 client credentials flow. No user, no interactive login, no refresh token.

Token request:

POST https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token
Content-Type: application/x-www-form-urlencoded

client_id={client_id}
client_secret={client_secret}
scope=https://graph.microsoft.com/.default
grant_type=client_credentials

Exact strings to configure:

Field	Exact value
grant_type	client_credentials
scope	https://graph.microsoft.com/.default
Token endpoint	https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token

Notes:

• {tenant_id} is the consuming practice's directory (tenant) ID, captured at consent time. It is NOT common or organizations: client credentials has no user to derive a tenant from, so the target tenant must be explicit.
• .default expands to the union of the application permissions already consented for that tenant. It is not a literal scope name.
• Response contains access_token and expires_in. Graph client credentials tokens are typically valid for about 60 minutes; treat expires_in as authoritative, never hardcode the TTL.
• Credential material: prefer an X.509 client certificate (assertion) over a client secret. A secret is acceptable for MVP but must be rotated before its maximum lifetime (24 months). Secrets live in Vaultwarden under the Wall-O project; only the encrypted reference enters Postgres or config.

1.4 Admin consent URL construction

Each practice admin must consent the app into their tenant before the first crawl. The v2.0 admin consent endpoint is constructed as:

https://login.microsoftonline.com/{tenant_id}/v2.0/adminconsent
  ?client_id={wall-o-app-client-id}
  &scope=https://graph.microsoft.com/.default
  &redirect_uri={configured-redirect-uri}

• {tenant_id}: the practice's directory ID, or organizations to let the signing admin's tenant be used automatically.
• redirect_uri: a registered reply URL on the app. Wall-O registers a no-op callback (e.g. https://wall-o.itpropartner.com/entra/callback) that returns a 200 and logs the consent result.
• The scope string is URL-encoded .default (i.e. https%3A%2F%2Fgraph.microsoft.com%2F.default).

Onboarding flow per practice:

1. Practice admin clicks the consent URL (delivered by the Wall-O onboarding UI or support).
2. Admin authenticates and approves the Sites.Selected application permission.
3. On success Entra redirects to the callback. The orchestrator records tenant_id, consent timestamp, and the admin's identity.
4. The site-level read grant (1.2) is then applied to the specific site collections mapped to that practice's channels.
5. Only now does the connector issue its first token and run a test crawl.

Consent revocation is detected at token time (see 1.5); the tenant is marked consent_revoked and the practice admin is prompted to re-consent.

1.5 Per-tenant token storage and refresh lifecycle

No refresh token exists in client credentials, so "refresh" means issuing a fresh token on demand. The cache is a memoization layer plus a lifecycle guard.

Storage (Postgres, tenant_credentials table):

Column	Purpose
tenant_id	Primary cache key
access_token	Encrypted at rest (AES-256-GCM, app-level, key in Vaultwarden)
expires_at	Absolute expiry derived from expires_in at issue time
last_error / error_code	Last failure for alerting (e.g. consent revoked, secret expired)
status	active, consent_revoked, secret_expired, suspended

Lifecycle rules:

• Token is considered usable while now < expires_at - 300s (5 minute safety margin). Outside that window the connector requests a fresh token before the next Graph call.
• On 401 Unauthorized with InvalidAuthenticationToken, or 403 with a consent/scope error, the connector retries once with a freshly issued token. A second failure escalates.
• Error code mapping:
  • AADSTS700016 (application not found in directory) or AADSTS7000112 (invalid client) means the app is not consented in that tenant: mark consent_revoked, alert practice admin.
  • AADSTS700082 or expired secret errors mean the secret is expired/rotated: mark secret_expired, page the Wall-O operator.
  • Graph throttling 429 is not an auth failure: honor Retry-After and back off; do not flag the tenant.
• Token issuance and refresh are serialized per tenant (single-flight lock) so concurrent sync workers do not stampede the token endpoint.
• Tokens are never logged or returned by any API. Logging redacts the Authorization header.

1.6 Publisher verification

Required so the multi-tenant consent prompt shows a verified publisher rather than "unverified", which materially reduces admin consent friction (and some tenants block unverified apps outright).

Requirements:

• A Microsoft AI Cloud Partner Program (MAICPP) account with an MPN ID for IT Pro Partner.
• The MPN ID associated with the app registration under Branding and properties.
• A verified custom domain (e.g. itpropartner.com) linked to the Entra tenant and set as the publisher domain.
• App registration Branding shows "Publisher verified" with the blue checkmark.

Publisher verification is a one-time per-publisher step, not per-tenant. It is a prerequisite to onboarding the first external practice; without it, practice admins see an unverified-publisher warning on the consent screen.

---

2. Sync and index pipeline

2.1 Crawl sources

Two source types, both driven by Microsoft Graph:

Source	Graph entry points
SharePoint document libraries	GET /sites/{site-id}/drives to enumerate libraries, then per-drive crawl
OneDrive for Business	GET /users/{user-id}/drive (a practice user's personal drive mapped to a channel)

Site resolution by human-friendly URL before crawling:

GET https://graph.microsoft.com/v1.0/sites/{hostname}:/{site-path}
GET https://graph.microsoft.com/v1.0/sites?search={query}

Each crawl source is registered in Postgres as part of a kb_scope, so a scope maps to one or more (site, drive) pairs. The connector crawls exactly those drives, never the whole tenant.

2.2 Crawl, extract, chunk, embed, write

Pipeline stages, all inside the FastAPI orchestrator (worker tasks, not the request path):

1. Crawl: enumerate items per drive using GET /sites/{site-id}/drive/root/children and GET /sites/{site-id}/drive/root:/{path}:/children for nested folders. Use $batch (up to 20 requests per batch) to reduce round trips and honor throttling.
2. Download: GET /sites/{site-id}/drive/items/{item-id}/content (or the item's @microsoft.graph.downloadUrl). Stream to disk, never into memory whole.
3. Extract: text extraction per format (see section 4.1). Produce plain text plus a metadata block.
4. Chunk: split with the strategy in 2.3.
5. Embed: embed each chunk with the configured model (section 5.2).
6. Write: upsert chunks into the {tenant}_{kb_scope} vector namespace in pgvector. One transaction per document: delete old chunks for that document_id, insert new, commit. Keeps the namespace consistent even if a sync crashes mid-document.

2.3 Chunking strategy

Concrete defaults:

Parameter	Value
Target chunk size	512 tokens (approx 2000 chars)
Overlap	64 tokens (approx 250 chars, 12.5%)
Splitter	Sentence-aware: split on paragraph then sentence boundaries, never mid-word
Hard max	1024 tokens for a single chunk (tables, bullet lists, malformed PDF text)
Min kept	Discard chunks under 20 tokens
Tokenizer	The embedding model's tokenizer (cl100k_base if using OpenAI text-embedding-3)

Per-chunk metadata written alongside the vector:

Metadata field	Source
tenant_id	canonical entity
channel_id	canonical entity
kb_scope_id	canonical entity
namespace	tenant_{tenant_id}__scope_{kb_scope_id}
document_id	Graph driveItem id
document_name	driveItem name
site_id / drive_id / item_id	Graph IDs for provenance and re-download
source_url	driveItem webUrl
modified_at	driveItem lastModifiedDateTime
chunk_index	position within document
mime / file_type	for format-aware handling
title / heading	nearest heading above the chunk, for context injection

Metadata is stored in the same Postgres chunk row so filtering (by channel, by scope) is a plain SQL predicate, not a secondary lookup.

2.4 Incremental sync (delta query + change notifications)

Full re-crawl on every sync is avoided with two complementary mechanisms.

Delta query (poll-based, authoritative):

GET https://graph.microsoft.com/v1.0/sites/{site-id}/drive/root/delta

• First call with no token returns the full set and a @odata.deltaLink (and @odata.nextLink while paging).
• Subsequent calls use the stored deltaLink and return only added, changed, and deleted items. Deleted items carry a deleted facet; the connector removes their chunks from the namespace.
• Persist the deltaLink per (site, drive) in Postgres.
• Delta tokens expire; a 410 Gone (or malformed delta token) means the connector must drop to a full re-crawl for that drive. The connector treats 410 as a normal control-flow event, logs it, and re-syncs fully.

Change notifications (push-based, reduces lag):

POST https://graph.microsoft.com/v1.0/subscriptions
Content-Type: application/json

{
  "changeType": "updated",
  "notificationUrl": "https://wall-o.itpropartner.com/entra/notifications",
  "resource": "/sites/{site-id}/drive/root",
  "expirationDateTime": "2026-08-18T18:00:00Z",
  "clientState": "tenant_{tenant_id}"
}

• changeType may be created,updated,deleted combined in one subscription.
• expirationDateTime maximum is 4230 minutes (3 days). The orchestrator renews every subscription before expiry (cron, every 6 hours, PATCH /subscriptions/{id} to extend).
• The notificationUrl must be HTTPS, publicly reachable, and answer the initial validationToken handshake: echo the token back as text/plain with HTTP 200 within 5 seconds, then process the notification asynchronously.
• clientState is echoed back on every notification so the orchestrator verifies the sender and does not act on forged payloads.
• On notification, the connector does NOT fetch blindly: it records a dirty (site, drive) and lets the next delta pass reconcile. This deduplicates burst notifications into one delta sweep.

Sync schedule (default): full crawl on onboarding and after a 410; delta sweep every 15 minutes; change notifications applied opportunistically between sweeps. Re-embed only changed/deleted chunks, not the whole namespace.

2.5 Sync observability

Per (tenant, scope) the orchestrator records: last_full_sync_at, last_delta_sync_at, items_seen, chunks_written, chunks_deleted, and last_sync_error. A dashboard flag is raised when last_delta_sync_at exceeds the 15 minute SLA by a factor of 3 (i.e. stale over 45 minutes), which also gates the stale-index retrieval fallback in section 3.3.

---

3. Retrieval

3.1 Semantic search (primary)

The agent's query is embedded with the same model and dimensionality used at index time. Retrieval runs against exactly one vector namespace: the agent's kb_scope namespace. Tenancy and channel scoping is therefore a hard SQL predicate, not an after-the-fact filter.

pgvector query shape:

SELECT chunk_id, document_id, document_name, source_url, text, chunk_index,
       1 - (embedding <=> $1) AS similarity
FROM chunks
WHERE namespace = $2
ORDER BY embedding <=> $1
LIMIT $3;

• <=> is cosine distance; 1 - (embedding <=> $1) yields cosine similarity. Embeddings are stored L2-normalized so cosine and inner product are equivalent.
• top-k for the candidate set is 20.

3.2 Rerank (optional second stage)

A cross-encoder reranks the 20 candidates against the raw query to 5 final passages. This is quality over recall: the cross-encoder sees query and passage jointly and scores true relevance, correcting embedding-only mistakes on synonyms and negations.

• Model: BAAI/bge-reranker-large (self-hosted on the app server) or a managed reranker via LiteLLM if one is added to admin-ai.
• Rerank is a config toggle per agent. If disabled, the top 5 by cosine similarity are used directly.

3.3 Graph live search fallback (cold or stale index)

When the vector namespace is cold (fewer than N chunks, e.g. onboarding in progress) or stale (delta sync behind, see 2.5), retrieval falls back to live Graph search so answers are never silently empty. The Graph search API is used as the fallback, not the primary, because it is slower, rate-limited, and returns passages without the tight namespace scoping.

POST https://graph.microsoft.com/v1.0/search/query
Content-Type: application/json

{
  "requests": [
    {
      "entityTypes": ["driveItem"],
      "query": { "queryString": "{user query}" },
      "from": 0,
      "size": 10,
      "fields": ["id", "name", "webUrl", "lastModifiedDateTime"]
    }
  ]
}

A simpler drive-scoped variant is available when the fallback is constrained to one site:

GET https://graph.microsoft.com/v1.0/sites/{site-id}/drive/root/search(q='{user query}')

The fallback returns file references (name, webUrl, snippet), which the orchestrator hands to the LLM as source citations rather than pre-chunked text.

3.4 Hybrid ranking approach

Two signals are fused when both are available:

Signal	Source	Weight role
Semantic	embedding cosine similarity	primary
Keyword	Postgres full-text search (tsvector) or pg_trgm over chunk text	secondary, for exact terms: policy IDs, benefit codes, acronyms

Fusion uses Reciprocal Rank Fusion (RRF):

score(d) = sum over signals of 1 / (k + rank_signal(d)),  k = 60

This preserves exact-term matches (e.g. a specific policy number) that pure embedding search can miss, without a separate vector backend. The keyword index is a generated tsvector column on the chunk table maintained by trigger or at write time.

3.5 Graph endpoints actually used (reference)

Purpose	Endpoint
Token	POST /oauth2/v2.0/token (login.microsoftonline.com)
Admin consent	GET /{tenant}/v2.0/adminconsent
Site resolution	GET /v1.0/sites/{hostname}:/{path}, GET /v1.0/sites?search=
Site read grant (Sites.Selected)	POST /v1.0/sites/{site-id}/permissions
List drives in a site	GET /v1.0/sites/{site-id}/drives
List folder children	GET /v1.0/sites/{site-id}/drive/root/children
Path-based children	GET /v1.0/sites/{site-id}/drive/root:/{path}:/children
Download content	GET /v1.0/sites/{site-id}/drive/items/{item-id}/content
Delta sync	GET /v1.0/sites/{site-id}/drive/root/delta
OneDrive (user)	GET /v1.0/users/{user-id}/drive/root/delta
Live search (fallback)	POST /v1.0/search/query
Drive-scoped search (fallback)	GET /v1.0/sites/{site-id}/drive/root/search(q='...')
Change notifications	POST /v1.0/subscriptions, PATCH /v1.0/subscriptions/{id}
Author resolution (optional)	GET /v1.0/users/{user-id}

---

4. Document format support, size limits, ACL mapping

4.1 Format support

Format	Extraction approach
PDF	PyMuPDF (fitz), per-page text extraction; flagged as text-only (embedded OCR is out of scope)
DOCX	python-docx: paragraphs + tables
XLSX	openpyxl: sheet-by-sheet, cell grid serialized with sheet name as heading context
PPTX	python-pptx: slide title + body text per slide
TXT / MD / CSV	plain read with encoding detection
HTML (optional)	trafilatura / BeautifulSoup text extraction

The connector downloads and parses locally. Graph does not return extracted text, only file content and metadata. Binary formats not in this table are skipped and counted in sync observability, never force-indexed.

4.2 File size limits

Limit	Value	Rationale
Max file for extraction	50 MB	Above this, office files and PDFs blow up parse time and memory; skipped and flagged
Max chunk text written per file	10 MB of extracted text	Mirrors SharePoint search's practical text-index ceiling; larger files are truncated with a notice chunk
Graph simple download ceiling	approx 250 MB via /content	Beyond this use the item's @microsoft.graph.downloadUrl or resumable download
SharePoint/OneDrive storage ceiling	250 GB per file	Microsoft limit; irrelevant to us because we cap extraction far lower

Files over 50 MB (or producing over 10 MB of text) are recorded as skipped_oversize with a pointer to their webUrl so an agent can still cite the source without the text being chunked.

4.3 Sites.Selected ACLs mapped to per-channel kb_scope

The security property is: an agent must only see documents its channel is authorized for. Two layers enforce this.

Layer 1: Graph ACLs (what the connector may even fetch). Sites.Selected grants the app read on a specific site collection only. A site the app cannot read produces 403 and is simply not crawlable. The connector can therefore never ingest content outside the consented sites.

Layer 2: kb_scope mapping (what an agent may search). Every crawl source is bound to a kb_scope, and every kb_scope maps to exactly one vector namespace. The mapping table in Postgres:

tenant_id	channel_id	kb_scope_id	source (site_id, drive_id)	vector namespace
wall-ortho	employee-resources	scope_er	site handbook.wallortho.com (drive A)	tenant_wall-ortho__scope_scope_er
wall-ortho	billing-procedures	scope_bp	site billing.wallortho.com (drive B)	tenant_wall-ortho__scope_scope_bp

Enforcement at query time:

• The agent is bound to one channel_id, which resolves to its kb_scope set.
• Retrieval queries only the namespace(s) in that kb_scope set. The namespace column is a mandatory equality filter; an agent with channel employee-resources can never match chunks in scope_bp.
• The same mapping gates the Graph fallback: the fallback only searches sites present in the agent's kb_scope sources.

The combination means authorization is enforced at both ingestion (Graph site grant) and retrieval (namespace filter). A misconfigured kb_scope cannot leak another channel's content because the namespace predicate is applied server-side and the connector never wrote that content into the agent's namespace in the first place.

---

5. Vector store and embedding model

5.1 Vector store: pgvector (recommended)

Criterion	pgvector	Qdrant	Chroma
Runs on existing stack	Yes: Postgres already in production	New stateful service on netcup	New stateful service (or embedded single-node)
Tenancy model	Namespace column + filtered query	Native collections (good)	Collections, weaker multi-tenant isolation
Operational overhead	Zero: reuse backups, HA, auth of Postgres	Extra container, memory, monitoring	Extra container; single-node design
Scale fit	Strong to low millions of chunks	Strong beyond that	Weak-moderate
Backup to Wasabi	Inherits existing Postgres S3 backup	Separate snapshot/export	Separate persistence
Metadata + filters	Native SQL join with tenant/channel tables	Payload filters (separate model)	Metadata filter, less mature

Rationale: ITPP already runs Postgres with backup to Wasabi S3, on self-hosted netcup servers. Wall-O's scale is moderate: a practice's internal staff knowledge base is hundreds to low thousands of documents, tens of thousands to low millions of chunks across all tenants. pgvector handles this comfortably with an HNSW index, adds zero new stateful services, keeps vectors and tenancy metadata in one transactional database (atomic upsert, consistent deletes), and inherits the existing backup and HA story. Qdrant would be justified only at very large scale or if vectors needed independent scaling from metadata; Chroma's single-node embedded design is the wrong fit for a multi-tenant production service.

Implementation notes:

• Extension: CREATE EXTENSION vector; (pgvector >= 0.5.0 for HNSW; newer for larger dimension support).
• Index: HNSW on the embedding column. Because queries are namespace-scoped, use a filtered/partial strategy: either a composite approach (HNSW per namespace table via partitioning) or a single HNSW with the namespace filter applied post-recall. At Wall-O scale, a single HNSW index plus a namespace equality filter in the WHERE clause is sufficient and simplest.
• Dimension: 1024 (see 5.2), well within pgvector's limits.
• Distance: cosine (<=>) with L2-normalized vectors.
• Chunk text and metadata live in the same table so retrieval returns citations in one query with no join to an object store for the common path. Wasabi S3 remains the archive for raw downloaded files and full-text backups, not the retrieval hot path.

5.2 Embedding model: text-embedding-3-large (Matryoshka to 1024), bge fallback

Model	Dims (used)	Hosting	Why / why not
OpenAI text-embedding-3-large	3072 native, truncated to 1024 via Matryoshka	Via admin-ai LiteLLM proxy	RECOMMENDED. Strong retrieval quality, already reachable through the existing LiteLLM gateway, no GPU. Matryoshka truncation to 1024 keeps pgvector index and storage small with negligible quality loss.
text-embedding-3-small	1536	Via LiteLLM	Acceptable budget option; slightly lower quality, still fine for internal KB.
BAAI/bge-large-en-v1.5	1024	Self-hosted on app server	FALLBACK. Zero external dependency and free, but adds model serving burden and slightly weaker than text-embedding-3-large on this task.
bge-m3	1024	Self-hosted	Only if multilingual staff content appears; out of scope for English-only Wall Orthodontics.

Rationale: Wall-O routes LLM calls through admin-ai (DeepSeek V4 Pro primary) already, so embeddings through the same LiteLLM gateway are the lowest-operational-overhead choice and keep spend attributable to the Wall-O virtual key. text-embedding-3-large with Matryoshka truncation to 1024 dimensions gives near-full quality at a quarter of the storage and index cost. The model is pinned and documented so index and query always use identical dimensionality and normalization; changing models requires a documented full re-embed of every namespace (a versioned embedding-model field on the chunk table gates this).

---

Non-negotiable constraints (summary)

• No patient records, no PHI, no HIPAA scope. The connector is configured per site and never pointed at clinical content.
• Sites.Selected read grants only; no tenant-wide read permission in production.
• Namespace (tenant + kb_scope) is a mandatory equality filter on every retrieval and every Graph fallback search.
• Client secret (or certificate) in Vaultwarden; tokens encrypted at rest in Postgres; nothing in logs.
• No em dashes, en dashes, or double hyphens in this document or any Wall-O docs.

Wall-O: Deployment, White-Label, and Phase 0 Build Plan

Scope: multi-tenant internal staff knowledge-base chat. Rocket.Chat is chat transport only (one workspace per tenant). The orchestrator is a multi-tenant FastAPI service that owns tenancy, agents, kb_scope, the M365 connector, retrieval, and LLM. Strictly internal staff knowledge. No patient records, no PHI.

Canonical entities: tenant_id (a practice), channel_id (a knowledge domain with one attached agent), agent_id (an AI persona bound to a channel), kb_scope (per-tenant plus per-channel document scope).

1. Deployment Topology

1.1 Component map

Component	Host	Port	Scope	Scales per tenant
Caddy reverse proxy	wall-o host	80 / 443	shared	single process, per-tenant site blocks
Orchestrator (FastAPI, ASGI via uvicorn worker)	wall-o host	127.0.0.1:8000	shared	1 instance; add workers or a second host behind a load balancer
Orchestrator DB (Postgres + pgvector)	wall-o host	127.0.0.1:5432	shared	1 primary; promote to a replica for read scale
Rocket.Chat workspace (tenant N)	wall-o host	127.0.0.1:3010N	per tenant	1 Docker Compose stack per tenant, no shared state
MongoDB (tenant N, Rocket.Chat native store)	wall-o host	127.0.0.1:2710N (loopback only)	per tenant	1 Mongo per tenant, isolated volume
Branding assets (logo, favicon, custom CSS, PWA shell)	wall-o host	/var/www/wall-o/	per tenant	static files, chmod 644, no process
M365 connector (orchestrator module)	wall-o host	n/a (in-process)	shared	horizontal with the orchestrator

Port formulas: tenant N Rocket.Chat = 30100 + N, tenant N MongoDB = 27100 + N. Tenant 1 (Wall Orthodontics) uses 30101 and 27101.

URL pattern (ITPP convention):

Surface	URL
Orchestrator API	https://api.wall-o.itpropartner.com
Orchestrator admin console	https://wall-o.itpropartner.com
Tenant N chat + branded PWA	https://.wall-o.itpropartner.com

Caddy is the TLS edge. Each tenant subdomain gets its own site block that reverse proxies to that tenant's Rocket.Chat host port. Site blocks are generated from the tenancy table by a small render script (single source of truth in the DB, not hand-edited Caddyfile).

1.2 Directory and config layout

/opt/wall-o/                                  # app code and compose, NOT /root
├── orchestrator/
│   ├── app/
│   │   ├── main.py                           # FastAPI app + routers
│   │   ├── tenancy.py                        # tenant CRUD + connection factory
│   │   ├── agents.py                         # agent personas, model binding
│   │   ├── kb_scope.py                       # per-tenant + per-channel scope enforcement
│   │   ├── m365.py                           # Microsoft Graph connector
│   │   ├── retrieval.py                      # pgvector query, RAG pipeline
│   │   └── llm.py                            # admin-ai (LiteLLM) client
│   ├── alembic/                              # schema migrations
│   ├── requirements.txt
│   └── .env.example                          # placeholders only, no secrets
├── tenants/
│   ├── wall-orthodontics/
│   │   ├── docker-compose.yml                # Rocket.Chat + Mongo for this tenant
│   │   ├── mongod.conf                       # replSetName: rs0
│   │   └── .env                              # gitignored, values from Vaultwarden
│   └── <tenant-slug>/ ...
├── caddy/
│   └── render-caddyfile.py                   # emits per-tenant site blocks from DB
└── backups/
    └── run-backups.sh                        # mongodump + pg_dump to Wasabi S3

Web-served static assets (branding, PWA shell, custom CSS) live under /var/www/wall-o/ and are mounted read-only into each Rocket.Chat container. All files under /var/www/wall-o/ are chmod 644, owned by a service account, never by root home.

1.3 Tenant Docker Compose (Rocket.Chat + MongoDB)

services:
  mongo:
    image: mongo:6.0
    command: ["mongod", "-f", "/etc/mongod.conf"]
    volumes:
      - ./mongod.conf:/etc/mongod.conf:ro
      - mongo-data:/data/db
    ports:
      - "127.0.0.1:27101:27017"          # loopback only, for mongodump
    restart: unless-stopped

  rocketchat:
    image: registry.rocket.chat/rocketchat/rocket.chat:7.4.0   # pinned; dev CE for Phase 0
    environment:
      ROOT_URL: ${ROOT_URL}               # https://wall-orthodontics.wall-o.itpropartner.com
      MONGO_URL: mongodb://mongo:27017/rocketchat?replicaSet=rs0
      MONGO_OPLOG_URL: mongodb://mongo:27017/local?replicaSet=rs0
      PORT: "3000"
      ADMIN_USERNAME: ${ADMIN_USERNAME}   # from Vaultwarden
      ADMIN_PASSWORD: ${ADMIN_PASSWORD}   # from Vaultwarden
      ADMIN_EMAIL: ${ADMIN_EMAIL}
      ADMIN_NAME: ${ADMIN_NAME}
    depends_on:
      - mongo
    ports:
      - "127.0.0.1:30101:3000"
    volumes:
      - /var/www/wall-o/wall-orthodontics/branding:/app/branding:ro   # chmod 644
    restart: unless-stopped

volumes:
  mongo-data:

mongod.conf:

storage:
  dbPath: /data/db
replication:
  replSetName: rs0

No secrets are committed in compose files. The per-tenant .env holds ADMIN_USERNAME, ADMIN_PASSWORD, and other values, and the .env values are populated at deploy time from Vaultwarden. .env is gitignored; .env.example carries placeholders only.

1.4 Orchestrator to N Rocket.Chat instances

The orchestrator does not hold one global Rocket.Chat credential. Each tenant row carries its own workspace base URL and admin identity (used for provisioning bots and rooms), keyed by tenant_id. Posting answers uses the per-agent bot identity on the agents table (Part 1, section 3.3).

CREATE TABLE tenants (
    tenant_id          UUID PRIMARY KEY,
    slug               TEXT UNIQUE NOT NULL,
    name               TEXT NOT NULL,
    rocket_chat_url            TEXT NOT NULL,
    rocket_chat_admin_user_id  TEXT NOT NULL,
    rocket_chat_admin_token_ref TEXT NOT NULL,
    status             TEXT NOT NULL DEFAULT 'provisioning',
    created_at         TIMESTAMPTZ NOT NULL DEFAULT now()
);

The provisioning client reads one row and builds an httpx client signed with the tenant admin identity. A separate factory builds the per-agent posting client from the agent's own bot token (agents.rocket_chat_bot_token_ref), so answers post as the channel's agent, never as the tenant admin. Tokens are never stored in the DB, only Vaultwarden item references.

def admin_client_for(tenant_id: str) -> httpx.AsyncClient:
    t = db.get_tenant(tenant_id)
    admin_cred = vaultwarden.get_item(t.rocket_chat_admin_token_ref).password   # admin auth token, not in DB
    return httpx.AsyncClient(
        base_url=t.rocket_chat_url,
        headers={"X-User-Id": t.rocket_chat_admin_user_id, "X-Auth-Token": admin_cred},
    )

Provisioning order per tenant: insert tenant row, render Caddy block, docker compose up, seed the workspace (admin identity, agent bot user, agent bot token), store admin and bot tokens in Vaultwarden, write the admin identity back into the tenants row and the agent bot identity into the agents row, flip status to ready.

2. White-Label Execution Plan

2.1 EE-strip decision: FOSS-only build (fossify), not stock CE

Facts that drive the decision:

Fact	Detail
Single codebase since 3.1.0	Community Edition core is MIT; Enterprise Edition (EE) code is source-available under a proprietary license and sits under apps/meteor/ee/
Official Docker image	registry.rocket.chat/rocketchat/rocket.chat bundles EE code in the build, even when no EE license key is applied
fossify script	Removes all non-MIT code to produce a pure FOSS build; no prebuilt FOSS image is published, so we build it ourselves

Decision: for a product we resell, use the FOSS-only build via the fossify script. MIT covers modification and commercial redistribution. The EE license restricts use and distribution, so shipping the stock image (which contains EE code) into a resold, white-labeled product is a licensing risk even if we never activate an EE key. We do not need EE features anyway: Rocket.Chat is chat transport only, and agents, retrieval, kb_scope, and the LLM all live in the orchestrator.

Consequence: maintain a private fork plus a CI job that runs fossify and builds a FOSS image (wall-o/rocketchat:foss). This is a Phase 1 gate, not a Phase 0 requirement. Phase 0 uses the stock CE image for speed, explicitly dev-only and never resold. No customer deployment ships before the FOSS image build is in CI.

2.2 Server rebrand steps (concrete)

Because we control the FOSS source, branding is a code patch plus admin settings, not a fragile CSS-only overlay.

1. Fork Rocket.Chat and run ./fossify.sh; build wall-o/rocketchat:foss.
2. Patch the footer and login strings in the fork: replace "Powered by Rocket.Chat" and the Rocket.Chat wordmark references with the Wall-O mark, and default the site name.
3. Replace bundled logo and favicon assets with per-tenant assets served from /var/www/wall-o//branding/ (chmod 644), mounted read-only into the container.
4. In the workspace admin (Settings, Layout): set Site Name, Site URL, language, and default roles; set the custom color scheme via Custom CSS.
5. Disable telemetry, the workspace registration prompt, and the "Register" gate in the fork for self-hosted tenants.
6. Bake default colors, logo, and favicon into the image; override per tenant via mounted branding assets and admin settings.
7. Confirm no "Powered by Rocket.Chat" string or Rocket.Chat logo remains in the rendered web client (grep the built bundle and spot-check the login page and footer).

2.3 PWA vs native app, and Phase 2 native fork roadmap

Phase 1 decision: server-side rebrand plus a branded PWA, no native app yet. The Rocket.Chat web client is already responsive and PWA-capable; we wrap it behind the tenant subdomain with our branding. PWA first matches the standing ITPP preference (validate PWA before native).

Phase 2 native fork roadmap:

1. Fork Rocket.Chat.ReactNative (MIT core), strip the app/ee/ directory (EE code) the same way as the server.
2. Rebrand the app (name, icon, splash, bundle id) and point the default server URL at the tenant subdomain.
3. Configure push: the app registers with a push gateway. Self-host the Rocket.Chat push gateway (open source) so push is white-label too, rather than routing through Rocket.Chat's cloud gateway.
4. Sign and publish: Google Play via the ITPP developer account; Apple App Store via an Apple Developer Program account (bundle id, provisioning profiles, certificates).
5. Per-tenant onboarding: the app resolves the tenant server from a single discovery host, so one app binary serves all tenants (tenant_id entered at first launch or chosen via a directory).

3. Phase 0 Build Plan

Goal: prove the core loop end to end: Rocket.Chat up, bot registered, channel created, agent attached, document uploaded, indexed, and mobile plus push verified.

Phase 0 uses the stock CE image for speed (dev-only). The FOSS image build from section 2.1 is a Phase 1 gate before any customer.

• 1. Provision host and install Docker. apt-get update && apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
• 2. Create service account and directories. useradd -r -s /usr/sbin/nologin wallo && mkdir -p /opt/wall-o/tenants/wall-orthodontics /var/www/wall-o/wall-orthodontics/branding
• 3. Write tenant compose and mongod.conf (section 1.3). Populate .env from Vaultwarden. No secrets inline.
• 4. Start the stack. docker compose -f /opt/wall-o/tenants/wall-orthodontics/docker-compose.yml up -d
• 5. Initialize the Mongo replica set (oplog). docker compose -f /opt/wall-o/tenants/wall-orthodontics/docker-compose.yml exec mongo mongosh -eval "rs.initiate()"
• 6. Verify workspace is reachable. curl -sI https://wall-orthodontics.wall-o.itpropartner.com returns 200 after Caddy cert issuance.
• 7. Login as admin and capture admin identity. curl -s https://wall-orthodontics.wall-o.itpropartner.com/api/v1/login -H "Content-Type: application/json" -d '{"user":"admin","password":"<from Vaultwarden>"}'
• 8. Register the agent bot user. curl -s .../api/v1/users.create -H "X-Auth-Token: <adminToken>" -H "X-User-Id: <adminUserId>" -d '{"name":"Wall-O Agent","username":"wallo.agent","email":"wallo-agent@wall-orthodontics.internal","password":"<from Vaultwarden>","roles":["bot"],"joinDefaultChannels":false,"verified":true}'
• 9. Create a personal access token for the bot. curl -s .../api/v1/users.createToken -H "X-Auth-Token: <adminToken>" -H "X-User-Id: <adminUserId>" -d '{"userId":"<botUserId>"}'; store botUserId plus botAuthToken in Vaultwarden.
• 10. Create the knowledge channel. curl -s .../api/v1/channels.create -H "X-Auth-Token: <adminToken>" -H "X-User-Id: <adminUserId>" -d '{"name":"employee-resources"}'
• 11. Attach the agent to the channel. curl -s .../api/v1/channels.addOwner -H "X-Auth-Token: <adminToken>" -H "X-User-Id: <adminUserId>" -d '{"roomId":"<roomId>","userId":"<botUserId>"}'
• 12. Confirm the bot can post. curl -s .../api/v1/chat.postMessage -H "X-Auth-Token: <botAuthToken>" -H "X-User-Id: <botUserId>" -d '{"roomId":"<roomId>","text":"Wall-O agent online."}'
• 13. Stand up the orchestrator skeleton plus Postgres + pgvector. docker compose -f /opt/wall-o/orchestrator/docker-compose.yml up -d with image pgvector/pgvector:pg16, port 127.0.0.1:5432, credentials from .env (Vaultwarden).
• 14. Run migrations to create tenants, channels, agents, kb_scope, documents, chunks tables (section 4.2 columns).
• 15. Seed the tenants row with rocket_chat_url, rocket_chat_admin_user_id, and rocket_chat_admin_token_ref for Wall Orthodontics, and the agents row with the bot user id and bot token reference; leave all tokens in Vaultwarden only.
• 16. Upload a test document. curl -s -F "file=@staff-handbook.pdf" -F "tenant_id=<tenantId>" -F "channel_id=<channelId>" https://api.wall-o.itpropartner.com/v1/documents; orchestrator stores the file and records kb_scope.
• 17. Index the document. Orchestrator chunks, embeds via admin-ai, and inserts vectors into pgvector with tenant_id plus channel_id on every row.
• 18. Retrieval smoke test. curl -s "https://api.wall-o.itpropartner.com/v1/retrieve?tenant_id=<tenantId>&channel_id=<channelId>&q=<question>" returns a scoped chunk.
• 19. Close the chat loop. Orchestrator polls the channel via channels.messages?roomId=<roomId> and replies through chat.postMessage using the bot identity; verify a staff question gets a grounded answer.
• 20. Verify mobile. Open the tenant subdomain in a mobile browser (install the PWA) and confirm chat renders and the bot replies.
• 21. Verify push. Configure Push settings in the workspace (gateway URL), send a direct message to a test user on a mobile device, confirm the notification arrives on the PWA or the Rocket.Chat mobile app pointed at the server.

4. Security and Compliance

4.1 No PHI

Scope is strictly internal staff knowledge. Hard controls: the M365 connector is limited to explicitly approved, non-patient document libraries; a pre-ingest filter rejects documents matching PHI markers (SSN, MRN, DOB plus name); no HIPAA features are enabled in Rocket.Chat; every ingestion and retrieval path is audited. If a document is suspected to contain PHI, ingestion is blocked and logged, not silently passed.

4.2 Per-tenant data isolation

Layer	Isolation
Chat	One Rocket.Chat workspace per tenant, with its own MongoDB instance and volume. No shared chat database.
Documents and vectors	Every chunks row carries tenant_id and channel_id; every retrieval query is prefixed with a tenant_id plus channel_id filter (kb_scope). Optional later hardening: schema-per-tenant.
Credentials	Bot tokens are per tenant and scoped to that workspace only; stored in Vaultwarden, referenced by id.
Routing	Tenant subdomains are isolated Caddy site blocks with no cross-tenant access.

4.3 Data residency

All components are self-hosted on netcup infrastructure in the chosen region. Chat, documents, and vectors never leave the self-hosted estate. LLM and embedding calls route through admin-ai, the internal LiteLLM proxy (DeepSeek V4 Pro primary), so prompts are processed inside the controlled stack and never carry PHI by scope.

4.4 Backups

Nightly job in /opt/wall-o/backups/run-backups.sh: per-tenant mongodump (via the loopback port) plus orchestrator pg_dump, encrypted, uploaded to Wasabi S3. Retention per the ITPP backup schedule with a documented RPO. Backups are restorable per tenant, matching the isolation boundary.

4.5 Secret management

All secrets live in Vaultwarden: per-tenant admin and bot credentials, Postgres passwords, M365 app credentials, admin-ai (LiteLLM) keys, and S3 keys. Repos and compose files carry .env.example placeholders only. No plaintext secrets in code, compose, or Caddy config. Deploy scripts pull values from Vaultwarden into .env at provision time; .env is gitignored.

---

Appendix: Cross-team reconciliation

Three parallel teams wrote one part each against a shared canonical model. The parent integrated them and reconciled the following cross-team decisions. These resolutions are authoritative and supersede any conflicting value in the parts above.

#	Topic	Team A (Part 1)	Team B / Team C	Resolution
1	Vector namespace format	tn_<tenant_id>/ch_<channel_id>	tenant_{tenant_id}__scope_{kb_scope_id}	Use tenant_{tenant_id}__scope_{kb_scope_id}. It is consistent with the 1:1 kb_scope-to-namespace mapping and matches the retrieval and ACL tables (Part 2). Part 1 schema updated.
2	Chunk size / overlap	1000 / 200 chars	512 tokens (~2000 chars) / 64 tokens (~250 chars)	Use token-based 512 / 64 (Part 2, section 2.3). Part 1 kb_scopes columns changed to token semantics.
3	Retrieval top-k	LIMIT 8	20 candidates, rerank to 5	Use 20 then rerank to 5 (Part 2, sections 3.1 and 3.2). Part 1 query updated.
4	Embedding dimension	configurable (embedding_dim)	1024 (text-embedding-3-large Matryoshka)	Pin 1024 (Part 2, section 5.2). embedding_dim carries the platform-wide value.
5	Bot identity location	per-agent on agents	tenant-level rocket_bot_* on tenants	Per-agent bot identity (agents.rocket_chat_bot_username / _user_id / _token_ref). The tenants table holds only the admin identity for provisioning (rocket_chat_url / _admin_user_id / _admin_token_ref). Part 3 section 1.4 updated.
6	M365 token cache table	not modeled	tenant_credentials table	Add tenant_credentials as an additive table (Part 2, section 1.5), complementing the 8-table schema in Part 1.
7	Phase 0 chat loop	webhook + websocket sidecar (production)	polling via channels.messages	Phase 0 uses polling to prove the loop quickly; production uses webhook + websocket (Part 1, sections 4 and 5).
8	Example channel name	"Employee Resources" / "Billing"	clinical-knowledge (Phase 0 step 10)	Renamed to employee-resources. A clinical-named channel contradicts the no-PHI scope.

Open items to confirm at Phase 0 time (not blockers)

• Rocket.Chat image tag 7.4.0 is a target pin; confirm the exact tag exists in registry.rocket.chat before Phase 0 step 4. Remember the stock official image bundles EE code (dev-only); the FOSS build (wall-o/rocketchat:foss) is the Phase 1 gate before any customer deployment.
• The tenant admin identity requires both an admin user id and an admin token; confirm the admin _id is captured at workspace seed time (Phase 0 step 7) and stored alongside the token reference.