docs: add Wall-O system architecture (integrated and reconciled)
Consolidated three parallel team sections into wall-o.md; source sections under projects/wall-o/. Reconciled cross-team discrepancies: namespace format, chunk params, top-k, bot identity location, token-cache table, Phase 0 loop, channel naming. No plaintext secrets; Vaultwarden refs only.
This commit is contained in:
root
@@ -0,0 +1,426 @@
|
||||
# Wall-O: Product Model and Data Architecture
|
||||
|
||||
Status: PLANNED
|
||||
First client: Wall Orthodontics
|
||||
Scope: STRICTLY internal staff knowledge (employee handbook, policies, procedures, billing questions). NOT patient records, NOT PHI, no HIPAA scope.
|
||||
|
||||
## 1. Core Primitive
|
||||
|
||||
Every channel is exactly three things bound together:
|
||||
|
||||
1. One knowledge domain (for example "Employee Resources", "Billing", "IT Help").
|
||||
2. One attached AI agent (a domain-tuned persona).
|
||||
3. One scoped knowledge source (a set of documents/indices that maps to one vector namespace).
|
||||
|
||||
This is a hard invariant. A channel has exactly one agent and exactly one knowledge scope. A channel cannot span two domains, and an agent cannot serve two channels. If a practice needs a second knowledge domain, it creates a second channel, a second agent, and a second scope.
|
||||
|
||||
## 2. Canonical Architecture Split
|
||||
|
||||
Do not redesign this split. It is the foundation of the tenancy model.
|
||||
|
||||
| Layer | Responsibility | Owns |
|
||||
|---|---|---|
|
||||
| Rocket.Chat | Chat transport only | One workspace per tenant (MIT core, EE stripped). Rooms, users, messages, DMs. Zero intelligence. |
|
||||
| Orchestrator | All intelligence | Multi-tenant Python FastAPI service. Tenancy, agents, kb_scope, M365 connector, retrieval, LLM, posting. |
|
||||
| Postgres + pgvector | State and vectors | Tenants, channels, agents, scopes, documents, chunks, messages. |
|
||||
| admin-ai | LLM | DeepSeek V4 Pro primary, configured fallback chain. |
|
||||
| Wasabi S3 | Object storage | M365 sync staging, backups, agent assets (avatars), audit exports. |
|
||||
|
||||
Deployment: Rocket.Chat in Docker on netcup Core/app servers; orchestrator is FastAPI behind Caddy; Postgres + pgvector on the app data tier; M365 connector runs as a worker inside the orchestrator.
|
||||
|
||||
### Entity relationship summary
|
||||
|
||||
```
|
||||
tenants 1:N channels 1:1 agents 1:1 kb_scopes
|
||||
| |
|
||||
| + 1:N documents 1:N chunks
|
||||
|
|
||||
+ 1:N messages (per channel)
|
||||
tenants 1:N users
|
||||
```
|
||||
|
||||
## 3. Relational Data Model
|
||||
|
||||
All identifiers are UUIDv4. Every table that holds tenant data carries `tenant_id` and is filtered by it on every query. See section 9 for isolation.
|
||||
|
||||
### 3.1 tenants
|
||||
|
||||
One row per practice. Root of the tenancy tree.
|
||||
|
||||
| Column | Type | Constraints | Notes |
|
||||
|---|---|---|---|
|
||||
| tenant_id | UUID | PK | Canonical identifier. |
|
||||
| slug | TEXT | UNIQUE, NOT NULL | URL-safe slug, used in the webhook URL path. |
|
||||
| name | TEXT | NOT NULL | Practice display name. |
|
||||
| logo_url | TEXT | NULL | Branding asset, stored on Wasabi S3. |
|
||||
| accent_color | TEXT | NULL | Branding hex color. |
|
||||
| m365_tenant_id | TEXT | NULL | Microsoft Entra directory (tenant) id. |
|
||||
| m365_client_id | TEXT | NULL | Entra app registration client id. |
|
||||
| m365_credential_ref | TEXT | NULL | Vaultwarden secret reference. Never inline the client secret. |
|
||||
| rocket_chat_url | TEXT | NULL | Workspace root URL for this tenant. |
|
||||
| rocket_chat_admin_token_ref | TEXT | NULL | Vaultwarden reference for the admin REST token used to provision bots/rooms. |
|
||||
| webhook_secret_ref | TEXT | NULL | Vaultwarden reference for the HMAC secret that signs webhook POSTs. |
|
||||
| status | TEXT | NOT NULL DEFAULT 'provisioning' | provisioning, active, suspended. |
|
||||
| created_at | TIMESTAMPTZ | NOT NULL DEFAULT now() | |
|
||||
| updated_at | TIMESTAMPTZ | NOT NULL DEFAULT now() | |
|
||||
|
||||
Indexes: `UNIQUE (slug)`.
|
||||
|
||||
### 3.2 channels
|
||||
|
||||
One row per knowledge domain. Belongs to exactly one tenant.
|
||||
|
||||
| Column | Type | Constraints | Notes |
|
||||
|---|---|---|---|
|
||||
| channel_id | UUID | PK | Canonical identifier. |
|
||||
| tenant_id | UUID | FK to tenants.tenant_id, NOT NULL | Multi-tenant isolation. |
|
||||
| name | TEXT | NOT NULL | Knowledge domain name, for example "Billing". |
|
||||
| slug | TEXT | NOT NULL | URL-safe, unique per tenant. |
|
||||
| description | TEXT | NULL | What this channel answers. |
|
||||
| rocket_chat_room_id | TEXT | NULL | Rocket.Chat room/team id this channel maps to. |
|
||||
| rocket_chat_room_name | TEXT | NULL | Human-readable room name. |
|
||||
| agent_id | UUID | FK to agents.agent_id, UNIQUE, NULL | Exactly one agent per channel. Null until the agent is attached. |
|
||||
| kb_scope_id | UUID | FK to kb_scopes.kb_scope_id, UNIQUE, NULL | Exactly one scope per channel. Null until bound. |
|
||||
| status | TEXT | NOT NULL DEFAULT 'draft' | draft, active, archived. |
|
||||
| created_at | TIMESTAMPTZ | NOT NULL DEFAULT now() | |
|
||||
| updated_at | TIMESTAMPTZ | NOT NULL DEFAULT now() | |
|
||||
|
||||
Indexes: `UNIQUE (tenant_id, slug)`, `INDEX (tenant_id)`, `UNIQUE (agent_id)`, `UNIQUE (kb_scope_id)`, `INDEX (rocket_chat_room_id)`.
|
||||
|
||||
The dual `agent_id`/`kb_scope_id` unique columns enforce the one-to-one invariant from both directions.
|
||||
|
||||
### 3.3 agents
|
||||
|
||||
One row per AI persona. Bound to exactly one channel.
|
||||
|
||||
| Column | Type | Constraints | Notes |
|
||||
|---|---|---|---|
|
||||
| agent_id | UUID | PK | Canonical identifier. |
|
||||
| tenant_id | UUID | FK to tenants.tenant_id, NOT NULL | Multi-tenant isolation. |
|
||||
| channel_id | UUID | FK to channels.channel_id, UNIQUE, NOT NULL | One agent per channel. |
|
||||
| name | TEXT | NOT NULL | Agent display name, for example "Wallo Billing". |
|
||||
| system_prompt | TEXT | NOT NULL | Domain-tuned persona and instructions. |
|
||||
| kb_scope_id | UUID | FK to kb_scopes.kb_scope_id, NULL | What this agent may retrieve. |
|
||||
| model_provider | TEXT | NOT NULL DEFAULT 'admin-ai' | LLM gateway. |
|
||||
| model_name | TEXT | NOT NULL DEFAULT 'deepseek-v4-pro' | Primary model. |
|
||||
| model_params | JSONB | NOT NULL DEFAULT '{}' | temperature, max_tokens, top_p. |
|
||||
| fallback_model | TEXT | NULL | Next model in the failover chain. |
|
||||
| rocket_chat_bot_username | TEXT | UNIQUE, NOT NULL | Bot username in Rocket.Chat. |
|
||||
| rocket_chat_bot_user_id | TEXT | NULL | Rocket.Chat internal _id, filled after provisioning. |
|
||||
| rocket_chat_bot_token_ref | TEXT | NULL | Vaultwarden reference for the bot personal access token. |
|
||||
| status | TEXT | NOT NULL DEFAULT 'draft' | draft, provisioning, active, disabled. |
|
||||
| created_at | TIMESTAMPTZ | NOT NULL DEFAULT now() | |
|
||||
| updated_at | TIMESTAMPTZ | NOT NULL DEFAULT now() | |
|
||||
|
||||
Indexes: `UNIQUE (channel_id)`, `UNIQUE (rocket_chat_bot_username)`, `INDEX (tenant_id)`.
|
||||
|
||||
### 3.4 kb_scopes
|
||||
|
||||
One row per knowledge scope. Maps to exactly one vector namespace and one channel.
|
||||
|
||||
| Column | Type | Constraints | Notes |
|
||||
|---|---|---|---|
|
||||
| kb_scope_id | UUID | PK | |
|
||||
| tenant_id | UUID | FK to tenants.tenant_id, NOT NULL | Multi-tenant isolation. |
|
||||
| channel_id | UUID | FK to channels.channel_id, UNIQUE, NOT NULL | One scope per channel. |
|
||||
| vector_namespace | TEXT | UNIQUE, NOT NULL | Logical pgvector namespace, for example `tenant_{tenant_id}__scope_{kb_scope_id}` (see Part 2, section 5). |
|
||||
| document_libraries | JSONB | NOT NULL DEFAULT '[]' | List of M365 document libraries this scope may search. Entries carry site_id, drive_id, list_id, and display name. |
|
||||
| index_refs | JSONB | NOT NULL DEFAULT '[]' | Search index names the scope may query (Graph fallback). |
|
||||
| embedding_model | TEXT | NOT NULL | Model used to embed chunks and queries. |
|
||||
| embedding_dim | INT | NOT NULL | Vector dimension. Must match the platform-wide column dimension. |
|
||||
| chunk_size | INT | NOT NULL DEFAULT 512 | Tokens per chunk (approx 2000 chars). See Part 2, section 2.3. |
|
||||
| chunk_overlap | INT | NOT NULL DEFAULT 64 | Overlap tokens between chunks (approx 250 chars). See Part 2, section 2.3. |
|
||||
| sync_policy | JSONB | NULL | Delta sync schedule and file-type filters. |
|
||||
| last_synced_at | TIMESTAMPTZ | NULL | Last successful M365 sync. |
|
||||
| created_at | TIMESTAMPTZ | NOT NULL DEFAULT now() | |
|
||||
| updated_at | TIMESTAMPTZ | NOT NULL DEFAULT now() | |
|
||||
|
||||
Indexes: `UNIQUE (channel_id)`, `UNIQUE (vector_namespace)`, `INDEX (tenant_id)`.
|
||||
|
||||
Constraint: pgvector stores a fixed dimension per column. The platform therefore standardizes on one embedding model and dimension across all scopes so `chunks.embedding` is a single `vector(N)` column. Changing the model requires a full re-embed and reindex.
|
||||
|
||||
### 3.5 documents
|
||||
|
||||
One row per synced source document. Always scoped to a tenant and a channel.
|
||||
|
||||
| Column | Type | Constraints | Notes |
|
||||
|---|---|---|---|
|
||||
| document_id | UUID | PK | |
|
||||
| tenant_id | UUID | FK to tenants.tenant_id, NOT NULL | Multi-tenant isolation. |
|
||||
| channel_id | UUID | FK to channels.channel_id, NOT NULL | |
|
||||
| kb_scope_id | UUID | FK to kb_scopes.kb_scope_id, NOT NULL | |
|
||||
| source_type | TEXT | NOT NULL | sharepoint, onedrive, manual_upload. |
|
||||
| m365_drive_id | TEXT | NULL | For Graph delta sync. |
|
||||
| m365_item_id | TEXT | NULL | For Graph delta sync. |
|
||||
| source_path | TEXT | NULL | Full source path, for example the SharePoint URL. |
|
||||
| title | TEXT | NULL | |
|
||||
| file_name | TEXT | NULL | |
|
||||
| mime_type | TEXT | NULL | |
|
||||
| content_hash | TEXT | NOT NULL | SHA256 of raw content, for change detection. |
|
||||
| size_bytes | BIGINT | NULL | |
|
||||
| metadata | JSONB | NOT NULL DEFAULT '{}' | author, last modified time, page count. |
|
||||
| status | TEXT | NOT NULL DEFAULT 'pending' | pending, extracting, indexing, indexed, failed, deleted. |
|
||||
| last_synced_at | TIMESTAMPTZ | NULL | |
|
||||
| indexed_at | TIMESTAMPTZ | NULL | |
|
||||
| error | TEXT | NULL | Last sync or index error. |
|
||||
| created_at | TIMESTAMPTZ | NOT NULL DEFAULT now() | |
|
||||
| updated_at | TIMESTAMPTZ | NOT NULL DEFAULT now() | |
|
||||
|
||||
Indexes: `INDEX (tenant_id, channel_id)`, `INDEX (kb_scope_id)`, `INDEX (status)`, partial `UNIQUE (tenant_id, m365_drive_id, m365_item_id) WHERE m365_item_id IS NOT NULL`.
|
||||
|
||||
### 3.6 chunks
|
||||
|
||||
One row per embedded chunk. Carries the vector and the denormalized tenant/channel/scope keys for fast, isolated retrieval.
|
||||
|
||||
| Column | Type | Constraints | Notes |
|
||||
|---|---|---|---|
|
||||
| chunk_id | UUID | PK | |
|
||||
| document_id | UUID | FK to documents.document_id ON DELETE CASCADE, NOT NULL | |
|
||||
| tenant_id | UUID | FK to tenants.tenant_id, NOT NULL | Denormalized for retrieval filtering. |
|
||||
| channel_id | UUID | FK to channels.channel_id, NOT NULL | |
|
||||
| kb_scope_id | UUID | FK to kb_scopes.kb_scope_id, NOT NULL | |
|
||||
| chunk_index | INT | NOT NULL | Position within the document. |
|
||||
| content | TEXT | NOT NULL | The chunk text. |
|
||||
| token_count | INT | NOT NULL | |
|
||||
| embedding | vector(N) | NOT NULL | pgvector column. N is the platform-wide dimension. |
|
||||
| metadata | JSONB | NOT NULL DEFAULT '{}' | page number, section heading, anchor text. |
|
||||
|
||||
Indexes: `INDEX (document_id)`, `INDEX (tenant_id, kb_scope_id)`, and a vector index:
|
||||
|
||||
```
|
||||
CREATE INDEX chunks_embedding_idx ON chunks
|
||||
USING hnsw (embedding vector_cosine_ops)
|
||||
WITH (m = 16, ef_construction = 64);
|
||||
```
|
||||
|
||||
For large tenants, partition `chunks` by `tenant_id` so each partition is its own physical namespace and the HNSW index is built per partition.
|
||||
|
||||
### 3.7 messages
|
||||
|
||||
One row per inbound question and per outbound answer. Doubles as the audit log and the idempotency ledger.
|
||||
|
||||
| Column | Type | Constraints | Notes |
|
||||
|---|---|---|---|
|
||||
| message_id | UUID | PK | |
|
||||
| tenant_id | UUID | FK to tenants.tenant_id, NOT NULL | Multi-tenant isolation. |
|
||||
| channel_id | UUID | FK to channels.channel_id, NOT NULL | |
|
||||
| agent_id | UUID | FK to agents.agent_id, NULL | Null for inbound user messages. |
|
||||
| rocket_chat_message_id | TEXT | UNIQUE, NOT NULL | Rocket.Chat message _id, used as the idempotency key. |
|
||||
| rocket_chat_room_id | TEXT | NOT NULL | |
|
||||
| rocket_chat_user_id | TEXT | NULL | Author _id on Rocket.Chat. |
|
||||
| direction | TEXT | NOT NULL | inbound, outbound. |
|
||||
| role | TEXT | NOT NULL | user, assistant. |
|
||||
| content | TEXT | NOT NULL | |
|
||||
| citations | JSONB | NULL | Provenance array, see section 7. |
|
||||
| prompt_tokens | INT | NULL | |
|
||||
| completion_tokens | INT | NULL | |
|
||||
| total_tokens | INT | NULL | |
|
||||
| model_name | TEXT | NULL | Model that produced the answer. |
|
||||
| latency_ms | INT | NULL | End to end latency. |
|
||||
| status | TEXT | NOT NULL | received, resolving, retrieving, prompting, posted, failed. |
|
||||
| error | TEXT | NULL | Failure detail. |
|
||||
| created_at | TIMESTAMPTZ | NOT NULL DEFAULT now() | |
|
||||
|
||||
Indexes: `UNIQUE (rocket_chat_message_id)`, `INDEX (tenant_id, channel_id, created_at)`, `INDEX (created_at)`.
|
||||
|
||||
### 3.8 users
|
||||
|
||||
One row per human staff member. Mirrors the Rocket.Chat user for identity and role mapping.
|
||||
|
||||
| Column | Type | Constraints | Notes |
|
||||
|---|---|---|---|
|
||||
| user_id | UUID | PK | |
|
||||
| tenant_id | UUID | FK to tenants.tenant_id, NOT NULL | Multi-tenant isolation. |
|
||||
| rocket_chat_user_id | TEXT | NOT NULL | Rocket.Chat user _id. |
|
||||
| email | TEXT | NULL | |
|
||||
| display_name | TEXT | NULL | |
|
||||
| role | TEXT | NOT NULL DEFAULT 'staff' | admin, staff. Controls orchestrator admin surface only. |
|
||||
| last_seen_at | TIMESTAMPTZ | NULL | |
|
||||
| created_at | TIMESTAMPTZ | NOT NULL DEFAULT now() | |
|
||||
| updated_at | TIMESTAMPTZ | NOT NULL DEFAULT now() | |
|
||||
|
||||
Indexes: `UNIQUE (tenant_id, rocket_chat_user_id)`, `INDEX (tenant_id)`.
|
||||
|
||||
## 4. Expanded Message Flow
|
||||
|
||||
The canonical 6-step flow, with exact mechanics.
|
||||
|
||||
### Step 1: User posts a question
|
||||
|
||||
A staff member either @mentions the agent bot in a channel, or DMs the bot directly (section 8).
|
||||
|
||||
### Step 2: Rocket.Chat fires the webhook
|
||||
|
||||
Channel mentions are captured by an outgoing webhook integration configured on the channel with trigger word equal to the bot username. Rocket.Chat POSTs to:
|
||||
|
||||
```
|
||||
POST https://orchestrator.wall-o.<domain>/api/v1/webhook/rocketchat/{tenant_slug}
|
||||
```
|
||||
|
||||
Headers:
|
||||
|
||||
| Header | Value |
|
||||
|---|---|
|
||||
| X-WallO-Signature | Hex HMAC-SHA256 of the raw request body, keyed by the tenant webhook secret. |
|
||||
| X-WallO-Timestamp | Unix seconds. Reject if skew is greater than 300 seconds. |
|
||||
| Content-Type | application/json |
|
||||
|
||||
Payload (Rocket.Chat outgoing webhook shape):
|
||||
|
||||
```json
|
||||
{
|
||||
"token": "<outgoing webhook token>",
|
||||
"channel_id": "<rocketchat room id>",
|
||||
"channel_name": "employee-resources",
|
||||
"timestamp": "2026-08-15T12:00:00.000Z",
|
||||
"user_id": "<rocketchat user id>",
|
||||
"user_name": "jane.doe",
|
||||
"text": "@wallo-billing How do I submit a PTO request?",
|
||||
"trigger_word": "@wallo-billing",
|
||||
"bot": false
|
||||
}
|
||||
```
|
||||
|
||||
The orchestrator strips the trigger word and mention prefix from `text` before treating the remainder as the question.
|
||||
|
||||
DM capture does not use this webhook. Rocket.Chat outgoing webhooks do not fire inside DMs, so the orchestrator's bot-listener sidecar (section 5) receives DM events over the Realtime API and normalizes them into the identical inbound message object.
|
||||
|
||||
### Step 3: Orchestrator resolves tenant, channel, agent, scope
|
||||
|
||||
Resolution order, all in one request context:
|
||||
|
||||
1. Authenticate: verify `X-WallO-Signature` against the tenant webhook secret looked up by `tenant_slug`.
|
||||
2. Idempotency: derive `rocket_chat_message_id` from a hash of (channel_id, user_id, timestamp, text). If a `messages` row already exists with that id, return HTTP 200 immediately and do nothing. This deduplicates webhook retries.
|
||||
3. Resolve channel: `SELECT * FROM channels WHERE tenant_id = ? AND rocket_chat_room_id = ?`, with a fallback match on `rocket_chat_room_name`/`slug`.
|
||||
4. Resolve agent: read `channel.agent_id`, then `SELECT * FROM agents WHERE agent_id = ?`.
|
||||
5. Resolve scope: read `agent.kb_scope_id`, then `SELECT vector_namespace, embedding_model FROM kb_scopes WHERE kb_scope_id = ?`.
|
||||
6. If the channel has no agent or no scope, reply with a configuration-error message (or a silent no-op, configurable per tenant) and stop.
|
||||
|
||||
### Step 4: Retrieval
|
||||
|
||||
Primary path: pgvector semantic search over the channel's vector namespace.
|
||||
|
||||
```
|
||||
SELECT chunk_id, document_id, content, metadata, 1 - (embedding <=> $query_vec) AS score
|
||||
FROM chunks
|
||||
WHERE tenant_id = $tenant_id AND kb_scope_id = $kb_scope_id
|
||||
ORDER BY embedding <=> $query_vec
|
||||
LIMIT 20; -- candidate set; rerank to 5 per Part 2, section 3.2
|
||||
```
|
||||
|
||||
The query is embedded with the scope's `embedding_model`. Both `tenant_id` and `kb_scope_id` filters are mandatory in the same query, which is what makes the namespace isolated.
|
||||
|
||||
Fallback path: if the top score is below a configured threshold (for example 0.70) or the query returns zero rows, the orchestrator calls Microsoft Graph `/search/query` over the scope's `document_libraries`, with the search entity type set to `driveItem` and the site/drive scoped to the scope's libraries. Results are optionally re-ranked before prompt assembly.
|
||||
|
||||
### Step 5: Prompt build and LLM call
|
||||
|
||||
The prompt is assembled from four parts:
|
||||
|
||||
1. `agents.system_prompt` (the domain-tuned persona).
|
||||
2. A retrieved-context block where each chunk is labeled with a source marker `[1]`, `[2]`, and so on, carrying its document title and section heading.
|
||||
3. A citation instruction: answer using only the provided context, cite sources inline as `[n]`, and if the context does not contain an answer, say so instead of guessing.
|
||||
4. The user question.
|
||||
|
||||
The call goes to admin-ai with `model_name` (DeepSeek V4 Pro primary) and `model_params`. A per-call timeout (for example 30 seconds) is enforced. On timeout or model error, the orchestrator retries once with `fallback_model`, then returns a canned "I could not reach the model" reply. It never fabricates an answer.
|
||||
|
||||
### Step 6: Post the answer back to Rocket.Chat
|
||||
|
||||
```
|
||||
POST https://<rocket_chat_url>/api/v1/chat.postMessage
|
||||
Headers: X-Auth-Token: <bot token>, X-User-Id: <bot user id>
|
||||
Body: { "roomId": "<room id>", "text": "<answer with [n] citations + sources footer>" }
|
||||
```
|
||||
|
||||
The answer is posted as the agent bot (section 7). On post failure the orchestrator retries with exponential backoff (up to 3 attempts), then marks the message `failed` and records the error.
|
||||
|
||||
### Error and timeout handling
|
||||
|
||||
- The webhook acknowledges immediately (HTTP 200/202) after persisting the inbound message and enqueuing async processing, so the Rocket.Chat webhook never blocks or times out on the LLM call. The answer is posted out of band via REST.
|
||||
- Retrieval empty and below threshold: reply "I could not find an answer in the knowledge base for this question" with no sources. Never hallucinate.
|
||||
- LLM timeout/error: failover model, then a canned error reply.
|
||||
- Rocket.Chat post failure: retry with backoff, then mark failed and surface to a tenant alert.
|
||||
- Poison messages go to a dead-letter status with the error retained; alerting fires on elevated failure rate.
|
||||
|
||||
## 5. Rocket.Chat Bot Integration
|
||||
|
||||
### Recommended mechanism
|
||||
|
||||
Primary path: (a) bot user type + REST API, with the outgoing webhook (b) as the canonical channel-mention trigger and a websocket listener for DMs. The Apps Engine (c) is not used.
|
||||
|
||||
Justification: the bot user gives each agent a stable, named identity (username, avatar, alias) inside Rocket.Chat and the REST API is versioned, stateless, and fully scriptable from the orchestrator, while the outgoing webhook delivers the canonical inbound POST with no code running inside the chat layer. The Apps Engine is rejected because it would place logic inside the transport layer, violating the architecture split, and would require maintaining a JS app per tenant. Incoming webhooks alone (b by itself) cannot provide a dynamic per-agent identity, so they are used only as an optional posting convenience, not as the identity layer.
|
||||
|
||||
### Register an agent bot
|
||||
|
||||
For each agent, the orchestrator (using the tenant admin token):
|
||||
|
||||
1. Create the bot user:
|
||||
`POST /api/v1/users.create` with:
|
||||
```json
|
||||
{
|
||||
"name": "<agent name>",
|
||||
"username": "<rocket_chat_bot_username>",
|
||||
"email": "<username>@<tenant-domain>",
|
||||
"password": "<random 32-char>",
|
||||
"roles": ["bot"],
|
||||
"joinDefaultChannels": false,
|
||||
"requirePasswordChange": false,
|
||||
"sendWelcomeEmail": false,
|
||||
"verified": true
|
||||
}
|
||||
```
|
||||
2. Create a personal access token:
|
||||
`POST /api/v1/users.createToken` with `{ "userId": "<bot _id>" }`. The returned `authToken` and `userId` are stored in Vaultwarden; the reference goes in `agents.rocket_chat_bot_token_ref` and the _id in `agents.rocket_chat_bot_user_id`.
|
||||
3. Optionally set the avatar:
|
||||
`POST /api/v1/users.setAvatar` (uploaded image) or `users.setAvatarFromUrl`.
|
||||
|
||||
### Join a channel
|
||||
|
||||
The orchestrator (via the tenant admin token) invites the bot to the channel's room:
|
||||
|
||||
`POST /api/v1/channels.invite` with `{ "roomId": "<rocket_chat_room_id>", "userId": "<bot _id>" }`.
|
||||
|
||||
Bots auto-accept invitations. After this the bot is a room member and can be @mentioned and post as itself.
|
||||
|
||||
### Route @mentions and commands
|
||||
|
||||
- @mention/trigger word: configure an outgoing webhook integration on the channel with trigger word set to the bot username. A user mentioning the bot (or typing the trigger word) causes Rocket.Chat to POST to the orchestrator webhook URL (section 4, step 2). The orchestrator strips the mention/trigger prefix.
|
||||
- Slash command (optional): `POST /api/v1/commands.create` to register a command such as `/ask` in the channel that routes to the same webhook. Useful when a channel hosts more than one purpose and explicit scoping is wanted.
|
||||
|
||||
### Route DMs
|
||||
|
||||
Because outgoing webhooks do not fire in DMs, the orchestrator runs a bot-listener sidecar that maintains a websocket (Realtime API) session per bot: connect, call `method: "login"` with the bot token, then subscribe to `stream-room-messages` for the bot's own user id. Both DMs and room mentions arrive as the same message event type. The sidecar normalizes them into the identical inbound message object as the webhook path and enqueues them through the same pipeline. The sidecar is a pure transport shim; it holds no intelligence.
|
||||
|
||||
## 6. Channel Lifecycle
|
||||
|
||||
| Step | Action | Performed by |
|
||||
|---|---|---|
|
||||
| 1. Create channel | Insert a `channels` row (tenant, name, slug, description, status = draft). Optionally create the Rocket.Chat room via `POST /api/v1/channels.create` and store `rocket_chat_room_id`. | Orchestrator API, tenant admin role |
|
||||
| 2. Attach agent | Insert an `agents` row (system_prompt, model config, bot username) with `channel_id` set. Provision the Rocket.Chat bot user and token (section 5). Invite the bot to the room. Set `channels.agent_id`. | Orchestrator API + Rocket.Chat REST |
|
||||
| 3. Bind kb_scope | Insert a `kb_scopes` row (vector_namespace, document_libraries, embedding model/dim, chunk params). Set `channels.kb_scope_id`. | Orchestrator API |
|
||||
| 4. Initial sync and index | The M365 connector pulls the named document libraries, extracts text, chunks, embeds, and writes `documents` + `chunks` rows under the namespace. | M365 connector worker |
|
||||
| 5. Go live | Set `channels.status = active`. Configure the outgoing webhook and optional slash command. Run an end-to-end smoke test question and confirm a cited answer posts back as the bot. | Orchestrator API + operator |
|
||||
|
||||
Steps 1 through 3 are idempotent API calls driven by an onboarding form in the orchestrator admin surface. Step 4 is the only long-running step; the channel stays in `draft` until `indexed` document count is nonzero. Step 5 flips status and does not require a redeploy.
|
||||
|
||||
## 7. Agent Identity and Response Citations
|
||||
|
||||
Identity: the bot posts with its own username and avatar (set at provisioning). Every answer is attributed to the bot user, never to a human, so staff can distinguish agent answers from colleague messages. The `system_prompt` instructs the agent to introduce itself as the channel's assistant (for example "I am the Wall-O assistant for Employee Resources") and to stay inside the domain.
|
||||
|
||||
Citations: retrieved chunks carry provenance in `metadata` (document title, section heading, page). During prompt assembly each chunk is labeled `[1]`, `[2]`, and so on. The orchestrator renders the final answer with inline `[n]` markers and appends a "Sources" footer listing each cited document title and a link (Microsoft Graph sharing link, or a Rocket.Chat file link). The full provenance (document_id, chunk_id, title, snippet, url) is stored in `messages.citations` as JSONB for audit and re-render. If the answer cites nothing, no footer is emitted.
|
||||
|
||||
## 8. User Interaction Model
|
||||
|
||||
| Mode | When to use | Trigger path |
|
||||
|---|---|---|
|
||||
| Channel @mention | Shared or discoverable questions, team-wide answers, anything others should see. | Outgoing webhook (section 4, step 2). |
|
||||
| DM the bot | Private follow-up, iterative clarification, personal-but-not-PHI questions, or a 1:1 thread the user does not want in the channel. | Bot-listener websocket (section 5). |
|
||||
|
||||
Guidance surfaced to staff: use the channel for Q&A that benefits the team (answers stay searchable in the room), and DM the bot for private or back-and-forth questions. Both paths produce the same cited answer format; only the destination differs.
|
||||
|
||||
## 9. Multi-tenant Isolation and Security
|
||||
|
||||
- Tenant_id on every table: tenants, channels, agents, kb_scopes, documents, chunks, messages, and users all carry `tenant_id`. The application derives the tenant from the webhook path and HMAC signature, never from a client-supplied body field alone, and scopes every query with it.
|
||||
- Postgres row-level security: enable RLS on all tenant tables with a policy `tenant_id = current_setting('app.tenant_id')`, set per request, as defense in depth under the application-level filter.
|
||||
- Vector namespace isolation: retrieval always filters `tenant_id` AND `kb_scope_id` in the same query, so a chunk can never leak across channels or tenants. Partitioning by `tenant_id` makes each tenant's vectors physically separate.
|
||||
- Secrets: M365 credentials, bot tokens, and webhook secrets live in Vaultwarden. The database stores references only. Never commit secrets; `.env.example` placeholders only.
|
||||
- Scope guard: the product never ingests patient records or PHI. The M365 connector's document_libraries are allowlisted per scope, and a content filter flags documents outside the allowlisted libraries before indexing.
|
||||
@@ -0,0 +1,453 @@
|
||||
# Wall-O: M365 Connector + Retrieval Architecture
|
||||
|
||||
Status: DESIGN (internal technical architecture)
|
||||
Owner: Wall-O build team
|
||||
Scope: Internal staff knowledge only. Employee handbook, policies, procedures, billing questions.
|
||||
Excluded: Patient records, PHI, any HIPAA-regulated data. This connector MUST NOT be pointed at clinical or patient data sources.
|
||||
|
||||
## Canonical entities used in this document
|
||||
|
||||
| Entity | Definition | Cardinality |
|
||||
|---|---|---|
|
||||
| tenant_id | A practice (e.g. Wall Orthodontics). The Microsoft 365 tenant boundary. | 1 M365 tenant per tenant_id |
|
||||
| channel_id | A knowledge domain within a tenant (e.g. "Employee Resources"). | many per tenant |
|
||||
| agent_id | An AI persona bound to a channel. One agent serves one channel. | 1:1 with channel_id |
|
||||
| kb_scope | The set of documents/indices an agent may search. Scoped per tenant AND per channel. Maps 1:1 to a vector namespace. | 1:1 with vector namespace |
|
||||
| documents/chunks | Files synced from M365, chunked and indexed into the tenant+channel vector namespace. | many per kb_scope |
|
||||
|
||||
The vector namespace is a logical concept. In pgvector it is implemented as a `namespace` column on the chunk table and filtered in the WHERE clause of every similarity query. In Qdrant it would be a native collection. The logical name format is fixed regardless of backend:
|
||||
|
||||
```
|
||||
tenant_{tenant_id}__scope_{kb_scope_id}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 1. Authentication model (Microsoft Graph, app-only)
|
||||
|
||||
### 1.1 Entra app registration
|
||||
|
||||
Single multi-tenant app registration in the IT Pro Partner (IPP) home tenant. Supported account type:
|
||||
|
||||
```
|
||||
Accounts in any organizational directory (Any Microsoft Entra ID tenant - Multitenant)
|
||||
```
|
||||
|
||||
This lets one app identity serve every practice tenant. Each practice admin consents the app into their own tenant; no app is registered inside the client's tenant.
|
||||
|
||||
### 1.2 Permission strategy: Sites.Selected over Sites.Read.All
|
||||
|
||||
Wall-O reads documents, never writes. Least privilege is achieved with the `Sites.Selected` application permission rather than the broad `Sites.Read.All`.
|
||||
|
||||
| Permission | Type | Scope | Why accepted / rejected |
|
||||
|---|---|---|---|
|
||||
| Sites.Selected | Application | Only sites explicitly granted via the site permissions API | ACCEPTED. Primary. Grants per-site read; the app sees nothing else in the tenant. |
|
||||
| Sites.Read.All | Application | Every site in the tenant | REJECTED for production. Violates least privilege; exposes all site collections including any we are not meant to index. |
|
||||
| Files.Read.All | Application | Every file in every drive | REJECTED. Superseded by Sites.Selected for site-scoped read. |
|
||||
| User.Read.All | Application | Read directory user profiles | OPTIONAL. Needed only to resolve author display names from OneDrive drive owner IDs. Not required for retrieval. |
|
||||
|
||||
`Sites.Selected` supports site-level roles: `read`, `write`, `fullcontrol`, `manage`. Wall-O requests `read` only. The grant is issued per site collection via:
|
||||
|
||||
```
|
||||
POST https://graph.microsoft.com/v1.0/sites/{site-id}/permissions
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"roles": ["read"],
|
||||
"grantedToIdentities": [
|
||||
{ "application": { "id": "{wall-o-app-client-id}", "displayName": "Wall-O" } }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
This request is made by the orchestrator using the app's own token for the tenant. Because the app already holds `Sites.Selected`, it can grant itself `read` on a specific site once a practice admin has approved the site in onboarding. A stricter variant has a practice global admin run the grant via Graph Explorer so the app never self-grants. Wall-O uses the admin-driven variant: the grant is issued during onboarding by the practice admin (or by the orchestrator on a one-time admin-approved site list), not by the app unprompted.
|
||||
|
||||
### 1.3 Client credentials grant
|
||||
|
||||
App-only OAuth 2.0 client credentials flow. No user, no interactive login, no refresh token.
|
||||
|
||||
Token request:
|
||||
|
||||
```
|
||||
POST https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token
|
||||
Content-Type: application/x-www-form-urlencoded
|
||||
|
||||
client_id={client_id}
|
||||
client_secret={client_secret}
|
||||
scope=https://graph.microsoft.com/.default
|
||||
grant_type=client_credentials
|
||||
```
|
||||
|
||||
Exact strings to configure:
|
||||
|
||||
| Field | Exact value |
|
||||
|---|---|
|
||||
| grant_type | `client_credentials` |
|
||||
| scope | `https://graph.microsoft.com/.default` |
|
||||
| Token endpoint | `https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token` |
|
||||
|
||||
Notes:
|
||||
|
||||
- `{tenant_id}` is the consuming practice's directory (tenant) ID, captured at consent time. It is NOT `common` or `organizations`: client credentials has no user to derive a tenant from, so the target tenant must be explicit.
|
||||
- `.default` expands to the union of the application permissions already consented for that tenant. It is not a literal scope name.
|
||||
- Response contains `access_token` and `expires_in`. Graph client credentials tokens are typically valid for about 60 minutes; treat `expires_in` as authoritative, never hardcode the TTL.
|
||||
- Credential material: prefer an X.509 client certificate (assertion) over a client secret. A secret is acceptable for MVP but must be rotated before its maximum lifetime (24 months). Secrets live in Vaultwarden under the Wall-O project; only the encrypted reference enters Postgres or config.
|
||||
|
||||
### 1.4 Admin consent URL construction
|
||||
|
||||
Each practice admin must consent the app into their tenant before the first crawl. The v2.0 admin consent endpoint is constructed as:
|
||||
|
||||
```
|
||||
https://login.microsoftonline.com/{tenant_id}/v2.0/adminconsent
|
||||
?client_id={wall-o-app-client-id}
|
||||
&scope=https://graph.microsoft.com/.default
|
||||
&redirect_uri={configured-redirect-uri}
|
||||
```
|
||||
|
||||
- `{tenant_id}`: the practice's directory ID, or `organizations` to let the signing admin's tenant be used automatically.
|
||||
- `redirect_uri`: a registered reply URL on the app. Wall-O registers a no-op callback (e.g. `https://wall-o.itpropartner.com/entra/callback`) that returns a 200 and logs the consent result.
|
||||
- The scope string is URL-encoded `.default` (i.e. `https%3A%2F%2Fgraph.microsoft.com%2F.default`).
|
||||
|
||||
Onboarding flow per practice:
|
||||
|
||||
1. Practice admin clicks the consent URL (delivered by the Wall-O onboarding UI or support).
|
||||
2. Admin authenticates and approves the `Sites.Selected` application permission.
|
||||
3. On success Entra redirects to the callback. The orchestrator records `tenant_id`, consent timestamp, and the admin's identity.
|
||||
4. The site-level `read` grant (1.2) is then applied to the specific site collections mapped to that practice's channels.
|
||||
5. Only now does the connector issue its first token and run a test crawl.
|
||||
|
||||
Consent revocation is detected at token time (see 1.5); the tenant is marked `consent_revoked` and the practice admin is prompted to re-consent.
|
||||
|
||||
### 1.5 Per-tenant token storage and refresh lifecycle
|
||||
|
||||
No refresh token exists in client credentials, so "refresh" means issuing a fresh token on demand. The cache is a memoization layer plus a lifecycle guard.
|
||||
|
||||
Storage (Postgres, `tenant_credentials` table):
|
||||
|
||||
| Column | Purpose |
|
||||
|---|---|
|
||||
| tenant_id | Primary cache key |
|
||||
| access_token | Encrypted at rest (AES-256-GCM, app-level, key in Vaultwarden) |
|
||||
| expires_at | Absolute expiry derived from `expires_in` at issue time |
|
||||
| last_error / error_code | Last failure for alerting (e.g. consent revoked, secret expired) |
|
||||
| status | `active`, `consent_revoked`, `secret_expired`, `suspended` |
|
||||
|
||||
Lifecycle rules:
|
||||
|
||||
- Token is considered usable while `now < expires_at - 300s` (5 minute safety margin). Outside that window the connector requests a fresh token before the next Graph call.
|
||||
- On `401 Unauthorized` with `InvalidAuthenticationToken`, or `403` with a consent/scope error, the connector retries once with a freshly issued token. A second failure escalates.
|
||||
- Error code mapping:
|
||||
- `AADSTS700016` (application not found in directory) or `AADSTS7000112` (invalid client) means the app is not consented in that tenant: mark `consent_revoked`, alert practice admin.
|
||||
- `AADSTS700082` or expired secret errors mean the secret is expired/rotated: mark `secret_expired`, page the Wall-O operator.
|
||||
- Graph throttling `429` is not an auth failure: honor `Retry-After` and back off; do not flag the tenant.
|
||||
- Token issuance and refresh are serialized per tenant (single-flight lock) so concurrent sync workers do not stampede the token endpoint.
|
||||
- Tokens are never logged or returned by any API. Logging redacts the `Authorization` header.
|
||||
|
||||
### 1.6 Publisher verification
|
||||
|
||||
Required so the multi-tenant consent prompt shows a verified publisher rather than "unverified", which materially reduces admin consent friction (and some tenants block unverified apps outright).
|
||||
|
||||
Requirements:
|
||||
|
||||
- A Microsoft AI Cloud Partner Program (MAICPP) account with an MPN ID for IT Pro Partner.
|
||||
- The MPN ID associated with the app registration under Branding and properties.
|
||||
- A verified custom domain (e.g. `itpropartner.com`) linked to the Entra tenant and set as the publisher domain.
|
||||
- App registration Branding shows "Publisher verified" with the blue checkmark.
|
||||
|
||||
Publisher verification is a one-time per-publisher step, not per-tenant. It is a prerequisite to onboarding the first external practice; without it, practice admins see an unverified-publisher warning on the consent screen.
|
||||
|
||||
---
|
||||
|
||||
## 2. Sync and index pipeline
|
||||
|
||||
### 2.1 Crawl sources
|
||||
|
||||
Two source types, both driven by Microsoft Graph:
|
||||
|
||||
| Source | Graph entry points |
|
||||
|---|---|
|
||||
| SharePoint document libraries | `GET /sites/{site-id}/drives` to enumerate libraries, then per-drive crawl |
|
||||
| OneDrive for Business | `GET /users/{user-id}/drive` (a practice user's personal drive mapped to a channel) |
|
||||
|
||||
Site resolution by human-friendly URL before crawling:
|
||||
|
||||
```
|
||||
GET https://graph.microsoft.com/v1.0/sites/{hostname}:/{site-path}
|
||||
GET https://graph.microsoft.com/v1.0/sites?search={query}
|
||||
```
|
||||
|
||||
Each crawl source is registered in Postgres as part of a `kb_scope`, so a scope maps to one or more (site, drive) pairs. The connector crawls exactly those drives, never the whole tenant.
|
||||
|
||||
### 2.2 Crawl, extract, chunk, embed, write
|
||||
|
||||
Pipeline stages, all inside the FastAPI orchestrator (worker tasks, not the request path):
|
||||
|
||||
1. Crawl: enumerate items per drive using `GET /sites/{site-id}/drive/root/children` and `GET /sites/{site-id}/drive/root:/{path}:/children` for nested folders. Use `$batch` (up to 20 requests per batch) to reduce round trips and honor throttling.
|
||||
2. Download: `GET /sites/{site-id}/drive/items/{item-id}/content` (or the item's `@microsoft.graph.downloadUrl`). Stream to disk, never into memory whole.
|
||||
3. Extract: text extraction per format (see section 4.1). Produce plain text plus a metadata block.
|
||||
4. Chunk: split with the strategy in 2.3.
|
||||
5. Embed: embed each chunk with the configured model (section 5.2).
|
||||
6. Write: upsert chunks into the `{tenant}_{kb_scope}` vector namespace in pgvector. One transaction per document: delete old chunks for that document_id, insert new, commit. Keeps the namespace consistent even if a sync crashes mid-document.
|
||||
|
||||
### 2.3 Chunking strategy
|
||||
|
||||
Concrete defaults:
|
||||
|
||||
| Parameter | Value |
|
||||
|---|---|
|
||||
| Target chunk size | 512 tokens (approx 2000 chars) |
|
||||
| Overlap | 64 tokens (approx 250 chars, 12.5%) |
|
||||
| Splitter | Sentence-aware: split on paragraph then sentence boundaries, never mid-word |
|
||||
| Hard max | 1024 tokens for a single chunk (tables, bullet lists, malformed PDF text) |
|
||||
| Min kept | Discard chunks under 20 tokens |
|
||||
| Tokenizer | The embedding model's tokenizer (cl100k_base if using OpenAI text-embedding-3) |
|
||||
|
||||
Per-chunk metadata written alongside the vector:
|
||||
|
||||
| Metadata field | Source |
|
||||
|---|---|
|
||||
| tenant_id | canonical entity |
|
||||
| channel_id | canonical entity |
|
||||
| kb_scope_id | canonical entity |
|
||||
| namespace | `tenant_{tenant_id}__scope_{kb_scope_id}` |
|
||||
| document_id | Graph driveItem `id` |
|
||||
| document_name | driveItem `name` |
|
||||
| site_id / drive_id / item_id | Graph IDs for provenance and re-download |
|
||||
| source_url | driveItem `webUrl` |
|
||||
| modified_at | driveItem `lastModifiedDateTime` |
|
||||
| chunk_index | position within document |
|
||||
| mime / file_type | for format-aware handling |
|
||||
| title / heading | nearest heading above the chunk, for context injection |
|
||||
|
||||
Metadata is stored in the same Postgres chunk row so filtering (by channel, by scope) is a plain SQL predicate, not a secondary lookup.
|
||||
|
||||
### 2.4 Incremental sync (delta query + change notifications)
|
||||
|
||||
Full re-crawl on every sync is avoided with two complementary mechanisms.
|
||||
|
||||
Delta query (poll-based, authoritative):
|
||||
|
||||
```
|
||||
GET https://graph.microsoft.com/v1.0/sites/{site-id}/drive/root/delta
|
||||
```
|
||||
|
||||
- First call with no token returns the full set and a `@odata.deltaLink` (and `@odata.nextLink` while paging).
|
||||
- Subsequent calls use the stored deltaLink and return only added, changed, and deleted items. Deleted items carry a `deleted` facet; the connector removes their chunks from the namespace.
|
||||
- Persist the deltaLink per (site, drive) in Postgres.
|
||||
- Delta tokens expire; a `410 Gone` (or malformed delta token) means the connector must drop to a full re-crawl for that drive. The connector treats 410 as a normal control-flow event, logs it, and re-syncs fully.
|
||||
|
||||
Change notifications (push-based, reduces lag):
|
||||
|
||||
```
|
||||
POST https://graph.microsoft.com/v1.0/subscriptions
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"changeType": "updated",
|
||||
"notificationUrl": "https://wall-o.itpropartner.com/entra/notifications",
|
||||
"resource": "/sites/{site-id}/drive/root",
|
||||
"expirationDateTime": "2026-08-18T18:00:00Z",
|
||||
"clientState": "tenant_{tenant_id}"
|
||||
}
|
||||
```
|
||||
|
||||
- `changeType` may be `created,updated,deleted` combined in one subscription.
|
||||
- `expirationDateTime` maximum is 4230 minutes (3 days). The orchestrator renews every subscription before expiry (cron, every 6 hours, `PATCH /subscriptions/{id}` to extend).
|
||||
- The notificationUrl must be HTTPS, publicly reachable, and answer the initial `validationToken` handshake: echo the token back as `text/plain` with HTTP 200 within 5 seconds, then process the notification asynchronously.
|
||||
- `clientState` is echoed back on every notification so the orchestrator verifies the sender and does not act on forged payloads.
|
||||
- On notification, the connector does NOT fetch blindly: it records a dirty (site, drive) and lets the next delta pass reconcile. This deduplicates burst notifications into one delta sweep.
|
||||
|
||||
Sync schedule (default): full crawl on onboarding and after a 410; delta sweep every 15 minutes; change notifications applied opportunistically between sweeps. Re-embed only changed/deleted chunks, not the whole namespace.
|
||||
|
||||
### 2.5 Sync observability
|
||||
|
||||
Per (tenant, scope) the orchestrator records: last_full_sync_at, last_delta_sync_at, items_seen, chunks_written, chunks_deleted, and last_sync_error. A dashboard flag is raised when `last_delta_sync_at` exceeds the 15 minute SLA by a factor of 3 (i.e. stale over 45 minutes), which also gates the stale-index retrieval fallback in section 3.3.
|
||||
|
||||
---
|
||||
|
||||
## 3. Retrieval
|
||||
|
||||
### 3.1 Semantic search (primary)
|
||||
|
||||
The agent's query is embedded with the same model and dimensionality used at index time. Retrieval runs against exactly one vector namespace: the agent's `kb_scope` namespace. Tenancy and channel scoping is therefore a hard SQL predicate, not an after-the-fact filter.
|
||||
|
||||
pgvector query shape:
|
||||
|
||||
```sql
|
||||
SELECT chunk_id, document_id, document_name, source_url, text, chunk_index,
|
||||
1 - (embedding <=> $1) AS similarity
|
||||
FROM chunks
|
||||
WHERE namespace = $2
|
||||
ORDER BY embedding <=> $1
|
||||
LIMIT $3;
|
||||
```
|
||||
|
||||
- `<=>` is cosine distance; `1 - (embedding <=> $1)` yields cosine similarity. Embeddings are stored L2-normalized so cosine and inner product are equivalent.
|
||||
- top-k for the candidate set is 20.
|
||||
|
||||
### 3.2 Rerank (optional second stage)
|
||||
|
||||
A cross-encoder reranks the 20 candidates against the raw query to 5 final passages. This is quality over recall: the cross-encoder sees query and passage jointly and scores true relevance, correcting embedding-only mistakes on synonyms and negations.
|
||||
|
||||
- Model: `BAAI/bge-reranker-large` (self-hosted on the app server) or a managed reranker via LiteLLM if one is added to admin-ai.
|
||||
- Rerank is a config toggle per agent. If disabled, the top 5 by cosine similarity are used directly.
|
||||
|
||||
### 3.3 Graph live search fallback (cold or stale index)
|
||||
|
||||
When the vector namespace is cold (fewer than N chunks, e.g. onboarding in progress) or stale (delta sync behind, see 2.5), retrieval falls back to live Graph search so answers are never silently empty. The Graph search API is used as the fallback, not the primary, because it is slower, rate-limited, and returns passages without the tight namespace scoping.
|
||||
|
||||
```
|
||||
POST https://graph.microsoft.com/v1.0/search/query
|
||||
Content-Type: application/json
|
||||
|
||||
{
|
||||
"requests": [
|
||||
{
|
||||
"entityTypes": ["driveItem"],
|
||||
"query": { "queryString": "{user query}" },
|
||||
"from": 0,
|
||||
"size": 10,
|
||||
"fields": ["id", "name", "webUrl", "lastModifiedDateTime"]
|
||||
}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
A simpler drive-scoped variant is available when the fallback is constrained to one site:
|
||||
|
||||
```
|
||||
GET https://graph.microsoft.com/v1.0/sites/{site-id}/drive/root/search(q='{user query}')
|
||||
```
|
||||
|
||||
The fallback returns file references (name, webUrl, snippet), which the orchestrator hands to the LLM as source citations rather than pre-chunked text.
|
||||
|
||||
### 3.4 Hybrid ranking approach
|
||||
|
||||
Two signals are fused when both are available:
|
||||
|
||||
| Signal | Source | Weight role |
|
||||
|---|---|---|
|
||||
| Semantic | embedding cosine similarity | primary |
|
||||
| Keyword | Postgres full-text search (tsvector) or pg_trgm over chunk text | secondary, for exact terms: policy IDs, benefit codes, acronyms |
|
||||
|
||||
Fusion uses Reciprocal Rank Fusion (RRF):
|
||||
|
||||
```
|
||||
score(d) = sum over signals of 1 / (k + rank_signal(d)), k = 60
|
||||
```
|
||||
|
||||
This preserves exact-term matches (e.g. a specific policy number) that pure embedding search can miss, without a separate vector backend. The keyword index is a generated `tsvector` column on the chunk table maintained by trigger or at write time.
|
||||
|
||||
### 3.5 Graph endpoints actually used (reference)
|
||||
|
||||
| Purpose | Endpoint |
|
||||
|---|---|
|
||||
| Token | `POST /oauth2/v2.0/token` (login.microsoftonline.com) |
|
||||
| Admin consent | `GET /{tenant}/v2.0/adminconsent` |
|
||||
| Site resolution | `GET /v1.0/sites/{hostname}:/{path}`, `GET /v1.0/sites?search=` |
|
||||
| Site read grant (Sites.Selected) | `POST /v1.0/sites/{site-id}/permissions` |
|
||||
| List drives in a site | `GET /v1.0/sites/{site-id}/drives` |
|
||||
| List folder children | `GET /v1.0/sites/{site-id}/drive/root/children` |
|
||||
| Path-based children | `GET /v1.0/sites/{site-id}/drive/root:/{path}:/children` |
|
||||
| Download content | `GET /v1.0/sites/{site-id}/drive/items/{item-id}/content` |
|
||||
| Delta sync | `GET /v1.0/sites/{site-id}/drive/root/delta` |
|
||||
| OneDrive (user) | `GET /v1.0/users/{user-id}/drive/root/delta` |
|
||||
| Live search (fallback) | `POST /v1.0/search/query` |
|
||||
| Drive-scoped search (fallback) | `GET /v1.0/sites/{site-id}/drive/root/search(q='...')` |
|
||||
| Change notifications | `POST /v1.0/subscriptions`, `PATCH /v1.0/subscriptions/{id}` |
|
||||
| Author resolution (optional) | `GET /v1.0/users/{user-id}` |
|
||||
|
||||
---
|
||||
|
||||
## 4. Document format support, size limits, ACL mapping
|
||||
|
||||
### 4.1 Format support
|
||||
|
||||
| Format | Extraction approach |
|
||||
|---|---|
|
||||
| PDF | PyMuPDF (fitz), per-page text extraction; flagged as text-only (embedded OCR is out of scope) |
|
||||
| DOCX | python-docx: paragraphs + tables |
|
||||
| XLSX | openpyxl: sheet-by-sheet, cell grid serialized with sheet name as heading context |
|
||||
| PPTX | python-pptx: slide title + body text per slide |
|
||||
| TXT / MD / CSV | plain read with encoding detection |
|
||||
| HTML (optional) | trafilatura / BeautifulSoup text extraction |
|
||||
|
||||
The connector downloads and parses locally. Graph does not return extracted text, only file content and metadata. Binary formats not in this table are skipped and counted in sync observability, never force-indexed.
|
||||
|
||||
### 4.2 File size limits
|
||||
|
||||
| Limit | Value | Rationale |
|
||||
|---|---|---|
|
||||
| Max file for extraction | 50 MB | Above this, office files and PDFs blow up parse time and memory; skipped and flagged |
|
||||
| Max chunk text written per file | 10 MB of extracted text | Mirrors SharePoint search's practical text-index ceiling; larger files are truncated with a notice chunk |
|
||||
| Graph simple download ceiling | approx 250 MB via `/content` | Beyond this use the item's `@microsoft.graph.downloadUrl` or resumable download |
|
||||
| SharePoint/OneDrive storage ceiling | 250 GB per file | Microsoft limit; irrelevant to us because we cap extraction far lower |
|
||||
|
||||
Files over 50 MB (or producing over 10 MB of text) are recorded as `skipped_oversize` with a pointer to their `webUrl` so an agent can still cite the source without the text being chunked.
|
||||
|
||||
### 4.3 Sites.Selected ACLs mapped to per-channel kb_scope
|
||||
|
||||
The security property is: an agent must only see documents its channel is authorized for. Two layers enforce this.
|
||||
|
||||
Layer 1: Graph ACLs (what the connector may even fetch). `Sites.Selected` grants the app `read` on a specific site collection only. A site the app cannot read produces 403 and is simply not crawlable. The connector can therefore never ingest content outside the consented sites.
|
||||
|
||||
Layer 2: kb_scope mapping (what an agent may search). Every crawl source is bound to a kb_scope, and every kb_scope maps to exactly one vector namespace. The mapping table in Postgres:
|
||||
|
||||
| tenant_id | channel_id | kb_scope_id | source (site_id, drive_id) | vector namespace |
|
||||
|---|---|---|---|---|
|
||||
| wall-ortho | employee-resources | scope_er | site `handbook.wallortho.com` (drive A) | tenant_wall-ortho__scope_scope_er |
|
||||
| wall-ortho | billing-procedures | scope_bp | site `billing.wallortho.com` (drive B) | tenant_wall-ortho__scope_scope_bp |
|
||||
|
||||
Enforcement at query time:
|
||||
|
||||
- The agent is bound to one channel_id, which resolves to its kb_scope set.
|
||||
- Retrieval queries only the namespace(s) in that kb_scope set. The namespace column is a mandatory equality filter; an agent with channel `employee-resources` can never match chunks in `scope_bp`.
|
||||
- The same mapping gates the Graph fallback: the fallback only searches sites present in the agent's kb_scope sources.
|
||||
|
||||
The combination means authorization is enforced at both ingestion (Graph site grant) and retrieval (namespace filter). A misconfigured kb_scope cannot leak another channel's content because the namespace predicate is applied server-side and the connector never wrote that content into the agent's namespace in the first place.
|
||||
|
||||
---
|
||||
|
||||
## 5. Vector store and embedding model
|
||||
|
||||
### 5.1 Vector store: pgvector (recommended)
|
||||
|
||||
| Criterion | pgvector | Qdrant | Chroma |
|
||||
|---|---|---|---|
|
||||
| Runs on existing stack | Yes: Postgres already in production | New stateful service on netcup | New stateful service (or embedded single-node) |
|
||||
| Tenancy model | Namespace column + filtered query | Native collections (good) | Collections, weaker multi-tenant isolation |
|
||||
| Operational overhead | Zero: reuse backups, HA, auth of Postgres | Extra container, memory, monitoring | Extra container; single-node design |
|
||||
| Scale fit | Strong to low millions of chunks | Strong beyond that | Weak-moderate |
|
||||
| Backup to Wasabi | Inherits existing Postgres S3 backup | Separate snapshot/export | Separate persistence |
|
||||
| Metadata + filters | Native SQL join with tenant/channel tables | Payload filters (separate model) | Metadata filter, less mature |
|
||||
|
||||
Rationale: ITPP already runs Postgres with backup to Wasabi S3, on self-hosted netcup servers. Wall-O's scale is moderate: a practice's internal staff knowledge base is hundreds to low thousands of documents, tens of thousands to low millions of chunks across all tenants. pgvector handles this comfortably with an HNSW index, adds zero new stateful services, keeps vectors and tenancy metadata in one transactional database (atomic upsert, consistent deletes), and inherits the existing backup and HA story. Qdrant would be justified only at very large scale or if vectors needed independent scaling from metadata; Chroma's single-node embedded design is the wrong fit for a multi-tenant production service.
|
||||
|
||||
Implementation notes:
|
||||
|
||||
- Extension: `CREATE EXTENSION vector;` (pgvector >= 0.5.0 for HNSW; newer for larger dimension support).
|
||||
- Index: HNSW on the embedding column. Because queries are namespace-scoped, use a filtered/partial strategy: either a composite approach (HNSW per namespace table via partitioning) or a single HNSW with the namespace filter applied post-recall. At Wall-O scale, a single HNSW index plus a namespace equality filter in the WHERE clause is sufficient and simplest.
|
||||
- Dimension: 1024 (see 5.2), well within pgvector's limits.
|
||||
- Distance: cosine (`<=>`) with L2-normalized vectors.
|
||||
- Chunk text and metadata live in the same table so retrieval returns citations in one query with no join to an object store for the common path. Wasabi S3 remains the archive for raw downloaded files and full-text backups, not the retrieval hot path.
|
||||
|
||||
### 5.2 Embedding model: text-embedding-3-large (Matryoshka to 1024), bge fallback
|
||||
|
||||
| Model | Dims (used) | Hosting | Why / why not |
|
||||
|---|---|---|---|
|
||||
| OpenAI text-embedding-3-large | 3072 native, truncated to 1024 via Matryoshka | Via admin-ai LiteLLM proxy | RECOMMENDED. Strong retrieval quality, already reachable through the existing LiteLLM gateway, no GPU. Matryoshka truncation to 1024 keeps pgvector index and storage small with negligible quality loss. |
|
||||
| text-embedding-3-small | 1536 | Via LiteLLM | Acceptable budget option; slightly lower quality, still fine for internal KB. |
|
||||
| BAAI/bge-large-en-v1.5 | 1024 | Self-hosted on app server | FALLBACK. Zero external dependency and free, but adds model serving burden and slightly weaker than text-embedding-3-large on this task. |
|
||||
| bge-m3 | 1024 | Self-hosted | Only if multilingual staff content appears; out of scope for English-only Wall Orthodontics. |
|
||||
|
||||
Rationale: Wall-O routes LLM calls through admin-ai (DeepSeek V4 Pro primary) already, so embeddings through the same LiteLLM gateway are the lowest-operational-overhead choice and keep spend attributable to the Wall-O virtual key. text-embedding-3-large with Matryoshka truncation to 1024 dimensions gives near-full quality at a quarter of the storage and index cost. The model is pinned and documented so index and query always use identical dimensionality and normalization; changing models requires a documented full re-embed of every namespace (a versioned embedding-model field on the chunk table gates this).
|
||||
|
||||
---
|
||||
|
||||
## Non-negotiable constraints (summary)
|
||||
|
||||
- No patient records, no PHI, no HIPAA scope. The connector is configured per site and never pointed at clinical content.
|
||||
- `Sites.Selected` read grants only; no tenant-wide read permission in production.
|
||||
- Namespace (tenant + kb_scope) is a mandatory equality filter on every retrieval and every Graph fallback search.
|
||||
- Client secret (or certificate) in Vaultwarden; tokens encrypted at rest in Postgres; nothing in logs.
|
||||
- No em dashes, en dashes, or double hyphens in this document or any Wall-O docs.
|
||||
@@ -0,0 +1,235 @@
|
||||
# Wall-O: Deployment, White-Label, and Phase 0 Build Plan
|
||||
|
||||
Scope: multi-tenant internal staff knowledge-base chat. Rocket.Chat is chat transport only (one workspace per tenant). The orchestrator is a multi-tenant FastAPI service that owns tenancy, agents, kb_scope, the M365 connector, retrieval, and LLM. Strictly internal staff knowledge. No patient records, no PHI.
|
||||
|
||||
Canonical entities: tenant_id (a practice), channel_id (a knowledge domain with one attached agent), agent_id (an AI persona bound to a channel), kb_scope (per-tenant plus per-channel document scope).
|
||||
|
||||
## 1. Deployment Topology
|
||||
|
||||
### 1.1 Component map
|
||||
|
||||
| Component | Host | Port | Scope | Scales per tenant |
|
||||
|---|---|---|---|---|
|
||||
| Caddy reverse proxy | wall-o host | 80 / 443 | shared | single process, per-tenant site blocks |
|
||||
| Orchestrator (FastAPI, ASGI via uvicorn worker) | wall-o host | 127.0.0.1:8000 | shared | 1 instance; add workers or a second host behind a load balancer |
|
||||
| Orchestrator DB (Postgres + pgvector) | wall-o host | 127.0.0.1:5432 | shared | 1 primary; promote to a replica for read scale |
|
||||
| Rocket.Chat workspace (tenant N) | wall-o host | 127.0.0.1:3010N | per tenant | 1 Docker Compose stack per tenant, no shared state |
|
||||
| MongoDB (tenant N, Rocket.Chat native store) | wall-o host | 127.0.0.1:2710N (loopback only) | per tenant | 1 Mongo per tenant, isolated volume |
|
||||
| Branding assets (logo, favicon, custom CSS, PWA shell) | wall-o host | /var/www/wall-o/ | per tenant | static files, chmod 644, no process |
|
||||
| M365 connector (orchestrator module) | wall-o host | n/a (in-process) | shared | horizontal with the orchestrator |
|
||||
|
||||
Port formulas: tenant N Rocket.Chat = 30100 + N, tenant N MongoDB = 27100 + N. Tenant 1 (Wall Orthodontics) uses 30101 and 27101.
|
||||
|
||||
URL pattern (ITPP convention):
|
||||
|
||||
| Surface | URL |
|
||||
|---|---|
|
||||
| Orchestrator API | https://api.wall-o.itpropartner.com |
|
||||
| Orchestrator admin console | https://wall-o.itpropartner.com |
|
||||
| Tenant N chat + branded PWA | https://<tenant-slug>.wall-o.itpropartner.com |
|
||||
|
||||
Caddy is the TLS edge. Each tenant subdomain gets its own site block that reverse proxies to that tenant's Rocket.Chat host port. Site blocks are generated from the tenancy table by a small render script (single source of truth in the DB, not hand-edited Caddyfile).
|
||||
|
||||
### 1.2 Directory and config layout
|
||||
|
||||
```text
|
||||
/opt/wall-o/ # app code and compose, NOT /root
|
||||
├── orchestrator/
|
||||
│ ├── app/
|
||||
│ │ ├── main.py # FastAPI app + routers
|
||||
│ │ ├── tenancy.py # tenant CRUD + connection factory
|
||||
│ │ ├── agents.py # agent personas, model binding
|
||||
│ │ ├── kb_scope.py # per-tenant + per-channel scope enforcement
|
||||
│ │ ├── m365.py # Microsoft Graph connector
|
||||
│ │ ├── retrieval.py # pgvector query, RAG pipeline
|
||||
│ │ └── llm.py # admin-ai (LiteLLM) client
|
||||
│ ├── alembic/ # schema migrations
|
||||
│ ├── requirements.txt
|
||||
│ └── .env.example # placeholders only, no secrets
|
||||
├── tenants/
|
||||
│ ├── wall-orthodontics/
|
||||
│ │ ├── docker-compose.yml # Rocket.Chat + Mongo for this tenant
|
||||
│ │ ├── mongod.conf # replSetName: rs0
|
||||
│ │ └── .env # gitignored, values from Vaultwarden
|
||||
│ └── <tenant-slug>/ ...
|
||||
├── caddy/
|
||||
│ └── render-caddyfile.py # emits per-tenant site blocks from DB
|
||||
└── backups/
|
||||
└── run-backups.sh # mongodump + pg_dump to Wasabi S3
|
||||
```
|
||||
|
||||
Web-served static assets (branding, PWA shell, custom CSS) live under /var/www/wall-o/ and are mounted read-only into each Rocket.Chat container. All files under /var/www/wall-o/ are chmod 644, owned by a service account, never by root home.
|
||||
|
||||
### 1.3 Tenant Docker Compose (Rocket.Chat + MongoDB)
|
||||
|
||||
```yaml
|
||||
services:
|
||||
mongo:
|
||||
image: mongo:6.0
|
||||
command: ["mongod", "-f", "/etc/mongod.conf"]
|
||||
volumes:
|
||||
- ./mongod.conf:/etc/mongod.conf:ro
|
||||
- mongo-data:/data/db
|
||||
ports:
|
||||
- "127.0.0.1:27101:27017" # loopback only, for mongodump
|
||||
restart: unless-stopped
|
||||
|
||||
rocketchat:
|
||||
image: registry.rocket.chat/rocketchat/rocket.chat:7.4.0 # pinned; dev CE for Phase 0
|
||||
environment:
|
||||
ROOT_URL: ${ROOT_URL} # https://wall-orthodontics.wall-o.itpropartner.com
|
||||
MONGO_URL: mongodb://mongo:27017/rocketchat?replicaSet=rs0
|
||||
MONGO_OPLOG_URL: mongodb://mongo:27017/local?replicaSet=rs0
|
||||
PORT: "3000"
|
||||
ADMIN_USERNAME: ${ADMIN_USERNAME} # from Vaultwarden
|
||||
ADMIN_PASSWORD: ${ADMIN_PASSWORD} # from Vaultwarden
|
||||
ADMIN_EMAIL: ${ADMIN_EMAIL}
|
||||
ADMIN_NAME: ${ADMIN_NAME}
|
||||
depends_on:
|
||||
- mongo
|
||||
ports:
|
||||
- "127.0.0.1:30101:3000"
|
||||
volumes:
|
||||
- /var/www/wall-o/wall-orthodontics/branding:/app/branding:ro # chmod 644
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
mongo-data:
|
||||
```
|
||||
|
||||
mongod.conf:
|
||||
|
||||
```yaml
|
||||
storage:
|
||||
dbPath: /data/db
|
||||
replication:
|
||||
replSetName: rs0
|
||||
```
|
||||
|
||||
No secrets are committed in compose files. The per-tenant .env holds ADMIN_USERNAME, ADMIN_PASSWORD, and other values, and the .env values are populated at deploy time from Vaultwarden. .env is gitignored; .env.example carries placeholders only.
|
||||
|
||||
### 1.4 Orchestrator to N Rocket.Chat instances
|
||||
|
||||
The orchestrator does not hold one global Rocket.Chat credential. Each tenant row carries its own workspace base URL and admin identity (used for provisioning bots and rooms), keyed by tenant_id. Posting answers uses the per-agent bot identity on the agents table (Part 1, section 3.3).
|
||||
|
||||
```sql
|
||||
CREATE TABLE tenants (
|
||||
tenant_id UUID PRIMARY KEY,
|
||||
slug TEXT UNIQUE NOT NULL,
|
||||
name TEXT NOT NULL,
|
||||
rocket_chat_url TEXT NOT NULL,
|
||||
rocket_chat_admin_user_id TEXT NOT NULL,
|
||||
rocket_chat_admin_token_ref TEXT NOT NULL,
|
||||
status TEXT NOT NULL DEFAULT 'provisioning',
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
```
|
||||
|
||||
The provisioning client reads one row and builds an httpx client signed with the tenant admin identity. A separate factory builds the per-agent posting client from the agent's own bot token (agents.rocket_chat_bot_token_ref), so answers post as the channel's agent, never as the tenant admin. Tokens are never stored in the DB, only Vaultwarden item references.
|
||||
|
||||
```python
|
||||
def admin_client_for(tenant_id: str) -> httpx.AsyncClient:
|
||||
t = db.get_tenant(tenant_id)
|
||||
admin_cred = vaultwarden.get_item(t.rocket_chat_admin_token_ref).password # admin auth token, not in DB
|
||||
return httpx.AsyncClient(
|
||||
base_url=t.rocket_chat_url,
|
||||
headers={"X-User-Id": t.rocket_chat_admin_user_id, "X-Auth-Token": admin_cred},
|
||||
)
|
||||
```
|
||||
|
||||
Provisioning order per tenant: insert tenant row, render Caddy block, docker compose up, seed the workspace (admin identity, agent bot user, agent bot token), store admin and bot tokens in Vaultwarden, write the admin identity back into the tenants row and the agent bot identity into the agents row, flip status to ready.
|
||||
|
||||
## 2. White-Label Execution Plan
|
||||
|
||||
### 2.1 EE-strip decision: FOSS-only build (fossify), not stock CE
|
||||
|
||||
Facts that drive the decision:
|
||||
|
||||
| Fact | Detail |
|
||||
|---|---|
|
||||
| Single codebase since 3.1.0 | Community Edition core is MIT; Enterprise Edition (EE) code is source-available under a proprietary license and sits under apps/meteor/ee/ |
|
||||
| Official Docker image | registry.rocket.chat/rocketchat/rocket.chat bundles EE code in the build, even when no EE license key is applied |
|
||||
| fossify script | Removes all non-MIT code to produce a pure FOSS build; no prebuilt FOSS image is published, so we build it ourselves |
|
||||
|
||||
Decision: for a product we resell, use the FOSS-only build via the fossify script. MIT covers modification and commercial redistribution. The EE license restricts use and distribution, so shipping the stock image (which contains EE code) into a resold, white-labeled product is a licensing risk even if we never activate an EE key. We do not need EE features anyway: Rocket.Chat is chat transport only, and agents, retrieval, kb_scope, and the LLM all live in the orchestrator.
|
||||
|
||||
Consequence: maintain a private fork plus a CI job that runs fossify and builds a FOSS image (wall-o/rocketchat:foss). This is a Phase 1 gate, not a Phase 0 requirement. Phase 0 uses the stock CE image for speed, explicitly dev-only and never resold. No customer deployment ships before the FOSS image build is in CI.
|
||||
|
||||
### 2.2 Server rebrand steps (concrete)
|
||||
|
||||
Because we control the FOSS source, branding is a code patch plus admin settings, not a fragile CSS-only overlay.
|
||||
|
||||
1. Fork Rocket.Chat and run ./fossify.sh; build wall-o/rocketchat:foss.
|
||||
2. Patch the footer and login strings in the fork: replace "Powered by Rocket.Chat" and the Rocket.Chat wordmark references with the Wall-O mark, and default the site name.
|
||||
3. Replace bundled logo and favicon assets with per-tenant assets served from /var/www/wall-o/<tenant>/branding/ (chmod 644), mounted read-only into the container.
|
||||
4. In the workspace admin (Settings, Layout): set Site Name, Site URL, language, and default roles; set the custom color scheme via Custom CSS.
|
||||
5. Disable telemetry, the workspace registration prompt, and the "Register" gate in the fork for self-hosted tenants.
|
||||
6. Bake default colors, logo, and favicon into the image; override per tenant via mounted branding assets and admin settings.
|
||||
7. Confirm no "Powered by Rocket.Chat" string or Rocket.Chat logo remains in the rendered web client (grep the built bundle and spot-check the login page and footer).
|
||||
|
||||
### 2.3 PWA vs native app, and Phase 2 native fork roadmap
|
||||
|
||||
Phase 1 decision: server-side rebrand plus a branded PWA, no native app yet. The Rocket.Chat web client is already responsive and PWA-capable; we wrap it behind the tenant subdomain with our branding. PWA first matches the standing ITPP preference (validate PWA before native).
|
||||
|
||||
Phase 2 native fork roadmap:
|
||||
|
||||
1. Fork Rocket.Chat.ReactNative (MIT core), strip the app/ee/ directory (EE code) the same way as the server.
|
||||
2. Rebrand the app (name, icon, splash, bundle id) and point the default server URL at the tenant subdomain.
|
||||
3. Configure push: the app registers with a push gateway. Self-host the Rocket.Chat push gateway (open source) so push is white-label too, rather than routing through Rocket.Chat's cloud gateway.
|
||||
4. Sign and publish: Google Play via the ITPP developer account; Apple App Store via an Apple Developer Program account (bundle id, provisioning profiles, certificates).
|
||||
5. Per-tenant onboarding: the app resolves the tenant server from a single discovery host, so one app binary serves all tenants (tenant_id entered at first launch or chosen via a directory).
|
||||
|
||||
## 3. Phase 0 Build Plan
|
||||
|
||||
Goal: prove the core loop end to end: Rocket.Chat up, bot registered, channel created, agent attached, document uploaded, indexed, and mobile plus push verified.
|
||||
|
||||
Phase 0 uses the stock CE image for speed (dev-only). The FOSS image build from section 2.1 is a Phase 1 gate before any customer.
|
||||
|
||||
- [ ] 1. Provision host and install Docker. `apt-get update && apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin`
|
||||
- [ ] 2. Create service account and directories. `useradd -r -s /usr/sbin/nologin wallo && mkdir -p /opt/wall-o/tenants/wall-orthodontics /var/www/wall-o/wall-orthodontics/branding`
|
||||
- [ ] 3. Write tenant compose and mongod.conf (section 1.3). Populate .env from Vaultwarden. No secrets inline.
|
||||
- [ ] 4. Start the stack. `docker compose -f /opt/wall-o/tenants/wall-orthodontics/docker-compose.yml up -d`
|
||||
- [ ] 5. Initialize the Mongo replica set (oplog). `docker compose -f /opt/wall-o/tenants/wall-orthodontics/docker-compose.yml exec mongo mongosh -eval "rs.initiate()"`
|
||||
- [ ] 6. Verify workspace is reachable. `curl -sI https://wall-orthodontics.wall-o.itpropartner.com` returns 200 after Caddy cert issuance.
|
||||
- [ ] 7. Login as admin and capture admin identity. `curl -s https://wall-orthodontics.wall-o.itpropartner.com/api/v1/login -H "Content-Type: application/json" -d '{"user":"admin","password":"<from Vaultwarden>"}'`
|
||||
- [ ] 8. Register the agent bot user. `curl -s .../api/v1/users.create -H "X-Auth-Token: <adminToken>" -H "X-User-Id: <adminUserId>" -d '{"name":"Wall-O Agent","username":"wallo.agent","email":"wallo-agent@wall-orthodontics.internal","password":"<from Vaultwarden>","roles":["bot"],"joinDefaultChannels":false,"verified":true}'`
|
||||
- [ ] 9. Create a personal access token for the bot. `curl -s .../api/v1/users.createToken -H "X-Auth-Token: <adminToken>" -H "X-User-Id: <adminUserId>" -d '{"userId":"<botUserId>"}'`; store botUserId plus botAuthToken in Vaultwarden.
|
||||
- [ ] 10. Create the knowledge channel. `curl -s .../api/v1/channels.create -H "X-Auth-Token: <adminToken>" -H "X-User-Id: <adminUserId>" -d '{"name":"employee-resources"}'`
|
||||
- [ ] 11. Attach the agent to the channel. `curl -s .../api/v1/channels.addOwner -H "X-Auth-Token: <adminToken>" -H "X-User-Id: <adminUserId>" -d '{"roomId":"<roomId>","userId":"<botUserId>"}'`
|
||||
- [ ] 12. Confirm the bot can post. `curl -s .../api/v1/chat.postMessage -H "X-Auth-Token: <botAuthToken>" -H "X-User-Id: <botUserId>" -d '{"roomId":"<roomId>","text":"Wall-O agent online."}'`
|
||||
- [ ] 13. Stand up the orchestrator skeleton plus Postgres + pgvector. `docker compose -f /opt/wall-o/orchestrator/docker-compose.yml up -d` with image pgvector/pgvector:pg16, port 127.0.0.1:5432, credentials from .env (Vaultwarden).
|
||||
- [ ] 14. Run migrations to create tenants, channels, agents, kb_scope, documents, chunks tables (section 4.2 columns).
|
||||
- [ ] 15. Seed the tenants row with rocket_chat_url, rocket_chat_admin_user_id, and rocket_chat_admin_token_ref for Wall Orthodontics, and the agents row with the bot user id and bot token reference; leave all tokens in Vaultwarden only.
|
||||
- [ ] 16. Upload a test document. `curl -s -F "file=@staff-handbook.pdf" -F "tenant_id=<tenantId>" -F "channel_id=<channelId>" https://api.wall-o.itpropartner.com/v1/documents`; orchestrator stores the file and records kb_scope.
|
||||
- [ ] 17. Index the document. Orchestrator chunks, embeds via admin-ai, and inserts vectors into pgvector with tenant_id plus channel_id on every row.
|
||||
- [ ] 18. Retrieval smoke test. `curl -s "https://api.wall-o.itpropartner.com/v1/retrieve?tenant_id=<tenantId>&channel_id=<channelId>&q=<question>"` returns a scoped chunk.
|
||||
- [ ] 19. Close the chat loop. Orchestrator polls the channel via `channels.messages?roomId=<roomId>` and replies through chat.postMessage using the bot identity; verify a staff question gets a grounded answer.
|
||||
- [ ] 20. Verify mobile. Open the tenant subdomain in a mobile browser (install the PWA) and confirm chat renders and the bot replies.
|
||||
- [ ] 21. Verify push. Configure Push settings in the workspace (gateway URL), send a direct message to a test user on a mobile device, confirm the notification arrives on the PWA or the Rocket.Chat mobile app pointed at the server.
|
||||
|
||||
## 4. Security and Compliance
|
||||
|
||||
### 4.1 No PHI
|
||||
|
||||
Scope is strictly internal staff knowledge. Hard controls: the M365 connector is limited to explicitly approved, non-patient document libraries; a pre-ingest filter rejects documents matching PHI markers (SSN, MRN, DOB plus name); no HIPAA features are enabled in Rocket.Chat; every ingestion and retrieval path is audited. If a document is suspected to contain PHI, ingestion is blocked and logged, not silently passed.
|
||||
|
||||
### 4.2 Per-tenant data isolation
|
||||
|
||||
| Layer | Isolation |
|
||||
|---|---|
|
||||
| Chat | One Rocket.Chat workspace per tenant, with its own MongoDB instance and volume. No shared chat database. |
|
||||
| Documents and vectors | Every chunks row carries tenant_id and channel_id; every retrieval query is prefixed with a tenant_id plus channel_id filter (kb_scope). Optional later hardening: schema-per-tenant. |
|
||||
| Credentials | Bot tokens are per tenant and scoped to that workspace only; stored in Vaultwarden, referenced by id. |
|
||||
| Routing | Tenant subdomains are isolated Caddy site blocks with no cross-tenant access. |
|
||||
|
||||
### 4.3 Data residency
|
||||
|
||||
All components are self-hosted on netcup infrastructure in the chosen region. Chat, documents, and vectors never leave the self-hosted estate. LLM and embedding calls route through admin-ai, the internal LiteLLM proxy (DeepSeek V4 Pro primary), so prompts are processed inside the controlled stack and never carry PHI by scope.
|
||||
|
||||
### 4.4 Backups
|
||||
|
||||
Nightly job in /opt/wall-o/backups/run-backups.sh: per-tenant mongodump (via the loopback port) plus orchestrator pg_dump, encrypted, uploaded to Wasabi S3. Retention per the ITPP backup schedule with a documented RPO. Backups are restorable per tenant, matching the isolation boundary.
|
||||
|
||||
### 4.5 Secret management
|
||||
|
||||
All secrets live in Vaultwarden: per-tenant admin and bot credentials, Postgres passwords, M365 app credentials, admin-ai (LiteLLM) keys, and S3 keys. Repos and compose files carry .env.example placeholders only. No plaintext secrets in code, compose, or Caddy config. Deploy scripts pull values from Vaultwarden into .env at provision time; .env is gitignored.
|
||||
Reference in New Issue
Block a user