itpp-infrastructure/projects/wall-o/02-m365-connector.md

Wall-O: M365 Connector + Retrieval Architecture

Status: DESIGN (internal technical architecture) Owner: Wall-O build team Scope: Internal staff knowledge only. Employee handbook, policies, procedures, billing questions. Excluded: Patient records, PHI, any HIPAA-regulated data. This connector MUST NOT be pointed at clinical or patient data sources.

Canonical entities used in this document

Entity	Definition	Cardinality
tenant_id	A practice (e.g. Wall Orthodontics). The Microsoft 365 tenant boundary.	1 M365 tenant per tenant_id
channel_id	A knowledge domain within a tenant (e.g. "Employee Resources").	many per tenant
agent_id	An AI persona bound to a channel. One agent serves one channel.	1:1 with channel_id
kb_scope	The set of documents/indices an agent may search. Scoped per tenant AND per channel. Maps 1:1 to a vector namespace.	1:1 with vector namespace
documents/chunks	Files synced from M365, chunked and indexed into the tenant+channel vector namespace.	many per kb_scope

The vector namespace is a logical concept. In pgvector it is implemented as a namespace column on the chunk table and filtered in the WHERE clause of every similarity query. In Qdrant it would be a native collection. The logical name format is fixed regardless of backend:

tenant_{tenant_id}__scope_{kb_scope_id}

---

1. Authentication model (Microsoft Graph, app-only)

1.1 Entra app registration

Single multi-tenant app registration in the IT Pro Partner (IPP) home tenant. Supported account type:

Accounts in any organizational directory (Any Microsoft Entra ID tenant - Multitenant)

This lets one app identity serve every practice tenant. Each practice admin consents the app into their own tenant; no app is registered inside the client's tenant.

1.2 Permission strategy: Sites.Selected over Sites.Read.All

Wall-O reads documents, never writes. Least privilege is achieved with the Sites.Selected application permission rather than the broad Sites.Read.All.

Permission	Type	Scope	Why accepted / rejected
Sites.Selected	Application	Only sites explicitly granted via the site permissions API	ACCEPTED. Primary. Grants per-site read; the app sees nothing else in the tenant.
Sites.Read.All	Application	Every site in the tenant	REJECTED for production. Violates least privilege; exposes all site collections including any we are not meant to index.
Files.Read.All	Application	Every file in every drive	REJECTED. Superseded by Sites.Selected for site-scoped read.
User.Read.All	Application	Read directory user profiles	OPTIONAL. Needed only to resolve author display names from OneDrive drive owner IDs. Not required for retrieval.

Sites.Selected supports site-level roles: read, write, fullcontrol, manage. Wall-O requests read only. The grant is issued per site collection via:

POST https://graph.microsoft.com/v1.0/sites/{site-id}/permissions
Content-Type: application/json

{
  "roles": ["read"],
  "grantedToIdentities": [
    { "application": { "id": "{wall-o-app-client-id}", "displayName": "Wall-O" } }
  ]
}

This request is made by the orchestrator using the app's own token for the tenant. Because the app already holds Sites.Selected, it can grant itself read on a specific site once a practice admin has approved the site in onboarding. A stricter variant has a practice global admin run the grant via Graph Explorer so the app never self-grants. Wall-O uses the admin-driven variant: the grant is issued during onboarding by the practice admin (or by the orchestrator on a one-time admin-approved site list), not by the app unprompted.

1.3 Client credentials grant

App-only OAuth 2.0 client credentials flow. No user, no interactive login, no refresh token.

Token request:

POST https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token
Content-Type: application/x-www-form-urlencoded

client_id={client_id}
client_secret={client_secret}
scope=https://graph.microsoft.com/.default
grant_type=client_credentials

Exact strings to configure:

Field	Exact value
grant_type	client_credentials
scope	https://graph.microsoft.com/.default
Token endpoint	https://login.microsoftonline.com/{tenant_id}/oauth2/v2.0/token

Notes:

• {tenant_id} is the consuming practice's directory (tenant) ID, captured at consent time. It is NOT common or organizations: client credentials has no user to derive a tenant from, so the target tenant must be explicit.
• .default expands to the union of the application permissions already consented for that tenant. It is not a literal scope name.
• Response contains access_token and expires_in. Graph client credentials tokens are typically valid for about 60 minutes; treat expires_in as authoritative, never hardcode the TTL.
• Credential material: prefer an X.509 client certificate (assertion) over a client secret. A secret is acceptable for MVP but must be rotated before its maximum lifetime (24 months). Secrets live in Vaultwarden under the Wall-O project; only the encrypted reference enters Postgres or config.

1.4 Admin consent URL construction

Each practice admin must consent the app into their tenant before the first crawl. The v2.0 admin consent endpoint is constructed as:

https://login.microsoftonline.com/{tenant_id}/v2.0/adminconsent
  ?client_id={wall-o-app-client-id}
  &scope=https://graph.microsoft.com/.default
  &redirect_uri={configured-redirect-uri}

• {tenant_id}: the practice's directory ID, or organizations to let the signing admin's tenant be used automatically.
• redirect_uri: a registered reply URL on the app. Wall-O registers a no-op callback (e.g. https://wall-o.itpropartner.com/entra/callback) that returns a 200 and logs the consent result.
• The scope string is URL-encoded .default (i.e. https%3A%2F%2Fgraph.microsoft.com%2F.default).

Onboarding flow per practice:

1. Practice admin clicks the consent URL (delivered by the Wall-O onboarding UI or support).
2. Admin authenticates and approves the Sites.Selected application permission.
3. On success Entra redirects to the callback. The orchestrator records tenant_id, consent timestamp, and the admin's identity.
4. The site-level read grant (1.2) is then applied to the specific site collections mapped to that practice's channels.
5. Only now does the connector issue its first token and run a test crawl.

Consent revocation is detected at token time (see 1.5); the tenant is marked consent_revoked and the practice admin is prompted to re-consent.

1.5 Per-tenant token storage and refresh lifecycle

No refresh token exists in client credentials, so "refresh" means issuing a fresh token on demand. The cache is a memoization layer plus a lifecycle guard.

Storage (Postgres, tenant_credentials table):

Column	Purpose
tenant_id	Primary cache key
access_token	Encrypted at rest (AES-256-GCM, app-level, key in Vaultwarden)
expires_at	Absolute expiry derived from expires_in at issue time
last_error / error_code	Last failure for alerting (e.g. consent revoked, secret expired)
status	active, consent_revoked, secret_expired, suspended

Lifecycle rules:

• Token is considered usable while now < expires_at - 300s (5 minute safety margin). Outside that window the connector requests a fresh token before the next Graph call.
• On 401 Unauthorized with InvalidAuthenticationToken, or 403 with a consent/scope error, the connector retries once with a freshly issued token. A second failure escalates.
• Error code mapping:
  • AADSTS700016 (application not found in directory) or AADSTS7000112 (invalid client) means the app is not consented in that tenant: mark consent_revoked, alert practice admin.
  • AADSTS700082 or expired secret errors mean the secret is expired/rotated: mark secret_expired, page the Wall-O operator.
  • Graph throttling 429 is not an auth failure: honor Retry-After and back off; do not flag the tenant.
• Token issuance and refresh are serialized per tenant (single-flight lock) so concurrent sync workers do not stampede the token endpoint.
• Tokens are never logged or returned by any API. Logging redacts the Authorization header.

1.6 Publisher verification

Required so the multi-tenant consent prompt shows a verified publisher rather than "unverified", which materially reduces admin consent friction (and some tenants block unverified apps outright).

Requirements:

• A Microsoft AI Cloud Partner Program (MAICPP) account with an MPN ID for IT Pro Partner.
• The MPN ID associated with the app registration under Branding and properties.
• A verified custom domain (e.g. itpropartner.com) linked to the Entra tenant and set as the publisher domain.
• App registration Branding shows "Publisher verified" with the blue checkmark.

Publisher verification is a one-time per-publisher step, not per-tenant. It is a prerequisite to onboarding the first external practice; without it, practice admins see an unverified-publisher warning on the consent screen.

---

2. Sync and index pipeline

2.1 Crawl sources

Two source types, both driven by Microsoft Graph:

Source	Graph entry points
SharePoint document libraries	GET /sites/{site-id}/drives to enumerate libraries, then per-drive crawl
OneDrive for Business	GET /users/{user-id}/drive (a practice user's personal drive mapped to a channel)

Site resolution by human-friendly URL before crawling:

GET https://graph.microsoft.com/v1.0/sites/{hostname}:/{site-path}
GET https://graph.microsoft.com/v1.0/sites?search={query}

Each crawl source is registered in Postgres as part of a kb_scope, so a scope maps to one or more (site, drive) pairs. The connector crawls exactly those drives, never the whole tenant.

2.2 Crawl, extract, chunk, embed, write

Pipeline stages, all inside the FastAPI orchestrator (worker tasks, not the request path):

1. Crawl: enumerate items per drive using GET /sites/{site-id}/drive/root/children and GET /sites/{site-id}/drive/root:/{path}:/children for nested folders. Use $batch (up to 20 requests per batch) to reduce round trips and honor throttling.
2. Download: GET /sites/{site-id}/drive/items/{item-id}/content (or the item's @microsoft.graph.downloadUrl). Stream to disk, never into memory whole.
3. Extract: text extraction per format (see section 4.1). Produce plain text plus a metadata block.
4. Chunk: split with the strategy in 2.3.
5. Embed: embed each chunk with the configured model (section 5.2).
6. Write: upsert chunks into the {tenant}_{kb_scope} vector namespace in pgvector. One transaction per document: delete old chunks for that document_id, insert new, commit. Keeps the namespace consistent even if a sync crashes mid-document.

2.3 Chunking strategy

Concrete defaults:

Parameter	Value
Target chunk size	512 tokens (approx 2000 chars)
Overlap	64 tokens (approx 250 chars, 12.5%)
Splitter	Sentence-aware: split on paragraph then sentence boundaries, never mid-word
Hard max	1024 tokens for a single chunk (tables, bullet lists, malformed PDF text)
Min kept	Discard chunks under 20 tokens
Tokenizer	The embedding model's tokenizer (cl100k_base if using OpenAI text-embedding-3)

Per-chunk metadata written alongside the vector:

Metadata field	Source
tenant_id	canonical entity
channel_id	canonical entity
kb_scope_id	canonical entity
namespace	tenant_{tenant_id}__scope_{kb_scope_id}
document_id	Graph driveItem id
document_name	driveItem name
site_id / drive_id / item_id	Graph IDs for provenance and re-download
source_url	driveItem webUrl
modified_at	driveItem lastModifiedDateTime
chunk_index	position within document
mime / file_type	for format-aware handling
title / heading	nearest heading above the chunk, for context injection

Metadata is stored in the same Postgres chunk row so filtering (by channel, by scope) is a plain SQL predicate, not a secondary lookup.

2.4 Incremental sync (delta query + change notifications)

Full re-crawl on every sync is avoided with two complementary mechanisms.

Delta query (poll-based, authoritative):

GET https://graph.microsoft.com/v1.0/sites/{site-id}/drive/root/delta

• First call with no token returns the full set and a @odata.deltaLink (and @odata.nextLink while paging).
• Subsequent calls use the stored deltaLink and return only added, changed, and deleted items. Deleted items carry a deleted facet; the connector removes their chunks from the namespace.
• Persist the deltaLink per (site, drive) in Postgres.
• Delta tokens expire; a 410 Gone (or malformed delta token) means the connector must drop to a full re-crawl for that drive. The connector treats 410 as a normal control-flow event, logs it, and re-syncs fully.

Change notifications (push-based, reduces lag):

POST https://graph.microsoft.com/v1.0/subscriptions
Content-Type: application/json

{
  "changeType": "updated",
  "notificationUrl": "https://wall-o.itpropartner.com/entra/notifications",
  "resource": "/sites/{site-id}/drive/root",
  "expirationDateTime": "2026-08-18T18:00:00Z",
  "clientState": "tenant_{tenant_id}"
}

• changeType may be created,updated,deleted combined in one subscription.
• expirationDateTime maximum is 4230 minutes (3 days). The orchestrator renews every subscription before expiry (cron, every 6 hours, PATCH /subscriptions/{id} to extend).
• The notificationUrl must be HTTPS, publicly reachable, and answer the initial validationToken handshake: echo the token back as text/plain with HTTP 200 within 5 seconds, then process the notification asynchronously.
• clientState is echoed back on every notification so the orchestrator verifies the sender and does not act on forged payloads.
• On notification, the connector does NOT fetch blindly: it records a dirty (site, drive) and lets the next delta pass reconcile. This deduplicates burst notifications into one delta sweep.

Sync schedule (default): full crawl on onboarding and after a 410; delta sweep every 15 minutes; change notifications applied opportunistically between sweeps. Re-embed only changed/deleted chunks, not the whole namespace.

2.5 Sync observability

Per (tenant, scope) the orchestrator records: last_full_sync_at, last_delta_sync_at, items_seen, chunks_written, chunks_deleted, and last_sync_error. A dashboard flag is raised when last_delta_sync_at exceeds the 15 minute SLA by a factor of 3 (i.e. stale over 45 minutes), which also gates the stale-index retrieval fallback in section 3.3.

---

3. Retrieval

3.1 Semantic search (primary)

The agent's query is embedded with the same model and dimensionality used at index time. Retrieval runs against exactly one vector namespace: the agent's kb_scope namespace. Tenancy and channel scoping is therefore a hard SQL predicate, not an after-the-fact filter.

pgvector query shape:

SELECT chunk_id, document_id, document_name, source_url, text, chunk_index,
       1 - (embedding <=> $1) AS similarity
FROM chunks
WHERE namespace = $2
ORDER BY embedding <=> $1
LIMIT $3;

• <=> is cosine distance; 1 - (embedding <=> $1) yields cosine similarity. Embeddings are stored L2-normalized so cosine and inner product are equivalent.
• top-k for the candidate set is 20.

3.2 Rerank (optional second stage)

A cross-encoder reranks the 20 candidates against the raw query to 5 final passages. This is quality over recall: the cross-encoder sees query and passage jointly and scores true relevance, correcting embedding-only mistakes on synonyms and negations.

• Model: BAAI/bge-reranker-large (self-hosted on the app server) or a managed reranker via LiteLLM if one is added to admin-ai.
• Rerank is a config toggle per agent. If disabled, the top 5 by cosine similarity are used directly.

3.3 Graph live search fallback (cold or stale index)

When the vector namespace is cold (fewer than N chunks, e.g. onboarding in progress) or stale (delta sync behind, see 2.5), retrieval falls back to live Graph search so answers are never silently empty. The Graph search API is used as the fallback, not the primary, because it is slower, rate-limited, and returns passages without the tight namespace scoping.

POST https://graph.microsoft.com/v1.0/search/query
Content-Type: application/json

{
  "requests": [
    {
      "entityTypes": ["driveItem"],
      "query": { "queryString": "{user query}" },
      "from": 0,
      "size": 10,
      "fields": ["id", "name", "webUrl", "lastModifiedDateTime"]
    }
  ]
}

A simpler drive-scoped variant is available when the fallback is constrained to one site:

GET https://graph.microsoft.com/v1.0/sites/{site-id}/drive/root/search(q='{user query}')

The fallback returns file references (name, webUrl, snippet), which the orchestrator hands to the LLM as source citations rather than pre-chunked text.

3.4 Hybrid ranking approach

Two signals are fused when both are available:

Signal	Source	Weight role
Semantic	embedding cosine similarity	primary
Keyword	Postgres full-text search (tsvector) or pg_trgm over chunk text	secondary, for exact terms: policy IDs, benefit codes, acronyms

Fusion uses Reciprocal Rank Fusion (RRF):

score(d) = sum over signals of 1 / (k + rank_signal(d)),  k = 60

This preserves exact-term matches (e.g. a specific policy number) that pure embedding search can miss, without a separate vector backend. The keyword index is a generated tsvector column on the chunk table maintained by trigger or at write time.

3.5 Graph endpoints actually used (reference)

Purpose	Endpoint
Token	POST /oauth2/v2.0/token (login.microsoftonline.com)
Admin consent	GET /{tenant}/v2.0/adminconsent
Site resolution	GET /v1.0/sites/{hostname}:/{path}, GET /v1.0/sites?search=
Site read grant (Sites.Selected)	POST /v1.0/sites/{site-id}/permissions
List drives in a site	GET /v1.0/sites/{site-id}/drives
List folder children	GET /v1.0/sites/{site-id}/drive/root/children
Path-based children	GET /v1.0/sites/{site-id}/drive/root:/{path}:/children
Download content	GET /v1.0/sites/{site-id}/drive/items/{item-id}/content
Delta sync	GET /v1.0/sites/{site-id}/drive/root/delta
OneDrive (user)	GET /v1.0/users/{user-id}/drive/root/delta
Live search (fallback)	POST /v1.0/search/query
Drive-scoped search (fallback)	GET /v1.0/sites/{site-id}/drive/root/search(q='...')
Change notifications	POST /v1.0/subscriptions, PATCH /v1.0/subscriptions/{id}
Author resolution (optional)	GET /v1.0/users/{user-id}

---

4. Document format support, size limits, ACL mapping

4.1 Format support

Format	Extraction approach
PDF	PyMuPDF (fitz), per-page text extraction; flagged as text-only (embedded OCR is out of scope)
DOCX	python-docx: paragraphs + tables
XLSX	openpyxl: sheet-by-sheet, cell grid serialized with sheet name as heading context
PPTX	python-pptx: slide title + body text per slide
TXT / MD / CSV	plain read with encoding detection
HTML (optional)	trafilatura / BeautifulSoup text extraction

The connector downloads and parses locally. Graph does not return extracted text, only file content and metadata. Binary formats not in this table are skipped and counted in sync observability, never force-indexed.

4.2 File size limits

Limit	Value	Rationale
Max file for extraction	50 MB	Above this, office files and PDFs blow up parse time and memory; skipped and flagged
Max chunk text written per file	10 MB of extracted text	Mirrors SharePoint search's practical text-index ceiling; larger files are truncated with a notice chunk
Graph simple download ceiling	approx 250 MB via /content	Beyond this use the item's @microsoft.graph.downloadUrl or resumable download
SharePoint/OneDrive storage ceiling	250 GB per file	Microsoft limit; irrelevant to us because we cap extraction far lower

Files over 50 MB (or producing over 10 MB of text) are recorded as skipped_oversize with a pointer to their webUrl so an agent can still cite the source without the text being chunked.

4.3 Sites.Selected ACLs mapped to per-channel kb_scope

The security property is: an agent must only see documents its channel is authorized for. Two layers enforce this.

Layer 1: Graph ACLs (what the connector may even fetch). Sites.Selected grants the app read on a specific site collection only. A site the app cannot read produces 403 and is simply not crawlable. The connector can therefore never ingest content outside the consented sites.

Layer 2: kb_scope mapping (what an agent may search). Every crawl source is bound to a kb_scope, and every kb_scope maps to exactly one vector namespace. The mapping table in Postgres:

tenant_id	channel_id	kb_scope_id	source (site_id, drive_id)	vector namespace
wall-ortho	employee-resources	scope_er	site handbook.wallortho.com (drive A)	tenant_wall-ortho__scope_scope_er
wall-ortho	billing-procedures	scope_bp	site billing.wallortho.com (drive B)	tenant_wall-ortho__scope_scope_bp

Enforcement at query time:

• The agent is bound to one channel_id, which resolves to its kb_scope set.
• Retrieval queries only the namespace(s) in that kb_scope set. The namespace column is a mandatory equality filter; an agent with channel employee-resources can never match chunks in scope_bp.
• The same mapping gates the Graph fallback: the fallback only searches sites present in the agent's kb_scope sources.

The combination means authorization is enforced at both ingestion (Graph site grant) and retrieval (namespace filter). A misconfigured kb_scope cannot leak another channel's content because the namespace predicate is applied server-side and the connector never wrote that content into the agent's namespace in the first place.

---

5. Vector store and embedding model

5.1 Vector store: pgvector (recommended)

Criterion	pgvector	Qdrant	Chroma
Runs on existing stack	Yes: Postgres already in production	New stateful service on netcup	New stateful service (or embedded single-node)
Tenancy model	Namespace column + filtered query	Native collections (good)	Collections, weaker multi-tenant isolation
Operational overhead	Zero: reuse backups, HA, auth of Postgres	Extra container, memory, monitoring	Extra container; single-node design
Scale fit	Strong to low millions of chunks	Strong beyond that	Weak-moderate
Backup to Wasabi	Inherits existing Postgres S3 backup	Separate snapshot/export	Separate persistence
Metadata + filters	Native SQL join with tenant/channel tables	Payload filters (separate model)	Metadata filter, less mature

Rationale: ITPP already runs Postgres with backup to Wasabi S3, on self-hosted netcup servers. Wall-O's scale is moderate: a practice's internal staff knowledge base is hundreds to low thousands of documents, tens of thousands to low millions of chunks across all tenants. pgvector handles this comfortably with an HNSW index, adds zero new stateful services, keeps vectors and tenancy metadata in one transactional database (atomic upsert, consistent deletes), and inherits the existing backup and HA story. Qdrant would be justified only at very large scale or if vectors needed independent scaling from metadata; Chroma's single-node embedded design is the wrong fit for a multi-tenant production service.

Implementation notes:

• Extension: CREATE EXTENSION vector; (pgvector >= 0.5.0 for HNSW; newer for larger dimension support).
• Index: HNSW on the embedding column. Because queries are namespace-scoped, use a filtered/partial strategy: either a composite approach (HNSW per namespace table via partitioning) or a single HNSW with the namespace filter applied post-recall. At Wall-O scale, a single HNSW index plus a namespace equality filter in the WHERE clause is sufficient and simplest.
• Dimension: 1024 (see 5.2), well within pgvector's limits.
• Distance: cosine (<=>) with L2-normalized vectors.
• Chunk text and metadata live in the same table so retrieval returns citations in one query with no join to an object store for the common path. Wasabi S3 remains the archive for raw downloaded files and full-text backups, not the retrieval hot path.

5.2 Embedding model: text-embedding-3-large (Matryoshka to 1024), bge fallback

Model	Dims (used)	Hosting	Why / why not
OpenAI text-embedding-3-large	3072 native, truncated to 1024 via Matryoshka	Via admin-ai LiteLLM proxy	RECOMMENDED. Strong retrieval quality, already reachable through the existing LiteLLM gateway, no GPU. Matryoshka truncation to 1024 keeps pgvector index and storage small with negligible quality loss.
text-embedding-3-small	1536	Via LiteLLM	Acceptable budget option; slightly lower quality, still fine for internal KB.
BAAI/bge-large-en-v1.5	1024	Self-hosted on app server	FALLBACK. Zero external dependency and free, but adds model serving burden and slightly weaker than text-embedding-3-large on this task.
bge-m3	1024	Self-hosted	Only if multilingual staff content appears; out of scope for English-only Wall Orthodontics.

Rationale: Wall-O routes LLM calls through admin-ai (DeepSeek V4 Pro primary) already, so embeddings through the same LiteLLM gateway are the lowest-operational-overhead choice and keep spend attributable to the Wall-O virtual key. text-embedding-3-large with Matryoshka truncation to 1024 dimensions gives near-full quality at a quarter of the storage and index cost. The model is pinned and documented so index and query always use identical dimensionality and normalization; changing models requires a documented full re-embed of every namespace (a versioned embedding-model field on the chunk table gates this).

---

Non-negotiable constraints (summary)

• No patient records, no PHI, no HIPAA scope. The connector is configured per site and never pointed at clinical content.
• Sites.Selected read grants only; no tenant-wide read permission in production.
• Namespace (tenant + kb_scope) is a mandatory equality filter on every retrieval and every Graph fallback search.
• Client secret (or certificate) in Vaultwarden; tokens encrypted at rest in Postgres; nothing in logs.
• No em dashes, en dashes, or double hyphens in this document or any Wall-O docs.