itpp-infrastructure/projects/wall-o/03-deployment-white-label.md

Wall-O: Deployment, White-Label, and Phase 0 Build Plan

Scope: multi-tenant internal staff knowledge-base chat. Rocket.Chat is chat transport only (one workspace per tenant). The orchestrator is a multi-tenant FastAPI service that owns tenancy, agents, kb_scope, the M365 connector, retrieval, and LLM. Strictly internal staff knowledge. No patient records, no PHI.

Canonical entities: tenant_id (a practice), channel_id (a knowledge domain with one attached agent), agent_id (an AI persona bound to a channel), kb_scope (per-tenant plus per-channel document scope).

1. Deployment Topology

1.1 Component map

Component	Host	Port	Scope	Scales per tenant
Caddy reverse proxy	wall-o host	80 / 443	shared	single process, per-tenant site blocks
Orchestrator (FastAPI, ASGI via uvicorn worker)	wall-o host	127.0.0.1:8000	shared	1 instance; add workers or a second host behind a load balancer
Orchestrator DB (Postgres + pgvector)	wall-o host	127.0.0.1:5432	shared	1 primary; promote to a replica for read scale
Rocket.Chat workspace (tenant N)	wall-o host	127.0.0.1:3010N	per tenant	1 Docker Compose stack per tenant, no shared state
MongoDB (tenant N, Rocket.Chat native store)	wall-o host	127.0.0.1:2710N (loopback only)	per tenant	1 Mongo per tenant, isolated volume
Branding assets (logo, favicon, custom CSS, PWA shell)	wall-o host	/var/www/wall-o/	per tenant	static files, chmod 644, no process
M365 connector (orchestrator module)	wall-o host	n/a (in-process)	shared	horizontal with the orchestrator

Port formulas: tenant N Rocket.Chat = 30100 + N, tenant N MongoDB = 27100 + N. Tenant 1 (Wall Orthodontics) uses 30101 and 27101.

URL pattern (ITPP convention):

Surface	URL
Orchestrator API	https://api.wall-o.itpropartner.com
Orchestrator admin console	https://wall-o.itpropartner.com
Tenant N chat + branded PWA	https://.wall-o.itpropartner.com

Caddy is the TLS edge. Each tenant subdomain gets its own site block that reverse proxies to that tenant's Rocket.Chat host port. Site blocks are generated from the tenancy table by a small render script (single source of truth in the DB, not hand-edited Caddyfile).

1.2 Directory and config layout

/opt/wall-o/                                  # app code and compose, NOT /root
├── orchestrator/
│   ├── app/
│   │   ├── main.py                           # FastAPI app + routers
│   │   ├── tenancy.py                        # tenant CRUD + connection factory
│   │   ├── agents.py                         # agent personas, model binding
│   │   ├── kb_scope.py                       # per-tenant + per-channel scope enforcement
│   │   ├── m365.py                           # Microsoft Graph connector
│   │   ├── retrieval.py                      # pgvector query, RAG pipeline
│   │   └── llm.py                            # admin-ai (LiteLLM) client
│   ├── alembic/                              # schema migrations
│   ├── requirements.txt
│   └── .env.example                          # placeholders only, no secrets
├── tenants/
│   ├── wall-orthodontics/
│   │   ├── docker-compose.yml                # Rocket.Chat + Mongo for this tenant
│   │   ├── mongod.conf                       # replSetName: rs0
│   │   └── .env                              # gitignored, values from Vaultwarden
│   └── <tenant-slug>/ ...
├── caddy/
│   └── render-caddyfile.py                   # emits per-tenant site blocks from DB
└── backups/
    └── run-backups.sh                        # mongodump + pg_dump to Wasabi S3

Web-served static assets (branding, PWA shell, custom CSS) live under /var/www/wall-o/ and are mounted read-only into each Rocket.Chat container. All files under /var/www/wall-o/ are chmod 644, owned by a service account, never by root home.

1.3 Tenant Docker Compose (Rocket.Chat + MongoDB)

services:
  mongo:
    image: mongo:6.0
    command: ["mongod", "-f", "/etc/mongod.conf"]
    volumes:
      - ./mongod.conf:/etc/mongod.conf:ro
      - mongo-data:/data/db
    ports:
      - "127.0.0.1:27101:27017"          # loopback only, for mongodump
    restart: unless-stopped

  rocketchat:
    image: registry.rocket.chat/rocketchat/rocket.chat:7.4.0   # pinned; dev CE for Phase 0
    environment:
      ROOT_URL: ${ROOT_URL}               # https://wall-orthodontics.wall-o.itpropartner.com
      MONGO_URL: mongodb://mongo:27017/rocketchat?replicaSet=rs0
      MONGO_OPLOG_URL: mongodb://mongo:27017/local?replicaSet=rs0
      PORT: "3000"
      ADMIN_USERNAME: ${ADMIN_USERNAME}   # from Vaultwarden
      ADMIN_PASSWORD: ${ADMIN_PASSWORD}   # from Vaultwarden
      ADMIN_EMAIL: ${ADMIN_EMAIL}
      ADMIN_NAME: ${ADMIN_NAME}
    depends_on:
      - mongo
    ports:
      - "127.0.0.1:30101:3000"
    volumes:
      - /var/www/wall-o/wall-orthodontics/branding:/app/branding:ro   # chmod 644
    restart: unless-stopped

volumes:
  mongo-data:

mongod.conf:

storage:
  dbPath: /data/db
replication:
  replSetName: rs0

No secrets are committed in compose files. The per-tenant .env holds ADMIN_USERNAME, ADMIN_PASSWORD, and other values, and the .env values are populated at deploy time from Vaultwarden. .env is gitignored; .env.example carries placeholders only.

1.4 Orchestrator to N Rocket.Chat instances

The orchestrator does not hold one global Rocket.Chat credential. Each tenant row carries its own workspace base URL and admin identity (used for provisioning bots and rooms), keyed by tenant_id. Posting answers uses the per-agent bot identity on the agents table (Part 1, section 3.3).

CREATE TABLE tenants (
    tenant_id          UUID PRIMARY KEY,
    slug               TEXT UNIQUE NOT NULL,
    name               TEXT NOT NULL,
    rocket_chat_url            TEXT NOT NULL,
    rocket_chat_admin_user_id  TEXT NOT NULL,
    rocket_chat_admin_token_ref TEXT NOT NULL,
    status             TEXT NOT NULL DEFAULT 'provisioning',
    created_at         TIMESTAMPTZ NOT NULL DEFAULT now()
);

The provisioning client reads one row and builds an httpx client signed with the tenant admin identity. A separate factory builds the per-agent posting client from the agent's own bot token (agents.rocket_chat_bot_token_ref), so answers post as the channel's agent, never as the tenant admin. Tokens are never stored in the DB, only Vaultwarden item references.

def admin_client_for(tenant_id: str) -> httpx.AsyncClient:
    t = db.get_tenant(tenant_id)
    admin_cred = vaultwarden.get_item(t.rocket_chat_admin_token_ref).password   # admin auth token, not in DB
    return httpx.AsyncClient(
        base_url=t.rocket_chat_url,
        headers={"X-User-Id": t.rocket_chat_admin_user_id, "X-Auth-Token": admin_cred},
    )

Provisioning order per tenant: insert tenant row, render Caddy block, docker compose up, seed the workspace (admin identity, agent bot user, agent bot token), store admin and bot tokens in Vaultwarden, write the admin identity back into the tenants row and the agent bot identity into the agents row, flip status to ready.

2. White-Label Execution Plan

2.1 EE-strip decision: FOSS-only build (fossify), not stock CE

Facts that drive the decision:

Fact	Detail
Single codebase since 3.1.0	Community Edition core is MIT; Enterprise Edition (EE) code is source-available under a proprietary license and sits under apps/meteor/ee/
Official Docker image	registry.rocket.chat/rocketchat/rocket.chat bundles EE code in the build, even when no EE license key is applied
fossify script	Removes all non-MIT code to produce a pure FOSS build; no prebuilt FOSS image is published, so we build it ourselves

Decision: for a product we resell, use the FOSS-only build via the fossify script. MIT covers modification and commercial redistribution. The EE license restricts use and distribution, so shipping the stock image (which contains EE code) into a resold, white-labeled product is a licensing risk even if we never activate an EE key. We do not need EE features anyway: Rocket.Chat is chat transport only, and agents, retrieval, kb_scope, and the LLM all live in the orchestrator.

Consequence: maintain a private fork plus a CI job that runs fossify and builds a FOSS image (wall-o/rocketchat:foss). This is a Phase 1 gate, not a Phase 0 requirement. Phase 0 uses the stock CE image for speed, explicitly dev-only and never resold. No customer deployment ships before the FOSS image build is in CI.

2.2 Server rebrand steps (concrete)

Because we control the FOSS source, branding is a code patch plus admin settings, not a fragile CSS-only overlay.

1. Fork Rocket.Chat and run ./fossify.sh; build wall-o/rocketchat:foss.
2. Patch the footer and login strings in the fork: replace "Powered by Rocket.Chat" and the Rocket.Chat wordmark references with the Wall-O mark, and default the site name.
3. Replace bundled logo and favicon assets with per-tenant assets served from /var/www/wall-o//branding/ (chmod 644), mounted read-only into the container.
4. In the workspace admin (Settings, Layout): set Site Name, Site URL, language, and default roles; set the custom color scheme via Custom CSS.
5. Disable telemetry, the workspace registration prompt, and the "Register" gate in the fork for self-hosted tenants.
6. Bake default colors, logo, and favicon into the image; override per tenant via mounted branding assets and admin settings.
7. Confirm no "Powered by Rocket.Chat" string or Rocket.Chat logo remains in the rendered web client (grep the built bundle and spot-check the login page and footer).

2.3 PWA vs native app, and Phase 2 native fork roadmap

Phase 1 decision: server-side rebrand plus a branded PWA, no native app yet. The Rocket.Chat web client is already responsive and PWA-capable; we wrap it behind the tenant subdomain with our branding. PWA first matches the standing ITPP preference (validate PWA before native).

Phase 2 native fork roadmap:

1. Fork Rocket.Chat.ReactNative (MIT core), strip the app/ee/ directory (EE code) the same way as the server.
2. Rebrand the app (name, icon, splash, bundle id) and point the default server URL at the tenant subdomain.
3. Configure push: the app registers with a push gateway. Self-host the Rocket.Chat push gateway (open source) so push is white-label too, rather than routing through Rocket.Chat's cloud gateway.
4. Sign and publish: Google Play via the ITPP developer account; Apple App Store via an Apple Developer Program account (bundle id, provisioning profiles, certificates).
5. Per-tenant onboarding: the app resolves the tenant server from a single discovery host, so one app binary serves all tenants (tenant_id entered at first launch or chosen via a directory).

3. Phase 0 Build Plan

Goal: prove the core loop end to end: Rocket.Chat up, bot registered, channel created, agent attached, document uploaded, indexed, and mobile plus push verified.

Phase 0 uses the stock CE image for speed (dev-only). The FOSS image build from section 2.1 is a Phase 1 gate before any customer.

• 1. Provision host and install Docker. apt-get update && apt-get install -y docker-ce docker-ce-cli containerd.io docker-compose-plugin
• 2. Create service account and directories. useradd -r -s /usr/sbin/nologin wallo && mkdir -p /opt/wall-o/tenants/wall-orthodontics /var/www/wall-o/wall-orthodontics/branding
• 3. Write tenant compose and mongod.conf (section 1.3). Populate .env from Vaultwarden. No secrets inline.
• 4. Start the stack. docker compose -f /opt/wall-o/tenants/wall-orthodontics/docker-compose.yml up -d
• 5. Initialize the Mongo replica set (oplog). docker compose -f /opt/wall-o/tenants/wall-orthodontics/docker-compose.yml exec mongo mongosh -eval "rs.initiate()"
• 6. Verify workspace is reachable. curl -sI https://wall-orthodontics.wall-o.itpropartner.com returns 200 after Caddy cert issuance.
• 7. Login as admin and capture admin identity. curl -s https://wall-orthodontics.wall-o.itpropartner.com/api/v1/login -H "Content-Type: application/json" -d '{"user":"admin","password":"<from Vaultwarden>"}'
• 8. Register the agent bot user. curl -s .../api/v1/users.create -H "X-Auth-Token: <adminToken>" -H "X-User-Id: <adminUserId>" -d '{"name":"Wall-O Agent","username":"wallo.agent","email":"wallo-agent@wall-orthodontics.internal","password":"<from Vaultwarden>","roles":["bot"],"joinDefaultChannels":false,"verified":true}'
• 9. Create a personal access token for the bot. curl -s .../api/v1/users.createToken -H "X-Auth-Token: <adminToken>" -H "X-User-Id: <adminUserId>" -d '{"userId":"<botUserId>"}'; store botUserId plus botAuthToken in Vaultwarden.
• 10. Create the knowledge channel. curl -s .../api/v1/channels.create -H "X-Auth-Token: <adminToken>" -H "X-User-Id: <adminUserId>" -d '{"name":"employee-resources"}'
• 11. Attach the agent to the channel. curl -s .../api/v1/channels.addOwner -H "X-Auth-Token: <adminToken>" -H "X-User-Id: <adminUserId>" -d '{"roomId":"<roomId>","userId":"<botUserId>"}'
• 12. Confirm the bot can post. curl -s .../api/v1/chat.postMessage -H "X-Auth-Token: <botAuthToken>" -H "X-User-Id: <botUserId>" -d '{"roomId":"<roomId>","text":"Wall-O agent online."}'
• 13. Stand up the orchestrator skeleton plus Postgres + pgvector. docker compose -f /opt/wall-o/orchestrator/docker-compose.yml up -d with image pgvector/pgvector:pg16, port 127.0.0.1:5432, credentials from .env (Vaultwarden).
• 14. Run migrations to create tenants, channels, agents, kb_scope, documents, chunks tables (section 4.2 columns).
• 15. Seed the tenants row with rocket_chat_url, rocket_chat_admin_user_id, and rocket_chat_admin_token_ref for Wall Orthodontics, and the agents row with the bot user id and bot token reference; leave all tokens in Vaultwarden only.
• 16. Upload a test document. curl -s -F "file=@staff-handbook.pdf" -F "tenant_id=<tenantId>" -F "channel_id=<channelId>" https://api.wall-o.itpropartner.com/v1/documents; orchestrator stores the file and records kb_scope.
• 17. Index the document. Orchestrator chunks, embeds via admin-ai, and inserts vectors into pgvector with tenant_id plus channel_id on every row.
• 18. Retrieval smoke test. curl -s "https://api.wall-o.itpropartner.com/v1/retrieve?tenant_id=<tenantId>&channel_id=<channelId>&q=<question>" returns a scoped chunk.
• 19. Close the chat loop. Orchestrator polls the channel via channels.messages?roomId=<roomId> and replies through chat.postMessage using the bot identity; verify a staff question gets a grounded answer.
• 20. Verify mobile. Open the tenant subdomain in a mobile browser (install the PWA) and confirm chat renders and the bot replies.
• 21. Verify push. Configure Push settings in the workspace (gateway URL), send a direct message to a test user on a mobile device, confirm the notification arrives on the PWA or the Rocket.Chat mobile app pointed at the server.

4. Security and Compliance

4.1 No PHI

Scope is strictly internal staff knowledge. Hard controls: the M365 connector is limited to explicitly approved, non-patient document libraries; a pre-ingest filter rejects documents matching PHI markers (SSN, MRN, DOB plus name); no HIPAA features are enabled in Rocket.Chat; every ingestion and retrieval path is audited. If a document is suspected to contain PHI, ingestion is blocked and logged, not silently passed.

4.2 Per-tenant data isolation

Layer	Isolation
Chat	One Rocket.Chat workspace per tenant, with its own MongoDB instance and volume. No shared chat database.
Documents and vectors	Every chunks row carries tenant_id and channel_id; every retrieval query is prefixed with a tenant_id plus channel_id filter (kb_scope). Optional later hardening: schema-per-tenant.
Credentials	Bot tokens are per tenant and scoped to that workspace only; stored in Vaultwarden, referenced by id.
Routing	Tenant subdomains are isolated Caddy site blocks with no cross-tenant access.

4.3 Data residency

All components are self-hosted on netcup infrastructure in the chosen region. Chat, documents, and vectors never leave the self-hosted estate. LLM and embedding calls route through admin-ai, the internal LiteLLM proxy (DeepSeek V4 Pro primary), so prompts are processed inside the controlled stack and never carry PHI by scope.

4.4 Backups

Nightly job in /opt/wall-o/backups/run-backups.sh: per-tenant mongodump (via the loopback port) plus orchestrator pg_dump, encrypted, uploaded to Wasabi S3. Retention per the ITPP backup schedule with a documented RPO. Backups are restorable per tenant, matching the isolation boundary.

4.5 Secret management

All secrets live in Vaultwarden: per-tenant admin and bot credentials, Postgres passwords, M365 app credentials, admin-ai (LiteLLM) keys, and S3 keys. Repos and compose files carry .env.example placeholders only. No plaintext secrets in code, compose, or Caddy config. Deploy scripts pull values from Vaultwarden into .env at provision time; .env is gitignored.