itpp-infrastructure/hermes-skills/delegation-pattern.md

name, description, version, author

name	description	version	author
delegation-pattern	When and how to delegate tasks to subagents so the main loop stays responsive. The user upgraded to 8C/15G specifically for parallel subagent workers. Long-running tasks MUST be delegated.	1.4.0	Sho'Nuff

Delegation Pattern

The user expects me to stay responsive at all times. Any task taking more than a few seconds blocks the main conversation loop and freezes the gateway for all profiles (including Anita's). The upgraded 8C/15G VPS was chosen specifically to enable parallel subagent workers.

When to delegate (MUST criteria)

Delegate when the task:

• Takes more than ~5 seconds
• Involves HTML/mockup builds (15KB+ files)
• Requires browser navigation (SPA scraping, API doc extraction)
• Is research / data collection across multiple sources
• Involves batch file operations or multi-file edit passes
• Requires interacting with external APIs (CRM, DocuSeal, LetterStream)
• Is deploys or installations that take 30s+

When NOT to delegate

• Quick lookups (single web_search, one terminal command)
• Short answers or clarifications
• Direct responses to the user

How to delegate

delegate_task(
    goal="Clear description of what the subagent should accomplish",
    context="File paths, error messages, constraints, style preferences"
)

Always include a content-brevity rule when the output will be read by Germaine:

context+="\nCRITICAL: When preparing content for Germaine to build himself (website copy, 
page content, etc.), deliver OUTLINES only — section headers, key points, bullet lists, 
table structures. Do NOT write full prose paragraphs. He writes the copy; you provide the skeleton."

Delegation Model Architecture (Jul 18, 2026)

Three-layer delegation pipeline:

DEEPSEEK v4 Pro (bulk workers, 95% of tasks)
    → 5.85M tokens yesterday, $0.17
    → Coding, research, file ops, builds
    
    ↓ output
    
GPT-5.5 PM (main agent, verification)
    → Reviews all subagent output
    → Fixes small issues directly
    → Routes critical work to QA
    
    ↓ critical path only (~5%)
    
CLAUDE SONNET 5 QA (final review gate)
    → $3/$15 per 1M tokens via anthropic provider
    → Complex PR review, architecture decisions
    → ~$0.90/day at 5% routing

Level	Model	Provider	Cost/1M in/out	Best For
Workers	deepseek-v4-pro	admin-ai	~$0.03 total	Bulk coding, debugging, research
PM	gpt-5.5	admin-ai	$5/$30	Verification, routing, escalation
QA	claude-sonnet-5	anthropic	$3/$15	Critical code review, architecture

Two expert teams: web-design-integration-team and systems-networks-team. Subagents run deepseek-v4-pro. Claude Sonnet 5 used explicitly for review passes via the anthropic provider.

Two expert teams: web-design-integration-team and systems-networks-team. Subagents run deepseek-v4-pro. Claude Sonnet 5 used explicitly for review passes via the anthropic provider.

Subagent model routing

Delegation config (current)

Config change does NOT require gateway restart (corrected Jul 12, 2026)

Changes to delegation.* config keys take effect immediately on new delegations — no restart needed. The hermes config set delegation.model "claude-sonnet-4-6" proved this: the next subagent dispatched used Claude without a gateway restart.

What DOES need a restart: Changes to model.default (the conversation model itself) require a restart because the running agent loop reads it at startup.

What does NOT need a restart: delegation.* (model, provider, fallback), auxiliary.* (vision, compression), and most other non-core settings are re-read from config on each tool invocation.

Aggressive delegation mandate (Jul 12, 2026)

Germaine explicitly corrected the delegation approach: "I know you love this stuff but you can't do everything. Delegations to qualified subs is necessary. Hire subject matter experts that are the right fit for our demanding projects. They need to represent you and must deliver as expected."

This is a standing directive, not a suggestion. Any task that:

• Takes more than a few seconds
• Involves research, design, UI work, or multi-file operations
• Could be done by a qualified subagent in parallel

MUST be delegated. The default answer is "I'll hand this off to a sub" not "I'll do this myself."

This applies to ALL task classes: server work, UI design, research, code builds, email composition, database operations. The only exceptions are tasks requiring interactive user input (clarify), direct responses, or tasks so trivial they'd take longer to delegate than execute.

Subagent quality verification (Jul 12, 2026): Germaine's directive: subs must represent you and deliver as expected. This means:

1. Verify output before reporting - A subagent's self-report is not sufficient. Always check the deliverable yourself: stat the file, hit the endpoint, read the content. If a subagent built a dashboard mockup, open it in the browser.
2. Reject substandard work - If a subagent returns incomplete, broken, or low-effort work, do NOT relay it to the user as success. Either fix it yourself or re-delegate with stricter requirements.
3. Provide adequate context - Subagents have NO memory of the current session. Include file paths, exact errors, relevant skills, and quality criteria.
4. Prefer SME role labels - UI design sub gets better results than generic task sub.

Subagent hiring directive: Use role-specific subagents for each class of work. UI work goes to a design sub. Research goes to a research sub. Docker deployments go to an infrastructure sub. Each is an SME who represents Sho'Nuff's quality standards.

Fail-safe: subagent inherits parent model

If delegation.model config is stale (e.g. still set to OpenRouter-style name that 402s), delegation.subagent_model: inherit (or omitting delegation config entirely) causes subagents to use the CONVERSATION model (deepseek-chat) instead. This is slower but always works because deepseek-chat is available through both providers. The platform's subagent inherit model is inherit, checked at config read time:

delegation:
  subagent_model: inherit

When in doubt about model name accuracy, set subagent_model: inherit and dispatch — the subagent will work even if crossing the model name mismatch.

The conversation model (deepseek-chat) is separate — the user stays on a cheap/fast model while subagents do heavy work on Opus.

Delegation provider API-key resolution pitfall

delegate_task resolves delegation.provider through Hermes' runtime provider system, not always through providers.<name>.api_key in config. A config can contain a working providers.openrouter.api_key while delegate_task still fails with:

Delegation provider 'openrouter' resolved but has no API key. Set the appropriate environment variable or run 'hermes auth'.

If this happens, do not keep retrying delegation. Fix one of these ways:

1. Put the provider's expected env var in ~/.hermes/.env (e.g. OPENROUTER_API_KEY) and restart/reload as needed; or
2. Use direct delegation endpoint fields: delegation.base_url + delegation.api_key; or
3. Remove delegation.provider so subagents inherit the parent provider/client.

Verify with one tiny delegate_task before relying on subagents for production work.

Vision model for subagents

Vision model follows the same pattern. Set via config.yaml:

auxiliary:
  vision:
    model: gemini-pro-latest  # NOT openrouter/openai/gpt-4o
    provider: admin-ai

• admin-ai model: gemini-pro-latest (works, no OpenRouter credits)
• OpenRouter model: openrouter/openai/gpt-4o (works but consumes credits)
• Use gemini-pro-latest to avoid credit depletion.

Subagent result verification

Subagent result verification

Subagent summaries are SELF-REPORTS, not verified facts. Always verify before reporting success:

• File writes: stat the file, read back content
• API calls: re-test the endpoint
• State changes: check the actual system state
• Deployments/builds: check the target server, not just the subagent's workspace. Subagents (especially DeepSeek Chat) frequently write code + run mock tests locally, report "completed," but never deploy to the target. The deliverable must exist on the target machine — ls /root/docker/mcp-<name>/ on app1, not just tests passing on the subagent's workstation. A subagent claiming "Complete" with "Next Steps" to run is NOT complete.
• Email sends: check IMAP Sent folder for copy. SMTP 250 does not equal sent.
• Purge passes: check if ALL test data was removed, not just the obvious rows. The subagent's purge may miss: chatbot responses with hardcoded names, console.log stubs with claim references, nav avatar initials, header greeting text, stat cards in a different section.
• Provisioning/config changes: confirm the API actually returned data. A subagent that reports "API accessible" may have only checked HTTP status, not whether the response contained actual data vs an auth wall. Always verify by inspecting the response body yourself.

File collision pitfall

When a subagent modifies a file the parent session has read (or a sibling subagent has modified), ALWAYS re-read the file before patching it. The subagent's version may differ from the cached version in the parent context. A patch that succeeds on the cached version can corrupt or duplicate content in the actual file.

Background-process handoff failover

When hermes chat -q (background user query) times out without returning results, fall back to:

1. Check if the session DB has the results (session_search with the session_id shown in the spawn output).
2. If results are partial, re-dispatch as a subagent task via delegate_task() with a shorter, more focused goal.
3. Do NOT retry hermes chat -q with the same timeout — if it dragged once it will drag again. Use a subagent instead.

Purge completeness — the full sweep

When a subagent is tasked to "purge test data" or "remove sample data" from a set of pages, the subagent's purge pass nearly always misses some spots. After the subagent reports done, ALWAYS run your own grep/near/find over the files for known test identifiers (company names, claim numbers, dollar amounts, sample stats) before telling the user the purge is complete. Common misses:

• Hardcoded chatbot responses referencing removed sample data
• Greeting text in nav bars, headers, or avatar initials
• Console.log stubs with claim/company references
• Stat cards in sections the subagent didn't examine
• Welcome messages containing previous client names

Cross-session recovery after context loss (CRITICAL — Jul 9, 2026)

When Germaine comes back after a session reset or context loss (especially after a long day where 7,000+ messages and 42+ /queue deferrals were logged):

Do NOT ask him to re-explain everything. The full conversation is preserved in /root/.hermes/state.db. Use this procedure:

1. Run DB integrity check first: sqlite3 /root/.hermes/state.db "PRAGMA integrity_check;" — confirm "ok" before proceeding.
2. Find the relevant session: Query for sessions from the lost timeframe:

   SELECT s.id, datetime(s.started_at, 'unixepoch', 'localtime'), COUNT(m.id) as msg_count
   FROM sessions s JOIN messages m ON m.session_id = s.id
   WHERE s.source='telegram' AND m.timestamp > <timestamp_of_interest>
   GROUP BY s.id ORDER BY s.started_at DESC LIMIT 3;
   

   Use strftime('%s', '2026-07-08 16:00:00', '-4 hours') for EDT timestamps.
3. Extract all user messages to reconstruct what was asked and queued:

   SELECT id, timestamp, datetime(timestamp, 'unixepoch', 'localtime') as dt, substr(content, 1, 120)
   FROM messages WHERE session_id = '<id>' AND timestamp > <cutoff> AND role = 'user'
   ORDER BY timestamp;
   

4. Identify /queue items specifically: grep the user messages output for lines containing /queue. Each /queue <topic> is a deferred task that was NOT executed. These must be re-presented to Germaine as a summary.
5. Identify delegation results: grep for [ASYNC DELEGATION BATCH COMPLETE — these mark subagent work that was completed. The parent's response after each one shows how the result was consumed.
6. Identify decisions/user preferences: grep for "yes", "no", "changed", "remember", "don't" in user messages — these are preference signals that should update skills/memory.
7. Report a compact summary — do NOT dump the raw messages. Group by project (shark game, ops portal, DRE, Apex, DR audit, etc.) with key decisions and queued items.
8. Offer to deep-dive any specific topic — don't pre-emptively reconstruct everything. Let Germaine pick where to restart.

Critical: The session DB is the source of truth for cross-session recovery. The Telegram export feature (24-hour delay) is a fallback, not the primary recovery path. Always use local SQLite first.

User preference: trust the user's verification (Jul 10, 2026)

Germaine personally verifies his infrastructure before reporting status. If he says an app or service is working, it IS working. Do NOT run your own checks to "confirm" and certainly do NOT contradict him based on a failed check. Your tool output may be stale (DNS cache, wrong server, wrong path) — his browser and his eyes are not.

Pattern from tonight: Germaine said n8n was working on app1. I ran docker ps and saw zero containers. I reported n8n wasn't there. He was right — n8n was running, my check was flawed. He called this out explicitly: "I won't tell you something about an app or service working or not working without verifying it for myself."

Rule: When Germaine states infrastructure status, accept it as ground truth. If your tool output contradicts him, the tool output is wrong — investigate why, don't report it as fact.

User preference: brevity and directness

Germaine has repeatedly corrected verbosity. When reporting results:

• Do not narrate your process — say what happened, not how you got there
• Do not pad answers — deliver the outcome, not the rationale
• Prefer tables over paragraphs for structured data
• If there are multiple items, use a concise list (never a wall of text)
• Let the user ask for details — don't pre-emptively explain everything
• A single line is better than a paragraph when the result is simple

This applies to ALL responses, not just delegation summaries. The user has explicitly called out over-explaining as a pattern they dislike.

Communication prefix system (established Jul 12, 2026)

Germaine established a structured prefix system for emailing shonuff@germainebrown.com. When he sends an email (or any message), the first line or subject prefix tells me how to handle it:

Prefix	Meaning	What to do
(none)	Immediate command	Do it now, report back
[bg] or [delegate]	Background task	Dispatch subagent, results come back async
[queue]	Queue for later	Acknowledge and shelve until next time he checks in
[lookup]	Quick research	One-shot search, report findings once
[note]	Just FYI	Acknowledge, save if relevant, no action required

Implementation: Apply this to ALL incoming channels (email, Telegram, SMS), not just email. When the Master sends [bg] Check disk on app2, delegate to a subagent. When he sends [queue] Set up monitoring for voipsimplicity.com, acknowledge it as queued and move on. When he sends bare text, execute immediately.

Communication prefix system

Multi-peer WireGuard tunnel management (Jul 9, 2026)

When Core + app1-bu both connect to the same router via WG, each needs its own peer entry on the router. The router's config may still reference OLD public keys from a previous setup. Always verify against LIVE state on both sides.

See mikrotik-onboarding skill's Multi-Peer WG Tunnel Management section for the full procedure.

Heritner server inventory (Jul 9, 2026)

Full 10-server Hetzner inventory cataloged at /root/.hermes/references/hetzner-server-inventory.md. Key findings:

• ai.itpropartner.com — 92% disk full (18G left on 226G). Runs LiteLLM, Ollama, OpenWebUI. CRITICAL.
• 9 servers runnning, 1 (ai) needs urgent disk cleanup
• 4 servers with Docker: ai (6 containers), unms (9), hudu (5), docker (8), n8n (2)
• Warm standby (app1-bu) has WG tunnel to router, Tailscale active

When investigating server issues, consult this inventory first before SSHing in.

Fabrication rule (zero tolerance)

NEVER fabricate content — titles, closings, quotes, data, lists, or claims about what the user said. This is a zero-tolerance violation.

• If you don't know something, say so and ask
• Reference files exist for a reason — read them instead of guessing
• Session search is the correct tool for finding cross-session context
• A truthful "I don't have that" is better than a confident lie

Previous violation pattern (Jul 8, 2026): fabricated 8 closing quotes when asked for the list. The user called it out as unacceptable. The fabricated content was removed and replaced with the real reference files. This is why the fabrication rule exists.

Dual event handler pitfall (Jul 12, 2026)

When a subagent adds UI event handlers: do NOT use both inline onclick AND addEventListener for the same action on the same element. When both fire on the same click event, they cancel each other out — e.g., classList.toggle('open') fires twice, resulting in no visible change.

This happened with the ops portal hamburger menu: the nav.html template had onclick="...classList.toggle('open')" AND initNav() added toggle.addEventListener('click', ...classList.toggle('open')). Both toggled the same class on every tap, making the menu appear broken.

Fix: Remove the addEventListener version and keep only the inline handler. The inline handler always fires first and is more reliable across browsers.

Prevention for UI work: When writing JS that modifies the DOM, check whether the target element already has an inline event handler before adding another through JS. If both exist, one must be removed.

Subagent timeout on multi-step builds (Jul 11, 2026)

The default child_timeout_seconds: 600 (10 min) is often insufficient for builds involving pip install, systemd installation, and gateway restart testing.

Prevention:

• For complex builds, note in the context that the subagent may need extra time (15-20 min).
• Break the build into phases: Phase 1 (subagent) writes code + installs deps. Phase 2 (manual) handles wiring + testing.

Rescue pattern: When a subagent times out on a build task, inspect what it left behind before re-delegating:

• Check for partial artifacts (venv, server.py, service file, docker-compose.yml)
• These can often be completed manually in minutes rather than a full re-delegation

Jul 11 example: Super Search MCP server subagent timed out at 600s after writing 192 lines of server.py, creating venv, and installing deps. All artifacts were valid — just needed systemctl enable --now + config wiring + testing. Rescue took 5 minutes.