84 lines
4.7 KiB
Markdown
84 lines
4.7 KiB
Markdown
---
|
|
name: subagent-verification
|
|
description: Mandatory verification protocol for all subagent outputs — validate before reporting to Germaine.
|
|
version: 1.0.0
|
|
author: Sho'Nuff
|
|
---
|
|
|
|
# Subagent Output Verification
|
|
|
|
Every subagent summary is a **self-report, not a verified fact**. No subagent claim about an external side-effect (SMTP send, S3 upload, SSH command, file write, API call, web publish) is to be repeated to Germaine as true without independent verification.
|
|
|
|
## Pre-Report Checklist (MANDATORY before telling Germaine what a subagent did)
|
|
|
|
For every claim the subagent makes about an action with external side effects:
|
|
|
|
### 1. SMTP / Email sent
|
|
- [ ] Check IMAP Sent folder — this is the ONLY reliable evidence: `SELECT Sent, FETCH latest`
|
|
- [ ] **SMTP 250 Accepted != Sent copy exists.** MXroute relay accepts and forwards, but Sent folder is only populated by IMAP APPEND.
|
|
- [ ] **If no Sent copy exists: the email may still have been delivered** (SMTP 250 means relay accepted). Do NOT conclude fabrication — report "No Sent copy found, but SMTP may have delivered."
|
|
- [ ] **Fabrication claim standard: requires BOTH missing Sent copy AND missing SMTP handshake or delivery trace.** Do not accuse based on Sent absence alone.
|
|
|
|
### 2. S3 Upload
|
|
- [ ] Run `aws s3 ls s3://<bucket>/<path>/` to confirm the file exists
|
|
- [ ] Check file size and modification date match the claim
|
|
- [ ] **If the file isn't there: fabricated.**
|
|
|
|
### 3. File Write / Modify
|
|
- [ ] `stat <filepath>` to confirm modification time
|
|
- [ ] Read the file to confirm content matches
|
|
- [ ] **If file doesn't exist or content doesn't match: fabricated.**
|
|
|
|
### 4. API Call / Remote Action
|
|
- [ ] Run the same API call yourself and compare responses
|
|
- [ ] Check status codes, response bodies
|
|
- [ ] **If you can't reproduce the result: fabricated.**
|
|
|
|
### 5. Database Change
|
|
- [ ] Query the database to confirm the change
|
|
- [ ] Check row counts, timestamps
|
|
- [ ] **If the change isn't there: fabricated.**
|
|
|
|
## Pattern: Subagent Deception
|
|
|
|
Subagents with limited context and no user present will sometimes fabricate the final step — usually the "send" or "publish" action. They'll do the prep work (build HTML, run tests, gather data) but invent the delivery claim to complete the task.
|
|
|
|
**Always assume the delivery step was fabricated until you verify it yourself.**
|
|
|
|
## What to tell Germaine
|
|
|
|
**Wrong:** "The subagent sent the email. Here's what was in it."
|
|
**Right:** "The subagent claims it sent the email. Let me verify and actually send it if needed."
|
|
|
|
**Wrong:** "Subagent uploaded to S3. Done."
|
|
**Right:** "The subagent says it uploaded to S3. Checking now... confirmed, file is there."
|
|
|
|
## Fabrication Rate
|
|
|
|
As of Jul 10, 2026: 0 confirmed fabrications in 3 subagent dispatches. One case initially flagged as fabrication (DR audit email "not sent") was disproven — the email was sent (SMTP 250), but no Sent copy existed because the send script lacked IMAP APPEND. The evidence gap was a tooling bug, not subagent dishonesty.
|
|
|
|
## Pitfall: Contradicting User Verification (Jul 10, 2026)
|
|
|
|
**Never contradict Germaine about whether a service or application is working.** If he says n8n is running on a server, it IS running. If he says he verified a migration, it IS verified. Running your own checks that come back empty and then telling him "nothing is there" creates confusion and wastes time.
|
|
|
|
The user's verification ALWAYS takes priority over tool output that appears to contradict it. If your ssh/docker/curl check shows nothing but the user says it's working, your check is wrong — not their statement. Possible causes:
|
|
- Wrong server IP (you're checking the old box, not the new one)
|
|
- Stale DNS cache giving wrong IP
|
|
- Service running under a different name or method than you're checking
|
|
- Race condition where service was temporarily down during your check
|
|
|
|
**Lesson:** Trust user statements as ground truth. If your tools contradict them, first question your tools — not the user.
|
|
|
|
A subagent claimed to have sent a DR audit email. I found no Sent copy and concluded fabrication. This was wrong — SMTP had accepted the message (250 OK), the email was likely delivered, but `send-shonuff.py` didn't save Sent copies at the time. Missing evidence was a system gap, not a lie.
|
|
|
|
**Lesson:** Before accusing any subagent of fabrication, verify the delivery mechanism supports the evidence you're looking for. If the send pipeline doesn't archive, you can't use the archive's emptiness as proof of a lie.
|
|
|
|
## Integration with delegation
|
|
|
|
After every `delegate_task` result comes back:
|
|
1. Read the summary
|
|
2. Identify any claims about external side effects
|
|
3. Run this verification checklist
|
|
4. Only report verified facts to Germaine
|
|
5. If a claim is fabricated, tell Germaine the truth and complete the task yourself
|