From 11a1110b81746244264c4903932134ad2780f29b Mon Sep 17 00:00:00 2001 From: root Date: Sat, 8 Aug 2026 12:55:30 -0400 Subject: [PATCH] 72hr audit: 4 new docs, port fixes, project-log updated --- api-master-list.md | 6 +- docs/72hr-review-2026-08-08.md | 191 ++++++++++++++++++ docs/ops-v1-retirement.md | 53 +++++ docs/osint-person-super-search-integration.md | 55 +++++ docs/project-log.md | 60 ++++++ docs/super-search-v2.4.0-client-tracking.md | 76 +++++++ 6 files changed, 438 insertions(+), 3 deletions(-) create mode 100644 docs/72hr-review-2026-08-08.md create mode 100644 docs/ops-v1-retirement.md create mode 100644 docs/osint-person-super-search-integration.md create mode 100644 docs/super-search-v2.4.0-client-tracking.md diff --git a/api-master-list.md b/api-master-list.md index ddf94d5..b61b9ee 100644 --- a/api-master-list.md +++ b/api-master-list.md @@ -1,6 +1,6 @@ # API Master List - IT Pro Partner -**Owner:** Sho'Nuff (Hermes) | **Last updated:** 2026-07-31 +**Owner:** Sho'Nuff (Hermes) | **Last updated:** 2026-08-08 **Purpose:** Single inventory of every API used across ITPP projects. Categories: Internal (self-hosted/our own), External (free or no key), Paid External (subscription/usage-based). No keys stored here - credentials live in `/root/.hermes/.env`, `.aws/credentials`, or Vaultwarden. --- @@ -30,8 +30,8 @@ | Camofox Browser | Camofox (self-hosted) | Core :9377 | Browser automation backend | | Twenty CRM | Twenty (self-hosted) | Core :3003 | CRM | | Uptime Kuma | Uptime Kuma (self-hosted) | Core :3001 | Service monitoring, status pages | -| Grafana | Grafana (self-hosted) | Core :3000 | Dashboards | -| Prometheus | Prometheus (self-hosted) | Core | Metrics collection | +| Grafana | Grafana (self-hosted) | Core :3002 | Dashboards | +| Prometheus | Prometheus (self-hosted) | Core :9090 | Metrics collection, Super Search scraping | | Telegraf | Telegraf (self-hosted) | Core | Metrics agent | | MikroTik Exporter | swoga (self-hosted) | Core :9436 | MikroTik router metrics | | SNMP HTTP Server | ITPP script | Core :9274 | SNMP device data | diff --git a/docs/72hr-review-2026-08-08.md b/docs/72hr-review-2026-08-08.md new file mode 100644 index 0000000..fa20815 --- /dev/null +++ b/docs/72hr-review-2026-08-08.md @@ -0,0 +1,191 @@ +# 72-Hour Project & Documentation Audit: Aug 5-8, 2026 + +**Report generated:** 2026-08-08 +**Scope:** All projects, infrastructure changes, and documentation health +**Methodology:** Session search + live infrastructure verification + documentation cross-reference + +--- + +## 1. Projects & Changes Cataloged (Aug 5-8) + +### Infrastructure Changes (Verified Live) + +| Change | Before | After | Verified | +|--------|--------|-------|----------| +| Super Search binding | 127.0.0.1:8899 | 0.0.0.0:8899 | ss -tlnp confirms 0.0.0.0 | +| UFW rule for Prometheus | none | allow 172.17.0.0/16 to :8899 | ufw status confirms | +| Prometheus scrape target | none | 172.17.0.1:8899/metrics @30s | prometheus.yml confirms | +| Grafana dashboard | none | "Super Search - Client Tracking" /d/ffuktvmgcpkhse | Grafana confirms | +| Grafana admin password | unknown | Reset to standard via grafana-cli | Login confirmed | +| /var/www/ops/ cleanup | *.html, css/, js/ present | data/ only | ls confirms | +| /var/www/ops/data/ | in /var/www/ops/ | migrated to /var/www/ops-v2/data/ | Files present | +| Caddy ops redirect | no redirect | / -> /v2/ 301 | curl confirms | +| 8 Python scripts | /var/www/ops/ paths | /var/www/ops-v2/ paths | Scripts updated | + +### New Deployments + +| Project | Host | Port/URL | Status | +|---------|------|----------|--------| +| Buzz Nostr Relay | app3 | buzz.iamgmb.com | Live, closed relay | +| Moore Sunny Daze (Beach Direct) | Core | :8911 | Backend built | + +### Features & Enhancements + +| Project | Change | Tracking | +|---------|--------|----------| +| Super Search v2.4.0 | Client-ID metrics via X-Client-Id middleware | Prometheus + Grafana | +| OSINT Person MCP | super_search.py MCP client module | Calls Super Search tools | +| Ops v1 Retirement | Orphaned HTML/CSS/JS removed, data migrated | Redirect to /v2/ | + +### Planning & Investigation + +| Topic | Status | +|-------|--------| +| Hermes Mission Control (Hermy HQ) | Scoped, pending host/domain decision | +| Grafana Dashboard Auth | Basic auth plugin investigated | +| Infrastructure Gap Assessment | 65+ services audited, 12 missing backups flagged | +| Git Structure Audit | 40 repos audited, credentials leak found | +| Hermes Conduit iOS integration | Investigated, on hold | + +--- + +## 2. Documentation Health + +### Docs Created (3 new) + +| Doc | Path | Covers | +|-----|------|--------| +| Super Search v2.4.0 | docs/super-search-v2.4.0-client-tracking.md | Client-ID tracking, Prometheus, Grafana, binding, UFW | +| Ops v1 Retirement | docs/ops-v1-retirement.md | File cleanup, data migration, Caddy redirect, script updates | +| OSINT Person MCP Integration | docs/osint-person-super-search-integration.md | super_search.py client module, MCP-to-MCP architecture | + +### Docs Updated (3 stale) + +| Doc | Stale Issue | Fix | +|-----|------------|-----| +| api-master-list.md | Grafana port listed as :3000 | Fixed to :3002 | +| api-master-list.md | Prometheus port blank | Added :9090 | +| api-master-list.md | Last updated: 2026-07-31 | Updated to 2026-08-08 | +| project-log.md | No entries past Jul 29 | Added 10 entries for Aug 5-8 | +| dependency-diagram.html | Generated July 6 | Updated to Aug 8, 4 fixes | +| dependency-diagram.html | app1-bu: CPX11, Offline | Fixed to CPX21, Warm Standby | +| dependency-diagram.html | Pending UISP/UniFi on app3 | Fixed to Running on App2 | +| dependency-diagram.html | 13 skills | Updated to 50+ skills | + +### Previously Existing Docs Confirmed Current + +| Doc | Coverage | Notes | +|-----|----------|-------| +| dns-records.md | All DNS records | app1-bu fix still pending (5.161.114.8 -> 5.161.225.131) | +| super-search-enhancement-plan.md | Super Search roadmap | Created just before audit window | +| infrastructure-gap-assessment-2026-08-04.md | Full infra audit | Aug 4, within window | +| git-audit-2026-08-07.md | Git repo audit | Aug 7, within window | +| projects/beachdirect.md | Beach Direct | Comprehensive | +| projects/buzz-agent-integration-spec.md | Buzz integration spec | 587 lines, thorough | +| projects/hotnow.md, hotnow-phase1.md | HotNow | Current | +| projects/intelsight.md | IntelSight | Current | +| projects/ops-portal*.md | Ops Portal | Current | +| backup-plan.md | Backup schedule | Current | + +### Remaining Stale Docs (Not Urgent) + +| Doc | Issue | Priority | +|-----|-------|----------| +| dr-issue-log.md | Last updated Jul 22, no Aug entries | Low (no new DR issues) | +| dns-records.md | Updated date: Jul 17, app1-bu still pending | Low (no DNS changes) | + +--- + +## 3. Infrastructure State Verification + +### Port Bindings (verified live) + +| Service | Expected | Actual | Match | +|---------|----------|--------|-------| +| Super Search | 0.0.0.0:8899 | 0.0.0.0:8899 | OK | +| OSINT Person MCP | 127.0.0.1:8902 | 127.0.0.1:8902 | OK | +| Ops Portal | 127.0.0.1:8090 | 127.0.0.1:8090 | OK | +| Grafana | :3002 | :3002 | OK | +| Prometheus | :9090 | Docker:9090 | OK | +| OSINT Person MCP client | super_search.py exists | /root/docker/osint-person-mcp/super_search.py | OK | + +### Firewall Rules (verified live) + +| Rule | Status | +|------|--------| +| 172.17.0.0/16 -> 8899/tcp ALLOW | OK | + +### Ops v1 Cleanup (verified live) + +| Path | Expected | Actual | +|------|----------|--------| +| /var/www/ops/ | data/ only | data/ only | +| /var/www/ops-v2/data/ | Has migrated files | ft360-devices.json, ft360-geocode-cache.json, ops-status.json, reolink-status.json, script-contents.json | + +### Prometheus Config (verified live) + +``` +- job_name: super-search + scrape_interval: 30s + static_configs: + - targets: + - 172.17.0.1:8899 + metrics_path: /metrics +``` + +--- + +## 4. Open Items & Recommendations + +### Immediate + +1. **app1-bu DNS record** -- Still pointing 5.161.114.8, should be 5.161.225.131. This is the oldest open item (since Jul 17). Needs Germaine to update at SiteGround. + +2. **Git audit findings** -- Hardcoded credentials in scripts repo and 13.6 MB blob in hermes-skills need remediation. See git-audit-2026-08-07.md. + +3. **Duplicate services** -- Twenty CRM on both Core and App1. SearXNG on both Core and App1. Gap assessment recommended shutting down stale Core instances. + +### This Week + +4. **Backup gaps** -- 12 services flagged with no backup in gap assessment. Top priority: Ragflow (App2), Mattermost (App1), Wazuh (App1). + +5. **Mission Control** -- Pending user decision on host and domain. Once decided, create project doc. + +6. **API master list** -- 14 services still missing. Add Super Search metrics endpoint, Super Search /metrics, and updated client list. + +### Documentation Gaps (from gap assessment -- not yet addressed) + +7. **22+ services** still have no project documentation. Most critical: Mattermost, n8n, Ragflow, Wazuh (production data, no docs, some missing backups). + +--- + +## 5. Files Modified During This Audit + +### Created + +- `/root/projects/itpp-infrastructure/docs/72hr-review-2026-08-08.md` -- This report +- `/root/projects/itpp-infrastructure/docs/super-search-v2.4.0-client-tracking.md` +- `/root/projects/itpp-infrastructure/docs/ops-v1-retirement.md` +- `/root/projects/itpp-infrastructure/docs/osint-person-super-search-integration.md` + +### Updated + +- `/root/projects/itpp-infrastructure/api-master-list.md` (Grafana port, Prometheus port, timestamp) +- `/root/projects/itpp-infrastructure/docs/project-log.md` (Aug 5-8 entries) +- `/root/portal-mockup/dependency-diagram.html` (app1-bu, App2 status, skill count, date) + +### Verified (read-only) + +- `/root/projects/itpp-infrastructure/dns-records.md` +- `/root/projects/itpp-infrastructure/backup-plan.md` +- `/root/.hermes/references/dr-issue-log.md` +- `/root/docker/monitoring/prometheus/prometheus.yml` +- All project docs in docs/ and projects/ + +--- + +## 6. Session Coverage + +The Aug 5-8 window yielded sessions on: Moore Sunny Daze/Beach Direct, Hermes Mission Control, Buzz Nostr relay, Git audit, Grafana dashboard auth, and Hermes Conduit. The specific infrastructure work (Super Search v2.4.0, Ops v1 retirement, Grafana password reset, osint-person MCP integration, binding change, Prometheus config) was mostly executed within an Aug 1 subagent delegation session and as subagent tasks -- all changes were verified live on infrastructure. + +**Session search hit rate:** 12 of 13 known projects found via 15+ queries. The Super Search v2.4.0 work was confirmed via infrastructure state, not session history. diff --git a/docs/ops-v1-retirement.md b/docs/ops-v1-retirement.md new file mode 100644 index 0000000..acd6a64 --- /dev/null +++ b/docs/ops-v1-retirement.md @@ -0,0 +1,53 @@ +# Ops v1 Retirement -- August 2026 + +**Created:** 2026-08-08 +**Status:** Complete + +--- + +## Summary + +Ops v1 (legacy HTML pages in `/var/www/ops/`) was retired and replaced by Ops v2 (SPA at `/var/www/ops-v2/`). All orphaned HTML, CSS, and JS files were removed. Data files were migrated. Caddy and 8 Python scripts were updated. Root redirect added: `ops.itpropartner.com` → `ops.itpropartner.com/v2`. + +## Changes + +### Files Removed +- `/var/www/ops/*.html` — all legacy dashboard pages +- `/var/www/ops/css/` — legacy stylesheets +- `/var/www/ops/js/` — legacy scripts + +### Files Migrated +- `/var/www/ops/data/` → `/var/www/ops-v2/data/` + - `ft360-devices.json`, `ft360-geocode-cache.json`, `ops-status.json`, `reolink-status.json`, `script-contents.json` + +### Caddy Config +```caddy +ops.itpropartner.com { + redir / /v2/ 301 + redir /v2 /v2/ 301 + handle_path /v2/* { + root * /var/www/ops-v2/ + file_server + } + reverse_proxy 127.0.0.1:8090 + ... +} +``` + +### Script Updates +8 Python scripts referencing `/var/www/ops/` paths were updated to use `/var/www/ops-v2/`. + +## Current State + +- Ops v2 SPA: `/var/www/ops-v2/index.html` +- Backend API: `127.0.0.1:8090` (ops-portal systemd service) +- Data directory: `/var/www/ops-v2/data/` +- Root redirect: `ops.itpropartner.com` → `/v2/` (301) + +## Verification + +``` +ls /var/www/ops/ -> data/ (only) +ls /var/www/ops-v2/ -> index.html, data/, ... +curl -I ops.itpropartner.com -> 301 -> /v2/ +``` diff --git a/docs/osint-person-super-search-integration.md b/docs/osint-person-super-search-integration.md new file mode 100644 index 0000000..c76f4a9 --- /dev/null +++ b/docs/osint-person-super-search-integration.md @@ -0,0 +1,55 @@ +# OSINT Person MCP -- Super Search Integration + +**Created:** 2026-08-08 +**Service:** OSINT Person MCP (Core, port 8902) +**Integration:** Super Search MCP (Core, port 8899) + +--- + +## Overview + +The OSINT Person MCP now integrates with Super Search via a dedicated client module. This mirrors the IntelSight pattern: an MCP server that calls Super Search tools through the local MCP endpoint at `http://127.0.0.1:8899/mcp`. + +## Architecture + +``` +OSINT Person MCP (port 8902) + -> super_search.py (MCP client module) + -> http://127.0.0.1:8899/mcp (Super Search MCP endpoint) + -> Super Search tools (web_search, web_extract, etc.) +``` + +## Files + +| File | Purpose | +|------|---------| +| `/root/docker/osint-person-mcp/super_search.py` | MCP client module (5.6K) | +| `/root/docker/osint-person-mcp/server.py` | Main OSINT Person server | +| `/root/docker/super-search/server.py` | Super Search MCP (referenced as dependency) | + +## Client Module (super_search.py) + +The module provides MCP client wrappers for Super Search tools: +- Call Super Search via `http://127.0.0.1:8899/mcp` +- Tool passthrough: any Super Search tool is available to OSINT Person +- Pattern mirrors IntelSight's `intelsight_api.py` + +## Clients + +| Client | Role | +|--------|------| +| `hermes` | Hermes Agent skip tracing tasks | +| `dre-osint` | DRE background research | + +## Service Status + +``` +systemctl is-active osint-person-mcp -> active +ss -tlnp | grep 8902 -> 127.0.0.1:8902 +``` + +## Related + +- Super Search v2.4.0: `/root/projects/itpp-infrastructure/docs/super-search-v2.4.0-client-tracking.md` +- IntelSight API: Core :8099 +- DRE MCP: Core :8900 diff --git a/docs/project-log.md b/docs/project-log.md index 6bcbaef..6692e69 100644 --- a/docs/project-log.md +++ b/docs/project-log.md @@ -67,6 +67,66 @@ ### Skills Updated - 10 skills patched: docker-service-deployment, home-lab-*, server-architecture-plan, twilio-10dlc, vaultwarden-management, voip-portal, hudu, syncromsp, recurring-information-scout +## 2026-08-05 through 2026-08-08 + +### Super Search v2.4.0 -- Client-ID Metrics Tracking +- Added Starlette middleware to intercept `X-Client-Id` header on every MCP call +- Prometheus counters per client (`hermes`, `intelsight`, `dre-osint`, `verdicttank`) and per tool +- Metrics exposed at `:8899/metrics`, scraped by Prometheus every 30s +- Grafana dashboard "Super Search - Client Tracking" at `/d/ffuktvmgcpkhse` on core:3002 +- Super Search binding changed from 127.0.0.1:8899 to 0.0.0.0:8899 for Docker access +- UFW rule added: allow 172.17.0.0/16 to port 8899 +- Prometheus scrape config added for super-search job at 172.17.0.1:8899/metrics +- Docs: `/root/projects/itpp-infrastructure/docs/super-search-v2.4.0-client-tracking.md` + +### OSINT Person MCP -- Super Search Integration +- Created `/root/docker/osint-person-mcp/super_search.py` MCP client module +- Calls Super Search tools via `http://127.0.0.1:8899/mcp` +- Mirrors IntelSight pattern for MCP-to-MCP tool delegation +- Docs: `/root/projects/itpp-infrastructure/docs/osint-person-super-search-integration.md` + +### Ops v1 Retirement +- Removed all orphaned `/var/www/ops/*.html`, `css/`, `js/` +- Migrated `/var/www/ops/data/` to `/var/www/ops-v2/data/` +- Updated Caddy: root redirect `ops.itpropartner.com` to `/v2/` (301) +- Updated 8 Python scripts referencing old ops paths +- Docs: `/root/projects/itpp-infrastructure/docs/ops-v1-retirement.md` + +### Grafana Admin Password Reset +- Reset admin password to standard credentials via `grafana-cli admin reset-admin-password` +- Grafana running on Core port 3002 (not 3000 as previously documented) + +### Moore Sunny Daze / Beach Direct +- Built internal product backend (FastAPI on port 8911, Core) for Moore Sunny Daze +- Fully documented Beach Direct as a standalone public product +- Project docs: `/root/projects/itpp-infrastructure/projects/beachdirect.md` +- Internal docs: `/root/projects/mooresunnydaze/docs/beach-direct-project.md` + +### Buzz Nostr Relay +- Deployed Buzz self-hosted relay on app3 (152.53.241.111) via CloudPanel Docker/Nginx +- Live at `https://buzz.iamgmb.com` +- Closed-relay membership, Postgres + Redis + MinIO backend +- Project spec: `/root/projects/itpp-infrastructure/projects/buzz-agent-integration-spec.md` + +### Hermes Mission Control (Planning) +- Investigated Sharbel's Hermes Mission Control template (Next.js dashboard + Postgres + Bridge) +- Architecture scoped: Dashboard host, Postgres setup, domain selection +- Pending user decision on host and domain before build + +### Git Structure Audit +- Full audit of all 40 Gitea repos + local repos under /root/projects/ +- Critical findings: hardcoded credentials in scripts repo, 13.6 MB blob in hermes-skills, missing .gitignore on 33/35 repos +- Docs: `/root/projects/itpp-infrastructure/docs/git-audit-2026-08-07.md` + +### Grafana Dashboard Auth +- Investigated Grafana basic auth plugin for external dashboard access +- Generated password hash for Hermes Conduit iOS app dashboard integration + +### Infrastructure Gap Assessment +- Subagent audit: 65+ services across 5 hosts, identified 12 services with no backup, 14 missing from API list +- Duplicate services found: Twenty CRM (Core + App1), SearXNG (Core + App1) +- Docs: `/root/projects/itpp-infrastructure/docs/infrastructure-gap-assessment-2026-08-04.md` + ## 2026-07-29 ### Village Express — Client Project diff --git a/docs/super-search-v2.4.0-client-tracking.md b/docs/super-search-v2.4.0-client-tracking.md new file mode 100644 index 0000000..e6f9d7c --- /dev/null +++ b/docs/super-search-v2.4.0-client-tracking.md @@ -0,0 +1,76 @@ +# Super Search v2.4.0 -- Client-ID Metrics Tracking + +**Created:** 2026-08-08 +**Service:** Super Search MCP (Core, port 8899) +**Feature:** Client-ID tracking via Prometheus metrics + Grafana dashboard + +--- + +## Overview + +Super Search v2.4.0 adds per-client usage tracking. A Starlette middleware intercepts the `X-Client-Id` header on every MCP call and increments Prometheus counters per client and per tool. Metrics are exposed at `:8899/metrics` and scraped by Prometheus every 30s. A Grafana dashboard visualizes usage. + +## Architecture + +``` +Client (hermes/intelsight/dre-osint/verdicttank) + -> X-Client-Id header + -> Super Search Middleware (intercepts, increments Prometheus counter) + -> MCP tool handler + -> :8899/metrics (Prometheus endpoint) + -> Prometheus (Docker, scrapes 172.17.0.1:8899/metrics every 30s) + -> Grafana (Dashboard: "Super Search - Client Tracking" at /d/ffuktvmgcpkhse) +``` + +## Clients Tracked + +| Client | Purpose | +|--------|---------| +| `hermes` | Hermes Agent's own Super Search usage | +| `intelsight` | IntelSight product backend | +| `dre-osint` | Debt Recovery Experts skip tracing | +| `verdicttank` | VerdictTank research | + +Fallback: calls without `X-Client-Id` header are logged as `anonymous`. + +## Key Changes + +### Super Search (server.py) +- Middleware added: intercepts `X-Client-Id` header on `/mcp` POST +- Prometheus counters: `ss_tool_calls_total{client, tool}`, `ss_tool_duration_seconds{client, tool}` +- `/metrics` endpoint exposed on port 8899 + +### Prometheus (prometheus.yml) +- Job: `super-search` +- Target: `172.17.0.1:8899` (Docker bridge to host) +- Scrape interval: 30s +- Config: `/root/docker/monitoring/prometheus/prometheus.yml` + +### Grafana +- Dashboard UID: `ffuktvmgcpkhse` +- Title: "Super Search - Client Tracking" +- Access: `https://core:3002/d/ffuktvmgcpkhse` +- Panels: tool calls per client, duration distribution, top tools + +### Firewall (UFW) +- Rule: allow 172.17.0.0/16 to port 8899/tcp +- Reason: Prometheus Docker container needs host access + +### Super Search Binding +- Changed from `127.0.0.1:8899` to `0.0.0.0:8899` +- Required because Docker containers (Prometheus) cannot reach 127.0.0.1 on the host + +## Verification + +``` +ss -tlnp | grep 8899 -> 0.0.0.0:8899 (bound to all interfaces) +curl -s 172.17.0.1:8899/metrics | grep ss_tool -> counters present +ufw status | grep 8899 -> ALLOW 172.17.0.0/16 +``` + +## Related Docs + +- Super Search Enhancement Plan: `/root/projects/itpp-infrastructure/docs/super-search-enhancement-plan.md` +- Server: `/root/docker/super-search/server.py` +- Systemd: `super-search.service` +- Prometheus config: `/root/docker/monitoring/prometheus/prometheus.yml`