ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: nest 19 files into audit/ clients/ infrastructure/ monitoring/ projects/ super-search/

Browse Source
This commit is contained in:
root
2026-08-08 13:04:55 -04:00
parent 11a1110b81
commit a522d11c35
20 changed files with 927 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/audit/72hr-review-2026-08-08.md
+191
View File
@@ -0,0 +1,191 @@
# 72-Hour Project & Documentation Audit: Aug 5-8, 2026
**Report generated:** 2026-08-08
**Scope:** All projects, infrastructure changes, and documentation health
**Methodology:** Session search + live infrastructure verification + documentation cross-reference
---
## 1. Projects & Changes Cataloged (Aug 5-8)
### Infrastructure Changes (Verified Live)
| Change | Before | After | Verified |
|--------|--------|-------|----------|
| Super Search binding | 127.0.0.1:8899 | 0.0.0.0:8899 | ss -tlnp confirms 0.0.0.0 |
| UFW rule for Prometheus | none | allow 172.17.0.0/16 to :8899 | ufw status confirms |
| Prometheus scrape target | none | 172.17.0.1:8899/metrics @30s | prometheus.yml confirms |
| Grafana dashboard | none | "Super Search - Client Tracking" /d/ffuktvmgcpkhse | Grafana confirms |
| Grafana admin password | unknown | Reset to standard via grafana-cli | Login confirmed |
| /var/www/ops/ cleanup | *.html, css/, js/ present | data/ only | ls confirms |
| /var/www/ops/data/ | in /var/www/ops/ | migrated to /var/www/ops-v2/data/ | Files present |
| Caddy ops redirect | no redirect | / -> /v2/ 301 | curl confirms |
| 8 Python scripts | /var/www/ops/ paths | /var/www/ops-v2/ paths | Scripts updated |
### New Deployments
| Project | Host | Port/URL | Status |
|---------|------|----------|--------|
| Buzz Nostr Relay | app3 | buzz.iamgmb.com | Live, closed relay |
| Moore Sunny Daze (Beach Direct) | Core | :8911 | Backend built |
### Features & Enhancements
| Project | Change | Tracking |
|---------|--------|----------|
| Super Search v2.4.0 | Client-ID metrics via X-Client-Id middleware | Prometheus + Grafana |
| OSINT Person MCP | super_search.py MCP client module | Calls Super Search tools |
| Ops v1 Retirement | Orphaned HTML/CSS/JS removed, data migrated | Redirect to /v2/ |
### Planning & Investigation
| Topic | Status |
|-------|--------|
| Hermes Mission Control (Hermy HQ) | Scoped, pending host/domain decision |
| Grafana Dashboard Auth | Basic auth plugin investigated |
| Infrastructure Gap Assessment | 65+ services audited, 12 missing backups flagged |
| Git Structure Audit | 40 repos audited, credentials leak found |
| Hermes Conduit iOS integration | Investigated, on hold |
---
## 2. Documentation Health
### Docs Created (3 new)
| Doc | Path | Covers |
|-----|------|--------|
| Super Search v2.4.0 | docs/super-search-v2.4.0-client-tracking.md | Client-ID tracking, Prometheus, Grafana, binding, UFW |
| Ops v1 Retirement | docs/ops-v1-retirement.md | File cleanup, data migration, Caddy redirect, script updates |
| OSINT Person MCP Integration | docs/osint-person-super-search-integration.md | super_search.py client module, MCP-to-MCP architecture |
### Docs Updated (3 stale)
| Doc | Stale Issue | Fix |
|-----|------------|-----|
| api-master-list.md | Grafana port listed as :3000 | Fixed to :3002 |
| api-master-list.md | Prometheus port blank | Added :9090 |
| api-master-list.md | Last updated: 2026-07-31 | Updated to 2026-08-08 |
| project-log.md | No entries past Jul 29 | Added 10 entries for Aug 5-8 |
| dependency-diagram.html | Generated July 6 | Updated to Aug 8, 4 fixes |
| dependency-diagram.html | app1-bu: CPX11, Offline | Fixed to CPX21, Warm Standby |
| dependency-diagram.html | Pending UISP/UniFi on app3 | Fixed to Running on App2 |
| dependency-diagram.html | 13 skills | Updated to 50+ skills |
### Previously Existing Docs Confirmed Current
| Doc | Coverage | Notes |
|-----|----------|-------|
| dns-records.md | All DNS records | app1-bu fix still pending (5.161.114.8 -> 5.161.225.131) |
| super-search-enhancement-plan.md | Super Search roadmap | Created just before audit window |
| infrastructure-gap-assessment-2026-08-04.md | Full infra audit | Aug 4, within window |
| git-audit-2026-08-07.md | Git repo audit | Aug 7, within window |
| projects/beachdirect.md | Beach Direct | Comprehensive |
| projects/buzz-agent-integration-spec.md | Buzz integration spec | 587 lines, thorough |
| projects/hotnow.md, hotnow-phase1.md | HotNow | Current |
| projects/intelsight.md | IntelSight | Current |
| projects/ops-portal*.md | Ops Portal | Current |
| backup-plan.md | Backup schedule | Current |
### Remaining Stale Docs (Not Urgent)
| Doc | Issue | Priority |
|-----|-------|----------|
| dr-issue-log.md | Last updated Jul 22, no Aug entries | Low (no new DR issues) |
| dns-records.md | Updated date: Jul 17, app1-bu still pending | Low (no DNS changes) |
---
## 3. Infrastructure State Verification
### Port Bindings (verified live)
| Service | Expected | Actual | Match |
|---------|----------|--------|-------|
| Super Search | 0.0.0.0:8899 | 0.0.0.0:8899 | OK |
| OSINT Person MCP | 127.0.0.1:8902 | 127.0.0.1:8902 | OK |
| Ops Portal | 127.0.0.1:8090 | 127.0.0.1:8090 | OK |
| Grafana | :3002 | :3002 | OK |
| Prometheus | :9090 | Docker:9090 | OK |
| OSINT Person MCP client | super_search.py exists | /root/docker/osint-person-mcp/super_search.py | OK |
### Firewall Rules (verified live)
| Rule | Status |
|------|--------|
| 172.17.0.0/16 -> 8899/tcp ALLOW | OK |
### Ops v1 Cleanup (verified live)
| Path | Expected | Actual |
|------|----------|--------|
| /var/www/ops/ | data/ only | data/ only |
| /var/www/ops-v2/data/ | Has migrated files | ft360-devices.json, ft360-geocode-cache.json, ops-status.json, reolink-status.json, script-contents.json |
### Prometheus Config (verified live)
```
- job_name: super-search
scrape_interval: 30s
static_configs:
- targets:
- 172.17.0.1:8899
metrics_path: /metrics
```
---
## 4. Open Items & Recommendations
### Immediate
1. **app1-bu DNS record** -- Still pointing 5.161.114.8, should be 5.161.225.131. This is the oldest open item (since Jul 17). Needs Germaine to update at SiteGround.
2. **Git audit findings** -- Hardcoded credentials in scripts repo and 13.6 MB blob in hermes-skills need remediation. See git-audit-2026-08-07.md.
3. **Duplicate services** -- Twenty CRM on both Core and App1. SearXNG on both Core and App1. Gap assessment recommended shutting down stale Core instances.
### This Week
4. **Backup gaps** -- 12 services flagged with no backup in gap assessment. Top priority: Ragflow (App2), Mattermost (App1), Wazuh (App1).
5. **Mission Control** -- Pending user decision on host and domain. Once decided, create project doc.
6. **API master list** -- 14 services still missing. Add Super Search metrics endpoint, Super Search /metrics, and updated client list.
### Documentation Gaps (from gap assessment -- not yet addressed)
7. **22+ services** still have no project documentation. Most critical: Mattermost, n8n, Ragflow, Wazuh (production data, no docs, some missing backups).
---
## 5. Files Modified During This Audit
### Created
- `/root/projects/itpp-infrastructure/docs/72hr-review-2026-08-08.md` -- This report
- `/root/projects/itpp-infrastructure/docs/super-search-v2.4.0-client-tracking.md`
- `/root/projects/itpp-infrastructure/docs/ops-v1-retirement.md`
- `/root/projects/itpp-infrastructure/docs/osint-person-super-search-integration.md`
### Updated
- `/root/projects/itpp-infrastructure/api-master-list.md` (Grafana port, Prometheus port, timestamp)
- `/root/projects/itpp-infrastructure/docs/project-log.md` (Aug 5-8 entries)
- `/root/portal-mockup/dependency-diagram.html` (app1-bu, App2 status, skill count, date)
### Verified (read-only)
- `/root/projects/itpp-infrastructure/dns-records.md`
- `/root/projects/itpp-infrastructure/backup-plan.md`
- `/root/.hermes/references/dr-issue-log.md`
- `/root/docker/monitoring/prometheus/prometheus.yml`
- All project docs in docs/ and projects/
---
## 6. Session Coverage
The Aug 5-8 window yielded sessions on: Moore Sunny Daze/Beach Direct, Hermes Mission Control, Buzz Nostr relay, Git audit, Grafana dashboard auth, and Hermes Conduit. The specific infrastructure work (Super Search v2.4.0, Ops v1 retirement, Grafana password reset, osint-person MCP integration, binding change, Prometheus config) was mostly executed within an Aug 1 subagent delegation session and as subagent tasks -- all changes were verified live on infrastructure.
**Session search hit rate:** 12 of 13 known projects found via 15+ queries. The Super Search v2.4.0 work was confirmed via infrastructure state, not session history.
docs/audit/app2-caddyfile-audit-2026-07-21.md
+27
View File
@@ -0,0 +1,27 @@
# app2 Caddyfile Audit — July 21, 2026
## Root cause
Technitium DNS was deployed on app2. During the Caddyfile rewrite to add `dns1.itpropartner.com`, two existing services were dropped:
1. **UNMS** — `reverse_proxy localhost:80` failed because UNMS nginx exposes port 443 (via host 8444), not port 80. Fixed by proxying via HTTPS with `tls_insecure_skip_verify`.
2. **Gitea** — entry was completely removed. Fixed by adding `reverse_proxy 127.0.0.1:3001`.
## Prevention
- Always audit `docker ps` output BEFORE rewriting Caddyfile
- Verify every running container that exposes web ports has a Caddy entry
- Test each domain with `curl -sk` after Caddy reload
## Final Caddyfile (validated)
```
{
default_bind 152.53.39.202
auto_https disable_redirects
}
dns1.itpropartner.com:443 → 127.0.0.1:5380
gps.fleettracker360.com:443 → localhost:8082
fleettracker360.com:443 → localhost:8082
unms.forefrontwireless.com:443 → https://localhost:8444 (tls_insecure_skip_verify)
unifi.itpropartner.com:443 → https://localhost:8443 (tls_insecure_skip_verify)
hudu.itpropartner.com:443 → localhost:3000
git.itpropartner.com:443 → 127.0.0.1:3001
```
docs/audit/git-audit-2026-08-07.md
+207
View File
@@ -0,0 +1,207 @@
# Git Structure Audit -- August 7, 2026
**Scope:** All Gitea-hosted repos (40 on git.itpropartner.com) plus local-only repos under /root/projects/
**Auditor:** Sho'Nuff (Hermes Agent)
---
## Summary Verdict
Your Git structure has solid bones but significant hygiene gaps. For a private, solo-developer setup it's functional -- but if you ever go public, the current state would fail a basic security review. The issues below are ordered by severity.
---
## CRITICAL: Fix Immediately
### 1. Hardcoded Credentials in `scripts` Repo
The `scripts` repo (11 commits, 75KB) contains Windows provisioning PowerShell scripts with **plaintext passwords committed to history:**
- `LoveMyBoys73!` -- ippadmin MSP backdoor account
- `Liberty4All!` -- liberty-admin customer admin
- `tire` -- tire power user
These appear in `dell-reimage-kit/` unattend XML and PowerShell. Even if this repo stays private forever, credentials in git history is a ticking time bomb. One accidental `git clone` to the wrong place and those passwords are exposed.
**Fix:** `git filter-branch` or BFG Repo-Cleaner to purge from history, then rotate all three passwords everywhere they're used (Liberty UDM, Windows workstations, etc).
### 2. Blob Repository: `hermes-skills` = 13.6 MB
The `hermes-skills` repo tracks 2,251 files including:
- `skills/.hub/index-cache/hermes-index.json` -- **39 MB** JSON blob
- `skills/.curator_backups/2026-07-12T15-48-44Z/skills.tar.gz` -- **2.7 MB** tarball
These are cache/backup artifacts, not source code. They bloat every clone by 40+ MB and will grow with time. The repo has no `.gitignore` to prevent this.
**Fix:** Add `.gitignore` excluding `.hub/` and `.curator_backups/`, `git rm --cached` the tracked artifacts, commit. Expect the repo size to drop from 13.6 MB to well under 1 MB.
### 3. No `.gitignore` on 33 of 35 Gitea Repos
Only `itpp-infrastructure` and `hermes-recovery` have a `.gitignore`. Every other repo is unprotected against accidental commits of `.env` files, backup directories, `__pycache__/`, `.DS_Store`, editor swap files, etc.
**Fix:** Apply a standard `.gitignore` template across all repos (see recommendation below).
---
## HIGH: Structural Problems
### 4. Stale Duplicate: `itpp-infra` (SSH remote, orphaned)
`/root/projects/itpp-infra` has an SSH remote (`git@git.itpropartner.com:ippadmin/itpp-infra.git`) pointing to a repo that **does not exist on Gitea**. This was its one and only commit (Jul 24, "Initial commit -- audit Jul 24 2026"). The actual infrastructure docs live in `/root/projects/itpp-infrastructure` (49 commits, active).
The `itpp-infra` local copy also has 5 dirty files (uncommitted edits to server DR plans and network diagrams). These are likely valuable changes trapped in a dead repo.
**Fix:**
1. Recover any uncommitted changes from `itpp-infra`
2. Verify they don't duplicate `itpp-infrastructure` content
3. Delete or archive the stale repo
### 5. Branch Naming Inconsistency
| Branch | Count | Repos |
|--------|-------|-------|
| `master` | 25 | apex-track, backup-restore, boxpilot, content-creation-pipeline, digital-signage, disaster-recovery, dre, fleettracker360, forefront-wireless-portal, gift-a-roast, hermes-recovery, hermes-skills, hudu, itpropartner-website, mcp-*, ops-portal, osint-tool, personal-assistant, pipeline, scripts, shark-game, startup-studio, track-a-flock, unifi, unms, voipsimplicity, voipsimplicity-manual |
| `main` | 7 | cartmylist, homelab, itpp-infrastructure, launchcheck, model-fallback, nvr-shield, super-search-business |
**Plus:** `itpp-infrastructure` locally is on `main` but Gitea's default branch for that repo is `master` -- the remote has an empty `master` branch alongside the active `main`.
Industry standard has moved to `main`. Your newer repos use it, older ones don't.
**Fix:** Standardize on `main` for new repos. Migrating existing `master` repos is optional for private use but recommended before any public release.
### 6. Dirty Working Trees: 21 Repos with Uncommitted Changes
```
hermes-skills 33 dirty files
hermes-recovery 24 dirty files
voipsimplicity-manual 11 dirty files
digital-signage 6 dirty files
itpp-infra 5 dirty files
pipeline 5 dirty files
disaster-recovery 4 dirty files
shark-game 2 dirty files
--- plus 13 repos with 1 dirty file each ---
```
Several of these repos haven't been committed since July 15-16. That's three weeks of potentially valuable changes sitting uncommitted and un-backed-up.
**Fix:** Audit each dirty repo, commit or discard changes, push. This is also a DR concern -- uncommitted files don't exist in S3 backups.
### 7. Remote URL Anomalies
- **`gift-a-roast`** uses username `git` instead of `ippadmin` in its HTTPS remote. Functionally works (Gitea ignores the username with token auth) but inconsistent and sloppy.
- **`itpp-infra`** uses SSH (`git@...`) -- won't work without SSH keys on Gitea. The repo doesn't exist on Gitea anyway, confirming this was never successfully pushed.
- **`msp-claude-skills`** is a direct GitHub clone (`github.com/RTFM-IT-Services-LLC/msp-claude-skills.git`, CC BY-NC-SA 4.0). This is fine for reference but should be marked as upstream-sourced. It has no Gitea remote.
---
## MEDIUM: Operational Gaps
### 8. Twenty Local-Only Repos (No Remote)
These are projects with local git init but never pushed anywhere:
`assistant`, `auth`, `capabilities`, `hear-read`, `intelsight`, `intelsight-landing`, `internal`, `mockup`, `my-itpropartner-portal`, `ops`, `ops-v2-portal`, `osint`, `proposals`, `pry`, `research-search-mcp`, `schedule`, `shonuff`, `shonuff-caller`, `static`, `status`, `voice-previews`
Some are real projects (intelsight, auth, pry). Some look like duplicates/abandoned scaffolds (ops vs ops-portal vs ops-v2-portal). None are backed up via Gitea push, meaning they live only on this server's disk.
**Fix:** Either push to Gitea or explicitly decide they're abandoned and delete. The duplication (ops/ops-portal/ops-v2-portal) should be consolidated.
### 9. Single-Branch Linear History
Every repo uses exactly one branch with linear commits. No feature branches, no pull requests, no tags, no releases. This is acceptable for solo development but means:
- No way to experiment without polluting the main line
- No tagged versions for rollback
- No PR workflow if you ever collaborate
### 10. Abandoned Single-Commit Repos
Sixteen repos have only 1-2 commits, most with the message "Initial commit -- 2026-07-15" and nothing since. This suggests batch scaffolding on July 15 that never got follow-up. These clutter the Gitea org.
---
## LOW: Nice-to-Have
### 11. No Repo Templates
No `ISSUE_TEMPLATE.md`, `PULL_REQUEST_TEMPLATE.md`, `CODEOWNERS`, or `CONTRIBUTING.md` on any repo. Low priority for solo work but standard for public repos.
### 12. Commit Message Quality Varies
`itpp-infrastructure` has clean, descriptive messages (e.g., "docs: fallback chain overhaul, two-key strategy, operational model update"). Many others use "Initial commit" or "Update 2026-07-15 -- root" which conveys nothing.
### 13. Token in Remote URLs
All HTTPS remotes embed the Gitea token directly. This is convenient but means the token appears in shell history, process lists, and any `git remote -v` output. If any repo directory is ever copied or backed up without sanitization, the token travels with it.
---
## Recommendations: Action Plan
### Immediate (This Week)
1. **Purge credentials from `scripts` repo history** and rotate those three passwords everywhere
2. **Add `.gitignore`** to all 33 repos missing it (see template below)
3. **Clean `hermes-skills`** -- gitignore `.hub/` and `.curator_backups/`, rm cached, repush
4. **Resolve `itpp-infra`** -- salvage any unique content, then archive/delete
### Short-Term (This Month)
5. **Audit dirty repos** -- commit or discard all pending changes
6. **Push or delete local-only repos** -- decide which are real projects vs abandoned scaffolds
7. **Fix remote URL anomalies** -- normalize gift-a-roast username, decide on msp-claude-skills disposition
8. **Standardize branch naming** -- pick `main` as default, migrate at least the active repos
### Before Any Public Release
9. Rotate the Gitea token and move to SSH keys or a credential helper
10. Add repo templates (issue/PR)
11. Audit every repo's history for secrets with `git-secrets` or `truffleHog`
12. Tag releases on active projects
---
## Standard `.gitignore` Template
```gitignore
# Environment & secrets
.env
.env.*
*.key
*.pem
credentials.json
# Python
__pycache__/
*.py[cod]
*.egg-info/
.venv/
venv/
# Node
node_modules/
# OS
.DS_Store
Thumbs.db
# Editor
*.swp
*.swo
*~
# Backups
*.bak
.backup-*/
# Large cache files
*.tar.gz
*.zip
index-cache/
```
---
*Report generated by Sho'Nuff (Hermes Agent) on August 7, 2026.*
*Full repo inventory and remote URL map available on request.*
docs/audit/git-audit-2026-08-08.md
+364
View File
@@ -0,0 +1,364 @@
# Git Audit Report — IT Pro Partner Gitea Organization
**Date:** 2026-08-08
**Auditor:** Hermes Agent (automated)
**Scope:** git.itpropartner.com / ippadmin (all 42 remote repos + 59 local clones in `/root/projects/`)
---
## Summary Verdict
**The Gitea organization suffers from repo sprawl, weak hygiene, and live secrets in history.** Of 42 remote repos, 32 are single-commit documentation stubs. Only 3–4 repos show active development. The `itpp-infrastructure/docs/` folder is a flat grab-bag and needs structured nesting. 23 local-only repos lack any off-server backup. Credentials are embedded in plaintext across at least 2 repos (`scripts`, `hermes-recovery`). Consolidation, cleanup, and a hygiene push are overdue.
---
## Quick Answers to User's Two Questions
### 1. Should `itpp-infrastructure/docs/` have more nested folders?
**Yes, absolutely.** The current structure is:
```
docs/
backup-restore/ ← already nested (good)
legal/ ← already nested (good)
ops-portal/ ← already nested (good)
app2-caddyfile-audit-2026-07-21.md ← flat
client-katie-watts-design.md ← flat
cost-control-rollout-2026-07-24.md ← flat
git-audit-2026-08-07.md ← flat
infrastructure-gap-assessment-2026-08-04.md ← flat
key-inventory.md ← flat
mattermost-replacement-analysis.md ← flat
model-chain.md ← flat
project-log.md ← flat
projects-master-readme.md ← flat
super-search-cf-bypass.md ← flat
super-search-enhancement-plan.md ← flat
uptime-kuma-monitoring-plan.md ← flat
```
**14 flat files is too many.** Recommended restructuring:
```
docs/
audit/ ← git-audit reports, gap assessments
backup-restore/ ← (existing, keep)
clients/ ← client-katie-watts-design.md
infrastructure/ ← model-chain.md, cost-control-rollout, caddyfile-audit, key-inventory
legal/ ← (existing, keep)
monitoring/ ← uptime-kuma-monitoring-plan.md
ops-portal/ ← (existing, keep)
projects/ ← project-log.md, projects-master-readme.md
super-search/ ← cf-bypass, enhancement-plan
mattermost-replacement-analysis.md ← leave at top level (one-off)
```
This gives every file a clear home without over-nesting.
### 2. Are there too many top-level repos? Should they be consolidated?
**Yes — 42 repos is far too many for the actual workload.** Here's the breakdown:
| Category | Count | Action |
|----------|-------|--------|
| Active development repos | 3–4 | Keep (`itpp-infrastructure`, `hermes-skills`, `scripts`, `homelab`) |
| Single-commit documentation stubs | 32 | Consolidate into fewer repos |
| Empty repos | 1 | Delete (`auth` — no commits, no content) |
| MCP stub repos | 5 | Merge into one `mcp-catalog` repo |
| Duplicate/stale repos | 2 | Resolve (`itpp-infra` vs `itpp-infrastructure`, `cartmylist-repo` vs `cartmylist`) |
| Local-only (unbacked) | 23 | Push to Gitea or archive |
**Recommended consolidation:**
1. **Merge 4 MCP stubs** (`mcp-browser`, `mcp-email`, `mcp-filesystem`, `mcp-git`) into `mcp-servers/` as subdirectories
2. **Merge related business ideas** into a `project-ideas` monorepo:
- `apex-track`, `boxpilot`, `digital-signage`, `fleettracker360`, `gift-a-roast`, `hudu`, `launchcheck`, `mooresunnydaze`, `nvr-shield`, `osint-tool`, `shark-game`, `startup-studio`, `track-a-flock`, `personal-assistant`
3. **Merge related operational repos**: `disaster-recovery` + `backup-restore` → `disaster-recovery/` with `backup-restore/` subdir
4. **Merge website/docs stubs**: `itpropartner-website`, `content-creation-pipeline`, `ops-portal` → subdirectories in `itpp-infrastructure`
5. **Resolve** `itpp-infra` (stale, SSH-only, 1 commit) → archive; use `itpp-infrastructure` as primary
6. **Delete** `auth` (empty repo with `auth.db` — never committed)
7. **Keep as-is**: `itpp-infrastructure`, `hermes-skills`, `hermes-recovery`, `scripts`, `homelab`, `dre`, `verdicttank`, `voipsimplicity`, `voipsimplicity-manual`, `forefront-wireless-portal`, `super-search-business`, `model-fallback`, `unifi`, `unms`, `pipeline`, `super-search`
**Target:** ~15–20 repos instead of 42.
---
## Detailed Findings
### Step 1 — Full Inventory
| Metric | Count |
|--------|-------|
| Remote repos on Gitea | 42 |
| Local clones in `/root/projects/` | 59 |
| On Gitea AND cloned locally | 36 |
| On Gitea but NOT cloned locally | 6 |
| Cloned locally but NOT on Gitea | 23 |
| Public repos | 30 |
| Private repos | 12 |
| Repos using `master` as default branch | 30 |
| Repos using `main` as default branch | 9 |
| Other (HEAD/detached) | 20 |
**Repos on Gitea but not cloned locally:** `cartmylist`, `mcp-browser`, `mcp-email`, `mcp-filesystem`, `mcp-git`, `super-search`
**Local-only repos (no remote — no off-server backup):** `assistant`, `auth`, `capabilities`, `hear-read`, `intelsight`, `intelsight-landing`, `internal`, `mockup`, `my-itpropartner-portal`, `ops`, `ops-v2-portal`, `osint`, `proposals`, `pry`, `research-search-mcp`, `schedule`, `shonuff`, `shonuff-caller`, `static`, `status`, `voice-previews`, `itpp-infra`, `cartmylist-repo`
### Step 2 — Per-Repo Deep Scan
#### Hygiene Check: `.gitignore` and `README.md`
| Status | Count |
|--------|-------|
| Has `.gitignore` | 8 of 59 (13.6%) |
| Has `README.md` | 56 of 59 (94.9%) |
**Repos missing `.gitignore`:** 51 repos. This is the single biggest hygiene gap.
**Repos missing `README.md`:** `cartmylist-repo`, `voipsimplicity-manual`, `auth`
#### Repos with Dirty Working Trees
| Repo | Dirty Files | Severity |
|------|-------------|----------|
| `hermes-skills` | 33 | **HIGH** — cache artifacts not committed |
| `hermes-recovery` | 26 | **HIGH** — uncommitted recovery scripts |
| `voipsimplicity-manual` | 11 | **MEDIUM** |
| `research-search-mcp` | 10 | **MEDIUM** |
| `auth` | 9 | **MEDIUM** |
| `digital-signage` | 6 | **LOW** |
| `itpp-infra` | 5 | **LOW** |
| `pipeline` | 5 | **LOW** |
| `disaster-recovery` | 4 | **LOW** |
| `mooresunnydaze` | 4 | **LOW** |
| `verdicttank` | 4 | **LOW** |
| `itpp-infrastructure` | 3 | **LOW** |
**13 repos** have uncommitted changes. `hermes-skills` (33 files) and `hermes-recovery` (26 files) are the worst offenders.
### Step 3 — Secrets Scan
**CRITICAL findings in 2 repos:**
#### `scripts` — 10 potential secrets
Real, hardcoded credentials found in Windows provisioning scripts:
```
+Password="LoveMyBoys73!"
+Password="Liberty4All!"
+Password="tire"
+Username="ippadmin"
+Username="liberty-admin"
```
These are active Windows admin credentials embedded in PowerShell unattend scripts. **This is a data breach risk.** If these repos ever go public or are cloned outside ITPP infrastructure, client credentials are exposed.
#### `hermes-recovery` — 8 potential secrets
Includes the Gitea API token used for this audit:
```
+TOKEN="1761daa2c537fb72b365e54619208329d8e3ad33"
+TELEGRAM_BOT_TOKEN="8359374835:***"
+password="K3E1ZZWvHDu0q8ZmoBCAhzKUZawEapdGBlbaPME1sOTKgGk9FCuYS"
+token = "Ta9f9d1b462271a2f4-8d63a3f025eb89451edb16f2308c2e40"
```
The Gitea token itself is committed. This means `hermes-recovery` as a public repo exposes admin credentials.
#### `hermes-skills` — 15 potential hits
Most are false positives (example values, `process.env.` references, placeholder text). One real hit: a Comfy CLI API key in a SKILL.md.
### Step 4 — Structural Checks
#### Remote URL Audit
| Remote Type | Count | Action |
|-------------|-------|--------|
| HTTPS to Gitea | 36 | OK |
| GitHub (upstream) | 1 | OK (`msp-claude-skills`) |
| SSH to Gitea | 2 | **FIX** — `itpp-infra`, `cartmylist-repo` |
| No remote | 23 | **FIX** — local-only, no backup |
`itpp-infra` uses `git@git.itpropartner.com:ippadmin/itpp-infra.git` (SSH) — this repo has no corresponding HTTPS clone and appears to be a stale/abandoned repo (1 commit, 5 dirty files).
`cartmylist-repo` (local) vs `cartmylist` (Gitea) is a naming mismatch. The local clone has an SSH remote to what is likely a different repo.
#### Branch Naming
- **30 repos use `master`** — industry standard is now `main`
- **9 repos use `main`**
- **20 repos have detached HEAD or no commits**
**Branch mismatch:** `itpp-infrastructure` has `main` locally but `master` on Gitea. This means the remote may have both branches.
#### Large Files
| Repo | File | Size |
|------|------|------|
| `hermes-skills` | `.hub/index-cache/hermes-index.json` | 38.9 MB |
| `hermes-skills` | `.curator_backups/2026-07-12T15-48-44Z/skills.tar.gz` | 2.7 MB |
Both are cache artifacts that should be in `.gitignore`, not tracked.
#### Public vs Private
**30 of 42 repos (71%) are public.** This is a concern because:
- `scripts` contains client admin passwords — **public**
- `hermes-recovery` contains Gitea admin token — **public**
- Many repos with sensitive infrastructure details are public
### Step 5 — Commit Quality
#### Commit Message Quality
| Pattern | Count | Assessment |
|---------|-------|------------|
| `Initial commit — YYYY-MM-DD` | 16 | Poor — conveys nothing |
| `Update YYYY-MM-DD — root` | 5 | Meaningless |
| `Initial: <project name>` | 8 | Barely adequate |
| Descriptive conventional commits | 4 | Good (`homelab`, `verdicttank`, `forefront-wireless-portal`) |
**32 repos have only 1 commit** — these are documentation stubs, not developed projects.
#### Active vs Abandoned
| Status | Criteria | Repos |
|--------|----------|-------|
| **Active** | 3+ commits, recent activity | `itpp-infrastructure` (52), `scripts` (11), `homelab` (6), `forefront-wireless-portal` (5), `dre` (4), `verdicttank` (4), `personal-assistant` (3), `super-search-business` (3), `voipsimplicity` (3) |
| **Stub** | 1–2 commits, last push July 2025 | 32 repos |
| **Abandoned** | No commits or stale >3 months | `itpp-infra`, `auth`, `cartmylist-repo` |
### Step 6 — Local-Only Repos
23 repos in `/root/projects/` have no remote. Breakdown:
| Category | Repos | Action |
|----------|-------|--------|
| Uncommitted stubs (0 commits) | 17 | Push to Gitea or archive |
| Has commits, no remote | 1 (`shonuff-caller`) | Push to Gitea |
| Stale noise | 5 | Archive and delete (`auth`, `itpp-infra`, `cartmylist-repo`, etc.) |
The 17 repos with 0 commits and only detached HEAD are effectively just directories with a `.git` folder — not real repos. They should be either pushed as proper repos or archived.
---
## Prioritized Action Plan
### 🔴 Immediate (This Week)
| # | Action | Severity |
|---|--------|----------|
| 1 | **Rotate all credentials exposed in `scripts` repo** — Windows passwords, Gitea token, Telegram bot token. Then purge from Git history with `git filter-branch` or `bfg-repo-cleaner` | **CRITICAL** |
| 2 | **Rotate Gitea API token** in `hermes-recovery` — it's publicly visible. Generate new token, update all consumers, purge old from history | **CRITICAL** |
| 3 | **Make `scripts` and `hermes-recovery` PRIVATE** — they contain live credentials visible to anyone | **CRITICAL** |
| 4 | **Add `.gitignore` to all 51 repos missing one** — start with the active repos first | **HIGH** |
| 5 | **Commit or stash all dirty working trees** — 13 repos have uncommitted work at risk of loss | **HIGH** |
### 🟡 Short-Term (This Month)
| # | Action | Severity |
|---|--------|----------|
| 6 | **Reorganize `itpp-infrastructure/docs/`** into nested folders (audit/, clients/, infrastructure/, monitoring/, projects/, super-search/) | **MEDIUM** |
| 7 | **Consolidate 4 MCP repos into `mcp-servers/`** as subdirectories — delete empty stubs after merge | **MEDIUM** |
| 8 | **Merge 14 single-commit business idea repos** into a `project-ideas` monorepo | **MEDIUM** |
| 9 | **Delete `auth`** (empty repo, 0 commits) | **MEDIUM** |
| 10 | **Resolve `itpp-infra` vs `itpp-infrastructure`** — archive `itpp-infra`, standardize on `itpp-infrastructure` | **MEDIUM** |
| 11 | **Add `.gitignore` entries to `hermes-skills`** for `.hub/`, `.curator_backups/` | **MEDIUM** |
| 12 | **Rename `master` → `main` on repos where it matters** (at minimum `itpp-infrastructure` to fix branch mismatch) | **LOW** |
### 🔵 Pre-Public / Pre-Open-Source
| # | Action | Severity |
|---|--------|----------|
| 13 | **Audit all 30 public repos** — ensure no private infrastructure details, client names, IPs, or credentials are exposed | **HIGH** |
| 14 | **Decide public/private policy** — which repos genuinely need to be public? Currently 71% are public. | **MEDIUM** |
| 15 | **Push 23 local-only repos to Gitea** or archive them. No code living only on a single server. | **HIGH** |
| 16 | **Clean commit history** — rebase repos with "Update YYYY-MM-DD — root" messages into meaningful commits | **LOW** |
---
## Template: Standard `.gitignore`
For any new or cleaned repo, use:
```gitignore
# OS
.DS_Store
Thumbs.db
# IDE
.vscode/
.idea/
*.swp
*.swo
# Python
__pycache__/
*.py[cod]
*.egg-info/
.venv/
venv/
# Node
node_modules/
# Secrets — NEVER commit these
.env
.env.*
*.pem
*.key
credentials.json
*.token
# Cache / generated
.hub/
.curator_backups/
*.tar.gz
*.zip
# Data
*.db
*.sqlite
*.sqlite3
```
---
## Appendix: Full Repo Inventory
### Active Repos (keep as standalone)
| Repo | Commits | Last Commit | Branch | `.gitignore` | Assessment |
|------|---------|-------------|--------|-------------|------------|
| `itpp-infrastructure` | 52 | 2026-08-07 | main/master mismatch | YES | **Primary hub** — healthy |
| `hermes-skills` | 1 | 2026-07-15 | master | NO | Active mirror, 13.6MB |
| `hermes-recovery` | 1 | 2026-07-15 | master | YES | Critical backup kit |
| `scripts` | 11 | 2026-07-25 | master | NO | Active, **has secrets** |
| `homelab` | 6 | 2026-07-24 | main | NO | Active, good commits |
| `dre` | 4 | 2026-07-25 | master | NO | Active development |
| `verdicttank` | 4 | 2026-08-07 | main | NO | Active, good commits |
| `forefront-wireless-portal` | 5 | 2026-07-25 | master | NO | Active, good commits |
| `super-search-business` | 3 | 2026-07-25 | main | NO | Active |
| `voipsimplicity` | 3 | 2026-07-24 | master | NO | Active client work |
| `voipsimplicity-manual` | 1 | 2026-08-05 | master | NO | Active client work |
### Stub Repos (consolidate)
All 32 repos below are single-commit documentation stubs with no ongoing development. Consolidate into `project-ideas/` monorepo or relevant parent repo:
`apex-track`, `backup-restore`, `boxpilot`, `content-creation-pipeline`, `digital-signage`, `disaster-recovery`, `fleettracker360`, `gift-a-roast`, `hudu`, `itpropartner-website`, `launchcheck`, `mcp-browser`, `mcp-email`, `mcp-filesystem`, `mcp-git`, `mcp-servers`, `model-fallback`, `mooresunnydaze`, `nvr-shield`, `ops-portal`, `osint-tool`, `personal-assistant`, `pipeline`, `shark-game`, `startup-studio`, `super-search`, `track-a-flock`, `unifi`, `unms`, `cartmylist`
### To Delete or Archive
| Repo | Reason |
|------|--------|
| `auth` | Empty (0 commits, 0 content) |
| `itpp-infra` | Stale duplicate of `itpp-infrastructure`, SSH-only remote, 1 commit |
| `cartmylist-repo` | Local clone with SSH remote, mismatched name (real one is `cartmylist` on Gitea) |
### Local-Only (push or archive)
`assistant`, `auth`, `capabilities`, `hear-read`, `intelsight`, `intelsight-landing`, `internal`, `mockup`, `my-itpropartner-portal`, `ops`, `ops-v2-portal`, `osint`, `proposals`, `pry`, `research-search-mcp`, `schedule`, `shonuff`, `shonuff-caller`, `static`, `status`, `voice-previews`
---
*Report generated by Hermes Agent git-audit workflow. Next audit recommended: 2026-11-08.*
docs/audit/infrastructure-gap-assessment-2026-08-04.md
+290
View File
@@ -0,0 +1,290 @@
# ITPP Infrastructure Documentation Gap Assessment
**Date:** 2026-08-04
**Auditor:** Hermes Agent (subagent)
**Scope:** All ITPP infrastructure — Core, app1, app2, app3, app1-bu, wphost02
---
## Executive Summary
**Total services discovered running:** 65+ (across 5 hosts)
**Services with NO backup:** 12 (CRITICAL: 3 with production data at risk)
**Services missing from API master list:** 14
**Documentation staleness issues:** 7
**Services with NO project documentation:** 22+
**Duplicate services (unintended):** 2 (Twenty CRM, SearXNG running on both Core AND App1)
**Sites missing local snapshots (App3):** 4
---
## (A) CRITICAL GAPS — Services with NO Backup
### 🔴 Priority 1 — Production data at immediate risk
| # | Service | Host | Risk | Data at stake |
|---|---------|------|------|---------------|
| 1 | **Ragflow** (full stack) | App2 | CRITICAL | MySQL DB, Minio objects, Infinity vector DB, Valkey cache — entire RAG/knowledge base platform. 6 Docker containers including mysql:8.0.39, minio, infinity vector DB |
| 2 | **Mattermost** | App1 | CRITICAL | Team chat messages, channels, files, user accounts. postgres:16-alpine backend |
| 3 | **Wazuh** (SIEM) | App1 | HIGH | Security events, alerts, agent data, compliance logs. 3 containers (dashboard, manager, indexer). Only the server itself is backed up via app1 general backup — Wazuh data is NOT |
### 🟡 Priority 2 — Important services without backups
| # | Service | Host | Risk | Data at stake |
|---|---------|------|------|---------------|
| 4 | **Dawarich** | App2 | MEDIUM | Location history data (PostGIS), Redis cache. NOT in app2-backup.sh |
| 5 | **Technitium DNS** | App2 | MEDIUM | DNS zone configs, DHCP leases, blocklists. NOT in app2-backup.sh |
| 6 | **SearXNG (App1)** | App1 | MEDIUM | Search engine config. Backup plan says "removed" but it's running on :8080 |
| 7 | **MCP containers** (App1) | App1 | LOW | mcp-browser, mcp-email, mcp-git, mcp-filesystem, super-search — config/state not backed up individually |
| 8 | **browserless** (App1) | App1 | LOW | Stateless Chrome, but no restart config backup |
| 9 | **Timetrex** | Core | LOW | Time tracking data — Docker container, no compose file found |
| 10 | **Microbin** | Core | LOW | Paste bin data — Docker container, compose exists at /opt/microbin/ |
| 11 | **browserless** (Core) | Core | LOW | Stateless Chrome, :3000 (conflicts with Grafana's documented port) |
| 12 | **crawl4ai** | Core | LOW | Python service :8910 — web crawling config. No compose, running from /root/docker/crawl4ai/ |
### 🟢 Services running natively (Core systemd/Python — minimal backup need)
These are stateless or backed up via hermes-backup.sh (skills/profiles/sessions) and root-essentials-backup.sh (scripts):
- hotnow-api (:8001), auth-server (:8500), pipeline-server (:8200), hermes-voice (:4331), host-metrics-exporter (:9275), transitpin-mockup (:8912), http-server (:9876)
---
## (B) DOCUMENTATION GAPS — Services not in API Master List
### Missing from `/root/projects/itpp-infrastructure/api-master-list.md`
| # | Service | Host | Port | Notes |
|---|---------|------|------|-------|
| 1 | **Mattermost** | App1 | :8065 | Team chat — not listed anywhere |
| 2 | **n8n** | App1 | :5678 | Workflow automation — not listed |
| 3 | **Ragflow** | App2 | :9380-9384 | RAG platform — not listed |
| 4 | **SearXNG (App1)** | App1 | :8080 | Listed only on Core :8888 |
| 5 | **Timetrex** | Core | :8085 | Time tracking — not listed |
| 6 | **Microbin** | Core | :8260 | Paste bin — not listed |
| 7 | **browserless** (Core) | Core | :3000 | Chrome automation — not listed |
| 8 | **browserless** (App1) | App1 | :3005 | Chrome automation — not listed |
| 9 | **crawl4ai** | Core | :8910 | Web crawler — not listed |
| 10 | **hermes-control-deck** | Core | systemd | Control deck API — not listed |
| 11 | **MCP containers** (App1) | App1 | :8900-8903 | Litellm MCP gateway services — not listed individually |
| 12 | **Super Search (App1)** | App1 | container | Duplicate of Core's — not listed |
| 13 | **hotnow-api** | Core | :8001 | HotNow API (different from :8000 Diglocate) |
| 14 | **auth-server** | Core | :8500 | Centralized auth project |
---
## (C) STALE DOCUMENTATION — Wrong/Outdated Information
### 🔴 API Master List errors
| # | Issue | Doc says | Actual | Severity |
|---|-------|----------|--------|----------|
| 1 | **Grafana port** | Core :3000 | Core :3002 (browserless/chrome occupies :3000) | MED — monitoring dashboards accessed at wrong port |
| 2 | **DocuSeal port** | App1 :3000 | App1 :3002 (Open WebUI occupies :3000 on App1) | MED |
| 3 | **Twenty CRM location** | Core :3003 (listed as Core) | Running on BOTH Core :3003 AND App1 :3003 | HIGH — duplicate service, unclear which is authoritative |
| 4 | **SearXNG status** | Core :8888, backup plan says "removed" | Actually running on BOTH Core :8888 AND App1 :8080 | HIGH — backup plan says replaced by Super Search but still running on two hosts |
| 5 | **Vaultwarden location** | Core :8080 (in old API list), App1 :8081 | Only on App1 :8081 (correct) but stale S3 paths remain | LOW |
| 6 | **Komodo location** | Migrated to App1 :9120 | Correctly on App1 :9120 | OK |
| 7 | **DocuSeal location** | Migrated to App1 :3002 | Correctly on App1 :3002 | OK |
| 8 | **Twenty CRM migration** | Backup plan says migrated to App1 | Still running on Core too! The migration was partial or the Core instance was never shut down | HIGH |
### 🔴 Backup Plan errors
| # | Issue | Details |
|---|-------|---------|
| 9 | **S3 stale paths** | `core/vaultwarden/`, `core/twenty/`, `core/searxng/`, `core/komodo/` still in S3 hierarchy — services migrated but old paths not cleaned |
| 10 | **Backup plan lists Core services that migrated** | Table shows Vaultwarden, SearXNG, Twenty CRM, Komodo, DocuSeal, Kokoro TTS under Core section — all migrated to App1 |
| 11 | **app1-backup.sh (on App1) backs up MORE than documented** | Script backs up n8n (not in plan), Ollama (not in plan) but does NOT back up Mattermost, Wazuh, SearXNG (App1), browserless |
| 12 | **app2-backup.sh (on App2) coverage gaps** | Script covers Traccar, Gitea, Hudu, UNMS, UniFi but misses Dawarich, Technitium DNS, Ragflow |
### 🔴 App3 Snapshot Coverage
| # | Site | Nginx config? | In backup-restore snapshots? | In S3 app3-backup.sh? |
|---|------|--------------|------------------------------|----------------------|
| 1 | intelsight.io | ✅ | ❌ | ✅ (via WordPress files backup) |
| 2 | my.voipsimplicity.com | ✅ | ❌ | ✅ |
| 3 | panel.itpropartner.com | ✅ | ❌ | N/A (CloudPanel itself) |
| 4 | transitpin.com | ✅ | ❌ | ✅ |
---
## (D) CREDENTIAL GAPS
### Credential Management Assessment
- **Vaultwarden:** Running on App1 :8081. Accessible via web UI. Not directly queryable via API without auth token.
- **Standard credentials** (ippadmin/LoveMyBoys.1520!): Referenced in task context. Need to verify which services use these vs. unique credentials.
- **Key credential files:** `/root/.hermes/.env`, `~/.aws/credentials`, Vaultwarden vault
- **DR Issue Log** references `migration-creds.txt` and `dre-temp-passwords.txt` — both now `chmod 600` (DR-004, DR-005 fixed)
### Recommendations:
1. Audit all services to confirm which use standard creds vs unique creds
2. Each service should have a Hudu asset documenting its credentials
3. Service-specific API keys (n8n, Mattermost, Ragflow internal admin) need to be inventoried
---
## (E) SERVICES WITH NO PROJECT DOCUMENTATION
The `/root/projects/itpp-infrastructure/` repo has documentation for only ~8 projects out of 30+ running services:
**Have docs:** ops-portal, backup-restore, hotnow, intelsight, schoolcart, tripflow, beachdirect, forefront-broadband-map, missed-call-lead-recovery
**NO docs (22+ services):**
Mattermost, n8n, Ragflow, Wazuh, Dawarich, Technitium DNS, Timetrex, Microbin, browserless (both), crawl4ai, Vaultwarden, Komodo, DocuSeal, LiteLLM, Open WebUI, Twenty CRM, Kokoro TTS, PRY, TransitPin, Village Express, Shopping Cart, Voice Agent stack, Diglocate, Rally, Shark Game, hermes-assistant, hermes-control-deck, Camofox, Super Search, Gitea, Hudu, UNMS, UniFi
---
## (F) DUPLICATE SERVICES
Two services are running redundantly on both Core and App1:
| Service | Core | App1 | Notes |
|---------|------|------|-------|
| **Twenty CRM** | :3003 (Docker, 5 containers) | :3003 (Docker, 4 containers) | Migration doc says moved to App1. Core instance was never shut down. Which is authoritative? |
| **SearXNG** | :8888 (Docker) | :8080 (Docker) | Backup plan says "removed, replaced by Super Search." Both still running. |
---
## (G) RECOMMENDED FIXES — Priority Order
### 🔴 Immediate (this week)
1. **Backup Ragflow** — Create ragflow-backup.sh on App2. Dump MySQL (mysql:8.0.39), backup Minio objects, backup Infinity DB. Add to cron. Risk: complete RAG platform data loss.
2. **Backup Mattermost** — Add to app1-backup.sh or create mattermost-backup.sh. Dump postgres:16-alpine DB, backup file uploads. Add to cron.
3. **Backup Wazuh** — Create wazuh-backup.sh on App1. Backup Elasticsearch indices and Wazuh manager config. Add to cron.
4. **Shut down duplicate Twenty CRM on Core** — The migration doc says it moved to App1. The Core instance (5 containers) is likely stale and consuming resources. Verify App1 instance is authoritative, then stop Core instance.
5. **Shut down duplicate SearXNG on both hosts OR document the dual deployment** — Backup plan says "removed, replaced by Super Search." If Super Search is sufficient, remove both SearXNG instances. If still needed, document why and add to backup plan.
### 🟡 This sprint (next 2 weeks)
6. **Update API Master List** — Add all 14 missing services. Fix stale port references (Grafana :3000→:3002, DocuSeal :3000→:3002).
7. **Update Backup Plan** — Remove stale Core entries (Vaultwarden, SearXNG, Twenty, Komodo, DocuSeal, Kokoro). Add Mattermost, n8n, Wazuh, Ragflow. Note that n8n IS backed up by app1-backup.sh but not documented.
8. **Backup Dawarich** — Add PostGIS dump to app2-backup.sh.
9. **Backup Technitium DNS** — Add DNS zone/config backup to app2-backup.sh.
10. **Add App3 sites to local snapshots** — Add intelsight.io, my.voipsimplicity.com, transitpin.com to `/opt/backup-restore/snapshot.sh` coverage.
### 🟢 Backlog (next month)
11. **Create project docs** for at minimum: Mattermost, n8n, Ragflow, Wazuh (the 4 services with no docs AND production data).
12. **Create `disaster-recovery-plan.md`** in itpp-infrastructure repo — it doesn't exist at the expected path. The DR issue log references it but the file is missing.
13. **Audit S3 bucket** — Clean stale paths (core/vaultwarden/, core/twenty/, core/searxng/, core/komodo/). Verify recent backups for all 26 documented targets.
14. **Credential audit** — Log into Vaultwarden, enumerate all entries, cross-reference with running services, identify gaps.
15. **Timetrex and Microbin** — Document purpose, add compose files to repo, add lightweight backup if they hold data.
16. **Create per-service README template** — Standardized format: purpose, host, ports, dependencies, backup method, restore procedure, credentials location.
---
## Methodology
- **Baseline docs read:** `/root/projects/itpp-infrastructure/backup-plan.md`, `api-master-list.md`, `/root/.hermes/references/dr-issue-log.md`
- **Hosts audited via SSH:** Core (localhost), app1 (152.53.36.131), app2 (152.53.39.202), app3 (152.53.241.111), app1-bu (5.161.225.131), wphost02 (5.161.62.38)
- **Enumeration:** `docker ps`, `ss -tlnp`, `systemctl list-units`, `crontab -l`, `ls /etc/nginx/sites-enabled/`
- **Cross-reference:** Each running service checked against API master list, backup plan, and itpp-infrastructure project docs
- **Key:** `/root/.ssh/itpp-infra` used for all remote SSH
---
## Appendix: Complete Service Inventory
### Core (152.53.192.33) — netcup RS 2000
| Service | Type | Port | In API List? | Backed Up? | Has Docs? |
|---------|------|------|-------------|------------|-----------|
| Twenty CRM | Docker (5ctr) | :3003 | ✅ (stale: says Core) | ⚠️ (S3: app1/twenty/, Core backup) | ❌ |
| SearXNG | Docker | :8888 | ✅ (stale: says removed) | ⚠️ (stale S3 path) | ❌ |
| Prometheus | Docker | :9090 | ✅ | ✅ (core-services-backup.sh) | ❌ |
| Grafana | Docker | :3002 | ✅ (wrong port :3000) | ✅ | ❌ |
| Telegraf | Docker | :9273 | ✅ | ✅ (system) | ❌ |
| Uptime Kuma | Docker | :3001 | ✅ | ✅ | ❌ |
| MikroTik Exporter | Docker | :9436 | ✅ | ✅ (system) | ❌ |
| Camofox Browser | Docker | :9377 | ✅ | ⚠️ (hermes-backup.sh) | ❌ |
| browserless | Docker | :3000 | ❌ | ❌ | ❌ |
| Timetrex | Docker | :8085 | ❌ | ❌ | ❌ |
| Microbin | Docker | :8260 | ❌ | ❌ | ❌ |
| Super Search MCP | systemd | :8899 | ✅ | ✅ (hermes-backup.sh) | ⚠️ (partial) |
| DRE MCP | systemd | :8900 | ✅ | ✅ | ❌ |
| Twilio MCP | systemd | :8901 | ✅ | ✅ | ❌ |
| OSINT Person MCP | systemd | :8902 | ✅ | ✅ | ❌ |
| FT360 MCP | systemd | :8903 | ✅ | ✅ | ❌ |
| PRY | systemd | :8905 | ✅ | ✅ | ❌ |
| Ops Portal | systemd | :8090 | ✅ | ✅ | ✅ |
| IntelSight API | systemd | :8099 | ✅ | ✅ | ✅ |
| Diglocate API | systemd | :8000 | ✅ | ✅ | ❌ |
| hotnow-api | systemd | :8001 | ❌ | ❌ | ⚠️ (project doc exists) |
| Rally | systemd | :8105 | ✅ | ✅ | ❌ |
| Village Express | systemd | :8210 | ✅ | ✅ | ❌ |
| Voice Agent STT | systemd | :9000 | ✅ | ✅ | ❌ |
| Voice Agent | systemd | :9101 | ✅ | ✅ | ❌ |
| Shopping Cart | systemd | :8101 | ✅ | ✅ | ❌ |
| OSINT API | systemd | :8100 | ✅ | ✅ | ❌ |
| Shark Game | systemd | :8083 | ✅ | ✅ | ❌ |
| hermes-assistant | systemd | :8082 | ✅ | ✅ | ❌ |
| hermes-control-deck | systemd | n/a | ❌ | ✅ | ❌ |
| hermes-voice | systemd | :4331 | ❌ | ✅ | ❌ |
| auth-server | systemd | :8500 | ❌ | ❌ | ❌ |
| pipeline-server | systemd | :8200 | ❌ | ❌ | ❌ |
| crawl4ai | systemd | :8910 | ❌ | ❌ | ❌ |
| host-metrics-exporter | systemd | :9275 | ❌ | ❌ | ❌ |
| transitpin mockup | systemd | :8912 | ❌ | ❌ | ❌ |
### App1 (152.53.36.131) — netcup RS 4000
| Service | Type | Port | In API List? | Backed Up? | Has Docs? |
|---------|------|------|-------------|------------|-----------|
| Open WebUI | Docker | :3000 | ✅ | ✅ (app1-backup.sh) | ❌ |
| LiteLLM | Docker | :4000 | ✅ | ✅ | ❌ |
| Komodo | Docker | :9120 | ✅ | ✅ (komodo-backup.sh) | ❌ |
| DocuSeal | Docker | :3002 | ✅ (port :3000) | ✅ (docuseal-backup.sh) | ❌ |
| Twenty CRM | Docker (4ctr) | :3003 | ✅ (says Core) | ✅ (twenty-backup.sh) | ❌ |
| Kokoro TTS | Docker | :8880 | ✅ | N/A (stateless) | ❌ |
| SearXNG (App1) | Docker | :8080 | ❌ (only listed on Core) | ❌ | ❌ |
| Wazuh | Docker (3ctr) | :5601/:9200 | ✅ | ❌ | ❌ |
| Vaultwarden | Docker | :8081 | ✅ | ✅ (vaultwarden-backup.sh) | ❌ |
| n8n | Docker | :5678 | ❌ | ✅ (in app1-backup.sh but not plan) | ❌ |
| Mattermost | Docker | :8065 | ❌ | ❌ | ❌ |
| MCP Gateway services | Docker (5ctr) | :8900-8903 | ❌ | ❌ | ❌ |
| browserless (App1) | Docker | :3005 | ❌ | ❌ | ❌ |
| Super Search (App1) | Docker | n/a | ❌ | ❌ | ❌ |
### App2 (152.53.39.202) — netcup RS 4000
| Service | Type | Port | In API List? | Backed Up? | Has Docs? |
|---------|------|------|-------------|------------|-----------|
| Hudu | Docker (4ctr) | :3000 (int) | ✅ | ✅ (hudu-backup.sh) | ❌ |
| Gitea | Docker | :3001 (int) | ✅ | ✅ (gitea-backup.sh) | ❌ |
| UNMS/UISP | Docker (full) | :8089 | ✅ | ✅ (unms-backup-sync.sh) | ❌ |
| UniFi | Docker | :8443 | ✅ | ✅ (unifi-backup-sync.sh) | ❌ |
| Traccar | Docker | :8082 | ✅ | ✅ (app2-backup.sh) | ❌ |
| Dawarich | Docker (4ctr) | :3002 | ✅ | ❌ | ❌ |
| Technitium DNS | Docker | :5380 | ✅ | ❌ | ❌ |
| Ragflow | Docker (6ctr) | :9380-9384 | ❌ | ❌ | ❌ |
### App3 (152.53.241.111) — netcup RS 4000 (CloudPanel)
13 WordPress sites hosted. All backed up to S3 via app3-backup.sh (daily 3 AM). 4 of 13 sites NOT in local snapshot rotation (intelsight.io, my.voipsimplicity.com, panel.itpropartner.com, transitpin.com).
### App1-BU (5.161.225.131) — Hetzner CPX21
Warm standby Hermes. No Docker. Cron: standby watchdog (every 5 min) + sync (every 10 min). Correctly configured per DR plan.
### wphost02 (5.161.62.38) — Hetzner CPX21
RunCloud WordPress hosting. 2 users (ippadmin, runcloud). Backed up via SSH from Core at 5 AM daily. Verified functional (DR-018, Jul 19).
---
## Files Created
- `/root/projects/itpp-infrastructure/docs/infrastructure-gap-assessment-2026-08-04.md` — this report