Initial skills documentation — 25 categories, all SKILL.md + references + scripts

This commit is contained in:
root
2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
+158
View File
@@ -0,0 +1,158 @@
---
name: security-audits
description: "Implement and maintain automated and periodic security audits for IT Pro Partner infrastructure — SSH attempt monitoring, package vulnerability scanning, config drift detection, port scanning, and failed login reviews."
version: 1.1.0
author: Sho'Nuff
platforms: [linux]
metadata:
hermes:
tags: [security, audit, monitoring, compliance, logging]
---
# Security Audits
Automated and periodic security audits for all ITPP servers. Results should eventually be surfaced through the IT Pro Partner operations portal.
## Lynis nightly scan (deployed Jul 6, 2026)
A `lynis` audit runs nightly at 3 AM on the netcup box via `~/.hermes/scripts/lynis-scan.sh`. It compares warnings count against the previous run and only alerts on NEW findings — silent if nothing changed.
**Installation:**
```bash
apt-get install lynis
lynis audit system --quick # initial baseline
```
**Cron setup:**
- Schedule: `0 3 * * *`
- Script: `lynis-scan.sh` (no_agent mode)
- Alert: only on warning count change, via Telegram
- Next step: install on wphost02 and Tony's box
**Verification:**
```bash
lynis audit system --quick | grep -E "Warnings|Hardening|Score"
grep "^warning\[\]" /var/log/lynis-report.dat
```
**Current baseline (Jul 6, 2026):**
- Netcup box: Hardening index 62/100, 3 warnings (1 vulnerable package, 2 DNS)
- wphost02: Hardening index 63/100, 3 warnings
| Type | Frequency | What it checks |
|---|---|---|
| **Automated** | Every tick | Open ports, recent failed SSH logins, disk usage alerts, service status |
| **Periodic** | Daily / Weekly | Package vulnerability scan, config drift vs known-good baseline, SSL cert expiry, user account audit |
## Wazuh vs simple cron approach
The user asked about Wazuh for security monitoring. Honest assessment:
**Wazuh pros:** Covers vulnerability scanning (CVEs), file integrity monitoring, intrusion detection (HIDS), log correlation, compliance checks (CIS), centralized dashboard — all in one.
**Wazuh cons:** Needs ~4GB RAM + 2 vCPU for the manager (CPX21 class). Takes a full day to deploy properly. Replaces the need for Graylog entirely since it handles log management.
**Simple cron approach (what we'd start with):** `lynis` weekly audits, SSH auth.log monitoring, automated port scans. Takes 30 minutes. Covers immediate gaps until Wazuh is ready.
**Recommendation:** Start with the cron approach for quick wins, plan Wazuh for when the portal goes customer-facing and compliance matters.
## Audit scope (per server)
### SSH audit
- Check `/var/log/auth.log` or `journalctl -u ssh` for failed login attempts in last 24h
- Count unique IPs attempting to brute force
- Check for non-root user accounts with shell access
- Verify `PermitRootLogin` and `PasswordAuthentication` settings
### Package audit
- `apt list --upgradable` for known security patches
- Check for unattended-upgrades status
- Verify kernel version matches expected image
### Service audit
- Check that critical services (Hermes gateway, cron, email) are running
- Verify Tailscale status
- Check Docker containers are running (if applicable)
### Network audit
- Scan for unexpected open ports with `ss -tlnp`
- Check firewall rules (ufw/iptables)
- Verify no services are listening on 0.0.0.0 that shouldn't be
## Alerting
- Security issues go to the user's Telegram immediately
- Low-urgency items (package updates, cert expiry > 30 days) go into a daily digest
- False positives should be tracked to prevent re-alerting on the same finding
- No alert is the expected state — silent means healthy
## Portal integration
When the portal is built, these audit results should appear on a dedicated Security Dashboard page with:
- Per-server health indicators
- Failed login trends over time
- Update status
- Last audit timestamp
**Pitfalls:**
- **auth.log rotation:** On Debian 13, auth logs may be under `/var/log/auth.log` or journalctl. Check both.
- **netcup SMTP block:** Automated email alerts may not send if the SMTP script uses port 587. Always use 2525.
- **Don't alert on the same thing twice:** Track last-reported state in a sentinel file under `/root/.hermes/scripts/.audit-*`.
- **For standalone scripts (no_agent), output must be self-explanatory:** The script output is delivered verbatim. No LLM is running to interpret it. Format alerts as plain text that the user can understand immediately.
- **Wazuh is on the roadmap** for full SIEM/XDR coverage across all servers (added Jul 6, 2026 to portal todo). Until deployed, Lynis + SSH monitoring are the active security audit tools.
- **Bitdefender GravityZone → Wazuh integration (Jul 15, 2026):** Germaine uses Bitdefender through SyncroMSP for Windows client endpoints. Official Wazuh integration path: Bitdefender GravityZone Event Push Service → `gz-evpsc` connector (Ubuntu VM) → Rsyslog → Wazuh manager. This enables unified Bitdefender + Wazuh events per MSP client, correlated timelines, and per-client compliance reporting (CIS/PCI). Bitdefender detects malware/EDR at the endpoint; Wazuh correlates with HIDS/FIM/SCA/network data for the full attack chain. Requires: Wazuh manager (CPX21), Bitdefender connector VM (~2GB), GravityZone API key for endpoint inventory.
- **Wazuh MCP plan (Jul 14, 2026):** Build `wazuh-mcp` as separate CPX21. Expose agent deployment, SCA/CIS checks, syscollector inventory, alert retrieval, and aggregated per-server security assessment tool. Note: Wazuh 4.8 removed the vulnerability API endpoint — CVE data accessible through Wazuh indexer (OpenSearch) side-channel.
- **VirusTotal integration (Jul 14, 2026):** Add to long-range plan as supplementary threat intel. Use cases: hash/file lookup, URL scan, IP/domain reputation. Integrate into security assessment reports alongside Wazuh + Lynis findings. API key pending.
- **Lynis is deployable on any Linux server** via a single `apt-get install lynis` + cron script. No subagent, tracking, or licensing needed. Done on netcup and wphost02. Tony's box is next.
## Operational security (approved user policy)
These rules govern how the agent operates on infrastructure, approved by Germaine on July 5, 2026.
### Read-only default
- Explore/check/read operations are automatic and need no approval
- Any state change (write, delete, modify, restart, reboot, config change) requires explicit user approval
### Destructive action approval
Before executing a destructive action, state the exact impact:
> "[action] on [server]. Impact: [what will happen]. Proceed?"
Then wait for "yes" or another affirmative. Do not proceed on implied consent.
### API key least-privilege
- Every API key used by the agent must be scoped to minimum permissions needed
- When a user provides a new key, verify its scope if possible and flag if it seems over-privileged
- Examples: Cloudflare = DNS edit only (not zone delete), Wasabi = write to backup buckets (not delete), Stripe = read-only (no refunds/charges)
### Session summaries
After any session that involved changes (config edits, server modifications, cron job changes, API key setup), provide a concise summary of every operation performed, its result, and any side effects.
### No assumed consent between actions
Each action stands alone. Completing task A does not imply consent for task B, even if B is closely related. Ask again.
## ITGC / Compliance Context
During a Jul 7, 2026 session, the user asked where we stand on ITGC controls (IT General Controls). Honest assessment for internal reference:
**What would pass a SOC 2 / ITGC review:**
- Access management: MFA on every admin access, quarterly access reviews, formal on/offboarding
- Change management: Documented tickets with approval, rollback plans
- Computer operations: Formal incident response plan, runbooks, documented backup restore testing
- Program development: Code review, source control
**What we actually have:**
- SSH key-only (no passwords), least-privilege API keys, Tailscale isolation
- Daily backups to S3 + warm standby + nightly Lynis scans
- No MFA on any admin access — biggest single gap
- No change tickets, no incident response plan, no access reviews
**The user's instruction: "Keep these audits/controls in mind as we build these systems."**
This means: when deploying new services (DRE portal, Zabbix, Wazuh, Mautic), build with audit readiness in mind from day one:
- Access logging on every service
- Least-privilege by default on every new service
- Documentation baked into deployment
- Change tracking in the project log
- MFA consideration at design time
@@ -0,0 +1,98 @@
---
name: lynis-security-scan
description: "Deploy and manage nightly Lynis security audits across all servers — package CVE scanning, misconfiguration detection, SSH brute-force monitoring. Silent when clean, alerts on new findings."
version: 1.0.0
author: Sho'Nuff
tags: [security, lynis, audit, cve, scanning, cron]
---
# Lynis Security Scan
Deploy the Lynis system audit tool across all servers. Nightly runs at 3 AM. Silent when no new warnings appear. Alerts if new findings emerge.
## Installation
```bash
apt-get install -y lynis
lynis --version # should be 3.1.x+
```
## First Run
```bash
lynis audit system --quick
```
Key output: Hardening index (0-100), Warnings count, Suggestions count.
Results are stored in `/var/log/lynis-report.dat` and `/var/log/lynis.log`.
## Cron Deployment
The script at `/root/.hermes/scripts/lynis-scan.sh`:
```bash
#!/bin/bash
# Nightly security audit via Lynis
# Runs at 3 AM, outputs report, only alerts on new warnings
LOG="/var/log/lynis-report.dat"
PREV="/root/.hermes/lynis-prev.dat"
lynis audit system --quick > /dev/null 2>&1
# Count warnings
WARNS=$(grep -c "^warning\[\]" "$LOG" 2>/dev/null || echo 0)
# Compare with previous run
if [ -f "$PREV" ]; then
OLD_WARNS=$(grep -c "^warning\[\]" "$PREV" 2>/dev/null || echo 0)
else
OLD_WARNS=0
fi
cp "$LOG" "$PREV"
# Only output if something changed
if [ "$WARNS" -ne "$OLD_WARNS" ]; then
echo "Lynis: $WARNS warnings (was $OLD_WARNS)"
grep "^warning\[\]" "$LOG" | sed 's/warning\[\]=//' | tr '|' ' '
else
echo "[SILENT]"
fi
```
Add as no_agent=True cron:
```bash
cronjob(action='create', name='lynis-scan', schedule='0 3 * * *',
script='lynis-scan.sh', no_agent=True, deliver='origin')
```
## Multi-Server Deployment
Copy the script to each server and add to crontab. **Use the server's native crontab** rather than the Hermes cron system — the Hermes cron scheduler only runs on the host box, so it cannot schedule jobs on remote servers. Use SSH + `crontab -l` to deploy remotely:
```bash
scp /root/.hermes/scripts/lynis-scan.sh root@<server>:/usr/local/bin/
ssh root@<server> 'chmod +x /usr/local/bin/lynis-scan.sh && \
(crontab -l 2>/dev/null; echo "0 3 * * * /usr/local/bin/lynis-scan.sh") | crontab -'
```
### Servers deployed
| Server | IP | Date deployed | Status |
|--------|----|-------------|--------|
| Core (netcup) | 152.53.192.33 | Jul 6, 2026 | ✅ Running via cron job |
| wphost02 (Hetzner) | 5.161.62.38 | Jul 7, 2026 | ✅ Running via crontab |
**Watchdog cron jobs should be no_agent=True.** The script itself handles the decision logic (silent unless something changed). Wrapping it in an LLM agent adds unnecessary tokens and causes phantom typing indicators in Telegram. Set `no_agent=True` when creating/updating the cron job for this script.
## Baseline Scores
| Server | Score (out of 100) | Warnings |
|--------|-------------------|----------|
| Netcup (core) | 62 | 3 (vulnerable package, IPv6 DNS) |
| Hetzner wphost02 | 63 | 3 (to review) |
Common warnings:
- **PKGS-7392**: One or more vulnerable packages. Fix with `apt-get update && apt-get upgrade`.
- **NETW-2704/NETW-2705**: Nameserver unreachable (often IPv6 DNS). Fix by disabling IPv6 DNS or adding working resolvers.
@@ -0,0 +1,34 @@
# Data Broker Removal — Action Plan (Jul 2026)
## Recommendation: Use a paid service
| Approach | Year 1 Cost | Time | Effectiveness |
|----------|------------|------|---------------|
| Manual DIY | $0 | 12-20 hrs initial + 2-4 hr/month | Medium |
| Kanary | **$84/yr** | 2 hrs setup, done | High (continuous) |
| DeleteMe | $129/yr | 2 hrs setup, done | High (premium) |
**Why paid:** Data respawns every 30-60 days. A service continuously re-scans and re-submits removals. Manual approaches fizzle because by the time you finish the 50-site list, the first sites have re-added you.
## Priority Tiers (50 sites)
### Priority 1 — The Big 10 (removal cascades to smaller sites)
Spokeo, Whitepages, Intelius, BeenVerified, TruthFinder, PeopleFinders, Radaris, MyLife, Nuwber, PeekYou
### Priority 2 — 20 high-traffic people-search sites
USSearch, ZabaSearch, CheckPeople, GoLookUp, PublicRecords, SocialCatfish, WhitePagesPrime, Addresses, PhoneNumber, InstantCheckmate, PeopleLooker, PeopleSearchNow, PeopleSmart, PublicReports, USPhoneBook, SearchPeopleFree, That'sThem, WhoIs, Yatedo, InfoTracer
### Priority 3 — 10 background check + public record sites
CriminalPages, FederalPrison, Mugshots, BustedMugshots, JailBase, TexasTribune, VineLink, PACER, UniCourt, Trellis
### Priority 4 — 10 marketing data brokers
Acxiom, Epsilon, Oracle Data Cloud, Experian Marketing, Neustar, Lotame, Merkle, Harte Hanks, BlueKai, Quantcast
## Breach Database Checks
- HaveIBeenPwned — https://haveibeenpwned.com
- Dehashed — https://dehashed.com (15B+ breach records)
- Firefox Monitor — https://monitor.firefox.com
- Google Dark Report — https://one.google.com
## Status
On the todo list as `kanary-setup` (pending user authorization to sign up).
@@ -0,0 +1,39 @@
# ITGC Build-Time Checklist
Keep this checklist in mind whenever deploying a NEW service or making significant infrastructure changes. Not every item applies to every project — use judgement.
## Pre-deployment
- [ ] **Access logging** — Does this service log admin actions? Who did what when?
- [ ] **Least privilege** — Is the API key / service account scoped to minimum permissions? Nothing more?
- [ ] **MFA** — Can this service support MFA for admin access? If not, document the gap.
- [ ] **Secrets** — Are credentials in `.env` (chmod 600), not hardcoded in config files?
- [ ] **Backup** — Is the data backed up independently of the server? How often? To where?
- [ ] **Restore test** — Has the backup been tested by actually restoring? (Quarterly minimum)
## Post-deployment
- [ ] **Document** — Is the deployment captured in the project log?
- [ ] **Access list** — Who has access? (SSH keys, API tokens, admin accounts)
- [ ] **Change ticket** — Is there a record of what was done, why, and by whom?
- [ ] **Runbook** — If this service goes down, does someone know how to bring it back?
## Semi-annual
- [ ] **Access review** — Are all SSH keys still needed? Any terminated employees/contractors still have access?
- [ ] **Token rotation** — Are long-lived API tokens still scoped correctly?
- [ ] **Backup restore test** — Actually restore from backup, not just verify backup ran.
## Current gaps (Jul 2026)
These are documented so we can track closure over time:
| Gap | Target | Status |
|-----|--------|--------|
| MFA on SSH/admin | All admin access | ❌ Missing |
| Access reviews | Quarterly | ❌ Missing |
| Change tickets | Every infra change | ❌ Missing |
| Incident response plan | Written + tested | ❌ Missing |
| Backup restore tests | Quarterly | ❌ Never done |
| Password policy | 14+ chars, rotated, MFA | ❌ Missing |
| Encryption at rest | Sensitive data stores | ❌ Missing |