ippadmin/hermes-skills
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial skills documentation — 25 categories, all SKILL.md + references + scripts

Browse Source
This commit is contained in:
root
2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
skills/devops/mikrotik-onboarding/references/dns-logging-onboarding.md
+32
View File
@@ -0,0 +1,32 @@
# DNS Query Logging — Router Onboarding
Added to the onboarding script after SSH key setup but before heartbeat:
```
/ip/dns/set allow-remote-requests=yes log-queries=yes log-fwd-queries=yes
/system/logging/add action=remote topics=dns remote=10.77.0.1:514 remote-log-prefix="rtr-<customer-id>"
```
## What it captures
| Field | Source |
|---|---|
| Domain | Every DNS query from client devices |
| Client IP | Which internal IP made the query |
| Time | When the query was made |
| Record type | A, AAAA, MX, TXT, etc. |
| Response | Resolved IP or NXDOMAIN |
## Portal display
The router detail page DNS tab shows:
- Total queries, unique domains, active clients, blocked count
- Top domains by query count (last 7 days)
- Top client IPs by query volume
- Individual query log with time, client, domain, type, response
## Limitations
- **HTTPS content is NOT visible** — only domain names from DNS lookups
- Search queries (google.com/q=..., etc.) are NOT visible — they're inside HTTPS
- Only shows queries that go through the router's DNS resolver; devices using external DNS (8.8.8.8) are not logged
skills/devops/mikrotik-onboarding/references/home-router-reference.md
+106
View File
@@ -0,0 +1,106 @@
# Home Router Reference — CCR2004-16G-2S+
Validated onboarding target for the IT Pro Partner WireGuard tunnel pattern.
## Hardware
| Field | Value |
|---|---|
| Board | CCR2004-16G-2S+ |
| Architecture | ARM64 (4 cores @ 1700MHz) |
| RAM | 4 GB |
| Storage | 128 MB NAND |
| RouterOS | 7.18.2 (stable) |
| ISP | AT&T residential fiber (CGNAT) |
## WireGuard Tunnel (Multi-Peer Topology)
The router has **two WireGuard peers** — one for Core (active Hermes) and one for app1-bu (warm standby). Both peers use the same interface `wg-itpp` on the router.
### Peer 1 — Core (netcup)
- **Interface:** wg-itpp
- **Listen port:** 13231 (same interface for both peers)
- **Local IP:** 10.77.0.2/24
- **Peer:** 10.77.0.1 (Core)
- **Router public key (CURRENT):** `1fPwdGQ20CxlZCQZQV134olDcE91hfp78yNDeaKJZzg=`
- **Core public key:** `WTwbuq3gwQDKaE8bdpBo/tFuO8fRcBlKX7ajSn5ry2k=`
- **Endpoint:** 152.53.192.33:51821
- **Keepalive:** 25s
- **MTU:** 1420
### Peer 2 — app1-bu (Hetzner standby)
- **Peer:** 10.77.0.3 (app1-bu)
- **app1-bu public key:** `aEDfY/XzE5m6van/NkvMnYZoYuhptvs1GGkbqmaFKBQ=`
- **Endpoint:** 5.161.114.8:51821
- **Keepalive:** 25s
**Latency:** 37ms (Ashburn → Savannah) for Core, 87ms for app1-bu (Hetzner → Savannah)
**Firewall rules (cleaned, no duplicates):**
- `Allow established/related input` — chain=input, connection-state=established,related, action=accept, place-before=1
- `IPsec IKE` — chain=input, protocol=udp, in-interface-list=WAN, dst-port=500
- `allow WG (ITPP)` — chain=input, protocol=udp, dst-port=13231, action=accept
- `IPsec NAT-T` — chain=input, protocol=udp, in-interface-list=WAN, dst-port=4500
- `L2TP` — chain=input, protocol=udp, in-interface-list=WAN, dst-port=1701
- `IPsec ESP` — chain=input, protocol=ipsec-esp, in-interface-list=WAN
Total firewall rules: 26 (cleaned from 31)
## SSH Access
- **Through WG tunnel (Core):** `ssh -i ~/.ssh/wisp_rsa shonuff@10.77.0.2`
- **SSH restricted to:** 10.1.0.0/24, 10.1.1.0/24, 192.168.88.0/24, 10.77.0.0/24
- **API restricted to:** 10.1.0.0/24, 10.1.1.0/24
- **API-SSL:** disabled
- **Winbox:** unrestricted (user requirement)
## SMTP Configuration (Status: Working)
**Applied settings:**
- Server: `mail.germainebrown.com`
- Port: `2525` (NOT 465 — times out from netcup infra)
- TLS: `starttls` (NOT yes/SSL)
- From/user: `g@germainebrown.com`
- Password: `LoveMyBoys1520!`
- Last status: **succeeded**
**DNS resolvers:** 10.1.1.14 (AdGuard primary), 10.1.1.10 (AdGuard backup)
**Previously on:** 192.168.1.254 (AT&T DHCP) — could not resolve `mail.germainebrown.com`
**Scheduled email jobs (daily at 4 AM, run-count was 0 until SMTP fixes):**
1. Email - Router Logs (`/log print file=home-rtr-logs`, sends `home-rtr-logs.txt`)
2. Email - Router Config (`/export file=home-rtr-config`, sends `home-rtr-config.rsc`)
3. Email - Router Backup (`/system backup save name=home-rtr`, sends `home-rtr.backup`)
**File naming fixed Jul 9:** All three scripts were using mismatched file names (created as `logs.txt` but emailed as `home-rtr-logs.txt`, etc.). Corrected to use consistent naming.
## DNS Logging
```
/ip/dns/set allow-remote-requests=yes log-queries=yes log-fwd-queries=yes
```
System logging outputs are set to `action=remote topics=dns remote=10.77.0.1:514 remote-log-prefix="home-rtr"`.
Note: Server-side syslog receiver on 10.77.0.1:514 is NOT yet configured (UDP 514 is not listening). This is a known gap — DNS logs are stored locally on the router but not yet shipped to the portal.
## Router Security Cleanup (Jul 9, 2026)
Removed:
- 6 duplicate firewall rules (WG rules pasted multiple times during troubleshooting)
- Remote Winbox SSTP client + scheduler + script (RWB_IP_RESOLVER)
- Stale user `52a95oKcepCRctu` (last login March 2026)
- shonuff L2TP PPP secret (WG covers server access now)
- PAP/MSCHAP1 L2TP auth (hardened to CHAP+MSCHAP2 only)
- API-SSL service (disabled)
- SSH/API address restrictions added
Added:
- Established/related INPUT chain rule (required for SMTP, outbound DNS)
## Notes
- The router is behind AT&T residential fiber with CGNAT. WireGuard tunnel with 25s keepalive survives NAT rebinds.
- The wisp_rsa SSH key (comment "wisp-backup") is the key with tunnel access.
- WireGuard keys: server private is `KkXVsdKyiYQVBbA5HbC9sF4dMX16WSmVOkVPYTq8mh8=` (pub: `WTwbuq3gwQDKaE8bdpBo/tFuO8fRcBlKX7ajSn5ry2k=`). Router private key may have been regenerated since initial setup — always verify actual public key via `/interface/wireguard/print detail where name=wg-itpp`, never trust an old config export.
- L2TP client `to-core` is still configured but disabled since WG handles it. Remove if/when L2TP is fully decommissioned.
skills/devops/mikrotik-onboarding/references/router-smtp-troubleshooting.md
+82
View File
@@ -0,0 +1,82 @@
# Router SMTP Troubleshooting Reference
Tested against `mail.germainebrown.com` (MXroute, port 2525) from a CCR2004-16G-2S+ (RouterOS 7.18.2) behind AT&T residential fiber via a netcup KVM.
## Validated Working Config
```
/tool/e-mail/set server=mail.germainebrown.com port=2525 tls=starttls \
from="g@germainebrown.com" password="<correct-password>"
/ip/dns/set servers=10.1.1.14,10.1.1.10
/ip/firewall/filter/add chain=input action=accept connection-state=established,related \
place-before=1 comment="Allow established/related input"
```
## Error Diagnosis Table
| Log Error (`/log/print where topics~"e-mail"`) | Root Cause | Fix |
|---|---|---|
| `DNS resolve failed` | Router DNS empty or AT&T DHCP (192.168.1.254) can't resolve mail.germainebrown.com | `/ip/dns/set servers=<internal-dns>` |
| `timeout occured` | Missing established/related INPUT rule — SMTP connects but return packets are dropped by WAN-DROP | Add input established/related rule (see above) |
| `AUTH failed` | Wrong SMTP password | `/tool/e-mail/set password="<correct>"` |
| `succeeded` | Working | Nothing |
## The established/related INPUT Rule
The router HAS an established/related rule on the **forward** chain (line 13), but unless explicitly added, the **INPUT** chain may lack one. Without it, any connection initiated BY the router (SMTP, DNS, API calls) has its response packets dropped by the WAN-DROP rule on the INPUT chain.
**Symptom:** DNS resolved fine, port was correct, AUTH didn't fail — just `timeout occured`. The router's outbound SYN reached the server, the server responded with SYN-ACK, but WAN-DROP on INPUT killed the response before the SMTP process saw it.
**Fix location:** Rule must be `chain=input` with `place-before=1` so it sits before WAN-DROP. NOT on the forward chain.
## Port Restrictions from netcup KVM
Only port **2525** works for outbound SMTP from this netcup box. Ports 25, 465, 587 all timeout. This is a netcup infrastructure restriction, not a router issue.
- TLS mode for port 2525: **starttls** (not `yes`/SSL)
- TLS mode for port 465: **yes** (SSL/TLS direct)
## DNS Resolution Path
The router's `dynamic-servers` comes from AT&T DHCP (192.168.1.254). AT&T's DNS cannot resolve `mail.germainebrown.com`. The static `servers` override must point to internal DNS (AdGuard at 10.1.1.14, 10.1.1.10).
## Verification Command
```
/tool/e-mail/print
```
Look for `last-status: succeeded`. If `failed`, check `/log/print where topics~"e-mail"` for the specific error and cross-reference the table above.
## Historical Context (Jul 9, 2026)
During the core migration, the home router's SMTP had been broken for months (run-count=0 on all schedulers). The port was still set to 465 (from Hetzner days where port 465 worked). The DNS was using AT&T's DHCP servers. The INPUT chain had no established/related rule. Fixing all three in sequence produced the error progression:
- DNS fix → `timeout occured` (was DNS resolve failed before)
- INPUT rule fix → `AUTH failed` (was timeout before)
- Password update → `succeeded`
## Router Scheduler SMTP (Email - Router Logs / Config / Backup)
Three schedulers on the home router send daily email backups at 4:00 AM. They had `run-count=0` which means they had never successfully executed. When investigating broken schedulers:
1. **Check SMTP first** — `/tool/e-mail/print` shows last-status. If failed, check `/log/print where topics~"e-mail"`
2. **File name mismatch** — Scheduler uses `file=home-rtr-logs.txt` but `/log print file=logs` creates `logs.txt`, not `home-rtr-logs.txt`. The file created and the file attached by the scheduler MUST match exactly. Fix by making `/log print file=home-rtr-logs` create the correct base name, then attach `.txt`.
3. **RouterOS 7.18.2 file naming quirks:**
- `/log print file=<name>` creates `<name>.txt`
- `/export file=<name>` creates `<name>.rsc` (auto-appends .rsc)
- `/system backup save name=<name>` creates `<name>.backup`
- The scheduler's `file=<name>` must include the correct extension
**Fix pattern for scheduler scripts:**
```
# Logs — file base name must match across creation and send
/system/scheduler/set [find where name="Email - Router Logs"] \
on-event="/log print file=home-rtr-logs\n:\delay 10\n/tool e-mail send \
to=\"g@germainebrown.com\" subject=\"...\" body=\"...\" file=home-rtr-logs.txt"
# Config — export auto-appends .rsc
/system/scheduler/set [find where name="Email - Router config (RCS)"] \
on-event="/export file=home-rtr-config\n/tool e-mail send \
to=\"g@germainebrown.com\" subject=\"...\" body=\"...\" file=home-rtr-config.rsc"
```
skills/devops/mikrotik-onboarding/references/ssh-key-deployment.md
+48
View File
@@ -0,0 +1,48 @@
# SSH Key Deployment on RouterOS 7 — Reference
## Methods (use in order of reliability)
### Method 1: /user/ssh-keys/add (ROS7 native, preferred for inline)
```
/user/ssh-keys/add user=admin key="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA..." comment="ITPP access key"
```
This accepts inline key text. More reliable over SSH pipes than `import`.
### Method 2: File-based via WinBox (most reliable)
```
/file/add name=itpp-key.pub contents="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA..."
/user/ssh-keys/import public-key-file=itpp-key.pub user=admin
/file/remove [find name="itpp-key.pub"]
```
**CRITICAL: The correct ROS7 parameter is `public-key-file=` NOT `file=`.** Using `file=` alone causes "expected end of command" errors.
## CRITICAL: Key content verification
This is the #1 failure mode. After importing a key but before declaring success:
1. Check which key files exist on the server: `ls /root/.ssh/*.pub`
2. Check the content of the key you INTEND to use: `cat ~/.ssh/<key>.pub`
3. Compare that base64 string to what was sent to the router
4. Test immediately: `ssh -i ~/.ssh/<key> admin@<tunnel-ip> "/system/identity/print"`
**The router accepts ANY key during import — it does NOT validate. A key mismatch is the most common reason "import said OK, but SSH still asks for password."**
## Lockout prevention
**Do NOT change the admin password to a temporary password for SSH key setup.** The correct sequence:
1. Verify key content (cat the .pub file)
2. Import the key (add or import command)
3. Test the key immediately (ssh -i command) — confirm keyless login works
4. Only then let the user change the password back
**Why:** Once an SSH key is imported, RouterOS disables password auth for that user. If the key doesn't match, and the password was changed to something unknown, the router is locked out from SSH until someone has WinBox access.
## ssh-copy-id does NOT work with RouterOS
RouterOS does not understand POSIX shell commands. Never use `ssh-copy-id` against a MikroTik. Always use native `/user/ssh-keys/add` or `/user/ssh-keys/import`.
## Strategy for SSH-only key setup (no WinBox available)
Use `/user/ssh-keys/add` which accepts inline key text. This is more likely to work over an SSH pipe than `import` (which expects a file reference). If you have PTY access (`ssh -tt`), `import` works fine — but for scripting/pipe mode, `add` is more reliable.