ippadmin/hermes-skills

Fork 0

Files

T

root 1b4fcd4427 Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00

36 KiB

Raw Permalink Blame History

name, description, version, author, tags

name

description

version

author

MikroTik Router Onboarding

Onboard a customer MikroTik router into the IT Pro Partner monitoring infrastructure. The router connects to the portal via WireGuard tunnel and checks in every 60 seconds.

CRITICAL: RouterOS 7 Syntax

RouterOS 7 uses slash-path syntax and equals-sign parameters. Spaced commands (/interface wireguard add name=...) sometimes work but slash-paths (/interface/wireguard/add name=...) are more reliable, especially when commands are piped or pasted via script.

CRITICAL syntax rules:

All ROS7 commands use /path/to/menu/action — forward slashes, not spaces between menu segments
Parameters use name=value with NO spaces around =
String values with spaces need double quotes INSIDE the parameter, not single quotes or no quotes
Line continuation (\ at end of line) works in WinBox terminal but is NOT reliable via SSH. For paste-able scripts, keep each command on one line
No non-command text (ASCII art, markdown, shell comments, separator lines) in the paste-able block — RouterOS will error on lines it can't parse

Correct ROS7 syntax:

/interface/wireguard/add name=wg-itpp private-key="KEY" comment="desc"

NOT (ROS6 style):

/interface wireguard add name=wg-itpp private-key=KEY

Workflow

1. Generate Server-Side WireGuard Config

apt-get install wireguard
wg genkey | tee /root/wg-server.key | wg pubkey > /root/wg-server.pub
wg genkey | tee /root/wg-ccr.key | wg pubkey > /root/wg-ccr.pub

Create /etc/wireguard/wg0.conf:

[Interface]
Address = 10.77.0.1/24
ListenPort = 51820
PrivateKey = <server-private-key>

[Peer]
PublicKey = <ccr-public-key>
AllowedIPs = 10.77.0.2/32, 10.10.10.0/24

Open UFW: ufw allow 51820/udp comment "WireGuard VPN"

2. Generate One-Shot MikroTik Registration Script (ROS7)

Build a single paste-able MikroTik script with NO separator lines, comments, or markdown outside valid RouterOS commands. Every line must be a valid ROS7 command. The complete validated script (tested on RouterOS 7.18.2, CCR2004-16G-2S+):

Tunnel setup (always included):

/interface/wireguard/add name=wg-itpp listen-port=13231 private-key="<key>" comment="IT Pro Partner Tunnel"
/interface/wireguard/peers/add interface=wg-itpp public-key="<server-pub>" endpoint-address=<server-ip> endpoint-port=51820 allowed-address=10.77.0.0/24 persistent-keepalive=25
/ip/address/add address=10.77.0.2/24 interface=wg-itpp
/ip/firewall/filter/add chain=input in-interface=wg-itpp action=accept comment="allow ITPP tunnel" place-before=1
/ip/firewall/filter/add chain=input protocol=udp dst-port=13231 action=accept comment="allow WG (ITPP)" place-before=1
/ip/service/set ssh address=10.77.0.0/24

DNS logging (enabled by default — tested on CCR2004-16G-2S+ RouterOS 7.18.2):

/ip/dns/set allow-remote-requests=yes log-queries=yes log-fwd-queries=yes
/system/logging/add action=remote topics=dns remote=10.77.0.1:514 remote-log-prefix="rtr-CUSTOMER-ID"

The server must be running a syslog receiver on port 514 to collect these. The DNS log captures every domain the router's clients resolve — useful for monitoring and blocked-domain detection without decrypting HTTPS traffic.

SSH key deployment on first-gen MikroTik (no /user/ssh-keys/add)

Some RouterOS 7 builds don't have the /user/ssh-keys/add user= key= command. If add user=admin key="..." gives a syntax error on the router, use the file-based method instead:

/file/add name=itpp-key.pub contents="ssh-ed25519 ..."
/user/ssh-keys/import public-key-file=itpp-key.pub user=admin

Port validity (netcup KVM):

SMTP: only port 2525 works from this netcup KVM — ports 25, 465, 587 all timeout from netcup infrastructure
WireGuard UDP: outbound UDP not blocked by AT&T residential fiber (tested on port 13231)
Keepalive interval: 25s reliably survives CGNAT rebinds on AT&T fiber

SSH key PROOF-OF-CONTENT check (critical): After importing a key but before declaring success, run ssh -i ~/.ssh/<key> admin@<tunnel-ip> "/system/identity/print". If it returns "Permission denied" despite a seemingly successful import, the key CONTENT was wrong. Always verify by running cat ~/.ssh/<key>.pub and visually comparing the base64 string to what was sent to the router. The router accepts ANY key during import — it does NOT validate that the key matches any local file. A key mismatch is the #1 reason for "import said OK, but SSH still asks for password."

CRITICAL: Check which key files exist. The server may have MULTIPLE key pairs (/root/.ssh/*.pub). The key you THINK you're importing (e.g. wisp_rsa) may not be the key you're actually testing with. Always run ls /root/.ssh/*.pub and cat /root/.ssh/<intended-key>.pub to confirm you have the correct file before sending to the router.

Lockout prevention — do NOT change the admin password to a temporary password for SSH key setup. If the key fails to import, and the password was changed to something the operator doesn't know, the router is locked out from SSH until someone has WinBox access. The correct sequence is: (1) verify the key content (cat the .pub file), (2) import the key (add or import command), (3) test the key immediately (ssh -i command), (4) only then let the user change the password back.

SSH key vs password interaction: Once an SSH key is imported for a user, RouterOS switches to key-only authentication for that user (password auth is disabled). If you change the admin password to a temporary one for setup and the key import fails, you will be locked out of SSH. The sequence should always be: import key → test key immediate success → THEN change anything else.

Key generation note

Both server and client WireGuard keys are generated on the SERVER (wg genkey) — the client private key is embedded in the onboarding script. For production at scale, consider generating keys ON the router and registering via API, but the current approach (generate both sides on server) is simpler and validated.

WireGuard on CCR2004-16G-2S+ tested parameters: The validated tunnel was established between a netcup KVM (152.53.192.33) and a home CCR2004-16G-2S+ running RouterOS 7.18.2:

Server port: 51820/udp (must be opened via UFW)
Router listen-port: 13231 (any high port works)
MTU: 1420 (default, fine)
Keepalive: 25s (reliable through NAT)
Latency: 37ms (Ashburn DC to Savannah, GA)
No performance issues after 2+ hours

SSH key import (included for remote management):

/file/add name=itpp-key.pub contents="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA..."
/user/ssh-keys/import public-key-file=itpp-key.pub user=admin

Heartbeat (60-second check-in):

/system/scheduler/add name=itpp-heartbeat interval=1m on-event="/tool/fetch url=\"https://portal.itpropartner.com/api/heartbeat\" http-method=post http-data=\"{\\\"id\\\":\\\"r7g3k9m2\\\"}\" output=none" start-time=startup
:log info "ITPP: Router RTR-CUSTOMER-01 registered"

Paste-able script template (complete)

When generating a one-shot onboarding script for a customer, assemble the following commands. Tested on RouterOS 7.18.2 (CCR2004-16G-2S+):

CRITICAL delivery rule: When presenting the paste-able script to the USER (who will paste into WinBox), include ONLY valid RouterOS commands. No separator lines, no ASCII art, no markdown, no shell comments. The user will copy-paste the ENTIRE block. If there is a single non-command line, RouterOS will error on it. Present the commands as a clean list (numbered or bulleted), then give the user a separate copy-paste block that is bare commands only.

WireGuard key generation rule: Generate BOTH server and client keys on the server side. The client (MikroTik) private key is embedded in the onboarding script. Use wg genkey on the server for both pairs. Document which key belongs to which peer. The server's /etc/wireguard/wg0.conf [Peer] section uses the client's public key.

Tunnel IP allocation rule: Server is always 10.77.0.1. Each new router gets the next available IP in the 10.77.0.0/24 range (.2, .3, .4, etc.). Track allocated IPs to avoid conflicts.

Per-customer script values that vary:

Router identity name (e.g. RTR-CUSTOMER-01)
WireGuard client private key
Tunnel IP (10.77.0.x)
Token (for heartbeat registration)
Remote log prefix (for DNS logging syslog ID)

/interface/wireguard/peers/add interface=wg-itpp public-key="<server-pub>" endpoint-address=<server-ip> endpoint-port=51820 allowed-address=10.77.0.0/24 persistent-keepalive=25
/ip/address/add address=10.77.0.2/24 interface=wg-itpp
/ip/firewall/filter/add chain=input in-interface=wg-itpp action=accept comment="allow ITPP tunnel" place-before=1
/ip/firewall/filter/add chain=input protocol=udp dst-port=13231 action=accept comment="allow WG (ITPP)" place-before=1
/ip/service/set ssh address=10.77.0.0/24

Use backslash line continuation in the documentation display, but for actual paste-able scripts, keep everything on one line per command.

DELIVERY RULE: When presenting the paste-able script to the user, do NOT include ASCII art dividers (===, ---, etc.) or non-RouterOS text in the same block. Paste it as bare commands only. Use a separate section for instructions/context. RouterOS will reject any line that isn't a valid command.

Router Security Hardening (Post-Onboarding Checklist)

After the tunnel is established and SSH key is working, run this hardening sequence. Tested on RouterOS 7.18.2:

1. Remove duplicate firewall rules

The onboarding sequence can create duplicate rules if instructions are pasted multiple times during troubleshooting. Check and clean:

/ip/firewall/filter/print count

If there are duplicate WG rules (same comment), remove extras:

/ip/firewall/filter/remove [find where comment="allow WG (ITPP)"]
/ip/firewall/filter/remove [find where comment="allow ITPP tunnel"]

RouterOS 7.18.2 quirk: Re-running the same command appends a new rule — it does NOT deduplicate by comment. The first paste creates rule A. Troubleshooting may create rules B, C, D with the same comment.

2. Add established/related input rule BEFORE WAN-DROP

The router WILL have forwarding established/related, but may lack it on the INPUT chain. Without this rule, outgoing connections initiated BY the router itself (SMTP, DNS, any outbound-initiated traffic) have their return traffic blocked by WAN-DROP:

Symptom without this rule: SMTP shows "timeout occurred" in /log/print where topics~\"e-mail\" even when DNS resolves and ports are correct. The router's outbound connection establishes, but response packets are dropped by the WAN-DROP rule on the INPUT chain before reaching the service.
Always add during post-onboarding hardening — don't wait for SMTP to fail.

/ip/firewall/filter/add chain=input action=accept connection-state=established,related place-before=1 comment="Allow established/related input"

Symptom without this rule: SMTP shows "timeout occurred" in /log/print where topics~"e-mail" even though DNS and ports are correct. The router can SEND but the response packets are dropped before reaching the service.

SMTP error triage table (check /log/print where topics~"e-mail"):

Log Error	Root Cause	Fix
`DNS resolve failed`	Router DNS empty or AT&T DHCP (192.168.1.254) can't resolve mail.germainebrown.com	`/ip/dns/set servers=10.1.1.14,10.1.1.10`
`timeout occured`	Missing established/related on INPUT chain — SMTP connects but response hits WAN-DROP	Add rule (step 2 above)
`AUTH failed`	Wrong SMTP password	`/tool/e-mail/set password="<correct>"`
`succeeded`	Working	Nothing

Other SMTP rules for netcup infra: Port 2525 only (not 465/587/25). TLS: starttls (not yes/SSL). See Router SMTP Configuration section.

3. Restrict IP services to trusted subnets

/ip/service/set ssh address=10.1.0.0/24,10.1.1.0/24,192.168.88.0/24,10.77.0.0/24
/ip/service/set api address=10.1.0.0/24,10.1.1.0/24
/ip/service/disable api-ssl

Keep Winbox unrestricted if the user accesses from external IPs, or restrict it too.

WARNING: Restricting SSH before adding 10.77.0.0/24 to the allowed list will LOCK OUT the WireGuard connection. Always include the WG tunnel subnet in the allowed list when running via SSH through the tunnel.

4. Disable insecure L2TP auth methods

/interface/l2tp-server/server/set authentication=chap,mschap2

Removes PAP (plaintext passwords) and MSCHAP1 (weak crypto).

5. Clean up stale services

Check for defunct SSTP clients, stale users, and old schedulers:

/interface/sstp-client/print  — check if RemoteWinboxVPN5 or similar exists
/user/print                   — check for users with last-login > 30 days
/system/scheduler/print       — check for defunct resolvers

Remove with:

/interface/sstp-client/remove [find where name="RemoteWinboxVPN5"]
/user/remove [find where name=<stale-user>]
/system/scheduler/remove [find where name=<stale-script>]
/system/script/remove [find where name=<stale-script>]

6. L2TP VPN cleanup

If the WireGuard tunnel replaces L2TP for server access, remove the old PPP secret:

/ppp/secret/remove [find where name=shonuff]

Keep gadmin's L2TP access if the user connects via L2TP from their devices.

7. Remove L2TP client (active outbound connection)

If you had an outbound L2TP connection from the router TO the server:

/interface/l2tp-client/disable [find where name=to-core]
/ppp/active/remove [find where name=shonuff]

The L2TP client will keep reconnecting until disabled.

Router SMTP Configuration

The home router sends scheduled email backups (logs, config, backup files). Two critical settings for this netcup KVM:

Port: 2525 (NOT 465). Port 465 times out from netcup infrastructure. Only 2525 works.
TLS: starttls (NOT yes/SSL). Port 465 uses SSL/TLS; port 2525 uses STARTTLS.
DNS servers: Must be set explicitly. The AT&T DHCP server (192.168.1.254) cannot resolve mail.germainebrown.com. Set to internal DNS:
```
/ip/dns/set servers=10.1.1.14,10.1.1.10
```

Symptom without correct DNS: e-mail,error Error sending e-mail <Subject>: DNS resolve failed

Symptom without established/related input rule: e-mail,error Error sending e-mail <Subject>: timeout occured (SMTP connects but return packets are dropped by WAN-DROP).

Symptom of wrong password: e-mail,error Error sending e-mail <Subject>: AUTH failed

The last-status command confirms delivery:

/tool/e-mail/print

Look for last-status: succeeded. If it shows failed, check /log/print where topics~"e-mail" for the specific error.

Common Pitfalls (Router Security Edition)

Stale WireGuard public key in config export

When you troubleshoot a 0-B-received wire, do NOT trust /export show-sensitive for WireGuard keys. The private key printed in the export was the private key AT THE TIME OF THE EXPORT. If the router regenerated its WG private key since then (e.g. after a config reset or manual update), the public key derived from that export is WRONG.

Always verify against live state:

/interface/wireguard/print detail where name=wg-itpp

The public-key= field in this output is the actual current key. Compare it to what the server's wg show wg0 shows for the peer. If mismatched, update the server's /etc/wireguard/wg0.conf with the correct public key and restart:

sed -i 's|PublicKey = <old-key>|PublicKey = <new-key>|' /etc/wireguard/wg0.conf
systemctl restart wg-quick@wg0

Symptom: wg show wg0 shows TX increasing but RX stays at 0. The server sends keepalives (TX goes up) but the router never responds.

Jul 9 post-migration: Three WG mismatches

When the server migrated from Hetzner (5.161.114.8) to netcup (152.53.192.33), the WG tunnel broke due to THREE simultaneous mismatches:

Stale router public key — Server's wg0.conf used old key from /root/wg-ccr.pub, but the router had regenerated its WG private key since
Stale server public key — Router's peer endpoint still had the Hetzner box's public key, not the new Core's key
Server listen-port mismatch — Router's endpoint-port was 51821 but new server was listening on 51820

Fix: All three had to be corrected at once:

Router's live public key (from /interface/wireguard/print detail) entered into server's wg0.conf
Server's live public key (from wg show) entered into router's peer
Server ListenPort changed to 51821, UFW rule updated, WG restarted

Lesson: When rebuilding a WG tunnel on a new server, DO NOT reuse the old server's keypair or the old router keypair from files. Verify both sides' current keys live and enter them fresh. The router may have rotated keys since the old export.

RouterOS 7.18.2 specific: script errors from SSH pipes

Running commands over ssh user@host <command> (non-interactive) can produce script errors on RouterOS 7.18.2 when the command contains variables, shell metacharacters, or line continuations. These errors don't affect the command execution (the change applies) but they flood the logs with script,error executing script from sshd failed. To minimize these:

Keep SSH commands single-line
Avoid mix of echo and RouterOS commands in the same SSH session
Use /ip/firewall/filter/add place-before=N with an explicit number rather than relying on position-based inserts that change

Backup Workflow (WireGuard tunnel target via wisp-backup.py pipeline)

The home CCR backup uses a two-layer transport: L2TP/IPsec connects the VPS to the router's LAN, then the backup script reaches the router at the WireGuard tunnel IP (10.77.0.2). The backup pipeline uses wisp-backup.py via the run-wisp-backup.sh wrapper (no_agent cron, daily at 6 AM):

Cron script: run-wisp-backup.sh (wrapper at ~/.hermes/scripts/) calls wisp-backup/wisp-backup.py
Config: wisp-backup/config.yaml — defines towers, SSH credentials, S3 bucket
Tower IP: Use the WireGuard tunnel IP (10.77.0.2), NOT the LAN IP (192.168.88.1). The router's SSH service is restricted to 10.77.0.0/24 via /ip/service/set ssh address=
Upload target: s3://mikrotik-ccr-backups/wisp-backups/ (Wasabi, endpoint https://s3.us-east-1.wasabisys.com)
SSH user: shonuff (key-based user, NOT admin)

CRITICAL: Tower IP must match router's SSH address restriction

The home CCR's /ip/service/set ssh address=10.77.0.0/24 means SSH connections are ONLY accepted from the WireGuard tunnel network. Connecting to the LAN IP (e.g. 192.168.88.1) produces "Error reading SSH protocol banner" because:

Even when connected via L2TP/IPsec VPN, the SSH service rejects connections from outside 10.77.0.0/24
paramiko's SSH client hangs at banner read and eventually times out

Fix: Use 10.77.0.2 (WireGuard tunnel IP) in config.yaml, never the LAN IP. Verify with ssh -i ~/.ssh/<key> shonuff@10.77.0.2 "/system/identity/print" — if that works but ssh ... 192.168.88.1 doesn't, the address restriction is the cause.

Pitfalls

Stale WireGuard public key in config export: /export show-sensitive prints the configured key at creation time. If the router regenerates its WG private key since then, the export is stale. Always verify via /interface/wireguard/print detail where name=wg-itpp. This was the root cause of the Jul 9 WireGuard outage on core — server kept sending (TX increased) but router never responded (RX=0) because the public key in server's peer config was from an old export, not the router's current key.
Stuck .in_progress export: If an /export terse file=... command crashes before completing, the export stays in .in_progress state and blocks subsequent SCP. Fix: ssh admin@10.77.0.2 "/file remove <file>.rsc.in_progress"
paramiko must be installed: python3 -c 'import paramiko' must succeed (v5.0.0+). No virtualenv.
VPN package dependency: apt-get install -y xl2tpd strongswan-starter are required on the VPS for the L2TP/IPsec transport layer the backup pipeline uses.
Cron must point to run-wisp-backup.sh: The old home-router-backup.sh is a dead workaround that uploaded to itpropartner-backups/ — verify with grep '"script"' ~/.hermes/cron/jobs.json | grep run-wisp-backup.
The watchdog is WireGuard-only: ping 10.77.0.2, silent unless down. Independent of the backup pipeline.
Logs accumulate in: ~/.hermes/scripts/logs/home-router-backup-YYYY-MM-DD.log

3. SSH Key Deployment on RouterOS 7

Method A — /user/ssh-keys/add (ROS7 native, preferred):

/user/ssh-keys/add user=admin key="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA..." comment="ITPP access key"

Method B — file-based import via WinBox (reliable):

/file/add name=itpp-key.pub contents="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA..."
/user/ssh-keys/import public-key-file=itpp-key.pub user=admin
/file/remove [find name="itpp-key.pub"]

CRITICAL: The correct ROS7 parameter is public-key-file= NOT file=. Using file= alone causes "expected end of command" syntax errors with RouterOS 7.

Multi-Peer WG Tunnel Management

When multiple management servers (Core + app1-bu) connect to the same router via WireGuard, each needs its own setup:

Each server peer gets its own peer entry on the router:

/interface/wireguard/peers/add interface=wg-itpp public-key="<server-pub>" allowed-address=10.77.0.3/32 endpoint-address=<server-ip> endpoint-port=51821 persistent-keepalive=25s comment="app1-bu (Hetzner standby)"

IP allocation: Server = 10.77.0.1. Each peer = next available (.2, .3, etc.). All IPs in 10.77.0.0/24.

When adding a new server peer whose WireGuard keypair was generated on a different box, the router's existing peer config may still reference that server's OLD public key from a previous setup. Always verify the actual current key on the router:

/interface/wireguard/peers/print detail where interface=wg-itpp

Compare each peer's public-key to the corresponding server's wg show output. If Core shows WTwbuq... and the 'Core (netcup)' peer shows aEDf..., the router has the wrong key for Core.

ROUTEROS 7 SSH KEY DEPLOYMENT PITFALLS:

ssh-copy-id does NOT work with RouterOS. RouterOS does not understand POSIX shell commands. Never use ssh-copy-id against a MikroTik. Always use native /user/ssh-keys/add or /user/ssh-keys/import commands. This is a hard failsafe — if the key did not import, verify with /user/ssh-keys/print rather than retrying with ssh-copy-id.
Key content must match exactly. When importing a key, double-check the actual public key on the server with cat ~/.ssh/<key>.pub. A mismatch (e.g. using a different key pair than intended) results in silent SSH denial even after a seemingly successful import — the router accepts the key, it just is not the right one. Verify by comparing the base64 fingerprint on both sides before sending to the router. This is the most common failure mode — re-running the same command with the wrong key produces no error.

CRITICAL: Check which key files exist. The server may have MULTIPLE key pairs (/root/.ssh/*.pub). The key you THINK you're importing (wisp_rsa) may not be the key you're actually testing with. Always run ls /root/.ssh/*.pub and cat /root/.ssh/<intended-key>.pub to confirm you have the correct file. In this session, the server had wisp_rsa.pub and itpp-infra.pub — and the content I was reading was from wisp_rsa.pub (ending in wisp-backup), but I was checking via cat and missing the mismatch. The import succeeded against a key that did NOT match the keyfile I was using for SSH auth. Run the key comparison as a TWO-STEP: (1) cat ~/.ssh/<test-key>.pub to confirm the content, (2) compare that what was sent to the router matches exactly before importing.

2.5 Strategy for SSH-only key setup when router has no WinBox. Use /user/ssh-keys/add (/user/ssh-keys/add user=admin key="ssh-ed25519 ..." user="admin") which accepts inline key text and is more likely to work over an SSH pipe than /user/ssh-keys/import. The import command expects a file reference and may behave differently when stdin is not a real terminal. If you have PTY access (ssh -tt), the import command works fine — but for scripting/pipe mode, add is more reliable.

If the admin user password was changed back after a temporary password was used for setup, SSH key auth may supersede password auth. Test the key immediately after importing BEFORE the user changes their password. If neither auth method actually works (wrong key + unknown password), the router is locked out from SSH until WinBox access is available. Test sequence: import key → ssh -i <key> admin@<ip> → confirm keyless login works → then let user change password.
For remote key setup (when you MUST do it over SSH, not WinBox): Use echo piped to an SSH session with -tt (PTY allocation). This is fragile — prefer having the user paste the add command directly in WinBox. The onboarding script should always include the key add step so the customer does it once.

### 4. Verify

After tunnel is established:

ping 10.77.0.1 from the router
wg show wg0 from the server to see handshake
Once SSH key is imported: ssh -i ~/.ssh/wisp_rsa admin@10.77.0.2 "/system/identity/print"

SSH Key Deployment on RouterOS 7 — Hard Lessons

This section consolidates EVERY pitfall discovered during live onboarding of a CCR2004-16G-2S+ (RouterOS 7.18.2). Follow this sequence religiously.

Fatal sequence (LOCKOUT): temporary password + key import

NEVER change the admin password to a temporary one for SSH key setup. The correct flow is:

Import the SSH key FIRST while the current password is still known
Test the key immediately: ssh -i ~/.ssh/<key> admin@<tunnel-ip> "/system/identity/print"
Confirm keyless login works before any password changes
Only then let the user change the password back

Why: Once an SSH key is imported for a user, RouterOS may switch to key-only authentication (password auth disabled). If the key import failed silently (wrong key content) AND the password was changed to something no one knows, SSH access is dead until someone gets WinBox access. This bit us — the admin password was temporarily changed, the wrong key was imported, and neither auth method worked.

Verify key content, not import success

The router ACCEPTS any key during import — it does NOT validate that the key matches any local file. The #1 failure mode:

Server has key pair: wisp_rsa.pub → content: ssh-ed25519 AAAAC3... wisp-backup
You imported: ssh-ed25519 AAAAC3... OTHER_KEY         ← different base64 string
Router reports: success
SSH test: Permission denied

PROTOCOL:

ls /root/.ssh/*.pub — see ALL available keys
cat ~/.ssh/<intended-key>.pub — read the raw content (compare the last word / comment)
Send that exact content to the router
ssh -i ~/.ssh/<intended-key> admin@<ip> "/system/identity/print" — confirm immediately

DO NOT: use ssh-copy-id against RouterOS (it doesn't work)
DO NOT: change the admin password before keyless login is confirmed
DO NOT: pipe the add user= key= command over SSH without PTY (-tt) — the router's stdin handling is fragile. Prefer WinBox for key import, or echo piped to ssh -tt.

DNS Logging Reference

See references/home-router-reference.md for the live CCR configuration that this onboarding pattern was validated against.

Validated Onboarding Session Reference (Jul 6, 2026, updated Jul 9 2026)

This onboarding was validated LIVE against a production CCR2004-16G-2S+ (RouterOS 7.18.2, home-rtr) via a WireGuard tunnel from netcup KVM (152.53.192.33). Key metrics and lessons:

Hardware:

CCR2004-16G-2S+, ARM64, 4 cores @ 1700MHz, 4GB RAM
RouterOS 7.18.2 (stable, build 2025-03-11)
Uptime: 14 weeks at time of onboarding

Tunnel performance:

Latency: 37ms (Ashburn, VA → Savannah, GA)
0% packet loss
Stable tunnel for entire session (4+ hours)

ACL/policy changes discovered during live test:

The home router had an existing /ip/service/set ssh address= rule locked to a specific subnet — had to be relaxed for the WireGuard tunnel network (10.77.0.0/24). When onboarding existing routers, check for pre-existing SSH address restrictions.
RouterOS had firewall filter rules that were rejecting incoming WireGuard UDP traffic. The place-before=1 directive is CRITICAL to ensure the allow rules apply before default deny rules that may be higher in the chain order. If the tunnel establishes but no data flows through, add allow rules before investigating further.

DNS logging validation:

/ip/dns/set log-queries=yes log-fwd-queries=yes works as documented on RouterOS 7.18.2.
Syslog remote logging destination (/system/logging/add action=remote) requires the server to run a syslog receiver on UDP 514. The netcup box had no syslog daemon listening, so DNS queries were logged locally but not received remotely. TODO: Set up rsyslog or similar on the server side for portal integration.

Port validity:

The SMTP port test against mail.germainebrown.com confirmed that 2525 is the only outbound SMTP port that works from this netcup KVM. Ports 25, 465, and 587 all timeout from netcup infrastructure. This applies to every email-enabled script running on this box.
The home router's ISP (AT&T) does not block outbound WireGuard UDP on port 13231 (the MikroTik listen-port). Other ISPs may differ — a persistent keepalive of 25s worked reliably.
CRITICAL: Check which key files exist. The server may have MULTIPLE key pairs (/root/.ssh/*.pub). The key you THINK you're importing (wisp_rsa) may not be the key you're actually testing with. Always run ls /root/.ssh/*.pub and cat /root/.ssh/<intended-key>.pub to confirm you have the correct file.

ROUTEROS 7 SYNTAX ONLY — ROS6 syntax (/interface wireguard add) may work inconsistently. When a customer copies a script, use ROS7 slash-path syntax for reliability.

user/ssh-keys/import public-key-file= NOT file= — The parameter name is public-key-file=. Using file= alone causes "expected end of command" errors.
ssh-copy-id does NOT work with RouterOS. RouterOS does not understand POSIX shell commands. Never use ssh-copy-id against a MikroTik. Always use native /user/ssh-keys/add or /user/ssh-keys/import commands instead. This is a hard failsafe — if it looks like the key did not import, verify with /user ssh-keys print rather than retrying with ssh-copy-id.
datacenter is not a valid API field — Use location with the two-letter code (ash not ash-dc1).
UFW blocks WireGuard — Always ufw allow 51820/udp before expecting handshakes.
No non-command lines in paste-able script — Divider lines, markdown, shell comments, or formatting will break the paste. Comments starting with # inside the script are fine in WinBox. DELIVERY RULE: Present the paste-able script as a CODE BLOCK with NO surrounding decoration. Do NOT include ASCII art (===, ---), headers, explanatory text, or non-RouterOS syntax in the same block the user copies. RouterOS rejects any line that isn't a valid command — even a markdown heading or separator line causes an "expected command name" error that stops the entire paste. Use a separate markdown section for instructions, then a clean code block for the commands.

When pasting a multiline script into WinBox: DO NOT include separator lines, ASCII art, markdown headings, or any non-RouterOS text in the same code block as the commands. The user copies the entire block. If a single non-RouterOS line is present (e.g. "=== Tunnel Setup ==="), RouterOS errors on that line and stops processing. Present instructions separately from the copy-paste block. Test yourself: if a line doesn't start with /, it probably won't parse. Rule of thumb: every line in the paste-able block must be a valid RouterOS command starting with /. No exceptions.

config export's public key can be stale: RouterOS /export show-sensitive prints the configured private key at file creation time, but if the router later regenerates its WireGuard private key (e.g. after a config reset or manual update), the export file is now stale. The actual public key is only visible via /interface wireguard print detail where name=<wg-interface>. Always verify the actual public key from the live router, not from an old export, when setting up the server-side peer. A stale export causes "0 B received" forever even when ports, IPs, and everything else is correct.
Peer keeps sending but never receives — 0 B received: This is the classic "wrong public key" symptom. The server sends keepalives (TX increases) but the router never responds (RX stays at 0). Diagnosis: /interface wireguard print detail where name=wg-itpp on the router, compare the public-key= value with the server's wg show peer public key. If mismatched, update the server-side config and restart wg-quick@wg0. This was the root cause of the Jul 9 WireGuard outage on core.
Router WG listen-port + server WG listen-port are independent numbers: The router listens on 13231 but connects to the server's 51820 (or 51821). They do NOT need to match. Each side's listen-port is its own inbound, while endpoint-port in the peer is the remote inbound. In this session, the server had moved from port 51820 to 51821 (to distinguish from a stale wg0) and the router peer needed endpoint-port=51821. After changing server listen-port, BOTH the router's peer endpoint-port AND the server's own ListenPort must be updated. The router's endpoint-port tells it WHICH PORT to connect to on the server. If the server's ListenPort is 51821 but the router's endpoint-port is still 51820, they never connect — handshake fails even though the server is listening.
Server public key in router's peer config must match wg show output, not wg pubkey from a stale key file: During the Jul 9 WG fix, the server was trying to connect with a public key from wg-ccr.pub (the OLD client key we generated during initial setup). But the router's private key had been regenerated at some point, producing a different actual public key. The server's peer config must match the router's CURRENT public key as shown by /interface wireguard print detail — not the key we generated alongside the initial config files.
Persistent keepalive: 25 — Prevents NAT timeouts keeping the tunnel alive.
Tunnel IP: 10.77.0.0/24 — Server is .1, router is .2.
Server key management — Keep generated keys in /root/ with .key and .pub extensions for future peer additions.
SSH key must match exactly — When importing a key, double-check the public key content on the server. A mismatch (e.g. using a different key pair than intended) results in silent denial even after a seemingly successful import. Verify with cat ~/.ssh/<key>.pub on the server before sending to the router.
RouterOS will reject password AND key if the password was changed back — Once the admin user has a new password and a key was imported, SSH key auth supersedes password auth. If neither auth method actually works (wrong key + unknown password), you are locked out until WinBox access is available. Always test the key immediately after importing.
Deliver the script as a single block with no separator lines — When presenting the paste-able script to the user, do NOT include ASCII art dividers (===, ---, etc.) or non-RouterOS text in the same block. Paste it as bare commands only. Use a separate section for instructions/context.

36 KiB Raw Permalink Blame History