|
|
|
@@ -0,0 +1,475 @@
|
|
|
|
|
---
|
|
|
|
|
name: mikrotik-onboarding
|
|
|
|
|
description: "Onboard MikroTik routers into IT Pro Partner infrastructure — WireGuard tunnel setup, one-shot registration script, heartbeat check-in, remote management access. RouterOS 7 syntax only."
|
|
|
|
|
version: 2.3.0
|
|
|
|
|
author: Sho'Nuff
|
|
|
|
|
tags: [mikrotik, wireguard, onboarding, vpn, infrastructure, routeros7]
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
# MikroTik Router Onboarding
|
|
|
|
|
|
|
|
|
|
Onboard a customer MikroTik router into the IT Pro Partner monitoring infrastructure. The router connects to the portal via WireGuard tunnel and checks in every 60 seconds.
|
|
|
|
|
|
|
|
|
|
## CRITICAL: RouterOS 7 Syntax
|
|
|
|
|
|
|
|
|
|
RouterOS 7 uses **slash-path syntax** and **equals-sign parameters**. Spaced commands (`/interface wireguard add name=...`) sometimes work but slash-paths (`/interface/wireguard/add name=...`) are more reliable, especially when commands are piped or pasted via script.
|
|
|
|
|
|
|
|
|
|
**CRITICAL syntax rules:**
|
|
|
|
|
- All ROS7 commands use `/path/to/menu/action` — forward slashes, not spaces between menu segments
|
|
|
|
|
- Parameters use `name=value` with NO spaces around `=`
|
|
|
|
|
- String values with spaces need double quotes INSIDE the parameter, not single quotes or no quotes
|
|
|
|
|
- Line continuation (`\` at end of line) works in WinBox terminal but is NOT reliable via SSH. For paste-able scripts, keep each command on one line
|
|
|
|
|
- No non-command text (ASCII art, markdown, shell comments, separator lines) in the paste-able block — RouterOS will error on lines it can't parse
|
|
|
|
|
|
|
|
|
|
Correct ROS7 syntax:
|
|
|
|
|
```
|
|
|
|
|
/interface/wireguard/add name=wg-itpp private-key="KEY" comment="desc"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
NOT (ROS6 style):
|
|
|
|
|
```
|
|
|
|
|
/interface wireguard add name=wg-itpp private-key=KEY
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Workflow
|
|
|
|
|
|
|
|
|
|
### 1. Generate Server-Side WireGuard Config
|
|
|
|
|
|
|
|
|
|
```bash
|
|
|
|
|
apt-get install wireguard
|
|
|
|
|
wg genkey | tee /root/wg-server.key | wg pubkey > /root/wg-server.pub
|
|
|
|
|
wg genkey | tee /root/wg-ccr.key | wg pubkey > /root/wg-ccr.pub
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Create `/etc/wireguard/wg0.conf`:
|
|
|
|
|
|
|
|
|
|
```ini
|
|
|
|
|
[Interface]
|
|
|
|
|
Address = 10.77.0.1/24
|
|
|
|
|
ListenPort = 51820
|
|
|
|
|
PrivateKey = <server-private-key>
|
|
|
|
|
|
|
|
|
|
[Peer]
|
|
|
|
|
PublicKey = <ccr-public-key>
|
|
|
|
|
AllowedIPs = 10.77.0.2/32, 10.10.10.0/24
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Open UFW: `ufw allow 51820/udp comment "WireGuard VPN"`
|
|
|
|
|
|
|
|
|
|
### 2. Generate One-Shot MikroTik Registration Script (ROS7)
|
|
|
|
|
|
|
|
|
|
Build a single paste-able MikroTik script with NO separator lines, comments, or markdown outside valid RouterOS commands. Every line must be a valid ROS7 command. The complete validated script (tested on RouterOS 7.18.2, CCR2004-16G-2S+):
|
|
|
|
|
|
|
|
|
|
**Tunnel setup (always included):**
|
|
|
|
|
```
|
|
|
|
|
/interface/wireguard/add name=wg-itpp listen-port=13231 private-key="<key>" comment="IT Pro Partner Tunnel"
|
|
|
|
|
/interface/wireguard/peers/add interface=wg-itpp public-key="<server-pub>" endpoint-address=<server-ip> endpoint-port=51820 allowed-address=10.77.0.0/24 persistent-keepalive=25
|
|
|
|
|
/ip/address/add address=10.77.0.2/24 interface=wg-itpp
|
|
|
|
|
/ip/firewall/filter/add chain=input in-interface=wg-itpp action=accept comment="allow ITPP tunnel" place-before=1
|
|
|
|
|
/ip/firewall/filter/add chain=input protocol=udp dst-port=13231 action=accept comment="allow WG (ITPP)" place-before=1
|
|
|
|
|
/ip/service/set ssh address=10.77.0.0/24
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**DNS logging (enabled by default — tested on CCR2004-16G-2S+ RouterOS 7.18.2):**
|
|
|
|
|
```
|
|
|
|
|
/ip/dns/set allow-remote-requests=yes log-queries=yes log-fwd-queries=yes
|
|
|
|
|
/system/logging/add action=remote topics=dns remote=10.77.0.1:514 remote-log-prefix="rtr-CUSTOMER-ID"
|
|
|
|
|
```
|
|
|
|
|
The server must be running a syslog receiver on port 514 to collect these. The DNS log captures every domain the router's clients resolve — useful for monitoring and blocked-domain detection without decrypting HTTPS traffic.
|
|
|
|
|
|
|
|
|
|
**SSH key deployment on first-gen MikroTik (no /user/ssh-keys/add)**
|
|
|
|
|
|
|
|
|
|
Some RouterOS 7 builds don't have the `/user/ssh-keys/add user= key=` command. If `add user=admin key="..."` gives a syntax error on the router, use the file-based method instead:
|
|
|
|
|
```
|
|
|
|
|
/file/add name=itpp-key.pub contents="ssh-ed25519 ..."
|
|
|
|
|
/user/ssh-keys/import public-key-file=itpp-key.pub user=admin
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Port validity (netcup KVM):**
|
|
|
|
|
- SMTP: only port 2525 works from this netcup KVM — ports 25, 465, 587 all timeout from netcup infrastructure
|
|
|
|
|
- WireGuard UDP: outbound UDP not blocked by AT&T residential fiber (tested on port 13231)
|
|
|
|
|
- Keepalive interval: 25s reliably survives CGNAT rebinds on AT&T fiber
|
|
|
|
|
|
|
|
|
|
**SSH key PROOF-OF-CONTENT check (critical):** After importing a key but before declaring success, run `ssh -i ~/.ssh/<key> admin@<tunnel-ip> "/system/identity/print"`. If it returns "Permission denied" despite a seemingly successful import, the key CONTENT was wrong. Always verify by running `cat ~/.ssh/<key>.pub` and visually comparing the base64 string to what was sent to the router. The router accepts ANY key during import — it does NOT validate that the key matches any local file. A key mismatch is the #1 reason for "import said OK, but SSH still asks for password."
|
|
|
|
|
|
|
|
|
|
**CRITICAL: Check which key files exist.** The server may have MULTIPLE key pairs (`/root/.ssh/*.pub`). The key you THINK you're importing (e.g. `wisp_rsa`) may not be the key you're actually testing with. Always run `ls /root/.ssh/*.pub` and `cat /root/.ssh/<intended-key>.pub` to confirm you have the correct file before sending to the router.
|
|
|
|
|
|
|
|
|
|
**Lockout prevention — do NOT change the admin password to a temporary password for SSH key setup.** If the key fails to import, and the password was changed to something the operator doesn't know, the router is locked out from SSH until someone has WinBox access. The correct sequence is: (1) verify the key content (cat the .pub file), (2) import the key (add or import command), (3) test the key immediately (ssh -i command), (4) only then let the user change the password back.
|
|
|
|
|
|
|
|
|
|
**SSH key vs password interaction:** Once an SSH key is imported for a user, RouterOS switches to key-only authentication for that user (password auth is disabled). If you change the admin password to a temporary one for setup and the key import fails, you will be locked out of SSH. The sequence should always be: import key → test key immediate success → THEN change anything else.
|
|
|
|
|
|
|
|
|
|
**Key generation note**
|
|
|
|
|
|
|
|
|
|
Both server and client WireGuard keys are generated on the SERVER (`wg genkey`) — the client private key is embedded in the onboarding script. For production at scale, consider generating keys ON the router and registering via API, but the current approach (generate both sides on server) is simpler and validated.
|
|
|
|
|
|
|
|
|
|
**WireGuard on CCR2004-16G-2S+ tested parameters:**
|
|
|
|
|
The validated tunnel was established between a netcup KVM (152.53.192.33) and a home CCR2004-16G-2S+ running RouterOS 7.18.2:
|
|
|
|
|
- Server port: 51820/udp (must be opened via UFW)
|
|
|
|
|
- Router listen-port: 13231 (any high port works)
|
|
|
|
|
- MTU: 1420 (default, fine)
|
|
|
|
|
- Keepalive: 25s (reliable through NAT)
|
|
|
|
|
- Latency: 37ms (Ashburn DC to Savannah, GA)
|
|
|
|
|
- No performance issues after 2+ hours
|
|
|
|
|
|
|
|
|
|
**SSH key import (included for remote management):**
|
|
|
|
|
```
|
|
|
|
|
/file/add name=itpp-key.pub contents="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA..."
|
|
|
|
|
/user/ssh-keys/import public-key-file=itpp-key.pub user=admin
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Heartbeat (60-second check-in):**
|
|
|
|
|
```
|
|
|
|
|
/system/scheduler/add name=itpp-heartbeat interval=1m on-event="/tool/fetch url=\"https://portal.itpropartner.com/api/heartbeat\" http-method=post http-data=\"{\\\"id\\\":\\\"r7g3k9m2\\\"}\" output=none" start-time=startup
|
|
|
|
|
:log info "ITPP: Router RTR-CUSTOMER-01 registered"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
## Paste-able script template (complete)
|
|
|
|
|
|
|
|
|
|
When generating a one-shot onboarding script for a customer, assemble the following commands. Tested on RouterOS 7.18.2 (CCR2004-16G-2S+):
|
|
|
|
|
|
|
|
|
|
**CRITICAL delivery rule:** When presenting the paste-able script to the USER (who will paste into WinBox), include ONLY valid RouterOS commands. No separator lines, no ASCII art, no markdown, no shell comments. The user will copy-paste the ENTIRE block. If there is a single non-command line, RouterOS will error on it. Present the commands as a clean list (numbered or bulleted), then give the user a separate copy-paste block that is bare commands only.
|
|
|
|
|
|
|
|
|
|
**WireGuard key generation rule:** Generate BOTH server and client keys on the server side. The client (MikroTik) private key is embedded in the onboarding script. Use `wg genkey` on the server for both pairs. Document which key belongs to which peer. The server's `/etc/wireguard/wg0.conf` [Peer] section uses the client's public key.
|
|
|
|
|
|
|
|
|
|
**Tunnel IP allocation rule:** Server is always 10.77.0.1. Each new router gets the next available IP in the 10.77.0.0/24 range (.2, .3, .4, etc.). Track allocated IPs to avoid conflicts.
|
|
|
|
|
|
|
|
|
|
**Per-customer script values that vary:**
|
|
|
|
|
- Router identity name (e.g. RTR-CUSTOMER-01)
|
|
|
|
|
- WireGuard client private key
|
|
|
|
|
- Tunnel IP (10.77.0.x)
|
|
|
|
|
- Token (for heartbeat registration)
|
|
|
|
|
- Remote log prefix (for DNS logging syslog ID)
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
/interface/wireguard/peers/add interface=wg-itpp public-key="<server-pub>" endpoint-address=<server-ip> endpoint-port=51820 allowed-address=10.77.0.0/24 persistent-keepalive=25
|
|
|
|
|
/ip/address/add address=10.77.0.2/24 interface=wg-itpp
|
|
|
|
|
/ip/firewall/filter/add chain=input in-interface=wg-itpp action=accept comment="allow ITPP tunnel" place-before=1
|
|
|
|
|
/ip/firewall/filter/add chain=input protocol=udp dst-port=13231 action=accept comment="allow WG (ITPP)" place-before=1
|
|
|
|
|
/ip/service/set ssh address=10.77.0.0/24
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
Use backslash line continuation in the documentation display, but for actual paste-able scripts, keep everything on one line per command.
|
|
|
|
|
|
|
|
|
|
**DELIVERY RULE:** When presenting the paste-able script to the user, do NOT include ASCII art dividers (===, ---, etc.) or non-RouterOS text in the same block. Paste it as bare commands only. Use a separate section for instructions/context. RouterOS will reject any line that isn't a valid command.
|
|
|
|
|
|
|
|
|
|
## Router Security Hardening (Post-Onboarding Checklist)
|
|
|
|
|
|
|
|
|
|
After the tunnel is established and SSH key is working, run this hardening sequence. Tested on RouterOS 7.18.2:
|
|
|
|
|
|
|
|
|
|
### 1. Remove duplicate firewall rules
|
|
|
|
|
The onboarding sequence can create duplicate rules if instructions are pasted multiple times during troubleshooting. Check and clean:
|
|
|
|
|
```
|
|
|
|
|
/ip/firewall/filter/print count
|
|
|
|
|
```
|
|
|
|
|
If there are duplicate WG rules (same comment), remove extras:
|
|
|
|
|
```
|
|
|
|
|
/ip/firewall/filter/remove [find where comment="allow WG (ITPP)"]
|
|
|
|
|
/ip/firewall/filter/remove [find where comment="allow ITPP tunnel"]
|
|
|
|
|
```
|
|
|
|
|
**RouterOS 7.18.2 quirk:** Re-running the same command appends a new rule — it does NOT deduplicate by comment. The first paste creates rule A. Troubleshooting may create rules B, C, D with the same comment.
|
|
|
|
|
|
|
|
|
|
### 2. Add established/related input rule BEFORE WAN-DROP
|
|
|
|
|
|
|
|
|
|
The router WILL have forwarding established/related, but may lack it on the INPUT chain. Without this rule, outgoing connections initiated BY the router itself (SMTP, DNS, any outbound-initiated traffic) have their return traffic blocked by WAN-DROP:
|
|
|
|
|
|
|
|
|
|
- **Symptom without this rule:** SMTP shows \"timeout occurred\" in `/log/print where topics~\"e-mail\"` even when DNS resolves and ports are correct. The router's outbound connection establishes, but response packets are dropped by the WAN-DROP rule on the INPUT chain before reaching the service.
|
|
|
|
|
- **Always add during post-onboarding hardening** — don't wait for SMTP to fail.
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
/ip/firewall/filter/add chain=input action=accept connection-state=established,related place-before=1 comment="Allow established/related input"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Symptom without this rule:** SMTP shows "timeout occurred" in `/log/print where topics~"e-mail"` even though DNS and ports are correct. The router can SEND but the response packets are dropped before reaching the service.
|
|
|
|
|
|
|
|
|
|
**SMTP error triage table (check `/log/print where topics~"e-mail"`):**
|
|
|
|
|
|
|
|
|
|
| Log Error | Root Cause | Fix |
|
|
|
|
|
|---|---|---|
|
|
|
|
|
| `DNS resolve failed` | Router DNS empty or AT&T DHCP (192.168.1.254) can't resolve mail.germainebrown.com | `/ip/dns/set servers=10.1.1.14,10.1.1.10` |
|
|
|
|
|
| `timeout occured` | Missing established/related on INPUT chain — SMTP connects but response hits WAN-DROP | Add rule (step 2 above) |
|
|
|
|
|
| `AUTH failed` | Wrong SMTP password | `/tool/e-mail/set password="<correct>"` |
|
|
|
|
|
| `succeeded` | Working | Nothing |
|
|
|
|
|
|
|
|
|
|
**Other SMTP rules for netcup infra:** Port **2525 only** (not 465/587/25). TLS: **starttls** (not yes/SSL). See Router SMTP Configuration section.
|
|
|
|
|
|
|
|
|
|
### 3. Restrict IP services to trusted subnets
|
|
|
|
|
```
|
|
|
|
|
/ip/service/set ssh address=10.1.0.0/24,10.1.1.0/24,192.168.88.0/24,10.77.0.0/24
|
|
|
|
|
/ip/service/set api address=10.1.0.0/24,10.1.1.0/24
|
|
|
|
|
/ip/service/disable api-ssl
|
|
|
|
|
```
|
|
|
|
|
Keep Winbox unrestricted if the user accesses from external IPs, or restrict it too.
|
|
|
|
|
|
|
|
|
|
**WARNING:** Restricting SSH before adding 10.77.0.0/24 to the allowed list will LOCK OUT the WireGuard connection. Always include the WG tunnel subnet in the allowed list when running via SSH through the tunnel.
|
|
|
|
|
|
|
|
|
|
### 4. Disable insecure L2TP auth methods
|
|
|
|
|
```
|
|
|
|
|
/interface/l2tp-server/server/set authentication=chap,mschap2
|
|
|
|
|
```
|
|
|
|
|
Removes PAP (plaintext passwords) and MSCHAP1 (weak crypto).
|
|
|
|
|
|
|
|
|
|
### 5. Clean up stale services
|
|
|
|
|
Check for defunct SSTP clients, stale users, and old schedulers:
|
|
|
|
|
```
|
|
|
|
|
/interface/sstp-client/print — check if RemoteWinboxVPN5 or similar exists
|
|
|
|
|
/user/print — check for users with last-login > 30 days
|
|
|
|
|
/system/scheduler/print — check for defunct resolvers
|
|
|
|
|
```
|
|
|
|
|
Remove with:
|
|
|
|
|
```
|
|
|
|
|
/interface/sstp-client/remove [find where name="RemoteWinboxVPN5"]
|
|
|
|
|
/user/remove [find where name=<stale-user>]
|
|
|
|
|
/system/scheduler/remove [find where name=<stale-script>]
|
|
|
|
|
/system/script/remove [find where name=<stale-script>]
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
### 6. L2TP VPN cleanup
|
|
|
|
|
If the WireGuard tunnel replaces L2TP for server access, remove the old PPP secret:
|
|
|
|
|
```
|
|
|
|
|
/ppp/secret/remove [find where name=shonuff]
|
|
|
|
|
```
|
|
|
|
|
Keep gadmin's L2TP access if the user connects via L2TP from their devices.
|
|
|
|
|
|
|
|
|
|
### 7. Remove L2TP client (active outbound connection)
|
|
|
|
|
If you had an outbound L2TP connection from the router TO the server:
|
|
|
|
|
```
|
|
|
|
|
/interface/l2tp-client/disable [find where name=to-core]
|
|
|
|
|
/ppp/active/remove [find where name=shonuff]
|
|
|
|
|
```
|
|
|
|
|
The L2TP client will keep reconnecting until disabled.
|
|
|
|
|
|
|
|
|
|
## Router SMTP Configuration
|
|
|
|
|
|
|
|
|
|
The home router sends scheduled email backups (logs, config, backup files). Two critical settings for this netcup KVM:
|
|
|
|
|
|
|
|
|
|
- **Port: 2525** (NOT 465). Port 465 times out from netcup infrastructure. Only 2525 works.
|
|
|
|
|
- **TLS: starttls** (NOT yes/SSL). Port 465 uses SSL/TLS; port 2525 uses STARTTLS.
|
|
|
|
|
- **DNS servers:** Must be set explicitly. The AT&T DHCP server (192.168.1.254) cannot resolve `mail.germainebrown.com`. Set to internal DNS:
|
|
|
|
|
```
|
|
|
|
|
/ip/dns/set servers=10.1.1.14,10.1.1.10
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Symptom without correct DNS:** `e-mail,error Error sending e-mail <Subject>: DNS resolve failed`
|
|
|
|
|
|
|
|
|
|
**Symptom without established/related input rule:** `e-mail,error Error sending e-mail <Subject>: timeout occured` (SMTP connects but return packets are dropped by WAN-DROP).
|
|
|
|
|
|
|
|
|
|
**Symptom of wrong password:** `e-mail,error Error sending e-mail <Subject>: AUTH failed`
|
|
|
|
|
|
|
|
|
|
The last-status command confirms delivery:
|
|
|
|
|
```
|
|
|
|
|
/tool/e-mail/print
|
|
|
|
|
```
|
|
|
|
|
Look for `last-status: succeeded`. If it shows `failed`, check `/log/print where topics~"e-mail"` for the specific error.
|
|
|
|
|
|
|
|
|
|
## Common Pitfalls (Router Security Edition)
|
|
|
|
|
|
|
|
|
|
### Stale WireGuard public key in config export
|
|
|
|
|
When you troubleshoot a 0-B-received wire, do NOT trust `/export show-sensitive` for WireGuard keys. The private key printed in the export was the private key AT THE TIME OF THE EXPORT. If the router regenerated its WG private key since then (e.g. after a config reset or manual update), the public key derived from that export is WRONG.
|
|
|
|
|
|
|
|
|
|
**Always verify against live state:**
|
|
|
|
|
```
|
|
|
|
|
/interface/wireguard/print detail where name=wg-itpp
|
|
|
|
|
```
|
|
|
|
|
The `public-key=` field in this output is the actual current key. Compare it to what the server's `wg show wg0` shows for the peer. If mismatched, update the server's `/etc/wireguard/wg0.conf` with the correct public key and restart:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
sed -i 's|PublicKey = <old-key>|PublicKey = <new-key>|' /etc/wireguard/wg0.conf
|
|
|
|
|
systemctl restart wg-quick@wg0
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Symptom:** `wg show wg0` shows TX increasing but RX stays at 0. The server sends keepalives (TX goes up) but the router never responds.
|
|
|
|
|
|
|
|
|
|
### Jul 9 post-migration: Three WG mismatches
|
|
|
|
|
When the server migrated from Hetzner (5.161.114.8) to netcup (152.53.192.33), the WG tunnel broke due to THREE simultaneous mismatches:
|
|
|
|
|
|
|
|
|
|
1. **Stale router public key** — Server's `wg0.conf` used old key from `/root/wg-ccr.pub`, but the router had regenerated its WG private key since
|
|
|
|
|
2. **Stale server public key** — Router's peer endpoint still had the Hetzner box's public key, not the new Core's key
|
|
|
|
|
3. **Server listen-port mismatch** — Router's `endpoint-port` was 51821 but new server was listening on 51820
|
|
|
|
|
|
|
|
|
|
**Fix:** All three had to be corrected at once:
|
|
|
|
|
- Router's live public key (from `/interface/wireguard/print detail`) entered into server's `wg0.conf`
|
|
|
|
|
- Server's live public key (from `wg show`) entered into router's peer
|
|
|
|
|
- Server `ListenPort` changed to 51821, UFW rule updated, WG restarted
|
|
|
|
|
|
|
|
|
|
**Lesson:** When rebuilding a WG tunnel on a new server, DO NOT reuse the old server's keypair or the old router keypair from files. Verify both sides' current keys live and enter them fresh. The router may have rotated keys since the old export.
|
|
|
|
|
|
|
|
|
|
### RouterOS 7.18.2 specific: script errors from SSH pipes
|
|
|
|
|
Running commands over `ssh user@host <command>` (non-interactive) can produce script errors on RouterOS 7.18.2 when the command contains variables, shell metacharacters, or line continuations. These errors don't affect the command execution (the change applies) but they flood the logs with `script,error executing script from sshd failed`. To minimize these:
|
|
|
|
|
- Keep SSH commands single-line
|
|
|
|
|
- Avoid mix of `echo` and RouterOS commands in the same SSH session
|
|
|
|
|
- Use `/ip/firewall/filter/add place-before=N` with an explicit number rather than relying on position-based inserts that change
|
|
|
|
|
|
|
|
|
|
## Backup Workflow (WireGuard tunnel target via wisp-backup.py pipeline)
|
|
|
|
|
|
|
|
|
|
The home CCR backup uses a two-layer transport: L2TP/IPsec connects the VPS to the router's LAN, then the backup script reaches the router at the **WireGuard tunnel IP** (10.77.0.2).
|
|
|
|
|
The backup pipeline uses `wisp-backup.py` via the `run-wisp-backup.sh` wrapper (no_agent cron, daily at 6 AM):
|
|
|
|
|
|
|
|
|
|
- **Cron script**: `run-wisp-backup.sh` (wrapper at `~/.hermes/scripts/`) calls `wisp-backup/wisp-backup.py`
|
|
|
|
|
- **Config**: `wisp-backup/config.yaml` — defines towers, SSH credentials, S3 bucket
|
|
|
|
|
- **Tower IP**: Use the **WireGuard tunnel IP** (10.77.0.2), NOT the LAN IP (192.168.88.1). The router's SSH service is restricted to `10.77.0.0/24` via `/ip/service/set ssh address=`
|
|
|
|
|
- **Upload target**: `s3://mikrotik-ccr-backups/wisp-backups/` (Wasabi, endpoint `https://s3.us-east-1.wasabisys.com`)
|
|
|
|
|
- **SSH user**: `shonuff` (key-based user, NOT admin)
|
|
|
|
|
|
|
|
|
|
### CRITICAL: Tower IP must match router's SSH address restriction
|
|
|
|
|
|
|
|
|
|
The home CCR's `/ip/service/set ssh address=10.77.0.0/24` means SSH connections are ONLY accepted from the WireGuard tunnel network. Connecting to the LAN IP (e.g. `192.168.88.1`) produces "Error reading SSH protocol banner" because:
|
|
|
|
|
|
|
|
|
|
1. Even when connected via L2TP/IPsec VPN, the SSH service rejects connections from outside 10.77.0.0/24
|
|
|
|
|
2. paramiko's SSH client hangs at banner read and eventually times out
|
|
|
|
|
|
|
|
|
|
**Fix: Use 10.77.0.2 (WireGuard tunnel IP) in config.yaml, never the LAN IP.** Verify with `ssh -i ~/.ssh/<key> shonuff@10.77.0.2 "/system/identity/print"` — if that works but `ssh ... 192.168.88.1` doesn't, the address restriction is the cause.
|
|
|
|
|
|
|
|
|
|
### Pitfalls
|
|
|
|
|
|
|
|
|
|
- **Stale WireGuard public key in config export:** `/export show-sensitive` prints the configured key at creation time. If the router regenerates its WG private key since then, the export is stale. Always verify via `/interface/wireguard/print detail where name=wg-itpp`. This was the root cause of the Jul 9 WireGuard outage on core — server kept sending (TX increased) but router never responded (RX=0) because the public key in server's peer config was from an old export, not the router's current key.
|
|
|
|
|
|
|
|
|
|
- **Stuck `.in_progress` export**: If an `/export terse file=...` command crashes before completing, the export stays in `.in_progress` state and blocks subsequent SCP. Fix: `ssh admin@10.77.0.2 "/file remove <file>.rsc.in_progress"`
|
|
|
|
|
- **paramiko must be installed**: `python3 -c 'import paramiko'` must succeed (v5.0.0+). No virtualenv.
|
|
|
|
|
- **VPN package dependency**: `apt-get install -y xl2tpd strongswan-starter` are required on the VPS for the L2TP/IPsec transport layer the backup pipeline uses.
|
|
|
|
|
- **Cron must point to `run-wisp-backup.sh`**: The old `home-router-backup.sh` is a dead workaround that uploaded to `itpropartner-backups/` — verify with `grep '"script"' ~/.hermes/cron/jobs.json | grep run-wisp-backup`.
|
|
|
|
|
- **The watchdog is WireGuard-only**: ping 10.77.0.2, silent unless down. Independent of the backup pipeline.
|
|
|
|
|
- **Logs accumulate in**: `~/.hermes/scripts/logs/home-router-backup-YYYY-MM-DD.log`
|
|
|
|
|
|
|
|
|
|
### 3. SSH Key Deployment on RouterOS 7
|
|
|
|
|
|
|
|
|
|
**Method A — /user/ssh-keys/add (ROS7 native, preferred):**
|
|
|
|
|
```
|
|
|
|
|
/user/ssh-keys/add user=admin key="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA..." comment="ITPP access key"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Method B — file-based import via WinBox (reliable):**
|
|
|
|
|
```
|
|
|
|
|
/file/add name=itpp-key.pub contents="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA..."
|
|
|
|
|
/user/ssh-keys/import public-key-file=itpp-key.pub user=admin
|
|
|
|
|
/file/remove [find name="itpp-key.pub"]
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**CRITICAL: The correct ROS7 parameter is `public-key-file=` NOT `file=`.** Using `file=` alone causes "expected end of command" syntax errors with RouterOS 7.
|
|
|
|
|
|
|
|
|
|
## Multi-Peer WG Tunnel Management
|
|
|
|
|
|
|
|
|
|
When multiple management servers (Core + app1-bu) connect to the same router via WireGuard, each needs its own setup:
|
|
|
|
|
|
|
|
|
|
**Each server peer gets its own peer entry on the router:**
|
|
|
|
|
```
|
|
|
|
|
/interface/wireguard/peers/add interface=wg-itpp public-key="<server-pub>" allowed-address=10.77.0.3/32 endpoint-address=<server-ip> endpoint-port=51821 persistent-keepalive=25s comment="app1-bu (Hetzner standby)"
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**IP allocation:** Server = 10.77.0.1. Each peer = next available (.2, .3, etc.). All IPs in 10.77.0.0/24.
|
|
|
|
|
|
|
|
|
|
**When adding a new server peer whose WireGuard keypair was generated on a different box, the router's existing peer config may still reference that server's OLD public key** from a previous setup. Always verify the actual current key on the router:
|
|
|
|
|
```
|
|
|
|
|
/interface/wireguard/peers/print detail where interface=wg-itpp
|
|
|
|
|
```
|
|
|
|
|
Compare each peer's public-key to the corresponding server's `wg show` output. If Core shows `WTwbuq...` and the 'Core (netcup)' peer shows `aEDf...`, the router has the wrong key for Core.
|
|
|
|
|
|
|
|
|
|
**ROUTEROS 7 SSH KEY DEPLOYMENT PITFALLS:**
|
|
|
|
|
|
|
|
|
|
1. **`ssh-copy-id` does NOT work with RouterOS.** RouterOS does not understand POSIX shell commands. Never use `ssh-copy-id` against a MikroTik. Always use native `/user/ssh-keys/add` or `/user/ssh-keys/import` commands. This is a hard failsafe — if the key did not import, verify with `/user/ssh-keys/print` rather than retrying with ssh-copy-id.
|
|
|
|
|
|
|
|
|
|
2. **Key content must match exactly.** When importing a key, double-check the actual public key on the server with `cat ~/.ssh/<key>.pub`. A mismatch (e.g. using a different key pair than intended) results in silent SSH denial even after a seemingly successful import — the router accepts the key, it just is not the right one. **Verify by comparing the base64 fingerprint on both sides before sending to the router.** This is the most common failure mode — re-running the same command with the wrong key produces no error.
|
|
|
|
|
|
|
|
|
|
**CRITICAL: Check which key files exist.** The server may have MULTIPLE key pairs (`/root/.ssh/*.pub`). The key you THINK you're importing (`wisp_rsa`) may not be the key you're actually testing with. Always run `ls /root/.ssh/*.pub` and `cat /root/.ssh/<intended-key>.pub` to confirm you have the correct file. In this session, the server had `wisp_rsa.pub` and `itpp-infra.pub` — and the content I was reading was from `wisp_rsa.pub` (ending in `wisp-backup`), but I was checking via `cat` and missing the mismatch. The import succeeded against a key that did NOT match the keyfile I was using for SSH auth. Run the key comparison as a TWO-STEP: (1) `cat ~/.ssh/<test-key>.pub` to confirm the content, (2) compare that what was sent to the router matches exactly before importing.
|
|
|
|
|
|
|
|
|
|
2.5 **Strategy for SSH-only key setup when router has no WinBox.** Use `/user/ssh-keys/add` (`/user/ssh-keys/add user=admin key="ssh-ed25519 ..." user="admin"`) which accepts inline key text and is more likely to work over an SSH pipe than `/user/ssh-keys/import`. The `import` command expects a file reference and may behave differently when stdin is not a real terminal. If you have PTY access (`ssh -tt`), the `import` command works fine — but for scripting/pipe mode, `add` is more reliable.
|
|
|
|
|
|
|
|
|
|
3. **If the admin user password was changed back after a temporary password was used for setup, SSH key auth may supersede password auth.** Test the key immediately after importing BEFORE the user changes their password. If neither auth method actually works (wrong key + unknown password), the router is locked out from SSH until WinBox access is available. **Test sequence: import key → `ssh -i <key> admin@<ip>` → confirm keyless login works → then let user change password.**
|
|
|
|
|
|
|
|
|
|
4. **For remote key setup (when you MUST do it over SSH, not WinBox):** Use `echo` piped to an SSH session with `-tt` (PTY allocation). This is fragile — prefer having the user paste the add command directly in WinBox. The onboarding script should always include the key add step so the customer does it once.
|
|
|
|
|
|
|
|
|
|
<a name="verify"></a>### 4. Verify
|
|
|
|
|
|
|
|
|
|
After tunnel is established:
|
|
|
|
|
- `ping 10.77.0.1` from the router
|
|
|
|
|
- `wg show wg0` from the server to see handshake
|
|
|
|
|
- Once SSH key is imported: `ssh -i ~/.ssh/wisp_rsa admin@10.77.0.2 "/system/identity/print"`
|
|
|
|
|
|
|
|
|
|
## SSH Key Deployment on RouterOS 7 — Hard Lessons
|
|
|
|
|
|
|
|
|
|
This section consolidates EVERY pitfall discovered during live onboarding of a CCR2004-16G-2S+ (RouterOS 7.18.2). Follow this sequence religiously.
|
|
|
|
|
|
|
|
|
|
### Fatal sequence (LOCKOUT): temporary password + key import
|
|
|
|
|
|
|
|
|
|
**NEVER change the admin password to a temporary one for SSH key setup.** The correct flow is:
|
|
|
|
|
|
|
|
|
|
1. Import the SSH key FIRST while the current password is still known
|
|
|
|
|
2. Test the key immediately: `ssh -i ~/.ssh/<key> admin@<tunnel-ip> "/system/identity/print"`
|
|
|
|
|
3. **Confirm keyless login works** before any password changes
|
|
|
|
|
4. Only then let the user change the password back
|
|
|
|
|
|
|
|
|
|
**Why:** Once an SSH key is imported for a user, RouterOS may switch to key-only authentication (password auth disabled). If the key import failed silently (wrong key content) AND the password was changed to something no one knows, SSH access is dead until someone gets WinBox access. This bit us — the admin password was temporarily changed, the wrong key was imported, and neither auth method worked.
|
|
|
|
|
|
|
|
|
|
### Verify key content, not import success
|
|
|
|
|
|
|
|
|
|
The router ACCEPTS any key during import — it does NOT validate that the key matches any local file. The #1 failure mode:
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
Server has key pair: wisp_rsa.pub → content: ssh-ed25519 AAAAC3... wisp-backup
|
|
|
|
|
You imported: ssh-ed25519 AAAAC3... OTHER_KEY ← different base64 string
|
|
|
|
|
Router reports: success
|
|
|
|
|
SSH test: Permission denied
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**PROTOCOL:**
|
|
|
|
|
1. `ls /root/.ssh/*.pub` — see ALL available keys
|
|
|
|
|
2. `cat ~/.ssh/<intended-key>.pub` — read the raw content (compare the last word / comment)
|
|
|
|
|
3. Send that exact content to the router
|
|
|
|
|
4. `ssh -i ~/.ssh/<intended-key> admin@<ip> "/system/identity/print"` — confirm immediately
|
|
|
|
|
|
|
|
|
|
**DO NOT:** use `ssh-copy-id` against RouterOS (it doesn't work)
|
|
|
|
|
**DO NOT:** change the admin password before keyless login is confirmed
|
|
|
|
|
**DO NOT:** pipe the `add user= key=` command over SSH without PTY (`-tt`) — the router's stdin handling is fragile. Prefer WinBox for key import, or `echo` piped to `ssh -tt`.
|
|
|
|
|
|
|
|
|
|
## DNS Logging Reference
|
|
|
|
|
|
|
|
|
|
See `references/home-router-reference.md` for the live CCR configuration that this onboarding pattern was validated against.
|
|
|
|
|
|
|
|
|
|
## Validated Onboarding Session Reference (Jul 6, 2026, updated Jul 9 2026)
|
|
|
|
|
|
|
|
|
|
This onboarding was validated LIVE against a production CCR2004-16G-2S+ (RouterOS 7.18.2, home-rtr) via a WireGuard tunnel from netcup KVM (152.53.192.33). Key metrics and lessons:
|
|
|
|
|
|
|
|
|
|
**Hardware:**
|
|
|
|
|
- CCR2004-16G-2S+, ARM64, 4 cores @ 1700MHz, 4GB RAM
|
|
|
|
|
- RouterOS 7.18.2 (stable, build 2025-03-11)
|
|
|
|
|
- Uptime: 14 weeks at time of onboarding
|
|
|
|
|
|
|
|
|
|
**Tunnel performance:**
|
|
|
|
|
- Latency: 37ms (Ashburn, VA → Savannah, GA)
|
|
|
|
|
- 0% packet loss
|
|
|
|
|
- Stable tunnel for entire session (4+ hours)
|
|
|
|
|
|
|
|
|
|
**ACL/policy changes discovered during live test:**
|
|
|
|
|
- The home router had an existing `/ip/service/set ssh address=` rule locked to a specific subnet — had to be relaxed for the WireGuard tunnel network (10.77.0.0/24). When onboarding existing routers, check for pre-existing SSH address restrictions.
|
|
|
|
|
- RouterOS had firewall filter rules that were rejecting incoming WireGuard UDP traffic. The `place-before=1` directive is CRITICAL to ensure the allow rules apply before default deny rules that may be higher in the chain order. If the tunnel establishes but no data flows through, add allow rules before investigating further.
|
|
|
|
|
|
|
|
|
|
**DNS logging validation:**
|
|
|
|
|
- `/ip/dns/set log-queries=yes log-fwd-queries=yes` works as documented on RouterOS 7.18.2.
|
|
|
|
|
- Syslog remote logging destination (`/system/logging/add action=remote`) requires the server to run a syslog receiver on UDP 514. The netcup box had no syslog daemon listening, so DNS queries were logged locally but not received remotely. **TODO: Set up rsyslog or similar on the server side** for portal integration.
|
|
|
|
|
|
|
|
|
|
**Port validity:**
|
|
|
|
|
- The SMTP port test against mail.germainebrown.com confirmed that **2525 is the only outbound SMTP port** that works from this netcup KVM. Ports 25, 465, and 587 all timeout from netcup infrastructure. This applies to every email-enabled script running on this box.
|
|
|
|
|
- The home router's ISP (AT&T) does not block outbound WireGuard UDP on port 13231 (the MikroTik listen-port). Other ISPs may differ — a persistent keepalive of 25s worked reliably.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- **CRITICAL: Check which key files exist.** The server may have MULTIPLE key pairs (`/root/.ssh/*.pub`). The key you THINK you're importing (`wisp_rsa`) may not be the key you're actually testing with. Always run `ls /root/.ssh/*.pub` and `cat /root/.ssh/<intended-key>.pub` to confirm you have the correct file.
|
|
|
|
|
|
|
|
|
|
**ROUTEROS 7 SYNTAX ONLY** — ROS6 syntax (`/interface wireguard add`) may work inconsistently. When a customer copies a script, use ROS7 slash-path syntax for reliability.
|
|
|
|
|
- **`user/ssh-keys/import public-key-file=` NOT `file=`** — The parameter name is `public-key-file=`. Using `file=` alone causes "expected end of command" errors.
|
|
|
|
|
- **ssh-copy-id does NOT work with RouterOS.** RouterOS does not understand POSIX shell commands. Never use `ssh-copy-id` against a MikroTik. Always use native `/user/ssh-keys/add` or `/user/ssh-keys/import` commands instead. This is a hard failsafe — if it looks like the key did not import, verify with `/user ssh-keys print` rather than retrying with ssh-copy-id.
|
|
|
|
|
- **`datacenter` is not a valid API field** — Use `location` with the two-letter code (`ash` not `ash-dc1`).
|
|
|
|
|
- **UFW blocks WireGuard** — Always `ufw allow 51820/udp` before expecting handshakes.
|
|
|
|
|
- **No non-command lines in paste-able script** — Divider lines, markdown, shell comments, or formatting will break the paste. Comments starting with `#` inside the script are fine in WinBox.
|
|
|
|
|
**DELIVERY RULE:** Present the paste-able script as a CODE BLOCK with NO surrounding decoration. Do NOT include ASCII art (===, ---), headers, explanatory text, or non-RouterOS syntax in the same block the user copies. RouterOS rejects any line that isn't a valid command — even a markdown heading or separator line causes an "expected command name" error that stops the entire paste. Use a separate markdown section for instructions, then a clean code block for the commands.
|
|
|
|
|
|
|
|
|
|
**When pasting a multiline script into WinBox:** DO NOT include separator lines, ASCII art, markdown headings, or any non-RouterOS text in the same code block as the commands. The user copies the entire block. If a single non-RouterOS line is present (e.g. "=== Tunnel Setup ==="), RouterOS errors on that line and stops processing. Present instructions separately from the copy-paste block. Test yourself: if a line doesn't start with `/`, it probably won't parse. Rule of thumb: every line in the paste-able block must be a valid RouterOS command starting with `/`. No exceptions.
|
|
|
|
|
- **config export's public key can be stale:** RouterOS `/export show-sensitive` prints the configured private key at file creation time, but if the router later regenerates its WireGuard private key (e.g. after a config reset or manual update), the export file is now stale. The actual public key is only visible via `/interface wireguard print detail where name=<wg-interface>`. Always verify the actual public key from the live router, not from an old export, when setting up the server-side peer. A stale export causes "0 B received" forever even when ports, IPs, and everything else is correct.
|
|
|
|
|
- **Peer keeps sending but never receives — 0 B received:** This is the classic "wrong public key" symptom. The server sends keepalives (TX increases) but the router never responds (RX stays at 0). Diagnosis: `/interface wireguard print detail where name=wg-itpp` on the router, compare the `public-key=` value with the server's `wg show` peer public key. If mismatched, update the server-side config and restart `wg-quick@wg0`. This was the root cause of the Jul 9 WireGuard outage on core.
|
|
|
|
|
- **Router WG listen-port + server WG listen-port are independent numbers:** The router listens on 13231 but connects to the server's 51820 (or 51821). They do NOT need to match. Each side's `listen-port` is its own inbound, while `endpoint-port` in the peer is the remote inbound. In this session, the server had moved from port 51820 to 51821 (to distinguish from a stale wg0) and the router peer needed `endpoint-port=51821`. **After changing server listen-port, BOTH the router's peer `endpoint-port` AND the server's own `ListenPort` must be updated.** The router's `endpoint-port` tells it WHICH PORT to connect to on the server. If the server's `ListenPort` is 51821 but the router's `endpoint-port` is still 51820, they never connect — handshake fails even though the server is listening.
|
|
|
|
|
- **Server public key in router's peer config must match `wg show` output, not `wg pubkey` from a stale key file:** During the Jul 9 WG fix, the server was trying to connect with a public key from `wg-ccr.pub` (the OLD client key we generated during initial setup). But the router's private key had been regenerated at some point, producing a different actual public key. The server's peer config must match the router's CURRENT public key as shown by `/interface wireguard print detail` — not the key we generated alongside the initial config files.
|
|
|
|
|
- **Persistent keepalive: 25** — Prevents NAT timeouts keeping the tunnel alive.
|
|
|
|
|
- **Tunnel IP: 10.77.0.0/24** — Server is .1, router is .2.
|
|
|
|
|
- **Server key management** — Keep generated keys in `/root/` with `.key` and `.pub` extensions for future peer additions.
|
|
|
|
|
- **SSH key must match exactly** — When importing a key, double-check the public key content on the server. A mismatch (e.g. using a different key pair than intended) results in silent denial even after a seemingly successful import. Verify with `cat ~/.ssh/<key>.pub` on the server before sending to the router.
|
|
|
|
|
- **RouterOS will reject password AND key if the password was changed back** — Once the admin user has a new password and a key was imported, SSH key auth supersedes password auth. If neither auth method actually works (wrong key + unknown password), you are locked out until WinBox access is available. Always test the key immediately after importing.
|
|
|
|
|
- **Deliver the script as a single block with no separator lines** — When presenting the paste-able script to the user, do NOT include ASCII art dividers (===, ---, etc.) or non-RouterOS text in the same block. Paste it as bare commands only. Use a separate section for instructions/context.
|
root