hermes-skills/skills/devops/infrastructure-automation/references/wisp-ccr-backup-pipeline.md

WISP CCR Daily Backup Pipeline — Session Detail

Overview

Automated daily backup of MikroTik CCR tower routers behind an L2TP/IPsec VPN. Connects to a WISP gateway, tunnels into the network, SSHs each tower, exports config + logs, gzips, and uploads to S3 organized by site/date.

Files on disk

All live under ~/.hermes/scripts/wisp-backup/:

File	Purpose
config.yaml	YAML with VPN creds, tower inventory, routed subnets, S3 config
wisp-vpn.sh	Shell script: L2TP/IPsec up/down/status + adds/removes static routes
wisp-backup.py	Python: SSH loop via paramiko, export, compress, upload to S3
~/.hermes/scripts/run-wisp-backup.sh	Cron wrapper: logs to file, silent on success, alert on failure
~/.ssh/wisp_rsa	Ed25519 key pair — one key deployed to all towers

config.yaml structure

vpn:
  server_ip: "PUBLIC_IP"
  psk: "<ipsec-psk>"
  username: "<l2tp-user>"
  password: "<l2tp-pass>"
  interface: "ppp0"
  routes:
    - "172.16.1.0/24"
    - "172.16.2.0/24"

s3:
  bucket: "my-wisp-backups"
  region: "us-east-1"
  prefix: "wisp-backups"
  endpoint: "https://s3.us-east-1.wasabisys.com"  # Wasabi; omit for real AWS S3

towers:
  - name: "tower-alpha"
    ip: "10.0.1.1"
    site: "black-creek"
    ssh_port: 22
    ssh_user: "wisp-backup"
    ssh_key: "/root/.ssh/wisp_rsa"

backup:
  log_lines: 500
  temp_dir: "/root/.hermes/.backups/wisp-backups"  # Disk-backed! NOT /tmp — tmpfs fills up
  compress: true
  timeout_seconds: 60

RouterOS v7 service user setup (VERIFIED against official manual)

Official docs: https://help.mikrotik.com/docs/spaces/ROS/pages/8978504/User (frozen but authoritative).

On each CCR tower — create SSH service user + import key

# Create read-only backup user (safest minimum)
/user add name=wisp-backup group=read disabled=no

# Option A: Import key from an uploaded file (needs SCP/FTP first)
/user ssh-keys import public-key-file=wisp_rsa.pub user=wisp-backup

# Option B: Paste key directly (no file transfer needed — v7+)
/user ssh-keys/add user=wisp-backup key="ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDnI4UwwPL8gJvtP/Jr7qiw0Qj/bQBwi2+f03p730xvn wisp-backup"

Default read group policies (v7): local,telnet,ssh,reboot,read,test,winbox,password,web,sniff,sensitive,api,romon,rest-api — with !ftp !write !policy

This means ssh and read are both enabled. /export terse works under read on v7 because it does not alter the router's configuration.

On the L2TP gateway — create tunnel user

# Local management user (for web/ssh to the gateway itself)
/user add name=wisp-backup group=read disabled=no

# PPP secret that authorizes the L2TP tunnel (separate from /user!)
# This is what actually lets the VPS connect via L2TP
/ppp secret add name=wisp-backup password="<strong-password>" service=l2tp

Important distinction: The /user entry controls local RouterOS management access. The /ppp secret entry authorizes the L2TP tunnel itself. Both need to exist for the backup pipeline to work — one for the tunnel, one in case you ever need to SSH directly to the gateway.

SSH key

Generated on VPS once, deployed to all towers:

ssh-keygen -t ed25519 -f /root/.ssh/wisp_rsa -N "" -C "wisp-backup"

Public key: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDnI4UwwPL8gJvtP/Jr7qiw0Qj/bQBwi2+f03p730xvn wisp-backup

Important: read_file and grep both mask secret-bearing config values as sk-xxxx...xxxx with literal ellipsis. To extract the real key value for copying into a profile's config.yaml, use xxd:

sed -n '<line>p' /root/.hermes/config.yaml | xxd

Each hex byte pair is the literal character — concatenate and clean up the whitespace.

Key design decisions from this session

Routed network architecture

• Each tower sits on its own subnet behind the gateway (e.g. 172.16.1.0/24, 172.16.2.0/24, etc.)
• After VPN connects, the VPS needs static routes added through ppp0 for each subnet — they don't appear automatically
• The VPN script (wisp-vpn.sh) parses vpn.routes from config.yaml and adds/deletes them in up/down handlers
• Verify with: ip route show 172.16.1.0/24 and ping 10.0.1.1

Separation of credentials

• L2TP user on the gateway — one service account for VPN connectivity
• SSH user per CCR tower — wisp-backup in group read, key-only auth
• The SSH key is shared across all towers for simplicity (trusted management network)

RouterOS v7 quirks

• ssh-keys is under /user ssh-keys on v7 (not a separate menu item)
• Supported key formats: PEM, PKCS#8, OpenSSH — Ed25519 preferred
• If password auth is attempted and a key exists, the key takes precedence; password auth fails silently
• Commands are NOT interactive — exec_command + recv_exit_status() works as expected
• Paramiko AutoAddPolicy is needed since CCRs don't have persistent host keys the VPS would recognize

Cron

• Schedule: daily at 2:00 AM ET (6:00 UTC) — 0 6 * * *
• Job ID: 9f04af6d75d0
• Type: no_agent (watchdog pattern — silent on success, alert on failure)
• Wrapper: ~/.hermes/scripts/run-wisp-backup.sh

First-time test sequence

# 1. Dry run (shows tower list without connecting)
python3 ~/.hermes/scripts/wisp-backup/wisp-backup.py --dry-run

# 2. Single tower test (leaves VPN connected for inspection)
python3 ~/.hermes/scripts/wisp-backup/wisp-backup.py --tower tower-alpha

# 3. Verify S3 upload
aws s3 ls s3://<bucket>/wisp-backups/configs/black-creek/

# 4. Full run
python3 ~/.hermes/scripts/wisp-backup/wisp-backup.py

# 5. Once verified, the cron handles it automatically

Troubleshooting

Symptom	Likely fix
VPN won't connect — IPsec negotiation reaches KE exchange then no shared key found	On strongSwan 6, the generic : PSK "..." format can fail. Use explicit IPs: 5.161.114.8 76.195.7.60 : PSK "...". Also ensure the secrets file doesn't have trailing YAML comments (use the sed strip-comment pattern).
NO_PROPOSAL_CHOSEN after IKE_SA established	ESP proposal mismatch. Add -modp1024 to esp line: esp=aes128-sha1-modp1024. MikroTik L2TP/IPsec expects MODP group in ESP proposals.
pppd exits with unrecognized option 'lock'	pppd 2.5.x removed the lock option. Omit it from ppp options files.
VPN connects but internet dies, can't reach VPS	You used replacedefaultroute in the ppp options. Remove it — use defaultroute + explicit static routes instead. If it's already happening, disable the ppp secret on the MikroTik to restore connectivity, then fix the options file.
L2TP call connects and immediately disconnects	Check journalctl -u xl2tpd for pppd exit codes. Code 2 = option error (usually lock or incompatible pppd version). Code 16 = modem hangup / peer disconnect.
Inline comments in YAML pollute parsed values	The bash get_cfg pattern: `grep -A50 "^vpn:" "$YAML"
[route] $subnet already exists from Already running a VPN.	Normal if the cron runs while a previous session left it up. VPN script's up handler checks for existing routes and skips dups.
SSH timeout	Tower might be offline OR the route isn't added. After VPN connects, check ip route show <tower-subnet>. If missing, the routes: list in config.yaml is wrong or empty.
/export terse returns nothing	Verify the SSH user has read policy. On v7, group=read should be sufficient.
S3 upload fails	Check ~/.aws/credentials and aws sts get-caller-identity.
Parallel SSH hangs	Reduce --parallel workers. CCRs sometimes throttle multiple connections. Start with 3.
Password in config has $ or "	Use single-quoted heredocs in bash or write credentials from Python directly. Shell will interpolate $VAR in double-quoted heredocs.