Initial skills documentation — 25 categories, all SKILL.md + references + scripts

2026-07-15 17:42:20 -04:00
commit 1b4fcd4427
976 changed files with 188344 additions and 0 deletions
@@ -0,0 +1,60 @@
+# Bulk node_exporter Install — July 2026 Rollout
+
+## Server inventory
+
+All 9 Hetzner servers received node_exporter v1.8.2 in a single session:
+
+| Hostname | IP | Status |
+|----------|-----|--------|
+| app1-bu.itpropartner.com | 5.161.114.8 | Running |
+| ai.itpropartner.com | 178.156.167.181 | Running |
+| hudu.itpropartner.com | 178.156.130.130 | Running |
+| (unifi) | 178.156.131.57 | Running |
+| unms.forefrontwireless.com | 5.161.225.131 | Running |
+| wphost02 | 5.161.62.38 | Running |
+| app1.itpropartner.com | 87.99.144.163 | Running |
+| (docker) | 178.156.168.35 | Running |
+| (fleettrack360) | 178.156.149.32 | Running |
+
+## Batch install pattern
+
+Write two scripts:
+
+1. **install-node-exporter.sh** — self-contained script that downloads, installs binary, creates systemd unit, enables/starts. The existing `scripts/install-node-exporter.sh` in this skill works.
+
+2. **Wrapper script** — iterates over a list and runs SCP + SSH for each:
+
+```bash
+#!/bin/bash
+SERVERS=(
+  "hostname1:1.2.3.4"
+  "hostname2:5.6.7.8"
+)
+KEY="/root/.ssh/itpp-infra"
+SCRIPT="/tmp/install-node-exporter.sh"
+
+for entry in "${SERVERS[@]}"; do
+  hostname="${entry%%:*}"
+  ip="${entry##*:}"
+  echo "=== Installing on $hostname ($ip) ==="
+  scp -o StrictHostKeyChecking=no -i "$KEY" "$SCRIPT" "root@$ip:/tmp/"
+  ssh -o StrictHostKeyChecking=no -i "$KEY" "root@$ip" "bash /tmp/install-node-exporter.sh"
+  echo "✓ $hostname"
+done
+```
+
+## Verifying all targets in Prometheus
+
+After the config is updated and Prometheus reloaded:
+
+```bash
+curl -s http://127.0.0.1:9090/api/v1/targets | python3 -c \
+  "import json,sys;d=json.load(sys.stdin);[print(t['labels']['instance'],t['health']) for t in d['data']['activeTargets']]"
+```
+
+## Pitfalls
+
+- **Security scanners blocking raw IPs** in SSH/SCP commands. Using hostnames where available bypasses false-positive security warnings.
+- **All 9 installs can run sequentially** in ~90 seconds total — no parallelism needed.
+- **UFW/firewall** must allow incoming port 9100 from Core's IP, or Prometheus must scrape via hostname/SSH tunnel. The existing servers had no restrictive UFW rules blocking port 9100.
+- **node_exporter listens on all interfaces by default** (0.0.0.0:9100). If restricting to localhost, add `--web.listen-address=127.0.0.1:9100` to ExecStart and scrape via a local proxy or Tailscale.