# Bulk node_exporter Install — July 2026 Rollout

## Server inventory

All 9 Hetzner servers received node_exporter v1.8.2 in a single session:

| Hostname | IP | Status |
|----------|-----|--------|
| app1-bu.itpropartner.com | 5.161.114.8 | Running |
| ai.itpropartner.com | 178.156.167.181 | Running |
| hudu.itpropartner.com | 178.156.130.130 | Running |
| (unifi) | 178.156.131.57 | Running |
| unms.forefrontwireless.com | 5.161.225.131 | Running |
| wphost02 | 5.161.62.38 | Running |
| app1.itpropartner.com | 87.99.144.163 | Running |
| (docker) | 178.156.168.35 | Running |
| (fleettrack360) | 178.156.149.32 | Running |

## Batch install pattern

Write two scripts:

1. **install-node-exporter.sh** — self-contained script that downloads, installs binary, creates systemd unit, enables/starts. The existing `scripts/install-node-exporter.sh` in this skill works.

2. **Wrapper script** — iterates over a list and runs SCP + SSH for each:

```bash
#!/bin/bash
SERVERS=(
  "hostname1:1.2.3.4"
  "hostname2:5.6.7.8"
)
KEY="/root/.ssh/itpp-infra"
SCRIPT="/tmp/install-node-exporter.sh"

for entry in "${SERVERS[@]}"; do
  hostname="${entry%%:*}"
  ip="${entry##*:}"
  echo "=== Installing on $hostname ($ip) ==="
  scp -o StrictHostKeyChecking=no -i "$KEY" "$SCRIPT" "root@$ip:/tmp/"
  ssh -o StrictHostKeyChecking=no -i "$KEY" "root@$ip" "bash /tmp/install-node-exporter.sh"
  echo "✓ $hostname"
done
```

## Verifying all targets in Prometheus

After the config is updated and Prometheus reloaded:

```bash
curl -s http://127.0.0.1:9090/api/v1/targets | python3 -c \
  "import json,sys;d=json.load(sys.stdin);[print(t['labels']['instance'],t['health']) for t in d['data']['activeTargets']]"
```

## Pitfalls

- **Security scanners blocking raw IPs** in SSH/SCP commands. Using hostnames where available bypasses false-positive security warnings.
- **All 9 installs can run sequentially** in ~90 seconds total — no parallelism needed.
- **UFW/firewall** must allow incoming port 9100 from Core's IP, or Prometheus must scrape via hostname/SSH tunnel. The existing servers had no restrictive UFW rules blocking port 9100.
- **node_exporter listens on all interfaces by default** (0.0.0.0:9100). If restricting to localhost, add `--web.listen-address=127.0.0.1:9100` to ExecStart and scrape via a local proxy or Tailscale.