Initial skills documentation — 25 categories, all SKILL.md + references + scripts
This commit is contained in:
root
@@ -0,0 +1,205 @@
|
||||
---
|
||||
name: tailscale-infrastructure-access
|
||||
description: "Set up and manage Tailscale for private infrastructure access — install, authenticate, Tailscale Serve for internal HTTPS, device enrollment, and the policy that internal services stay behind the tailnet."
|
||||
version: 1.0.0
|
||||
author: ShoNuff
|
||||
license: MIT
|
||||
platforms: [linux, macos, ios]
|
||||
metadata:
|
||||
hermes:
|
||||
tags: [tailscale, vpn, networking, security, internal-services]
|
||||
related_skills: [docker-service-deployment, infrastructure-automation]
|
||||
---
|
||||
|
||||
# Tailscale Infrastructure Access
|
||||
|
||||
Tailscale provides a zero-config VPN for private infrastructure access. Internal services (Vaultwarden, future databases, admin panels) live behind the tailnet — no public ports, no DNS, no TLS cert worries.
|
||||
|
||||
## Policy
|
||||
|
||||
**Internal services stay behind Tailscale.** No public exposure. No DNS records for admin-only services. Access is via:
|
||||
|
||||
1. **Tailscale Serve** — HTTPS proxy within the tailnet (for web UIs)
|
||||
2. **Direct Tailscale IP + port** — for API/CLI access
|
||||
3. **SSH over Tailscale** — once public SSH is locked down
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
curl -fsSL https://tailscale.com/install.sh | sh
|
||||
```
|
||||
|
||||
### Authentication
|
||||
|
||||
```bash
|
||||
tailscale up
|
||||
# Prints: https://login.tailscale.com/a/XXXXX
|
||||
# Open in browser, sign in with any identity provider (Google, Microsoft, GitHub, Apple)
|
||||
# A self-hosted email domain works fine — the email is just a login identity
|
||||
```
|
||||
|
||||
### Verify
|
||||
|
||||
```bash
|
||||
tailscale status
|
||||
# Shows: <tailscale-ip> <hostname> <user> <os>
|
||||
```
|
||||
|
||||
## Device Enrollment
|
||||
|
||||
Users install the Tailscale app on each device:
|
||||
|
||||
| Device | Where to get it |
|
||||
|---|---|
|
||||
| macOS | tailscale.com/download-mac or Mac App Store |
|
||||
| iOS/iPad | App Store → "Tailscale" |
|
||||
| iPhone | App Store → "Tailscale" |
|
||||
| Linux | `curl -fsSL https://tailscale.com/install.sh | sh` |
|
||||
| Windows | tailscale.com/download-windows |
|
||||
|
||||
All devices log in with the same identity provider (Google Workspace account recommended).
|
||||
|
||||
### Renaming the server hostname for clarity
|
||||
|
||||
```bash
|
||||
# Default hostname is the OS hostname (e.g. v2202607377162478911)
|
||||
# Rename to something readable:
|
||||
tailscale set --hostname app1
|
||||
|
||||
# Verify:
|
||||
tailscale status
|
||||
# → 100.71.155.7 app1 user@email.com linux
|
||||
```
|
||||
|
||||
## Tailscale Serve (internal HTTPS)
|
||||
|
||||
Expose a local port as HTTPS within your tailnet — no public domain needed, no Let's Encrypt, automatic TLS certs.
|
||||
|
||||
```bash
|
||||
tailscale serve --bg --https 443 --set-path / http://127.0.0.1:<port>
|
||||
|
||||
# Result:
|
||||
# https://<hostname>.tail<XXXXX>.ts.net/
|
||||
# |-- proxy http://127.0.0.1:<port>
|
||||
```
|
||||
|
||||
The URL is only resolvable within the tailnet. Anyone outside gets a DNS failure.
|
||||
|
||||
### Updating the Serve config after renaming
|
||||
|
||||
If you rename the hostname, Tailscale Serve doesn't move automatically. Remove and re-add:
|
||||
|
||||
```bash
|
||||
# Old approach:
|
||||
tailscale serve --https=443 off
|
||||
# Then re-add with --bg --https 443
|
||||
```
|
||||
|
||||
### "Web page keeps spinning / 404" troubleshooting
|
||||
|
||||
If the page loads but the app hangs, check:
|
||||
1. **DOMAIN environment variable** on the service matches the actual URL
|
||||
2. For Bitwarden-compatible apps, the API must advertise the same URL users type
|
||||
3. `curl http://localhost:<port>` from the server to confirm the service is healthy
|
||||
|
||||
### "Password incorrect" after changing domain
|
||||
|
||||
Changing the URL (and thus `DOMAIN`) changes KDF/key derivation parameters cached by the client app. Fix:
|
||||
- In the Bitwarden app: Settings → Account → "Delete account from device" (not "Delete account")
|
||||
- Re-add server with new URL
|
||||
- The app fetches fresh parameters before prompting for password
|
||||
|
||||
### Ambiguous characters in the URL — mobile app can't connect
|
||||
|
||||
The auto-generated hostname (e.g. `vaultwarden.tailc2f3b0.ts.net`) can contain characters that mobile apps confuse (e.g. `0` vs `O`, `1` vs `l`, `5` vs `S`). If a mobile Bitwarden app shows "This is not a recognized Bitwarden server" or just spins:
|
||||
|
||||
**Fix — rename hostname, re-add Serve, update DOMAIN:**
|
||||
|
||||
```bash
|
||||
# 1. Rename hostname to something with no ambiguous chars
|
||||
tailscale set --hostname vaultwarden
|
||||
|
||||
# 2. Re-add Tailscale Serve (it auto-generates new URL)
|
||||
tailscale serve --https=443 off 2>/dev/null || true
|
||||
tailscale serve --bg --https 443 --set-path / http://127.0.0.1:<port>
|
||||
|
||||
# 3. Update the service's DOMAIN env var to match new URL
|
||||
# In docker-compose.yml, update DOMAIN=https://new-hostname.tailXXXXX.ts.net
|
||||
docker compose up -d
|
||||
|
||||
# 4. Force the client app to re-fetch config
|
||||
# Delete the server entry from device, force-close app, re-add fresh URL
|
||||
```
|
||||
|
||||
**If the Bitwarden app still says "not a recognized server":** The API config at `/api/config` advertises what the server expects. What you type in the app must match the `vault` URL exactly — including http vs https and any port. Verify with `curl https://new-hostname.tailXXXXX.ts.net/api/config | jq '.environment'`.
|
||||
|
||||
## Accessing Services
|
||||
|
||||
### Vaultwarden example
|
||||
|
||||
```bash
|
||||
# On any enrolled device:
|
||||
# Open: https://<hostname>.tail<XXXXX>.ts.net/
|
||||
# Or if using raw Tailscale IP:
|
||||
# Open: http://100.71.155.7:8080
|
||||
```
|
||||
|
||||
The Bitwarden app configuration:
|
||||
- Server URL: `https://vaultwarden.tailcXXXXX.ts.net` (or IP)
|
||||
- Self-hosted environment → enter URL once, app fetches API/identity/sso URLs from `/api/config`
|
||||
|
||||
### SSH over Tailscale
|
||||
|
||||
Once Tailscale is the only access method:
|
||||
|
||||
```bash
|
||||
ssh root@100.71.155.7
|
||||
# No public IP needed. No port forwarding.
|
||||
```
|
||||
|
||||
## Firewall
|
||||
|
||||
UFW should block service ports on all interfaces and open them only on `tailscale0`:
|
||||
|
||||
```bash
|
||||
# Block all external access
|
||||
ufw deny <port>/tcp
|
||||
|
||||
# Allow only from tailscale interface
|
||||
ufw allow in on tailscale0 to any port <port> proto tcp comment '<Service> via Tailscale'
|
||||
```
|
||||
|
||||
Current rules on app1:
|
||||
- 22 (SSH) — all interfaces (public, may lock down later)
|
||||
- 80, 443 — all interfaces (public web)
|
||||
- 8080 — tailscale0 only (Vaultwarden)
|
||||
|
||||
## Security safeguards: user preferences
|
||||
|
||||
This user asked explicitly about safeguards against hallucinations and unauthorized actions. See `references/security-preferences.md` for the full document.
|
||||
|
||||
Hard rules carried forward from user feedback:
|
||||
- **Before any destructive action** (reboot, config deploy, DNS change, firewall edit, container destroy), state target hostname + IP to the user and wait for confirmation. Never act on opaque IDs alone.
|
||||
- **Default to read-only.** Unless told to enter "write mode," explore only.
|
||||
- **No inventing config keys or features.** If a Hermes feature isn't confirmed in docs/skill/source, say "I don't know."
|
||||
- **API keys get minimum permissions** — Cloudflare DNS-only, Wasabi write-only to backup buckets, Stripe read-only.
|
||||
- **Kill switch:** `/lockdown` → remove SSH key, stop all cron jobs, stop email processing, report done.
|
||||
|
||||
## Pitfalls
|
||||
|
||||
- **Tailscale Serve and hostname are linked.** If you change the server's Tailscale hostname, the Serve URL changes. Re-add Serve to pick up the new name, then update any service DOMAIN config.
|
||||
- **Tailscale can conflict with local Caddy on port 443.** When Tailscale is installed and running, `tailscaled` binds port 443 on its Tailnet IP (e.g. `100.71.155.7:443`). If you also run a local Caddy HTTPS server on :443, Caddy fails with `bind: address already in use` because `:443` captures all interfaces including the Tailscale one. Check with `ss -tlnp | grep 443` — if `tailscaled` holds the port on a specific IP (not `0.0.0.0`), the clean fix is:
|
||||
|
||||
**Add `default_bind <public-ip>` to the Caddyfile global block** — this tells Caddy to bind only to the public IP, avoiding the Tailscale interface. Both services then run on port 443 simultaneously without conflict. See `references/caddy-tailscale-port-443-conflict.md` for the full fix.
|
||||
|
||||
If `default_bind` isn't suitable, fallback options:
|
||||
- Run Caddy on an alt port (e.g., 8443) and use iptables/redirect
|
||||
- Disable Tailscale's port 443 usage: `tailscale serve --https=443 off`
|
||||
- Accept that Caddy can't serve :443 alongside Tailscale
|
||||
- Update the `service-health-check.sh` script to skip the Caddy check if Tailscale port 443 is intentional
|
||||
- **Public ports + Tailscale = redundant.** Every port open on all interfaces that's also served via Tailscale is a wasted attack surface. Lock down to `tailscale0` once you confirm the tunnel works.
|
||||
- **Tailscale is not a replacement for backups.** It's a network layer — doesn't protect against data loss. The 15-min S3 sync is still the backup strategy.
|
||||
- **Standby server should also have Tailscale.** If the live box dies and the standby takes over, you need a way to reach it. Install Tailscale on the standby too so it's reachable via tailnet IP even if public SSH is locked down.
|
||||
- **The Tailscale auth email doesn't need to be a real email domain.** A self-hosted domain (`info@itpropartner.com`) works — Tailscale uses it only as an identity label. Sign up with any Google/Microsoft/GitHub/Apple account.
|
||||
- **Unlinking Tailscale from the box.** If you need to remove a server from your tailnet: `tailscale logout` on the server, then remove it from the Tailscale admin console.
|
||||
- **Telegram IPv6 timeout on netcup.** This netcup KVM box prefers IPv6 DNS resolution, but Telegram's IPv6 endpoint (`2001:67c:4e8:f004::9`) is unreachable. The Hermes gateway gets stuck in connection retries indefinitely. Fix: `sysctl -w net.ipv6.conf.all.disable_ipv6=1` and restart the gateway. See `references/telegram-ipv6-timeout.md`.
|
||||
+89
@@ -0,0 +1,89 @@
|
||||
# Caddy / Tailscale Port 443 Conflict
|
||||
|
||||
## Symptom
|
||||
|
||||
`systemctl status caddy` shows:
|
||||
```
|
||||
listen tcp :443: bind: address already in use
|
||||
```
|
||||
|
||||
## Root Cause
|
||||
|
||||
`tailscaled` (the Tailscale daemon) binds port 443 on **its Tailnet IP** (e.g. `100.71.155.7:443`). When Caddy tries to bind `:443` (all interfaces — `0.0.0.0:443`), the kernel blocks it because `100.71.155.7:443` is already taken. The public IP (`152.53.192.33:443`) is completely free — Caddy just can't reach it via the blanket `:443` bind.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
```bash
|
||||
# Check what's on port 443
|
||||
ss -tlnp | grep 443
|
||||
# → tailscaled PID 897 bound to 100.71.155.7:443 (NOT 0.0.0.0:443)
|
||||
|
||||
# Confirm Caddy config is trying to use all-interfaces :443
|
||||
grep ':443' /etc/caddy/Caddyfile
|
||||
|
||||
# Confirm Tailscale serve is active
|
||||
tailscale serve status
|
||||
```
|
||||
|
||||
## Resolution — Use `default_bind` in Caddyfile
|
||||
|
||||
Tailscale binds to its specific Tailnet interface IP. Caddy can bind to the **public IP** instead, and both coexist:
|
||||
|
||||
Add a global options block at the **top** of `/etc/caddy/Caddyfile`:
|
||||
|
||||
```caddy
|
||||
{
|
||||
default_bind 152.53.192.33
|
||||
}
|
||||
```
|
||||
|
||||
This tells Caddy to bind ALL site blocks to the public IP only, avoiding the Tailscale interface entirely. Leases port 443 for public HTTPS without fighting Tailscale.
|
||||
|
||||
**Verify Caddy parses the config before restarting:**
|
||||
|
||||
```bash
|
||||
caddy validate --config /etc/caddy/Caddyfile
|
||||
# → "Valid configuration"
|
||||
|
||||
# Format it
|
||||
caddy fmt --overwrite /etc/caddy/Caddyfile
|
||||
|
||||
# Start it
|
||||
systemctl start caddy
|
||||
|
||||
# Confirm it's listening
|
||||
systemctl is-active caddy
|
||||
# → active
|
||||
|
||||
# Test a site
|
||||
curl -sS -o /dev/null -w 'HTTP %{http_code}' https://sign.itpropartner.com/
|
||||
# → HTTP 200
|
||||
```
|
||||
|
||||
**After the fix, verify the health check passes:**
|
||||
```bash
|
||||
bash /root/.hermes/scripts/service-health-check.sh
|
||||
echo "Exit code: $?" # Should be 0 (silent = all healthy)
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
- Tailscale binds `100.71.155.7:443` (its Tailnet IP, interface `tailscale0`)
|
||||
- Caddy with `default_bind 152.53.192.33` binds `152.53.192.33:443` (the public IP, interface `eth0`)
|
||||
- No overlap. Both serve HTTPS on port 443 to their respective audiences.
|
||||
|
||||
Without `default_bind`, Caddy tries `0.0.0.0:443` (all interfaces), which includes the Tailscale IP → conflict.
|
||||
|
||||
## Resolution Options (ranked)
|
||||
|
||||
| Option | Effort | Effect |
|
||||
|--------|--------|--------|
|
||||
| **`default_bind <public-ip>`** (recommended) | Add 3 lines to Caddyfile | Full Caddy + Tailscale coexistence on :443 |
|
||||
| **iptables redirect** from Tailscale IP | Moderate | Works if Tailscale isn't actively serving |
|
||||
| **Disable Tailscale :443** | `tailscale serve --https=443 off` | Loses Tailscale Serve HTTPS on that box |
|
||||
| **Run Caddy on alt port** (e.g., 8443) | Update Caddy config + DNS | `sign.itpropartner.com` needs custom port |
|
||||
| **Let Caddy fail** | Update health check to skip Caddy | Acceptable if services reachable another way |
|
||||
|
||||
## Current State (Jul 11, 2026)
|
||||
|
||||
On **Core** (netcup, 152.53.192.33): **Fixed — Caddy running with `default_bind 152.53.192.33`.** Both Caddy (public HTTPS) and Tailscale (tailnet HTTPS) coexist cleanly.
|
||||
@@ -0,0 +1,25 @@
|
||||
# Security Safeguards: User Preferences
|
||||
|
||||
Germaine explicitly asked "what safeguards do you have to not run amok?" and "what can we put in place to make sure hallucinations don't happen?"
|
||||
|
||||
## Approval Tiers
|
||||
|
||||
| Level | Examples | Requires user approval? |
|
||||
|---|---|---|
|
||||
| **Read** | Check status, read logs, search files, inspect config | Never |
|
||||
| **Scoped write** | Edit portal CSS, create draft pages, run backup scripts | Only for production systems |
|
||||
| **Dangerous** | DNS changes, server reboot, config edits, file deletion, firewall rule changes, API key operations | **Always** — state hostname+IP+impact first |
|
||||
|
||||
## Hard Rules
|
||||
|
||||
1. **Before any destructive action**, state the target hostname AND IP. Never act on opaque IDs.
|
||||
2. **Default to read-only** unless explicitly told "write mode."
|
||||
3. **No invented config keys** — if a feature isn't confirmed, say "I don't know."
|
||||
4. **API keys get minimum permissions** — Cloudflare DNS-only, Wasabi write-only, Stripe read-only.
|
||||
5. **Kill switch:** `/lockdown` → remove SSH key from all servers, stop all cron jobs, stop email triage, report done.
|
||||
|
||||
## Carried Forward
|
||||
|
||||
- "Don't be making shit up" — verify before claiming success
|
||||
- Caution over speed for infrastructure operations
|
||||
- State hostname + IP and confirm before any destructive action
|
||||
@@ -0,0 +1,43 @@
|
||||
# Telegram IPv6 Timeout on Netcup
|
||||
|
||||
On this netcup box, the Hermes gateway (and specifically the Telegram polling adapter) can get stuck trying to connect via IPv6 to `api.telegram.org` (resolves to `2001:67c:4e8:f004::9`). The connection attempt times out, and the gateway appears stuck at "Connecting to Telegram (attempt 1/8)…" indefinitely.
|
||||
|
||||
## Symptoms
|
||||
|
||||
- Gateway starts and runs, but Telegram never connects
|
||||
- `journalctl` shows only: "Discovering Telegram API fallback IPs via DNS-over-HTTPS…" and "Connecting to Telegram (attempt 1/8)…"
|
||||
- Gateway zombies accumulate after repeated restarts — a PID is running but no actual TCP connection established
|
||||
- `ss -tnp | grep <pid>` shows no ESTABLISHED connections to Telegram's IPs
|
||||
- Direct connection test: `timeout 5 bash -c 'echo > /dev/tcp/api.telegram.org/443 && echo "OK"'` fails, but connecting via IPv4 IP (149.154.166.110:443) succeeds
|
||||
|
||||
## Root Cause
|
||||
|
||||
The netcup server prefers IPv6 DNS resolution (AAAA record) over IPv4 (A record). Telegram's IPv6 endpoint at `2001:67c:4e8:f004::9` is unreachable from this provider, likely due to netcup's IPv6 routing or firewall on the KVM hypervisor.
|
||||
|
||||
The Hermes Telegram adapter has an internal "fallback IP discovery via DNS-over-HTTPS" mechanism, but the timeout on the initial IPv6 connection exceeds the gateway's connection budget.
|
||||
|
||||
## Fix
|
||||
|
||||
Disable IPv6 system-wide on the netcup box:
|
||||
```bash
|
||||
sysctl -w net.ipv6.conf.all.disable_ipv6=1
|
||||
sysctl -w net.ipv6.conf.default.disable_ipv6=1
|
||||
```
|
||||
|
||||
To make permanent:
|
||||
```bash
|
||||
cat >> /etc/sysctl.d/99-disable-ipv6.conf << 'EOF'
|
||||
net.ipv6.conf.all.disable_ipv6 = 1
|
||||
net.ipv6.conf.default.disable_ipv6 = 1
|
||||
EOF
|
||||
```
|
||||
|
||||
## After Fix
|
||||
|
||||
1. Restart the profile's gateway
|
||||
2. Confirm clean connection via `ss -tnp | grep <pid>` — should show `ESTABLISHED` connections to `149.154.166.110:443`
|
||||
3. `journalctl` should no longer show connection retries
|
||||
|
||||
## Profiles Affected
|
||||
|
||||
This affects ANY profile gateway trying to reach Telegram from this netcup box. The `default` profile's gateway was originally started before this was noticed; Anita's profile was the one that exposed it during the migration because her gateway was started fresh.
|
||||
Reference in New Issue
Block a user