Initial skills documentation — 25 categories, all SKILL.md + references + scripts
This commit is contained in:
root
+89
@@ -0,0 +1,89 @@
|
||||
# Caddy / Tailscale Port 443 Conflict
|
||||
|
||||
## Symptom
|
||||
|
||||
`systemctl status caddy` shows:
|
||||
```
|
||||
listen tcp :443: bind: address already in use
|
||||
```
|
||||
|
||||
## Root Cause
|
||||
|
||||
`tailscaled` (the Tailscale daemon) binds port 443 on **its Tailnet IP** (e.g. `100.71.155.7:443`). When Caddy tries to bind `:443` (all interfaces — `0.0.0.0:443`), the kernel blocks it because `100.71.155.7:443` is already taken. The public IP (`152.53.192.33:443`) is completely free — Caddy just can't reach it via the blanket `:443` bind.
|
||||
|
||||
## Diagnosis
|
||||
|
||||
```bash
|
||||
# Check what's on port 443
|
||||
ss -tlnp | grep 443
|
||||
# → tailscaled PID 897 bound to 100.71.155.7:443 (NOT 0.0.0.0:443)
|
||||
|
||||
# Confirm Caddy config is trying to use all-interfaces :443
|
||||
grep ':443' /etc/caddy/Caddyfile
|
||||
|
||||
# Confirm Tailscale serve is active
|
||||
tailscale serve status
|
||||
```
|
||||
|
||||
## Resolution — Use `default_bind` in Caddyfile
|
||||
|
||||
Tailscale binds to its specific Tailnet interface IP. Caddy can bind to the **public IP** instead, and both coexist:
|
||||
|
||||
Add a global options block at the **top** of `/etc/caddy/Caddyfile`:
|
||||
|
||||
```caddy
|
||||
{
|
||||
default_bind 152.53.192.33
|
||||
}
|
||||
```
|
||||
|
||||
This tells Caddy to bind ALL site blocks to the public IP only, avoiding the Tailscale interface entirely. Leases port 443 for public HTTPS without fighting Tailscale.
|
||||
|
||||
**Verify Caddy parses the config before restarting:**
|
||||
|
||||
```bash
|
||||
caddy validate --config /etc/caddy/Caddyfile
|
||||
# → "Valid configuration"
|
||||
|
||||
# Format it
|
||||
caddy fmt --overwrite /etc/caddy/Caddyfile
|
||||
|
||||
# Start it
|
||||
systemctl start caddy
|
||||
|
||||
# Confirm it's listening
|
||||
systemctl is-active caddy
|
||||
# → active
|
||||
|
||||
# Test a site
|
||||
curl -sS -o /dev/null -w 'HTTP %{http_code}' https://sign.itpropartner.com/
|
||||
# → HTTP 200
|
||||
```
|
||||
|
||||
**After the fix, verify the health check passes:**
|
||||
```bash
|
||||
bash /root/.hermes/scripts/service-health-check.sh
|
||||
echo "Exit code: $?" # Should be 0 (silent = all healthy)
|
||||
```
|
||||
|
||||
## How It Works
|
||||
|
||||
- Tailscale binds `100.71.155.7:443` (its Tailnet IP, interface `tailscale0`)
|
||||
- Caddy with `default_bind 152.53.192.33` binds `152.53.192.33:443` (the public IP, interface `eth0`)
|
||||
- No overlap. Both serve HTTPS on port 443 to their respective audiences.
|
||||
|
||||
Without `default_bind`, Caddy tries `0.0.0.0:443` (all interfaces), which includes the Tailscale IP → conflict.
|
||||
|
||||
## Resolution Options (ranked)
|
||||
|
||||
| Option | Effort | Effect |
|
||||
|--------|--------|--------|
|
||||
| **`default_bind <public-ip>`** (recommended) | Add 3 lines to Caddyfile | Full Caddy + Tailscale coexistence on :443 |
|
||||
| **iptables redirect** from Tailscale IP | Moderate | Works if Tailscale isn't actively serving |
|
||||
| **Disable Tailscale :443** | `tailscale serve --https=443 off` | Loses Tailscale Serve HTTPS on that box |
|
||||
| **Run Caddy on alt port** (e.g., 8443) | Update Caddy config + DNS | `sign.itpropartner.com` needs custom port |
|
||||
| **Let Caddy fail** | Update health check to skip Caddy | Acceptable if services reachable another way |
|
||||
|
||||
## Current State (Jul 11, 2026)
|
||||
|
||||
On **Core** (netcup, 152.53.192.33): **Fixed — Caddy running with `default_bind 152.53.192.33`.** Both Caddy (public HTTPS) and Tailscale (tailnet HTTPS) coexist cleanly.
|
||||
@@ -0,0 +1,25 @@
|
||||
# Security Safeguards: User Preferences
|
||||
|
||||
Germaine explicitly asked "what safeguards do you have to not run amok?" and "what can we put in place to make sure hallucinations don't happen?"
|
||||
|
||||
## Approval Tiers
|
||||
|
||||
| Level | Examples | Requires user approval? |
|
||||
|---|---|---|
|
||||
| **Read** | Check status, read logs, search files, inspect config | Never |
|
||||
| **Scoped write** | Edit portal CSS, create draft pages, run backup scripts | Only for production systems |
|
||||
| **Dangerous** | DNS changes, server reboot, config edits, file deletion, firewall rule changes, API key operations | **Always** — state hostname+IP+impact first |
|
||||
|
||||
## Hard Rules
|
||||
|
||||
1. **Before any destructive action**, state the target hostname AND IP. Never act on opaque IDs.
|
||||
2. **Default to read-only** unless explicitly told "write mode."
|
||||
3. **No invented config keys** — if a feature isn't confirmed, say "I don't know."
|
||||
4. **API keys get minimum permissions** — Cloudflare DNS-only, Wasabi write-only, Stripe read-only.
|
||||
5. **Kill switch:** `/lockdown` → remove SSH key from all servers, stop all cron jobs, stop email triage, report done.
|
||||
|
||||
## Carried Forward
|
||||
|
||||
- "Don't be making shit up" — verify before claiming success
|
||||
- Caution over speed for infrastructure operations
|
||||
- State hostname + IP and confirm before any destructive action
|
||||
@@ -0,0 +1,43 @@
|
||||
# Telegram IPv6 Timeout on Netcup
|
||||
|
||||
On this netcup box, the Hermes gateway (and specifically the Telegram polling adapter) can get stuck trying to connect via IPv6 to `api.telegram.org` (resolves to `2001:67c:4e8:f004::9`). The connection attempt times out, and the gateway appears stuck at "Connecting to Telegram (attempt 1/8)…" indefinitely.
|
||||
|
||||
## Symptoms
|
||||
|
||||
- Gateway starts and runs, but Telegram never connects
|
||||
- `journalctl` shows only: "Discovering Telegram API fallback IPs via DNS-over-HTTPS…" and "Connecting to Telegram (attempt 1/8)…"
|
||||
- Gateway zombies accumulate after repeated restarts — a PID is running but no actual TCP connection established
|
||||
- `ss -tnp | grep <pid>` shows no ESTABLISHED connections to Telegram's IPs
|
||||
- Direct connection test: `timeout 5 bash -c 'echo > /dev/tcp/api.telegram.org/443 && echo "OK"'` fails, but connecting via IPv4 IP (149.154.166.110:443) succeeds
|
||||
|
||||
## Root Cause
|
||||
|
||||
The netcup server prefers IPv6 DNS resolution (AAAA record) over IPv4 (A record). Telegram's IPv6 endpoint at `2001:67c:4e8:f004::9` is unreachable from this provider, likely due to netcup's IPv6 routing or firewall on the KVM hypervisor.
|
||||
|
||||
The Hermes Telegram adapter has an internal "fallback IP discovery via DNS-over-HTTPS" mechanism, but the timeout on the initial IPv6 connection exceeds the gateway's connection budget.
|
||||
|
||||
## Fix
|
||||
|
||||
Disable IPv6 system-wide on the netcup box:
|
||||
```bash
|
||||
sysctl -w net.ipv6.conf.all.disable_ipv6=1
|
||||
sysctl -w net.ipv6.conf.default.disable_ipv6=1
|
||||
```
|
||||
|
||||
To make permanent:
|
||||
```bash
|
||||
cat >> /etc/sysctl.d/99-disable-ipv6.conf << 'EOF'
|
||||
net.ipv6.conf.all.disable_ipv6 = 1
|
||||
net.ipv6.conf.default.disable_ipv6 = 1
|
||||
EOF
|
||||
```
|
||||
|
||||
## After Fix
|
||||
|
||||
1. Restart the profile's gateway
|
||||
2. Confirm clean connection via `ss -tnp | grep <pid>` — should show `ESTABLISHED` connections to `149.154.166.110:443`
|
||||
3. `journalctl` should no longer show connection retries
|
||||
|
||||
## Profiles Affected
|
||||
|
||||
This affects ANY profile gateway trying to reach Telegram from this netcup box. The `default` profile's gateway was originally started before this was noticed; Anita's profile was the one that exposed it during the migration because her gateway was started fresh.
|
||||
Reference in New Issue
Block a user