hermes-skills/skills/devops/tailscale-infrastructure-access/SKILL.md

---
name: tailscale-infrastructure-access
description: "Set up and manage Tailscale for private infrastructure access — install, authenticate, Tailscale Serve for internal HTTPS, device enrollment, and the policy that internal services stay behind the tailnet."
version: 1.0.0
author: ShoNuff
license: MIT
platforms: [linux, macos, ios]
metadata:
  hermes:
    tags: [tailscale, vpn, networking, security, internal-services]
    related_skills: [docker-service-deployment, infrastructure-automation]
---

# Tailscale Infrastructure Access

Tailscale provides a zero-config VPN for private infrastructure access. Internal services (Vaultwarden, future databases, admin panels) live behind the tailnet — no public ports, no DNS, no TLS cert worries.

## Policy

**Internal services stay behind Tailscale.** No public exposure. No DNS records for admin-only services. Access is via:

1. **Tailscale Serve** — HTTPS proxy within the tailnet (for web UIs)
2. **Direct Tailscale IP + port** — for API/CLI access
3. **SSH over Tailscale** — once public SSH is locked down

## Installation

```bash
curl -fsSL https://tailscale.com/install.sh | sh
```

### Authentication

```bash
tailscale up
# Prints: https://login.tailscale.com/a/XXXXX
# Open in browser, sign in with any identity provider (Google, Microsoft, GitHub, Apple)
# A self-hosted email domain works fine — the email is just a login identity
```

### Verify

```bash
tailscale status
# Shows: <tailscale-ip>    <hostname>    <user>  <os>
```

## Device Enrollment

Users install the Tailscale app on each device:

| Device | Where to get it |
|---|---|
| macOS | tailscale.com/download-mac or Mac App Store |
| iOS/iPad | App Store → "Tailscale" |
| iPhone | App Store → "Tailscale" |
| Linux | `curl -fsSL https://tailscale.com/install.sh | sh` |
| Windows | tailscale.com/download-windows |

All devices log in with the same identity provider (Google Workspace account recommended).

### Renaming the server hostname for clarity

```bash
# Default hostname is the OS hostname (e.g. v2202607377162478911)
# Rename to something readable:
tailscale set --hostname app1

# Verify:
tailscale status
# → 100.71.155.7    app1    user@email.com    linux
```

## Tailscale Serve (internal HTTPS)

Expose a local port as HTTPS within your tailnet — no public domain needed, no Let's Encrypt, automatic TLS certs.

```bash
tailscale serve --bg --https 443 --set-path / http://127.0.0.1:<port>

# Result:
# https://<hostname>.tail<XXXXX>.ts.net/
# |-- proxy http://127.0.0.1:<port>
```

The URL is only resolvable within the tailnet. Anyone outside gets a DNS failure.

### Updating the Serve config after renaming

If you rename the hostname, Tailscale Serve doesn't move automatically. Remove and re-add:

```bash
# Old approach:
tailscale serve --https=443 off
# Then re-add with --bg --https 443
```

### "Web page keeps spinning / 404" troubleshooting

If the page loads but the app hangs, check:
1. **DOMAIN environment variable** on the service matches the actual URL
2. For Bitwarden-compatible apps, the API must advertise the same URL users type
3. `curl http://localhost:<port>` from the server to confirm the service is healthy

### "Password incorrect" after changing domain

Changing the URL (and thus `DOMAIN`) changes KDF/key derivation parameters cached by the client app. Fix:
- In the Bitwarden app: Settings → Account → "Delete account from device" (not "Delete account")
- Re-add server with new URL
- The app fetches fresh parameters before prompting for password

### Ambiguous characters in the URL — mobile app can't connect

The auto-generated hostname (e.g. `vaultwarden.tailc2f3b0.ts.net`) can contain characters that mobile apps confuse (e.g. `0` vs `O`, `1` vs `l`, `5` vs `S`). If a mobile Bitwarden app shows "This is not a recognized Bitwarden server" or just spins:

**Fix — rename hostname, re-add Serve, update DOMAIN:**

```bash
# 1. Rename hostname to something with no ambiguous chars
tailscale set --hostname vaultwarden

# 2. Re-add Tailscale Serve (it auto-generates new URL)
tailscale serve --https=443 off 2>/dev/null || true
tailscale serve --bg --https 443 --set-path / http://127.0.0.1:<port>

# 3. Update the service's DOMAIN env var to match new URL
#    In docker-compose.yml, update DOMAIN=https://new-hostname.tailXXXXX.ts.net
docker compose up -d

# 4. Force the client app to re-fetch config
#    Delete the server entry from device, force-close app, re-add fresh URL
```

**If the Bitwarden app still says "not a recognized server":** The API config at `/api/config` advertises what the server expects. What you type in the app must match the `vault` URL exactly — including http vs https and any port. Verify with `curl https://new-hostname.tailXXXXX.ts.net/api/config | jq '.environment'`.

## Accessing Services

### Vaultwarden example

```bash
# On any enrolled device:
# Open: https://<hostname>.tail<XXXXX>.ts.net/
# Or if using raw Tailscale IP:
# Open: http://100.71.155.7:8080
```

The Bitwarden app configuration:
- Server URL: `https://vaultwarden.tailcXXXXX.ts.net` (or IP)
- Self-hosted environment → enter URL once, app fetches API/identity/sso URLs from `/api/config`

### SSH over Tailscale

Once Tailscale is the only access method:

```bash
ssh root@100.71.155.7
# No public IP needed. No port forwarding.
```

## Firewall

UFW should block service ports on all interfaces and open them only on `tailscale0`:

```bash
# Block all external access
ufw deny <port>/tcp

# Allow only from tailscale interface
ufw allow in on tailscale0 to any port <port> proto tcp comment '<Service> via Tailscale'
```

Current rules on app1:
- 22 (SSH) — all interfaces (public, may lock down later)
- 80, 443 — all interfaces (public web)
- 8080 — tailscale0 only (Vaultwarden)

## Security safeguards: user preferences

This user asked explicitly about safeguards against hallucinations and unauthorized actions. See `references/security-preferences.md` for the full document.

Hard rules carried forward from user feedback:
- **Before any destructive action** (reboot, config deploy, DNS change, firewall edit, container destroy), state target hostname + IP to the user and wait for confirmation. Never act on opaque IDs alone.
- **Default to read-only.** Unless told to enter "write mode," explore only.
- **No inventing config keys or features.** If a Hermes feature isn't confirmed in docs/skill/source, say "I don't know."
- **API keys get minimum permissions** — Cloudflare DNS-only, Wasabi write-only to backup buckets, Stripe read-only.
- **Kill switch:** `/lockdown` → remove SSH key, stop all cron jobs, stop email processing, report done.

## Pitfalls

- **Tailscale Serve and hostname are linked.** If you change the server's Tailscale hostname, the Serve URL changes. Re-add Serve to pick up the new name, then update any service DOMAIN config.
- **Tailscale can conflict with local Caddy on port 443.** When Tailscale is installed and running, `tailscaled` binds port 443 on its Tailnet IP (e.g. `100.71.155.7:443`). If you also run a local Caddy HTTPS server on :443, Caddy fails with `bind: address already in use` because `:443` captures all interfaces including the Tailscale one. Check with `ss -tlnp | grep 443` — if `tailscaled` holds the port on a specific IP (not `0.0.0.0`), the clean fix is:

  **Add `default_bind <public-ip>` to the Caddyfile global block** — this tells Caddy to bind only to the public IP, avoiding the Tailscale interface. Both services then run on port 443 simultaneously without conflict. See `references/caddy-tailscale-port-443-conflict.md` for the full fix.

  If `default_bind` isn't suitable, fallback options:
  - Run Caddy on an alt port (e.g., 8443) and use iptables/redirect
  - Disable Tailscale's port 443 usage: `tailscale serve --https=443 off`
  - Accept that Caddy can't serve :443 alongside Tailscale
  - Update the `service-health-check.sh` script to skip the Caddy check if Tailscale port 443 is intentional
- **Public ports + Tailscale = redundant.** Every port open on all interfaces that's also served via Tailscale is a wasted attack surface. Lock down to `tailscale0` once you confirm the tunnel works.
- **Tailscale is not a replacement for backups.** It's a network layer — doesn't protect against data loss. The 15-min S3 sync is still the backup strategy.
- **Standby server should also have Tailscale.** If the live box dies and the standby takes over, you need a way to reach it. Install Tailscale on the standby too so it's reachable via tailnet IP even if public SSH is locked down.
- **The Tailscale auth email doesn't need to be a real email domain.** A self-hosted domain (`info@itpropartner.com`) works — Tailscale uses it only as an identity label. Sign up with any Google/Microsoft/GitHub/Apple account.
- **Unlinking Tailscale from the box.** If you need to remove a server from your tailnet: `tailscale logout` on the server, then remove it from the Tailscale admin console.
- **Telegram IPv6 timeout on netcup.** This netcup KVM box prefers IPv6 DNS resolution, but Telegram's IPv6 endpoint (`2001:67c:4e8:f004::9`) is unreachable. The Hermes gateway gets stuck in connection retries indefinitely. Fix: `sysctl -w net.ipv6.conf.all.disable_ipv6=1` and restart the gateway. See `references/telegram-ipv6-timeout.md`.