hermes-recovery/references/shark-game-recovery-manual.md

# 🦈 Shark Attack Fantasy League — Offline Recovery Manual

**Author:** Sho'Nuff | **Audience:** Germaine  
**Last updated:** July 9, 2026  
**Purpose:** Recover the full shark game system without Sho'Nuff being online.

---

## 1. What Is This? (Quick Overview)

The Shark Attack Fantasy League is a game where players draft shark-attack regions and score points when real shark events hit those regions. It has three parts:

| Part | What it does | Where it lives |
|------|-------------|----------------|
| **Backend** | Python web server + database | `/root/shark-game/backend/` — port 8083 |
| **Frontend** | Web pages you see in the browser | `/var/www/shark-game/` |
| **Scraper** | Fetches shark news and scores points | `/root/shark-game/scraper/` — runs via cron |

**Domains:** `shark.iamgmb.com` (the main site) and `shark.itpropartner.com` (backup URL)

---

## 2. Prerequisites — What You MUST Have Installed

If you're reading this on the server, check these are available:

```
which sqlite3       # Should show /usr/bin/sqlite3
which python3       # Should show /usr/bin/python3
which aws           # Should show /opt/awscli-venv/bin/aws
which curl          # Should show /usr/bin/curl
which systemctl     # Should show /usr/bin/systemctl
which journalctl    # Should show /usr/bin/journalctl
```

If any of those are missing, install them:

```bash
# Debian/Ubuntu
apt-get install -y sqlite3 curl systemd

# AWS CLI (if missing)
apt-get install -y python3-pip
pip3 install awscli
```

---

## 3. System Architecture — What Runs Where

```
┌─────────────────────────────────────────────────────────┐
│                      Internet                            │
└──────────┬──────────────────────────────┬────────────────┘
           │                              │
    shark.iamgmb.com              shark.itpropartner.com
           │                              │
           ▼                              ▼
    ┌──────────────────────────────────────────┐
    │            Caddy (reverse proxy)          │
    │   /etc/caddy/Caddyfile                   │
    │   shark.iamgmb.com → 127.0.0.1:8083     │
    │   shark.itpropartner.com → static files  │
    └──────────────┬───────────────────────────┘
                   │
    ┌──────────────▼───────────────────────────┐
    │   FastAPI Backend (port 8083)             │
    │   /root/shark-game/backend/server.py      │
    │   Runs as: shark-game.service (systemd)   │
    └──────────────┬───────────────────────────┘
                   │
    ┌──────────────▼───────────────────────────┐
    │   SQLite Database                         │
    │   /root/shark-game/backend/game.db        │
    └───────────────────────────────────────────┘
```

**Frontend files** are served by Caddy from `/var/www/shark-game/`:
- `index.html` — Main game page
- `draft-room.html` — Draft interface
- `league.html` — League dashboard
- `settings.html` — User settings
- `how-to-play.html` — Rules
- `help.html` — Help page
- `sw.js` — Service worker (push notifications)

---

## 4. Daily Operations — How to Check the Game

### 4.1 Is the game running?

```bash
systemctl status shark-game
```

**What to look for:**
- **Active: active (running)** ✅ — everything is fine
- **Active: inactive (dead)** ❌ — the service has stopped
- **Active: failed** ❌ — something crashed

### 4.2 View recent logs

```bash
# Last 50 lines
journalctl -u shark-game -n 50

# Follow live logs (like tail -f)
journalctl -u shark-game -n 50 -f

# View today's logs
journalctl -u shark-game --since today

# Filter for errors
journalctl -u shark-game -n 200 | grep -i "error\|traceback\|exception"
```

### 4.3 Check the database

```bash
# Open interactive SQLite shell
sqlite3 /root/shark-game/backend/game.db

# Or run a query directly
sqlite3 /root/shark-game/backend/game.db "SELECT * FROM users;"

# Useful queries:
sqlite3 /root/shark-game/backend/game.db "SELECT id, email, display_name, is_admin FROM users;"
sqlite3 /root/shark-game/backend/game.db "SELECT id, name, status, invite_code FROM leagues;"
sqlite3 /root/shark-game/backend/game.db "SELECT * FROM scores ORDER BY created_at DESC LIMIT 10;"
```

### 4.4 Check disk space

```bash
df -h /root/shark-game/
```

### 4.5 Restart the game

```bash
systemctl restart shark-game
```

Then verify it came back up:

```bash
sleep 3 && systemctl status shark-game
```

---

## 5. Backup & Restore

### 5.1 How backups work

The game database (`game.db`) is NOT backed up separately — it's included in the **Hermes live-sync** backup, which runs every 15 minutes and uploads to Wasabi S3.

### 5.2 Checking backups exist

```bash
aws s3 ls s3://hermes-vps-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com
```

You should see folders like `live/`, `live-sync/`, `standby/`.

### 5.3 Restoring the database from S3

**Step 1:** Stop the game (so it doesn't write to the DB while restoring)

```bash
systemctl stop shark-game
```

**Step 2:** Download the latest state.db backup

```bash
aws s3 cp s3://hermes-vps-backups/live-sync/state.db /root/shark-game/backend/game.db \
  --endpoint-url https://s3.us-east-1.wasabisys.com
```

*(Note: The backup is named `state.db` on S3 but we save it locally as `game.db`)*

**Step 3:** Restart the game

```bash
systemctl start shark-game
```

### 5.4 Restoring the frontend files

The frontend files are in `/var/www/shark-game/`. If they get deleted, you can rebuild them from:

1. **If the git repo exists:** `cd /root/shark-game && git pull`
2. **If the S3 backup has them:** Download them from the `live/` prefix and copy to `/var/www/shark-game/`
3. **If neither exists:** Contact me — I'll need to regenerate the frontend files

### 5.5 Taking a manual backup (for safety)

```bash
cp /root/shark-game/backend/game.db /root/shark-game/backend/game.db.backup.$(date +%Y%m%d-%H%M%S)
```

---

## 6. Common Issues & How to Fix Them

### 6.1 "Port 8083 already in use"

**Symptom:** `systemctl start shark-game` fails, or you see "Address already in use" in logs.

**Fix:**
```bash
# Kill whatever is using port 8083, then restart
fuser -k 8083/tcp
systemctl restart shark-game
```

### 6.2 "Database locked" errors

**Symptom:** Users see errors when trying to join leagues, make picks, or view scores. Logs show "database is locked".

**Fix:**
```bash
# Remove the lock file
rm -f /root/shark-game/backend/game.db.lock
rm -f /root/shark-game/backend/game.db-shm
rm -f /root/shark-game/backend/game.db-wal

# Restart the service
systemctl restart shark-game
```

### 6.3 "500 Internal Server Error" when joining a league

**Symptom:** Users can't join leagues. You see "500" errors in the browser.

**Fix:**
```bash
# Check the server logs for details
journalctl -u shark-game -n 50 | grep -A10 "500\|Error\|Traceback"
```

Common causes:
- Missing environment variables (see Section 13)
- Database schema mismatch (after restoring wrong backup)
- Invalid invite code

### 6.4 Push notifications not sending

**Symptom:** Users report they're not getting push notifications.

**Check 1:** Is the service worker loaded?
- Open `https://shark.iamgmb.com/sw.js` in a browser
- You should see JavaScript code, not an error

**Check 2:** Are the VAPID keys set?
```bash
# Check if VAPID keys are in the environment
systemctl show shark-game | grep -i vapid
```

**Check 3:** Test push
```bash
curl -X POST https://shark.iamgmb.com/api/push/test
```

**Fix:** If VAPID keys are missing, restart the service (env vars are loaded at startup):
```bash
systemctl restart shark-game
```

### 6.5 Website shows blank page / 502 Bad Gateway

**Symptom:** `shark.iamgmb.com` shows a blank page or 502 error.

**Fix:**
```bash
# Step 1: Check if backend is running
systemctl status shark-game

# Step 2: Check if Caddy is running
systemctl status caddy

# Step 3: Restart both
systemctl restart shark-game
systemctl restart caddy
```

### 6.6 SSL certificate expired

**Symptom:** Browser shows "Your connection is not private" warning.

**Fix:** Caddy auto-renews certificates. If it didn't:
```bash
systemctl restart caddy
```

---

## 7. Scraper & Scoring

### 7.1 How scoring works

The scraper searches for shark news articles, scores them, and writes pending scores to:
`/root/shark-game/data/pending-scores.json`

The backend picks up these files and awards points to players who drafted the affected regions.

### 7.2 Scheduled runs

The scraper runs automatically via system cron at **8:00 AM ET** every day:

```
0 8 * * * /root/shark-game/scraper/run.sh
```

There's also a **draft reminder** that runs every 15 minutes during draft season.

### 7.3 Manual scrape

If you need to run the scraper right now:

```bash
cd /root/shark-game
source .venv/bin/activate
python3 scraper/scrape.py >> data/scraper.log 2>&1
deactivate
```

Or use the wrapper script:

```bash
cd /root/shark-game && bash scraper/run.sh
```

### 7.4 Check scraper results

```bash
# View the last 50 lines of the scraper log
tail -50 /root/shark-game/data/scraper.log

# Check pending scores
cat /root/shark-game/data/pending-scores.json
```

### 7.5 View the current cron jobs

```bash
crontab -l
```

You should see entries for the scraper and draft reminder.

---

## 8. Push Notifications — How They Work

The game uses **Web Push API** (browser push notifications):

1. **VAPID keys** authenticate the server to send pushes
2. **Service worker** (`/var/www/shark-game/sw.js`) receives pushes in the browser
3. **Users** subscribe via the browser's permission prompt when they visit the site

**VAPID keys** are stored as environment variables and embedded in the server code at `/root/shark-game/backend/server.py`. They include:
- `VAPID_PRIVATE_KEY`
- `VAPID_PUBLIC_KEY`
- `VAPID_CLAIMS` → `mailto:email@redacted`

**Push subscription data** is stored in the `push_subscriptions` table in the database.

### Push notification test

```bash
# Send a test push to all subscribed users
curl -X POST https://shark.iamgmb.com/api/push/test
```

### Check if users are subscribed

```bash
sqlite3 /root/shark-game/backend/game.db "SELECT COUNT(*) FROM push_subscriptions;"
```

---

## 9. Quick Command Reference

### Service Management
```bash
systemctl status shark-game    # Check if running
systemctl restart shark-game    # Restart the backend
systemctl stop shark-game       # Stop the backend
systemctl start shark-game      # Start the backend
```

### Logs
```bash
journalctl -u shark-game -n 100          # Last 100 log lines
journalctl -u shark-game -n 100 -f       # Live tail
journalctl -u shark-game --since today   # Today's logs
journalctl -u shark-game -n 200 | grep -i error  # Errors only
```

### Database
```bash
sqlite3 /root/shark-game/backend/game.db "SELECT * FROM users;"
sqlite3 /root/shark-game/backend/game.db "SELECT * FROM leagues;"
sqlite3 /root/shark-game/backend/game.db "SELECT * FROM scores ORDER BY created_at DESC LIMIT 20;"
sqlite3 /root/shark-game/backend/game.db "SELECT id, display_name, email FROM users WHERE is_admin=1;"
sqlite3 /root/shark-game/backend/game.db "PRAGMA integrity_check;"   # Check DB health
```

### Scraper
```bash
cd /root/shark-game && bash scraper/run.sh             # Force scrape now
tail -50 /root/shark-game/data/scraper.log             # View scraper log
cat /root/shark-game/data/pending-scores.json          # View pending scores
crontab -l                                             # List scheduled jobs
```

### Backups
```bash
# Manual backup
cp /root/shark-game/backend/game.db /root/shark-game/backend/game.db.backup.$(date +%Y%m%d)

# Check S3 backups
aws s3 ls s3://hermes-vps-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com

# Restore from S3
systemctl stop shark-game
aws s3 cp s3://hermes-vps-backups/live-sync/state.db /root/shark-game/backend/game.db \
  --endpoint-url https://s3.us-east-1.wasabisys.com
systemctl start shark-game
```

### Port / Network
```bash
fuser -k 8083/tcp         # Kill anything on port 8083
ss -tlnp | grep 8083      # See what's listening on 8083
curl -I https://shark.iamgmb.com  # Test site response
curl http://127.0.0.1:8083/       # Test backend directly
```

### Push Notifications
```bash
curl -X POST https://shark.iamgmb.com/api/push/test               # Test push
curl https://shark.iamgmb.com/api/push/vapid-public-key           # Check VAPID key
sqlite3 /root/shark-game/backend/game.db "SELECT COUNT(*) FROM push_subscriptions;"  # Check subs
```

---

## 10. Complete Recovery Flow — If Everything Is Broken

Follow these steps in order if the entire game is down.

### Step 1: Check server is reachable
```bash
ping -c 2 shark.iamgmb.com
```

### Step 2: Check web server (Caddy)
```bash
systemctl status caddy
# If not running: systemctl restart caddy
```

### Step 3: Check backend service
```bash
systemctl status shark-game
# If not running: systemctl start shark-game
```

### Step 4: Check port 8083
```bash
ss -tlnp | grep 8083
# If nothing listening: systemctl restart shark-game
# If wrong process: fuser -k 8083/tcp && systemctl restart shark-game
```

### Step 5: Check the database
```bash
sqlite3 /root/shark-game/backend/game.db "PRAGMA integrity_check;"
# Should return "ok"
```

### Step 6: Check the scraper ran today
```bash
tail -20 /root/shark-game/data/scraper.log
# If last run was yesterday, run it manually
cd /root/shark-game && bash scraper/run.sh
```

### Step 7: Test the site
```bash
curl -I https://shark.iamgmb.com
# Should return HTTP/2 200
```

### Step 8: Restore from backup (if DB is corrupted)
```bash
systemctl stop shark-game
aws s3 cp s3://hermes-vps-backups/live-sync/state.db /root/shark-game/backend/game.db \
  --endpoint-url https://s3.us-east-1.wasabisys.com
systemctl start shark-game
```

---

## 11. File Locations Map

| What | Path |
|------|------|
| Backend server | `/root/shark-game/backend/server.py` |
| Database | `/root/shark-game/backend/game.db` |
| Python virtual environment | `/root/shark-game/backend/venv/` |
| Frontend files | `/var/www/shark-game/` |
| Service worker (push) | `/var/www/shark-game/sw.js` |
| Scraper script | `/root/shark-game/scraper/scrape.py` |
| Scraper wrapper | `/root/shark-game/scraper/run.sh` |
| Scraper dependencies | `/root/shark-game/scraper/requirements.txt` |
| Scraper log | `/root/shark-game/data/scraper.log` |
| Pending scores | `/root/shark-game/data/pending-scores.json` |
| Systemd service | `/etc/systemd/system/shark-game.service` |
| Caddy config | `/etc/caddy/Caddyfile` |

---

## 12. Database Tables Reference

| Table | Purpose |
|-------|---------|
| `users` | Players — email, display name, admin status, notification prefs |
| `leagues` | Leagues — name, invite code, status (draft/active/closed), max players |
| `league_members` | Which users belong to which leagues |
| `league_excluded_regions` | Regions excluded from a league's draft |
| `regions` | Shark attack regions (geo-areas players draft) |
| `draft_picks` | Which player drafted which region in which league |
| `scores` | Score events — what happened, how many points, which region |
| `user_scores` | Running total scores per user |
| `push_subscriptions` | Web push notification subscriptions |

---

## 13. Environment Variables

The game backend needs these environment variables. They're set in the systemd service file or in `/root/.hermes/.env`:

| Variable | Purpose |
|----------|---------|
| `VAPID_PRIVATE_KEY` | Private key for Web Push notifications |
| `VAPID_PUBLIC_KEY` | Public key for Web Push notifications |

To check if they're loaded:

```bash
systemctl show shark-game -P Environment
```

---

## 14. Contact / Escalation

If you've tried everything in this manual and the game is still broken:

1. **Save the logs:**
   ```bash
   journalctl -u shark-game -n 500 > /tmp/shark-game-logs.txt
   ```

2. **Save the database:**
   ```bash
   cp /root/shark-game/backend/game.db /root/shark-game/backend/game.db.crash
   ```

3. **Send me the files.** I can work from logs alone if needed.

---

> **🦈 Remember:** The most common fix is simply `systemctl restart shark-game`. Try that first before doing anything complicated!