hermes-recovery/references/shark-game-recovery-manual.md

🦈 Shark Attack Fantasy League — Offline Recovery Manual

Author: Sho'Nuff | Audience: Germaine
Last updated: July 9, 2026
Purpose: Recover the full shark game system without Sho'Nuff being online.

---

1. What Is This? (Quick Overview)

The Shark Attack Fantasy League is a game where players draft shark-attack regions and score points when real shark events hit those regions. It has three parts:

Part	What it does	Where it lives
Backend	Python web server + database	/root/shark-game/backend/ — port 8083
Frontend	Web pages you see in the browser	/var/www/shark-game/
Scraper	Fetches shark news and scores points	/root/shark-game/scraper/ — runs via cron

Domains: shark.iamgmb.com (the main site) and shark.itpropartner.com (backup URL)

---

2. Prerequisites — What You MUST Have Installed

If you're reading this on the server, check these are available:

which sqlite3       # Should show /usr/bin/sqlite3
which python3       # Should show /usr/bin/python3
which aws           # Should show /opt/awscli-venv/bin/aws
which curl          # Should show /usr/bin/curl
which systemctl     # Should show /usr/bin/systemctl
which journalctl    # Should show /usr/bin/journalctl

If any of those are missing, install them:

# Debian/Ubuntu
apt-get install -y sqlite3 curl systemd

# AWS CLI (if missing)
apt-get install -y python3-pip
pip3 install awscli

---

3. System Architecture — What Runs Where

┌─────────────────────────────────────────────────────────┐
│                      Internet                            │
└──────────┬──────────────────────────────┬────────────────┘
           │                              │
    shark.iamgmb.com              shark.itpropartner.com
           │                              │
           ▼                              ▼
    ┌──────────────────────────────────────────┐
    │            Caddy (reverse proxy)          │
    │   /etc/caddy/Caddyfile                   │
    │   shark.iamgmb.com → 127.0.0.1:8083     │
    │   shark.itpropartner.com → static files  │
    └──────────────┬───────────────────────────┘
                   │
    ┌──────────────▼───────────────────────────┐
    │   FastAPI Backend (port 8083)             │
    │   /root/shark-game/backend/server.py      │
    │   Runs as: shark-game.service (systemd)   │
    └──────────────┬───────────────────────────┘
                   │
    ┌──────────────▼───────────────────────────┐
    │   SQLite Database                         │
    │   /root/shark-game/backend/game.db        │
    └───────────────────────────────────────────┘

Frontend files are served by Caddy from /var/www/shark-game/:

• index.html — Main game page
• draft-room.html — Draft interface
• league.html — League dashboard
• settings.html — User settings
• how-to-play.html — Rules
• help.html — Help page
• sw.js — Service worker (push notifications)

---

4. Daily Operations — How to Check the Game

4.1 Is the game running?

systemctl status shark-game

What to look for:

• Active: active (running) ✅ — everything is fine
• Active: inactive (dead) ❌ — the service has stopped
• Active: failed ❌ — something crashed

4.2 View recent logs

# Last 50 lines
journalctl -u shark-game -n 50

# Follow live logs (like tail -f)
journalctl -u shark-game -n 50 -f

# View today's logs
journalctl -u shark-game --since today

# Filter for errors
journalctl -u shark-game -n 200 | grep -i "error\|traceback\|exception"

4.3 Check the database

# Open interactive SQLite shell
sqlite3 /root/shark-game/backend/game.db

# Or run a query directly
sqlite3 /root/shark-game/backend/game.db "SELECT * FROM users;"

# Useful queries:
sqlite3 /root/shark-game/backend/game.db "SELECT id, email, display_name, is_admin FROM users;"
sqlite3 /root/shark-game/backend/game.db "SELECT id, name, status, invite_code FROM leagues;"
sqlite3 /root/shark-game/backend/game.db "SELECT * FROM scores ORDER BY created_at DESC LIMIT 10;"

4.4 Check disk space

df -h /root/shark-game/

4.5 Restart the game

systemctl restart shark-game

Then verify it came back up:

sleep 3 && systemctl status shark-game

---

5. Backup & Restore

5.1 How backups work

The game database (game.db) is NOT backed up separately — it's included in the Hermes live-sync backup, which runs every 15 minutes and uploads to Wasabi S3.

5.2 Checking backups exist

aws s3 ls s3://hermes-vps-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com

You should see folders like live/, live-sync/, standby/.

5.3 Restoring the database from S3

Step 1: Stop the game (so it doesn't write to the DB while restoring)

systemctl stop shark-game

Step 2: Download the latest state.db backup

aws s3 cp s3://hermes-vps-backups/live-sync/state.db /root/shark-game/backend/game.db \
  --endpoint-url https://s3.us-east-1.wasabisys.com

(Note: The backup is named state.db on S3 but we save it locally as game.db)

Step 3: Restart the game

systemctl start shark-game

5.4 Restoring the frontend files

The frontend files are in /var/www/shark-game/. If they get deleted, you can rebuild them from:

1. If the git repo exists: cd /root/shark-game && git pull
2. If the S3 backup has them: Download them from the live/ prefix and copy to /var/www/shark-game/
3. If neither exists: Contact me — I'll need to regenerate the frontend files

5.5 Taking a manual backup (for safety)

cp /root/shark-game/backend/game.db /root/shark-game/backend/game.db.backup.$(date +%Y%m%d-%H%M%S)

---

6. Common Issues & How to Fix Them

6.1 "Port 8083 already in use"

Symptom: systemctl start shark-game fails, or you see "Address already in use" in logs.

Fix:

# Kill whatever is using port 8083, then restart
fuser -k 8083/tcp
systemctl restart shark-game

6.2 "Database locked" errors

Symptom: Users see errors when trying to join leagues, make picks, or view scores. Logs show "database is locked".

Fix:

# Remove the lock file
rm -f /root/shark-game/backend/game.db.lock
rm -f /root/shark-game/backend/game.db-shm
rm -f /root/shark-game/backend/game.db-wal

# Restart the service
systemctl restart shark-game

6.3 "500 Internal Server Error" when joining a league

Symptom: Users can't join leagues. You see "500" errors in the browser.

Fix:

# Check the server logs for details
journalctl -u shark-game -n 50 | grep -A10 "500\|Error\|Traceback"

Common causes:

• Missing environment variables (see Section 13)
• Database schema mismatch (after restoring wrong backup)
• Invalid invite code

6.4 Push notifications not sending

Symptom: Users report they're not getting push notifications.

Check 1: Is the service worker loaded?

• Open https://shark.iamgmb.com/sw.js in a browser
• You should see JavaScript code, not an error

Check 2: Are the VAPID keys set?

# Check if VAPID keys are in the environment
systemctl show shark-game | grep -i vapid

Check 3: Test push

curl -X POST https://shark.iamgmb.com/api/push/test

Fix: If VAPID keys are missing, restart the service (env vars are loaded at startup):

systemctl restart shark-game

6.5 Website shows blank page / 502 Bad Gateway

Symptom: shark.iamgmb.com shows a blank page or 502 error.

Fix:

# Step 1: Check if backend is running
systemctl status shark-game

# Step 2: Check if Caddy is running
systemctl status caddy

# Step 3: Restart both
systemctl restart shark-game
systemctl restart caddy

6.6 SSL certificate expired

Symptom: Browser shows "Your connection is not private" warning.

Fix: Caddy auto-renews certificates. If it didn't:

systemctl restart caddy

---

7. Scraper & Scoring

7.1 How scoring works

The scraper searches for shark news articles, scores them, and writes pending scores to: /root/shark-game/data/pending-scores.json

The backend picks up these files and awards points to players who drafted the affected regions.

7.2 Scheduled runs

The scraper runs automatically via system cron at 8:00 AM ET every day:

0 8 * * * /root/shark-game/scraper/run.sh

There's also a draft reminder that runs every 15 minutes during draft season.

7.3 Manual scrape

If you need to run the scraper right now:

cd /root/shark-game
source .venv/bin/activate
python3 scraper/scrape.py >> data/scraper.log 2>&1
deactivate

Or use the wrapper script:

cd /root/shark-game && bash scraper/run.sh

7.4 Check scraper results

# View the last 50 lines of the scraper log
tail -50 /root/shark-game/data/scraper.log

# Check pending scores
cat /root/shark-game/data/pending-scores.json

7.5 View the current cron jobs

crontab -l

You should see entries for the scraper and draft reminder.

---

8. Push Notifications — How They Work

The game uses Web Push API (browser push notifications):

1. VAPID keys authenticate the server to send pushes
2. Service worker (/var/www/shark-game/sw.js) receives pushes in the browser
3. Users subscribe via the browser's permission prompt when they visit the site

VAPID keys are stored as environment variables and embedded in the server code at /root/shark-game/backend/server.py. They include:

• VAPID_PRIVATE_KEY
• VAPID_PUBLIC_KEY
• VAPID_CLAIMS → mailto:email@redacted

Push subscription data is stored in the push_subscriptions table in the database.

Push notification test

# Send a test push to all subscribed users
curl -X POST https://shark.iamgmb.com/api/push/test

Check if users are subscribed

sqlite3 /root/shark-game/backend/game.db "SELECT COUNT(*) FROM push_subscriptions;"

---

9. Quick Command Reference

Service Management

systemctl status shark-game    # Check if running
systemctl restart shark-game    # Restart the backend
systemctl stop shark-game       # Stop the backend
systemctl start shark-game      # Start the backend

Logs

journalctl -u shark-game -n 100          # Last 100 log lines
journalctl -u shark-game -n 100 -f       # Live tail
journalctl -u shark-game --since today   # Today's logs
journalctl -u shark-game -n 200 | grep -i error  # Errors only

Database

sqlite3 /root/shark-game/backend/game.db "SELECT * FROM users;"
sqlite3 /root/shark-game/backend/game.db "SELECT * FROM leagues;"
sqlite3 /root/shark-game/backend/game.db "SELECT * FROM scores ORDER BY created_at DESC LIMIT 20;"
sqlite3 /root/shark-game/backend/game.db "SELECT id, display_name, email FROM users WHERE is_admin=1;"
sqlite3 /root/shark-game/backend/game.db "PRAGMA integrity_check;"   # Check DB health

Scraper

cd /root/shark-game && bash scraper/run.sh             # Force scrape now
tail -50 /root/shark-game/data/scraper.log             # View scraper log
cat /root/shark-game/data/pending-scores.json          # View pending scores
crontab -l                                             # List scheduled jobs

Backups

# Manual backup
cp /root/shark-game/backend/game.db /root/shark-game/backend/game.db.backup.$(date +%Y%m%d)

# Check S3 backups
aws s3 ls s3://hermes-vps-backups/ --endpoint-url https://s3.us-east-1.wasabisys.com

# Restore from S3
systemctl stop shark-game
aws s3 cp s3://hermes-vps-backups/live-sync/state.db /root/shark-game/backend/game.db \
  --endpoint-url https://s3.us-east-1.wasabisys.com
systemctl start shark-game

Port / Network

fuser -k 8083/tcp         # Kill anything on port 8083
ss -tlnp | grep 8083      # See what's listening on 8083
curl -I https://shark.iamgmb.com  # Test site response
curl http://127.0.0.1:8083/       # Test backend directly

Push Notifications

curl -X POST https://shark.iamgmb.com/api/push/test               # Test push
curl https://shark.iamgmb.com/api/push/vapid-public-key           # Check VAPID key
sqlite3 /root/shark-game/backend/game.db "SELECT COUNT(*) FROM push_subscriptions;"  # Check subs

---

10. Complete Recovery Flow — If Everything Is Broken

Follow these steps in order if the entire game is down.

Step 1: Check server is reachable

ping -c 2 shark.iamgmb.com

Step 2: Check web server (Caddy)

systemctl status caddy
# If not running: systemctl restart caddy

Step 3: Check backend service

systemctl status shark-game
# If not running: systemctl start shark-game

Step 4: Check port 8083

ss -tlnp | grep 8083
# If nothing listening: systemctl restart shark-game
# If wrong process: fuser -k 8083/tcp && systemctl restart shark-game

Step 5: Check the database

sqlite3 /root/shark-game/backend/game.db "PRAGMA integrity_check;"
# Should return "ok"

Step 6: Check the scraper ran today

tail -20 /root/shark-game/data/scraper.log
# If last run was yesterday, run it manually
cd /root/shark-game && bash scraper/run.sh

Step 7: Test the site

curl -I https://shark.iamgmb.com
# Should return HTTP/2 200

Step 8: Restore from backup (if DB is corrupted)

systemctl stop shark-game
aws s3 cp s3://hermes-vps-backups/live-sync/state.db /root/shark-game/backend/game.db \
  --endpoint-url https://s3.us-east-1.wasabisys.com
systemctl start shark-game

---

11. File Locations Map

What	Path
Backend server	/root/shark-game/backend/server.py
Database	/root/shark-game/backend/game.db
Python virtual environment	/root/shark-game/backend/venv/
Frontend files	/var/www/shark-game/
Service worker (push)	/var/www/shark-game/sw.js
Scraper script	/root/shark-game/scraper/scrape.py
Scraper wrapper	/root/shark-game/scraper/run.sh
Scraper dependencies	/root/shark-game/scraper/requirements.txt
Scraper log	/root/shark-game/data/scraper.log
Pending scores	/root/shark-game/data/pending-scores.json
Systemd service	/etc/systemd/system/shark-game.service
Caddy config	/etc/caddy/Caddyfile

---

12. Database Tables Reference

Table	Purpose
users	Players — email, display name, admin status, notification prefs
leagues	Leagues — name, invite code, status (draft/active/closed), max players
league_members	Which users belong to which leagues
league_excluded_regions	Regions excluded from a league's draft
regions	Shark attack regions (geo-areas players draft)
draft_picks	Which player drafted which region in which league
scores	Score events — what happened, how many points, which region
user_scores	Running total scores per user
push_subscriptions	Web push notification subscriptions

---

13. Environment Variables

The game backend needs these environment variables. They're set in the systemd service file or in /root/.hermes/.env:

Variable	Purpose
VAPID_PRIVATE_KEY	Private key for Web Push notifications
VAPID_PUBLIC_KEY	Public key for Web Push notifications

To check if they're loaded:

systemctl show shark-game -P Environment

---

14. Contact / Escalation

If you've tried everything in this manual and the game is still broken:

1. Save the logs:

   journalctl -u shark-game -n 500 > /tmp/shark-game-logs.txt
   

2. Save the database:

   cp /root/shark-game/backend/game.db /root/shark-game/backend/game.db.crash
   

3. Send me the files. I can work from logs alone if needed.

---

🦈 Remember: The most common fix is simply systemctl restart shark-game. Try that first before doing anything complicated!