ippadmin/itpp-infrastructure
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

standby sync: flock single-instance lock, stop listing cron/output (190k objects), --only-show-errors

Browse Source
This commit is contained in:
root
2026-09-15 13:21:55 -04:00
parent 128c7b34b6
commit 6e0c89de26
1 changed files with 8 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+8
View File
@@ -15,6 +15,14 @@ Verified after the swap, not assumed:
**Trade-off accepted:** both DR nodes (Core and core-bu) are now netcup. Provider diversity is gone until the US replacement lands. app1-bu remains the only non-netcup node but is no longer armed.
**Sync fixes found only after arming (the arm is what exposed them):**
- **Two `aws s3 sync` instances were running concurrently.** The first full pull took longer than the 10-minute cron interval, so every tick started another one. Fixed with an `flock` single-instance lock. Proven: a second launch logs `Another sync is already running (lock held), exiting`.
- **The sync pulled `live/cron/` as a whole prefix**, which means LISTING 190k+ objects under `cron/output/` (Core's job-log history, 192,716 files on disk at last count). Even with `--exclude`, the listing is the cost, and that listing is what blew past the interval. Fixed by dropping cron from the prefix sync and copying only `cron/jobs.json` and `cron/executions.db`.
- **The sync log had reached 30 MB** of per-object download lines, and aws's partial progress lines merged into the following log line. That corrupted an audit read: a filename containing "error" (e.g. `errors.py`) looked like a real error, and a merged line made a single run look like it downloaded 17,927 objects. Switched to `--only-show-errors`.
Result: a full cycle now completes in **13 seconds** with a 9-line log and **zero** `cron/output` files touched (verified by mtime), versus 10+ minutes and overlapping runs before. Job definitions still refresh: `cron/jobs.json` advanced from 12:49:37 to 13:19:40 on the test run.
**Operational note:** with core-bu armed, a Core Hermes outage lasting 2 consecutive 5-minute cycles (about 10 minutes) fences Core and triggers takeover. Planned gateway work expected to exceed about 10 minutes will cause a failover, then an automatic failback when Core returns. Re-disarm core-bu (`touch /root/.hermes/standby.disarmed`) before any long maintenance window.
---