Retune the connection-acquisition breaker threshold - 5-in-30s was structurally unreachable
Circuit breaker trip condition (5 ECHECKOUTTIMEOUT in 30s, db.ts:107) is structurally unreachable on the observed failure distribution: 13 events/5 days, densest cluster only 2-in-30s, counter is per-container (db.ts:95) so arrivals split further across warm containers. Retune the threshold/window to reliably trip on a real cluster like the observed pairs, without adding cross-container shared state (infra cost not justified by current event rate - revisit only if bursts start correlating across containers). Purpose is fail-fast/UX (avoid stacking a full 15s hang per request when Supavisor is degraded), not load-shedding - tripping doesn't relieve Supabase's own bottleneck. Do not remove the breaker: a genuine per-container saturation burst is still the scenario it exists to catch, this is a tuning gap not a design failure. Cite MARS-691 (measurement) as the source.
Activity
-
coder-mars-cc
-
DONE at a7dd9e5, v2.79.28. PASS:a7dd9e5eba0cfa4e509bb1c8d51ede5d45639c1d deep from audit-mars-ca, no findings. CHANGE: BREAKER_FAILURE_THRESHOLD 5 -> 2 in src/lib/db.ts. Window unchanged at 30s - it is the interval the observed clusters were measured in, and widening to 60s would pair two unrelated one-off failures a minute apart for no measured gain. Breaker NOT removed, no cross-container shared state, per the PM call on this WI. SOURCE (MARS-691): 13 ECHECKOUTTIMEOUT events 09-04..09-09, densest cluster TWO within 30s (09-04 22:45:37/22:45:40 and 09-09 23:23:15/23:23:24). Combined with db.ts:95's per-container bound, per-container arrivals sit near 1, so 5-in-30s was structurally unreachable rather than untuned - zero trips was the old rule working correctly. 13 is a FLOOR, not a count (appEvents is a WRITE and is blind during a write outage, MARS-700), but the floor does not rescue the old threshold: the cleanest window, 09-04 22:24-22:45 with no write outage, is still only 4 events in 21 minutes. TRADEOFF STATED IN THE CODE: a false positive costs one BREAKER_BASE_COOLDOWN_MS (5s) of fast-failing that self-clears on the half-open probe's first success; a false negative costs every affected request a full 15s Supavisor hang. ~3x asymmetry, and the false positive is the recoverable direction, so erring trippy is tuning for the actual goal (fail-fast/UX, not load-shedding). TRAP FOUND AND FIXED IN THE SAME PUSH - worth carrying forward: src/lib/db.test.ts encoded the old threshold as LITERAL CALL COUNTS (failCheckout(5), a for-i<5 loop) instead of importing BREAKER_FAILURE_THRESHOLD. A grep for the constant NAME across src/ finds none of them, so the constant reads as having exactly two call sites when it effectively has ten. Changing db.ts alone breaks ~8 tests, and .gitpush-pre.sh runs vitest, so it would have burned a version number. Any future retune must edit both files in one commit. 16/16 pass locally and audit reproduced that independently. VERIFY: live /api/health version 2.79.28 == REMOTE package.json on main read at check time via gh api (2.79.28), deployment dpl_J5c5jAbV7eTk3J4VUJDXNm4Q3xtD, audit read dbOk:true on ?db=1. NOT CLAIMED: no trip has been observed in production at the new threshold. This is a tuning change verified by unit test and deploy, not by a live trip. The next real ECHECKOUTTIMEOUT pair is the first evidence either way, and it will write one db-breaker-open row per trip - MARS-691's re-measure should now look for that row.
-
Retune the connection-acquisition breaker threshold - 5-in-30s was structurally unreachable