Connection-acquisition circuit breaker in src/lib/db.ts (2026-09-04 pool exhaustion): trip on the Supavisor ECHECKOUTTIMEOUT signature, not on pool.waitingCount
Local SHAs a29ec9fd1a7295b09072544aa13084ca4fd43c69 + fixup 06eece98b2ab441e009195455beea135e12188a4, Local-Author coder02-mars-cc, awaiting publication by coder-mars-cc (coder02 is not a push-lane agent). Files: src/lib/db.ts + new src/lib/db.test.ts (14 tests). tsc clean on BOTH tsconfig.json and tsconfig.test.json; full vitest 758/758; lint 0 errors. WHY THE OBVIOUS DESIGN WAS WRONG. The three incident rows (appEvents 43c26f80, fb08ab44, 256f05c9, all /mi-cursada/[id] RSC renders) carry '(ECHECKOUTTIMEOUT) unable to check out connection from the pool after 15000ms in Transaction mode'. That is NOT our client: pg-pool 3.14.0 emits 'timeout exceeded when trying to connect' and 'Connection terminated due to connection timeout', and 15000ms is not our connectionTimeoutMillis of 10000. ECHECKOUTTIMEOUT, the 15s budget and the 'in Transaction mode' phrasing are Supavisor's. So our client DID take one of its own 6 slots and reach the pooler; SUPAVISOR could not check out a server connection from its own pool. Consequences: (a) max:6 was not the binding constraint; (b) pool.waitingCount reads LOW during this failure, so the originally-briefed waitingCount-threshold trip would have been permanently inert - green tests, breaker that can never fire, outage recurs; (c) the saturated pool is upstream and shared across every warm container, so no process sees the global picture. WHAT SHIPPED. Breaker trips on the ERROR SIGNATURE, 5 acquisition failures in 30s. Open = fail fast with a typed DbUnavailableError and NO checkout at all (removes load rather than reshaping it). Half-open after 5s admits exactly ONE real request as the probe, slot claimed synchronously before any await; close resets the cooldown to base (so the doubling cannot ratchet across unrelated incidents to the 60s cap permanently); a failed probe doubles it, capped at 60s. Matcher covers Supavisor's ECHECKOUTTIMEOUT and both node-pg checkout messages, and deliberately NOT ordinary SQL errors - those prove a connection was obtained. Gates query() AND withTransaction() (the latter calls pool.connect() directly and would otherwise bypass it). Telemetry, deliberately narrowed from the brief: per-failure pool counters go to stderr; appEvents gets ONE row per breaker TRIP, not one per ECHECKOUTTIMEOUT. Error-tier rows page the live PM + Elazar through the applog rail, so per-failure is a page storm, and each row is itself another query against the pool that is already failing. logEvent is imported lazily (logger.ts imports query() from db.ts - a static import is a cycle) and never awaited, with its rejection caught: logEvent RETHROWS on insert failure, which during this exact outage is the likely path, and an uncaught rejection is the worker-crash mode the pool 'error' listener exists to prevent. Also in scope: withTransaction's ROLLBACK is now wrapped, so a rollback failure on a dead connection can no longer REPLACE the original error in the throw (the caller was seeing 'Connection terminated' instead of the constraint violation that actually failed); release(true) destroys the client only when the rollback itself failed, since a clean rollback leaves a healthy connection and destroying it forces a needless reconnect through the scarce resource. Stale header comment corrected - migration 122 folded the layout's 3-wide read into get_layout_state. NOT IN SCOPE, see the sibling WI: statement_timeout. NOT DONE EITHER: no retry wrapper - rejected 4-of-4 independently on blackboard bb-coder-mars-cc-mtny0cwv-k8v8a6, because a retry re-enters the same contended queue as a brand-new waiter and adds offered load exactly when the upstream pool is full. BOUND, documented in the file rather than hidden: on Vercel the breaker is PER CONTAINER. Several warm containers each keep their own state and their own 6 slots, so it sheds most but not all load. Limit to state, not a reason to skip it. Close this only after coder-mars-cc publishes and the post-push live-version confirm + audit verdict land.
Activity
-
Follow-up P1 committed locally: 68d428baf9b50e80c4b007e08285290583d42eca (Local-Author coder02-mars-cc, awaiting coder-mars-cc push). Bounds the half-open probe. DEFECT IN THE SHIPPED BREAKER, surfaced by MARS-698's flapping outage rather than by review. A half-open probe is a REAL request: the checkout has already succeeded, so connectionTimeoutMillis no longer applies, and Mars has no statement_timeout (MARS-696 - unsettable from app code under Supavisor transaction pooling). A probe issued into a flap can therefore hang with no upper bound. probeInFlight was cleared ONLY on settlement, so a probe that never settles left admit() returning false forever: the breaker wedged OPEN permanently for that container and stayed dark long after the database recovered. That converts a transient upstream outage into a permanent local one - strictly worse than the failure the breaker exists to soften. coder-mars-cc reported the risk as degraded recovery; it is worse than that, it is a permanent wedge, and the mechanism is that nothing else in the module ever clears the flag. FIX. Probes carry a 15s deadline, evaluated lazily inside admit() rather than by a timer - a serverless container can be frozen between invocations, so wall-clock measured at the next admission is the honest reading and a timer is not. An expired probe is abandoned: slot released, cooldown doubled and capped as before, and the request that OBSERVES the expiry is itself shed rather than admitted, so the deadline un-wedges the breaker without opening a stampede at the instant it expires. Probe tickets are generation-numbered so a probe settling after abandonment cannot double the cooldown a second time nor clear a slot a newer probe now holds; a late SUCCESS is still honoured, because reachability is reachability and closing the breaker makes any newer in-flight probe moot rather than wrong. VERIFICATION. 16/16 in src/lib/db.test.ts, full suite 767/767, tsc clean on BOTH tsconfig.json and tsconfig.test.json, lint 0 errors. The wedge test was confirmed to FAIL against the previous db.ts by reverting the file and re-running - it pins a real regression, not just intent. STILL OPEN, not addressed here and not mine to close: MARS-696 (statement_timeout at role level) remains the only thing that bounds hold time at the source; this deadline bounds the damage, not the cause. MARS-698 also measured that appEvents held ZERO error rows across the whole outage - the app cannot log its own unavailability because logging is a database write - so the one-row-per-trip telemetry this breaker emits is unreliable during exactly the event it describes. The stderr counters are the surviving signal.
-
Shipped: BREAKER_FAILURE_THRESHOLD=2 (src/lib/db.ts:132), PROBE_DEADLINE_MS=15_000 (src/lib/db.ts:156), both verified live in tree 2026-09-11. Threshold retuned from 5->2 at a7dd9e5 (MARS-704) after the densest real cluster measured 2-in-30s not 5; probe deadline added at 68d428b so a hung half-open probe can't wedge the breaker permanently open. No PASS verdict was ever issued on the four SHAs: audit-mars-ca acked 'working' then was retired under the 2026-09-11 three-agent roster before a verdict landed. Closing on coder-mars-cc's authorization; recorded unreviewed-and-accepted per evolutiva-coder-commons.md Push-to-Done item 8.5, not as a passed review.