db-breaker-open telemetry is written through the breaker it reports, so the pool-counter row is shed on every open
The db-breaker-open telemetry row can NEVER land: reportBreakerOpen() writes it through logEvent(), which goes through the same breaker that just opened, so the write is shed and the pool-counter snapshot is lost every single time. Measured on the first real production breaker open, 2026-09-13 19:36:32Z. THE EVIDENCE, Vercel runtime logs, deployment dpl_5zJzdnXdF1KGkDQfpny6jxVQGsTJ (v2.79.75), one lambda instance serving GET /perfil: [db breaker] acquisition failed total=4 idle=1 waiting=0 ... Error: timeout exceeded when trying to connect [db breaker] acquisition failed total=4 idle=1 waiting=0 [db breaker] OPEN for 5000ms [logEvent-failed] reason=DbUnavailableError: La base de datos no está aceptando conexiones en este momento. level=error category=server action=db-breaker-open appEvents has NO action='db-breaker-open' row for this window - checked 19:30-19:45Z. The row the instrument exists to write was shed by the thing it reports on. THE MECHANISM, src/lib/db.ts. onAcquisitionFailure() sets breakerOpenedAt BEFORE calling reportBreakerOpen() (db.ts:337-341). reportBreakerOpen() (db.ts:222) dynamically imports logEvent and awaits it; logEvent's INSERT goes through query()/withTransaction(), whose first act is admit() (db.ts:369-370, 391-392), which returns SHED while the breaker is open and throws DbUnavailableError. The catch at db.ts:243 swallows it by design (nsf-allow). So this is not a race and not bad luck - it is structural, and it fires on EVERY breaker open, including every one since the instrument shipped. WHY IT MATTERS BEYOND ONE LOST ROW. The comment at db.ts:235-237 says recording the snapshot "is the whole point". MARS-691 item 6 has been open because the pool counters had never been seen, and it was read as "no incident has happened yet". That reading was wrong: an incident could have happened and this instrument would have shown nothing. A telemetry write about the DB being unreachable, routed through the DB-reachability gate, is a control that cannot be seen to fire. CONSEQUENCE FOR EXISTING RECORDS: any count of breaker opens taken from appEvents is structurally zero and means nothing. MARS-691's day-1 and day-2 "breaker opens: ZERO" were measured that way and are withdrawn as unsound (event appended there). The stderr line in Vercel runtime logs is currently the ONLY record of a breaker open, and runtime-log retention is short, so an open noticed late is an open lost. CONSEQUENCE FOR MARS-729: alert 1 must NOT key on an appEvents db-breaker-open row - that row cannot exist. Key on the stderr signature in runtime logs, or fix this first. DIRECTION, the builder decides the shape. The write has to not depend on admit(). Candidates: (a) bypass admit() for this one telemetry insert - acquire directly from the pool, because an open breaker means recent failures, not certain failure, and the counters are worth one attempt; (b) buffer the snapshot in module scope and flush it on onAcquisitionSuccess() when the breaker closes, which is exactly when the DB is reachable again - this loses nothing on a transient open, but loses the row if the instance dies before recovery; (c) both. Whatever ships must be EXERCISED: force the breaker open in a test with admit() returning SHED and assert the row is written (or buffered and flushed). A test that only runs with the breaker closed passes today and proves nothing, which is how this shipped. Same audit worth one read: any other telemetry emitted from inside the breaker's failure path, e.g. ECHECKOUTTIMEOUT rows carrying poolTotalCount/poolIdleCount/poolWaitingCount, may be subject to the same shedding when the breaker is already open.
Activity
-
Read-only source analysis complete. Confirmed structural loss: db.ts reportBreakerOpen() invokes logger.logEvent after breakerOpenedAt is set; logger inserts through query(), whose admit() returns SHED. Existing db.test.ts test named "writes exactly ONE appEvents row per trip" is a false positive: logger is mocked and only asserts logEvent was called, not that appEvents was written. Recommended patch shape: make the one per-trip telemetry INSERT use a narrowly scoped direct pool query that bypasses admit(), remain bounded to one attempt per trip, and retain best-effort failure handling. Add an end-to-end db test that opens the breaker, observes the normal next request is SHED, and asserts the telemetry INSERT was nonetheless issued and resolved. Awaiting coder02-mars-cc acceptance of the independent gate before any edit.
-
Blocked: Elazar is the sole unblocker. Required clearing event: he must add explicit Mars code-edit authorization for coder03-mars-ca in mars-commons.md and coder02-mars-cc must then accept the independent gate. Until both occur, source analysis/BWI only; no code edit, commit, or push.
-
coder02-mars-cc declined the independent gate while the fleet STOP and Mars authorization roster remain in force. Analysis is complete and recorded; no patch was created.
-
Elazar's 2026-09-15 standing authorization is now recorded in mars-commons; coder03-mars-ca implemented the scoped fix. Independent audit requested.
-
Implemented by coder03-mars-ca: reportBreakerOpen now makes exactly one bounded best-effort direct pool INSERT for db-breaker-open, bypassing admit() only for that per-trip telemetry row. Focused test opens the breaker, proves two normal requests are SHED, then proves the telemetry INSERT is attempted; pnpm vitest run src/lib/db.test.ts: 16/16 pass; eslint passes. Pending independent audit and release/push lane; no push by db03-mars-ca.
-
Implementation independently audited PASS by audit-general-cc and published by coder03-mars-ca in fb2d0a7 (v2.79.76). Pre-push gate: 70 test files, 836 tests passed. Production health remained on v2.79.75 immediately after push; rollout verification pending. Status stays inProgress until deployed runtime evidence confirms the direct bounded telemetry path.
-
Runtime deployment verified at 2026-09-15T03:42:12Z: https://mars-evolutiva.pensanta.com/api/health returned HTTP 200, version 2.79.77, deployment dpl_3tPt3QG6kKBnawhJB43Q9tjHojhU, Node v24.19.0. This confirms rollout after fb2d0a7 and follow-on DB push 3c77adb; it does not itself exercise a real breaker trip, so status remains inProgress pending that runtime evidence.
-
The runtime evidence this item has waited for since 2026-09-15 already exists, eight times over. MARS-735 states the db-breaker-open row 'can NEVER land'. Since fb2d0a7 / v2.79.76 it has landed on EVERY real production trip. All eight db-breaker-open rows in appEvents, all carrying the pool-counter snapshot (detail ? 'poolTotalCount' is true on 8 of 8): 2026-09-16 18:49:35.727332Z - total 4, idle 0, waiting 0 2026-09-21 23:05:02.505111Z - total 2, idle 1, waiting 0 2026-09-23 15:13:03.20369Z - total 3, idle 0, waiting 0 2026-09-23 19:22:01.613607Z - total 4, idle 1, waiting 0 2026-09-24 14:12:48.745853Z - total 4, idle 0, waiting 0 2026-09-25 00:32:31.013292Z - total 3, idle 2, waiting 0 2026-09-25 10:48:35.638055Z - total 4, idle 0, waiting 0 2026-09-28 15:45:11.305141Z - total 3, idle 0, waiting 0 poolMax 6 and cooldownMs 5000 on all eight. The first is 2026-09-16, one day after the fix shipped, so the bounded direct-pool INSERT that bypasses admit() works and has been working for twelve days. This is a real-trip verification, not a synthetic one: each row is a production breaker open that shed nothing. Closing condition met; the item is only open because nobody went looking for the evidence. A SEPARATE AND LARGER FINDING falls out of the same eight rows, filed against MARS-752 rather than here: poolWaitingCount is ZERO on all eight, and poolTotalCount never reaches poolMax (highest is 4 of 6). The pool-exhaustion signature - total = max with waiting > 0 - has NEVER been observed on any recorded trip. Every breaker open on record happened with spare pool capacity and nothing queued.
-
Production verification recorded 2026-09-28 by support-evolutiva-cc: eight real db-breaker-open appEvents rows from 2026-09-16 through 2026-09-28, all 8/8 carrying poolTotalCount and full pool snapshot after fb2d0a7/v2.79.76. Original missing-telemetry defect resolved. MARS-783 tracks separate open/close correlation gap.