MARS-145 follow-up: daytime-peak 53300 re-scan of the txn-pooler runtime pool

Sub-tickets

Activity

• wi-cli-venus created · 2026-07-03
• wi-cli-venus assigned · 5w ago
  db-mars-cc
• wi-cli-venus verified · 5w ago
  SWEEP DONE, db-mars-cc 2026-09-11, covering the FULL 10 weeks since MARS-145 rather than one peak window - the row asked for 'tomorrow's daytime peak' and that tomorrow was 2026-07-04, so a single-window sweep now would answer a narrower question than the data can. NOT CLOSEABLE YET, and the reason is a two-day population, not a bad result. THE ROW'S LITERAL QUESTION IS ANSWERED AND IT IS CLEAN: ZERO 53300, zero 'too many clients', zero 'too many connections', zero 'remaining connection slots' in appEvents since 2026-07-03. The SERVER-side ceiling was never reached. max_connections is 60 with 3 superuser-reserved, and the live picture right now (13:45 ART, in the daytime window this row names) is 9 client backends total, of which mars_app_runtime holds ONE. The Supavisor/default_pool_size revisit the row contemplates is not needed on this evidence. BUT THE THING THE ROW WAS ACTUALLY WORRIED ABOUT DID HAPPEN, 15 TIMES. MARS-145 lowered pg.Pool max 8->6 and the row says the pool CEILING under concurrent load was not exercised. It was, and it failed: 15 CLIENT-side exhaustions, 14 of them '(ECHECKOUTTIMEOUT) unable to check out connection from the pool after 15000ms in Transaction mode' plus one 'Connection terminated due to connection timeout'. That is the max:6 client ceiling, not the server's. Dates: 08-22 (1), 09-04 (4), 09-05 (1), 09-07 (2), 09-09 (7). Every one is category=server, action=unhandled-server-error, i.e. a student got a 500. ROUTES NAME THE CAUSE AND IT IS ALREADY FIXED ELSEWHERE: all 15 are /mi-cursada/<id> or /practicas/<id> - the practica detail bundle. That is MARS-680's measured hot-practica burst (8 concurrent calls for one practicaId, durations climbing), which exhausted max:6 and returned 500s to every viewer. MARS-680's in-flight coalescer shipped 8155822 on 2026-09-10, and MARS-694 put statement_timeout=10000 on the runtime role the same day, which bounds the pathological query instead of letting it hold a connection indefinitely. THE TIMELINE SUPPORTS THE FIX AND I CHECKED IT AGAINST TRAFFIC RATHER THAN AGAINST THE CALENDAR, because 'zero failures' on a quiet day is not evidence. Practica-detail request volume by day: 09-04 187 hits/4 failures, 09-05 171/1, 09-06 299/0, 09-07 325/2, 09-08 137/0, 09-09 7/7 (that day's appEvents are near-empty because the disk-full outage rejected the writes - MARS-703's finding, visible here), 09-10 286/0, 09-11 116/0 so far. SO THE FIRST FULL POST-FIX DAY CARRIED MORE PRACTICA-DETAIL TRAFFIC THAN EITHER DAY THAT FAILED AND PRODUCED ZERO FAILURES. That is a real signal, not an artifact of quiet. TWO CORRECTIONS TO THIS ROW'S PREMISES, both measurable. (1) 'Daytime peak' is the wrong window: by ART hour the 15 failures fall at 19:00 (6), 16:00 (2), 20:00 (2), 18:00 (1), 15:00 (1), 11:00 (1), 10:00 (1), 06:00 (1). The failure peak is ~19:00-20:00 ART, evening, not midday. Anyone re-running this should sweep the evening. (2) The row predates MARS-694: 'the running deployment' is no longer the MARS-145 deployment connecting as postgres, it is mars_app_runtime through the same :6543 pooler since 2026-09-10, so the pool being measured is not quite the pool the row was written about. CLOSE CONDITION, stated so this is not re-opened on vibes: seven consecutive days from 2026-09-10 with at least one day above 300 practica-detail hits and ZERO ECHECKOUTTIMEOUT rows. Two of those seven are in (09-10 at 286 hits, 09-11 partial), both clean. RE-CHECK 2026-09-18 with the query on this row; if still zero, close as validated by the MARS-680/MARS-694 pair rather than by the original sizing, and say so - the cutover was not proven safe, it was made safe by two later changes.
• wi-cli-venus assigned · 4w ago
  db03-mars-ca