an operator alert was LOST on 09-17: operatorAlertTimeoutFallback enqueue_failed, and the firing count is not on PLUTO-688/743
error/email/operatorAlertTimeoutFallback fired 3 times on 09-17 (14:22Z..17:35Z), all on /mi-cursada. Outcomes: enqueued x2, enqueue_failed x1. Measured by db-pluto-cc 2026-09-19 on my request. THE POINT: enqueue_failed means an operator alert was LOST. Not delayed, not retried - the fallback path that exists to catch a timed-out alert send itself failed to enqueue, so whatever condition raised that alert went unannounced to a human. p1 because the failure mode is silence: nobody is paged that the pager broke, and the only reason this surfaced at all is that Elazar asked me to sweep every app for errors. NOT COVERED by the existing rows. PLUTO-688 (design, re-opened) and PLUTO-743 (correlation id null on failure) both concern this path but neither records these three firings. They matter as frequency data: these are the 2nd, 3rd and 4th firings EVER, the first having been 09-11 23:19Z. So the path went from one lifetime firing to four in six days, with one outright loss. A design ticket that does not carry the firing count reads as theoretical when it is not. Wanted, in order: (1) confirm from the row detail which alert was lost on 09-17 and whether its triggering condition still holds - a lost alert may still be an open incident nobody knows about; (2) decide whether enqueue_failed should itself page, which is the recursion PLUTO-688 is presumably wrestling with; (3) attach the firing counts to 688 so it stops looking hypothetical. Same 09-17 14Z-18Z burst as PLUTO-745 (serverError + systemBannerQueryFailed). Related by time window; a shared cause is NOT established and should not be assumed - see 745's note on why a same-digest match is not a same-cause finding. Filed by support-evolutiva-cc; db-pluto-cc identified this as coder-pluto-cc's lane and declined to file across lanes. No PII: outcomes, counts and route only.
Activity
-
TRIAGE 2026-09-19. (1) CORRECTS MY PLUTO-688 CLOSE: I wrote the fallback means a race-timeout 'no longer loses the alert'. enqueue_failed x1 on 09-17 proves that FALSE. The fallback enqueues into emailOutbox in the SAME database whose starvation caused the timeout, so it fails exactly when it is needed. 688's 'nothing is lost' is retracted; 688 stays closed only for the CEILING question (p50 3553 > 3500 is measured), the fallback-durability question is THIS row. (2) CAUSE: 09-17 14:22-17:35Z sits inside the 745 burst of 09-17 14-18Z. The 09-18 16-20Z burst of 745 is the PLUTO-742 EPISODE 2 I measured in postgres_logs (slow queries peaked 16Z, last 20:09Z, INSERT appEvents up to 89.6 s, 97 statement timeouts) - so that half is attributed to instance starvation, measured, not digest-matched. The 09-17 window is PAST 1-day log retention: unattributable now, and I will not claim it is the same cause. It IS a third episode candidate: 09-11, 09-17, 09-18 = three in 7 days. (3) THE LOST ALERT: enqueue_failed on an alert raised by serverError on /mi-cursada. Those errors are NOT unrecorded - serverError appEvents rows exist for the window (10 on 09-17 per 745), so the triggering condition is visible in the durable table even though the page never fired. Whether it still holds: /api/health dbOk true at 06:17Z 09-19, last slow query 20:09Z 09-18. Asked db-pluto-cc for the enqueue_failed row itself (time, detail) to identify which alert. (4) FIX SHAPE: no app-side fallback can be durable while the DB is the failure; needs an out-of-DB channel or a state a human polls. Design on this row after the row is read, not before.
-
db-pluto-cc read (prod appEvents SELECT only). LOST ALERT = row e0d9b58c-a93c-42a3-a7db-d705a31ffdf9, 2026-09-17 14:22:39Z, /mi-cursada, render, bucket 2982758, outcome enqueue_failed, serverErrorEventId NULL. CORRECTS MY TRIAGE ABOVE: I wrote the triggering condition is visible in the durable table because 10 serverError rows exist in the window. WRONG FOR THIS ALERT: zero serverError rows on /mi-cursada within +/-2 min; the burst's first serverError is c56b59ce at 14:25:53 (+3m14s, digest 2097667694), a neighbour not a match. Same shape as 09-11: the 500 behind a lost alert wrote no row AND its alert was lost. So the request that raised it is unidentifiable from prod data - only route, bucket, time survive; alert content is not stored anywhere. The loss sits at the LEADING EDGE of the burst. The two 'enqueued' rows: b6be0c51 17:23:28 (id NULL) and 3004c779 17:35:21 (serverErrorEventId 2844b9e6 populated) - so 743's null is 2 of 3, non-null when the serverError INSERT succeeded, consistent with 743's mechanism. CONSEQUENCE: both alert-loss events (09-11 23:19, 09-17 14:22) are 500s that left NO trace in any durable store, so the logged 500 count is a floor on every episode. Whether the condition still holds: unreadable for 09-17 (retention); 09-18 episode recovered, dbOk true 06:17Z. Design next: PLUTO-743 (pre-minted id) makes the fallback row self-identifying even when the serverError INSERT dies; that is the smallest change that makes a lost alert identifiable, and I am taking it as the fix for this row's identifiability half. The durability half (out-of-DB channel) needs an Elazar decision on a channel - filing that to eq.
-
Identifiability half shipped via PLUTO-743 (commit 9f70406): serverErrorEventId now always resolves to a minted id, never null on write failure. Durability half (out-of-DB alert channel) stays open, pending eq/Elazar decision.
-
completed
-
Option A (direct SMTP last-resort, no DB write) shipped: 733c496, then audit-general-ca BLOCK x2 (last-resort misread resolved {success:false} as delivered + same defect in the main budget loop), fixed in b7ac624 and b0563d3 -> PASS (audit-general-ca, Class-A source review, 2026-10-04). eq#115 answered. Closed.