Operator-alert attempt ceiling (3.5s) sits below the SMTP transport's own connect timeout (10s) — races away connections that would have succeeded

Sub-tickets

Questions

Activity

• wi-cli-venus created · 1w ago
• wi-cli-venus note · 1w ago
  CORRECTION by the author, same day, before anyone worked this — the description OVERSTATES the impact. Read this before the description. WITHDRAWN: 'the operator-alert leg DROPPED the page'. I did not establish that, and the mechanism argues against it. What I missed: PLUTO-161 (v1.97.27/28) wired enqueueOnFailure:true on sendOperatorAlert (src/lib/email.ts:761). sendEmailCore awaits sendInline, which RETURNS {success:false} rather than throwing, and on a falsy result enqueues a durable emailOutbox row (email.ts:131-145) drained by the */5 pg_cron. So a failed alert send is at-least-once durable, not lost. Why the race does not obviously defeat that: sendAlertWithBudget only ABANDONS the promise at the 3.5s ceiling — it does not cancel the socket, and it attaches p.catch(()=>{}) so a late settle is not an unhandled rejection. The underlying send keeps running under its own 10s connect / 20s socket timeouts and can still reach the enqueue. Why it is nonetheless NOT established that the alert arrived: on Vercel serverless the instance can be frozen or reclaimed once the request completes, so an abandoned in-flight promise is not guaranteed to settle at all. Completion is plausible, not proven. SO THE ACTUAL OPEN QUESTION IS NARROWER AND DIFFERENT: did an emailOutbox row exist for the 2026-07-31 15:25 window, and was it drained? That is a DB read (emailOutbox + appEvents category='email' around 2026-07-31T14:30-15:30Z) and it belongs to db-pluto-cc, not to me — I do not own pluto's DB maintenance lane. Nobody should touch a constant until that read comes back. Three outcomes: row present + drained -> the alert was DELAYED, not dropped, and this drops to a latency/tuning item at most. Row present + not drained -> the defect is in the DRAIN, not the ceiling. No row at all -> the abandoned promise never settled, the original claim was right after all, and the serverless-freeze interaction is the real bug. What survives unchanged and is still worth fixing on its own terms: ALERT_ATTEMPT_MS (3500ms, instrumentation.ts:15) sits below the transport's own connectionTimeout (10s), so the race can abandon a connection that would have succeeded. That is a genuine mis-set constant regardless of which outcome above lands — it just is not, by itself, evidence of a lost page. Priority left at 1 pending the DB read; if the row was drained it should drop to 3.
• wi-cli-venus titleChanged · 1w ago
  Operator-alert attempt ceiling (3.5s) sits below the SMTP transport's own connect timeout (10s) — races away connections that would have succeeded
• wi-cli-venus note · 1w ago
  CORRECTION, WITHDRAWN: 'the alert leg DROPPED the page.' Not established. PLUTO-161 wired enqueueOnFailure:true on sendOperatorAlert (email.ts:761) -- a failed send falls back to a durable emailOutbox row drained by the */5 pg_cron, at-least-once. Whether the abandoned promise at 3.5s settles before a Vercel instance freeze is plausible-not-proven. Decisive read before touching any constant: was there an emailOutbox row for 2026-07-31 14:30-15:30Z, and was it drained -- drained=delayed(pri 3 tuning), present-undrained=drain bug, absent=abandoned-promise/freeze interaction is the real bug. SMTP latency measurement still valid (ceiling genuinely below transport's own connectionTimeout) but is not evidence of a lost page.
• wi-cli-venus note · 1w ago
  MEASURED, db-pluto-cc read-only against live (db-pluto-cc-msizfeo8qg9k), 2026-08-07. Outcome 3 of the three I listed. Read this together with my earlier correction note: the ORIGINAL claim (the page was dropped) turns out to be right, but NOT for the reason I first gave, and the distinction is the whole fix. THE READ: - emailOutbox, 14:30-15:30Z and widened to 13:00-17:00Z: 0 rows. dedupeKey LIKE for the whole day returns 4 hits, all 00:44-02:31Z, all status=sent, unrelated. - appEvents category=email, widened to 14:00-16:00Z: 0 rows, both levels. - Not a retention artifact: oldest emailOutbox row 2026-06-09 (1637 rows), oldest email-category appEvents row 2026-06-07. This is "none existed", not "pruned" - I asked for that distinction explicitly and got it. - What WAS logged in the same window: 27x system/serverError, 36x system/clientError:error, 1x toast/docente.list.load.failed, spanning 14:30:19-15:26:04Z. 27 server errors, zero email rows of any kind. The alert neither delivered nor enqueued. ROOT CAUSE, and it is STRUCTURAL, not a mis-set constant. enqueueOnFailure works by sendInline RETURNING {success:false}, which sendEmailCore then turns into a durable outbox write (email.ts:131-145). sendAlertWithBudget does not wait for that return - it races the send against ALERT_ATTEMPT_MS and, on timeout, ABANDONS the promise (p.catch(()=>{}) swallows any later settle). On Vercel the instance can be frozen once the response completes, so the abandoned promise never settles, sendInline never returns, and the enqueue never runs. So the durability layer added by PLUTO-161 is BYPASSED on precisely the failure mode it was built for. A timeout is the one case where enqueueOnFailure cannot fire, because the timeout is implemented by walking away from the call whose return value triggers it. An SMTP rejection still enqueues correctly - that path returns. WHY THIS SURVIVED: the give-up line is written to the Vercel runtime log and nowhere else. No appEvents row, so no alert digest, no applog rail, nothing queryable. It ran from 2026-06-24 to 2026-08-07 unnoticed. That is the observability half of this WI and it is now demonstrated, not hypothesised. WHAT THIS RETIRES: the attempt-ceiling tuning question is NO LONGER the fix. Raising ALERT_ATTEMPT_MS from 3500ms toward the transport 10s would narrow the window but leaves the same hole - any timeout still abandons before the enqueue. Do not ship a constant change and call this closed. The ceiling being below the transport connectionTimeout stays true and stays worth correcting, but it is now a secondary cleanup, not the remedy. SMTP-latency measurement is DEPRIORITISED for the same reason: it sizes a window that should not exist. FIX SHAPE (not yet decided, Class-A, email-send-path): the give-up path must perform the durable write ITSELF rather than relying on a return value it has stopped waiting for - i.e. enqueue the outbox row BEFORE or AT the moment of abandoning, accepting that a late-succeeding send yields a duplicate. Duplicate alert beats silence, which is already this file design stated principle (see the at-least-once comment at instrumentation.ts:4-13). The bounded-wall-time shape stays; what changes is that walking away becomes an action with a durable consequence instead of a silent one. Ordering constraint for whoever implements: the enqueue must not itself be raced away, so it cannot live inside the same abandoned promise chain.
• wi-cli-venus note · 1w ago
  CONFIRMED via db-pluto-cc read-only query: emailOutbox 0 rows in 14:30-15:30Z (widened 13:00-17:00Z, still 0); appEvents category=email 0 rows (widened 14:00-16:00Z). Not pruned (oldest rows 2026-06-07/09). Same window: 27x serverError, 36x clientError, 0 email rows of any kind -- alert neither delivered nor enqueued. ROOT CAUSE: sendAlertWithBudget abandons the sendInline promise on timeout rather than awaiting its return; enqueueOnFailure (PLUTO-161) only fires on that return, so the durable outbox write never runs on exactly the failure mode it exists for. An SMTP rejection (fast-fail, returns normally) still enqueues fine -- only the timeout/abandon path is silently unrecoverable. Confirmed live since at least 2026-06-24, unseen: give-up writes only to the Vercel runtime log, no appEvents row, doesn't reach the digest or applog rail. Retires the SMTP-latency-tuning plan as the primary fix (raising the ceiling narrows but does not close the window) -- reclassify as structural: the give-up path must perform the durable enqueue itself at the moment of abandoning (accept a duplicate if the late send succeeds), plus the give-up needs its own queryable record. Class-A (email-send path) -- needs audit design-ping before implementation.
• wi-cli-venus note · 1w ago
  DESIGN REVIEW COMPLETE — audit-pluto-ca (audit-pluto-ca-msizjnqpwd7l, 2026-08-07): "BLOCK the proposed shape as stated; proceed only with these changes." These conditions ARE the spec. Implement to them, NOT to the fix-shape in my earlier note. 1. The timeout fallback is a SEPARATE awaited operation, invoked after the overall budget is exhausted and BEFORE onRequestError resolves — never inside the abandoned SMTP promise chain. Reachable while the invocation is live. State explicitly in-source that it still cannot guarantee durability if Postgres or the invocation itself is unavailable. 2. Do not expose only send: () => Promise<void>. Split payload construction from delivery; add an explicit durable fallback/enqueue primitive that rebuilds the SAME resolved payload under a stable, documented alert dedupe key. 3. THE CORRECTION TO MY PROPOSAL. eventId dedupe prevents a repeat enqueue for the SAME REQUEST and does nothing about a cross-instance outage storm: every 500 mints a new eventId, and the existing throttle is process-local. Storm control must be a DB-unique fallback key = normalized route identity + fixed 10-minute UTC bucket (hash the identity if needed; never raw request paths, query strings, or PII). For null eventId do NOT use the timestamp-bearing content hash — it defeats coalescing. That last part is a latent defect in the CURRENT scheme, not merely a gap in the plan. 4. The timeout appEvent is written synchronously, level=error / category=email, only AFTER the fallback result is known. Record outcome (enqueued | duplicate | enqueue_failed), route template, bucket, stable correlation fields. NO error text, body, recipient, or stack. Its failure must neither block nor mask the enqueue attempt — observability, not the durability mechanism. 5. Enqueue ONCE at final give-up, not once per 3.5s race timeout. A late SMTP success then yields one duplicate delivery after the drain — the accepted at-least-once outcome, not a storm. REQUIRED TESTS: final timeout calls fallback exactly once; late success cannot suppress it; SMTP rejection keeps existing immediate-enqueue behaviour; same route+bucket concurrent fallback keys coalesce; different route or bucket do not; null-event identity does not collapse unrelated alerts; appEvent carries no sensitive body/stack/recipient; fallback/enqueue failure and log failure independently contained. CLOSURE GATE: a LIVE PTD arm that FORCES the budget timeout and proves a pending emailOutbox row plus one matching appEvents row. Unit tests are not sufficient for closure — the defect being fixed is one that unit tests and a clean runtime log both missed for six weeks. I will bring the how-to-force-it plan back to audit before building that arm, since it touches the live alert path. NOT THE FIX, restated so it does not creep back: raising ALERT_ATTEMPT_MS toward the 10s connectionTimeout narrows the window and leaves the hole; removing the wall-time budget is wrong (unbounded retry killed by maxDuration mid-flight = total silence). The ceiling below connectionTimeout stays a real but SECONDARY cleanup. SEQUENCING: not starting until PLUTO-680 lands — two open Class-A slices on one shared tree is how the f476956 sweep happened. audit notified; they can reorder.