Server-error email from instrumentation.ts onRequestError never delivers (8x SMTP ETIMEDOUT since 2026-06-10; SMTP healthy, instrumentation-hook teardown timing)

Sub-tickets

Questions

Activity

• wi-cli-venus created · 2026-06-20
• wi-cli-venus commented · 2026-06-20
  db-pluto appEvents read (30d): all 8 server-error email attempts since 2026-06-10 = sendEmailFailed/smtpCode=ETIMEDOUT/smtpResponseCode=null; ZERO successes. Other sendOperatorAlert paths (login-rejected, security, practica, KPI) land fine (hundreds of email-sent) -> SMTP config healthy; failure isolated to instrumentation.ts onRequestError (blocking SMTP handshake in a tearing-down Vercel serverless context, insufficient remaining budget). NOTE real-time error visibility is NOT lost: the same errors land in appEvents via logError and the applog-listen@pluto rail DMs pm-pluto-cc (proven 18:42 2026-06-20). Fix options: (A) DROP the direct-SMTP send in the instrumentation hook; rely on appEvents+applog rail (DM real-time + applog-pull digest email) -- leanest, removes the fragile path; (B) enqueue to existing email-outbox.ts (out-of-band drainer) instead of blocking SMTP in onRequestError -- preserves an immediate dedicated per-error email. Touches email-send path -> audit design-ping before implement + diff review before push (commons trigger). Awaiting Elazar pick A vs B.
• wi-cli-venus commented · 2026-06-20
  PARKED at todo pending Elazar's fleet A-vs-B decision (generalpm rolling Mars+Venus into ONE decision — same onRequestError antipattern likely fleet-wide; uniform fix). Live question for Elazar: does he want IMMEDIATE per-error email at all, or do real-time DM (working) + batched digest email suffice? Yes→B (email-outbox), No/suffice→A (drop direct-SMTP). No coder dispatch until generalpm relays the call.
• wi-cli-venus commented · 6w ago
  Mars working-config diff (pm-mars 2026-07-03), 3-part fix: (a) SMTP host smtp.gmail.com PORT 587 secure:false STARTTLS — NOT 465/SSL; if Pluto is on 465 + egress blocks it = ETIMEDOUT by itself → verify Pluto's port first. (b) BOUNDED timeouts load-bearing: connectionTimeout/greetingTimeout 10s, socketTimeout 20s; retry transient-only {ECONNRESET,421,450,451}×3, ETIMEDOUT/ECONNECTION deliberately NOT retried. (c) send pattern: onRequestError alert AWAITED IN-REQUEST not after()-hook (after() teardown kills the socket=ESOCKET) — NOTE Pluto mem says our alert email is ALREADY fully-awaited-synchronous, so (c) likely already satisfied here; the real Pluto gap is the DURABLE BACKSTOP: mars enqueues to emailOutbox + pg_cron drain (MARS-59) so a CONN-timeout stays pending + re-drains instead of dropping. That backstop IS PLUTO-126 (email retry/failure gap after the PLUTO-38 drain-cron drop). So 161 fix = verify port 587 + implement the 126 outbox+cron. Mars coder offered to walk Pluto coder through the outbox schema.
• wi-cli-venus statusChanged · 6w ago
  Design converged (coder+db, Elazar-greenlit, held for audit pre-impl ping). ROOT: onRequestError operator-alert is fire-and-forget — sendEmailCore{immediate:true} bypasses enqueueEmail, ETIMEDOUT caught+console.error-dropped, never persisted (only mail path w/o outbox durability; PLUTO-166 made immediate:true deliberate for real-time crash visibility where after() is unavailable). Port(587/STARTTLS)+timeouts already correct — no-op. FIX (coder, src/lib/email.ts only, no schema): IMMEDIATE-FIRST, ENQUEUE-ON-FAILURE — keep synchronous attempt, on failure enqueue resolved payload to existing emailOutbox; dedupeKey=eventId+date. Delivers only WITH the 126 drain re-enable.
• wi-cli-venus completed · 6w ago
  SHIPPED v1.97.27 SHA d4ce038, audit PASS (Vercel READY, prod alias attached, live /api/app-version==1.97.27 deploymentId-matched, zero new runtime errors). Root cause: onRequestError operator-alert path was fire-and-forget — ETIMEDOUT returned {success:false}, console.error-dropped, never enqueued (only mail path without outbox durability). Fix: immediate-first, enqueue-on-failure — keep the synchronous immediate attempt (real-time crash visibility + after()-teardown-ESOCKET immunity per PLUTO-166), on failure enqueue the resolved payload to the existing emailOutbox. Wired enqueueOnFailure:true on sendOperatorAlert (covers sendServerErrorNotification+sendErrorNotification) + sendAccessRequestNotification. Ticket-reply deliberately NOT wired (returns delivery-truth per PLUTO-218). email-admin stays drop-on-failure by design (its own alert channel). Coverage note: failure→enqueue branch is inert until 049 cron live; the failure→drain round-trip is validated by 049's smoke-proof (authenticated drain 200 + 5 stuck rows drained).