Restore Mars error->email-to-Elazar loop (bs-mqn8g0rhso5 fleet decision): B ESOCKET operator-email hotfix (after->emailOutbox enqueue) + C instant-error-email onRequestError->enqueue direct-to-Elazar, throttled

Sub-tickets

Questions

Activity

• wi-cli-venus created · 2026-06-21
• wi-cli-venus commented · 2026-06-21
  Scope + fleet decision (bs-mqn8g0rhso5 closed + bs-mqn8ogqjpze harden-and-unify). Restores Elazar's error→DM-in-aro→email→fly-solo loop. FLEET STANDARD: separate TRIGGER (instrumentation.ts onRequestError, in-lambda, relay-independent) from DELIVERY (emailOutbox enqueue + Supabase pg_cron */5 drain+retry, MARS-76); the hook ENQUEUES (local INSERT, can't ESOCKET in-hook), never raw-awaits SMTP; per-route throttle = dedupeKey(route+sig) at insert. Mars owns the reference DELIVERY shape (emailOutbox). LEVER B (LIVE INCIDENT, ships FIRST, in flight w/ coder-mars): operator-alert email in src/app/api/log-error/route.ts:~243 fires via after()/teardown → 10/10 ESOCKET tonight = instant error-email DOWN. Fix = route off after() onto awaited emailOutbox enqueue. Recipient UNCHANGED for B (support inbox soporte-evolutiva-mars@pensanta.com). Covers CLIENT-reported errors (logError calls). Audit pre-push (email path) + PTD + verify: genuine error → outbox ROW + actual DELIVERY (not 'queued'). Then run 48h ESOCKET-vs-success appEvents scan (serverError↔email-success join) as send-health proof. LEVER C (queued AFTER B green): add the canonical TRIGGER — instrumentation.ts onRequestError (today LOGS ONLY, no email) → enqueue to same outbox → recipient DIRECT-TO-ELAZAR (elazarpimentel@gmail.com), per close DEFAULT decision (reversible, Elazar-veto-able; veto window = B deploy). Throttle 1/route/10min via dedupeKey + try/catch belt-and-suspenders. Coordinate helper shape w/ coder-pluto (but Pluto's leg also lacks try/catch+throttle — do NOT clone as-is, harden). Daily error-digest (MARS-148) STAYS as rollup. Mars ends: TWO triggers (onRequestError=C server-side, log-error=B client) → ONE outbox → one dedupe. LEG-2 DM rail already alive (DMed PM 3x tonight, error-gated).
• wi-cli-venus commented · 2026-06-21
  TWO binding corrections to the scope (fold before building C). (1) DB-DOWN DELIVERY CORRECTION (pm-venus/audit-venus C2, decisive): you CANNOT deliver a DB-down alert through a DB-resident queue — the dominant server-500 IS the DB being down, so onRequestError→enqueue(emailOutbox) fails the INSERT→no row→silent on the exact outage. So DELIVERY MUST MATCH FAULT: lever C trigger (ii) instrumentation.ts onRequestError → DIRECT in-process SMTP to Elazar, DB-INDEPENDENT, in-memory module-scope throttle 1/route/10min, never-throw try/catch — NOT outbox. Lever B (log-error, client-reported, DB provably up) → emailOutbox enqueue STAYS correct. So Mars is two-trigger→TWO-delivery matched to fault, not one delivery. In-memory throttle is per-instance (a multi-instance loop sends a few/10min — bounded, accepted; a pg-backed throttle would reintroduce DB-down silence). (2) LEG-2 ELAZAR AMENDMENT (generalpm, binding all apps): alert must POST TO aro:mars (aro-tagged), NOT private DM to PM, so Elazar sees the in-room notification; PM runs the ENTIRE fly-solo troubleshoot IN aro:mars (coder/db/audit dispatch aro-tagged+visible), not private DMs; verify Elazar is a member of aro:mars. Rail target change DM-pm-mars→post-aro:mars is infra (nw-venus/coder-venus). Dedup/throttle still applies (no aro flood).
• wi-cli-venus commented · 2026-06-21
  bs-mqn8ogqjpze CLOSED (fleet-binding). Final deltas to Mars scope: (1) C RECIPIENT = WORK email, env-driven (pensanta.com), NOT gmail — my earlier 'elazarpimentel@gmail.com' was wrong; the email-outbox.ts firePersistentFailureAlert gmail hardcode is a SEPARATE low-pri cleanup WI, do not propagate it. (2) C delivery = DIRECT in-process SMTP (DB-independent, in-memory module-scope throttle 1/route/10min, never-throw), confirmed NOT outbox — a DB-down 500 can't INSERT into a DB-resident queue. B (log-error client-reported, DB-up) → outbox enqueue STAYS. (3) LEG-2 aro requirements for Mars: (a) daemon target DM→aro:mars = nw-venus infra; (b) fan-out RE-ENABLED on aro:mars = hub aro_fanout_allow config, owner pm-llmmsgsrv/llmmsg-srv; (c) Elazar ∈ aro:mars VERIFIED ✓ (aro_list: members coder/db/elazar/pm). (4) PM runs the ENTIRE incident troubleshoot IN aro:mars (coder/db/audit dispatch aro-tagged+visible), routine non-incident dispatch stays DM. Close-proof: genuine serverError → alert VISIBLE in aro:mars + PM visibly debugs there + email lands to work address.
• wi-cli-venus commented · 2026-06-21
  STATE (fleet standard RATIFIED+frozen, bs-mqn8vka8iky CLOSED). B FINISHED + delivery-verified: v2.16.80, audit PASS:8d345ba, outbox row SENT 3.4s, 0 send-failed — inline-SMTP-in-after() ESOCKET (10/10) → outbox enqueue+pg_cron drain. WI stays OPEN for C. C (leg-1 instant server-error email, Mars NET-NEW) GREENLIT + building by coder-mars, corrected shape confirmed: instrumentation.ts onRequestError → DIRECT in-process SMTP (NOT outbox), in-memory module-scope throttle 1/route/10min, never-throw, ERROR_NOTIFY_ENABLED default-ON, recipient=WORK email env-driven (pensanta.com), daily digest stays; coordinating Pluto's proven sendEmailCore-immediate helper; ETA ~30-40min; PTD incl genuine serverError round-trip (DM-to-aro + email). A/EVO-12 (MARS-155, server-side 404 capture + own-route→ERROR w/ referer-path≠requested-path discriminator + bot-probe→WARN carve-out) dispatched to audit-ca as 2nd coder, parallel (A=log-404 vs C=instrumentation, no collision). LEG-2: Elazar∈aro:mars✓; nw-venus restarted Mars listener on keepalive 60cf5e6 (clean 03:50:18 UTC); APPLOG_ARO=aro:mars daemon flip gated on (1) pm-llmmsgsrv re-enabling hub fan-out on aro:mars (requested) + (2) the applog-listen.ts APPLOG_ARO code change. Drainer-gate: Mars */5 pg_cron drainer VERIFIED ✓ so the outbox fallback branch is safe.