Restore Mars error->email-to-Elazar loop (bs-mqn8g0rhso5 fleet decision): B ESOCKET operator-email hotfix (after->emailOutbox enqueue) + C instant-error-email onRequestError->enqueue direct-to-Elazar, throttled

Sub-tickets

Activity

• wi-cli-venus created · 2026-06-21
• wi-cli-venus commented · 2026-06-21
  Scope + fleet decision (bs-mqn8g0rhso5 closed + bs-mqn8ogqjpze harden-and-unify). Restores Elazar's error→DM-in-aro→email→fly-solo loop. FLEET STANDARD: separate TRIGGER (instrumentation.ts onRequestError, in-lambda, relay-independent) from DELIVERY (emailOutbox enqueue + Supabase pg_cron */5 drain+retry, MARS-76); the hook ENQUEUES (local INSERT, can't ESOCKET in-hook), never raw-awaits SMTP; per-route throttle = dedupeKey(route+sig) at insert. Mars owns the reference DELIVERY shape (emailOutbox). LEVER B (LIVE INCIDENT, ships FIRST, in flight w/ coder-mars): operator-alert email in src/app/api/log-error/route.ts:~243 fires via after()/teardown → 10/10 ESOCKET tonight = instant error-email DOWN. Fix = route off after() onto awaited emailOutbox enqueue. Recipient UNCHANGED for B (support inbox soporte-evolutiva-mars@pensanta.com). Covers CLIENT-reported errors (logError calls). Audit pre-push (email path) + PTD + verify: genuine error → outbox ROW + actual DELIVERY (not 'queued'). Then run 48h ESOCKET-vs-success appEvents scan (serverError↔email-success join) as send-health proof. LEVER C (queued AFTER B green): add the canonical TRIGGER — instrumentation.ts onRequestError (today LOGS ONLY, no email) → enqueue to same outbox → recipient DIRECT-TO-ELAZAR (elazarpimentel@gmail.com), per close DEFAULT decision (reversible, Elazar-veto-able; veto window = B deploy). Throttle 1/route/10min via dedupeKey + try/catch belt-and-suspenders. Coordinate helper shape w/ coder-pluto (but Pluto's leg also lacks try/catch+throttle — do NOT clone as-is, harden). Daily error-digest (MARS-148) STAYS as rollup. Mars ends: TWO triggers (onRequestError=C server-side, log-error=B client) → ONE outbox → one dedupe. LEG-2 DM rail already alive (DMed PM 3x tonight, error-gated).
• wi-cli-venus commented · 2026-06-21
  TWO binding corrections to the scope (fold before building C). (1) DB-DOWN DELIVERY CORRECTION (pm-venus/audit-venus C2, decisive): you CANNOT deliver a DB-down alert through a DB-resident queue — the dominant server-500 IS the DB being down, so onRequestError→enqueue(emailOutbox) fails the INSERT→no row→silent on the exact outage. So DELIVERY MUST MATCH FAULT: lever C trigger (ii) instrumentation.ts onRequestError → DIRECT in-process SMTP to Elazar, DB-INDEPENDENT, in-memory module-scope throttle 1/route/10min, never-throw try/catch — NOT outbox. Lever B (log-error, client-reported, DB provably up) → emailOutbox enqueue STAYS correct. So Mars is two-trigger→TWO-delivery matched to fault, not one delivery. In-memory throttle is per-instance (a multi-instance loop sends a few/10min — bounded, accepted; a pg-backed throttle would reintroduce DB-down silence). (2) LEG-2 ELAZAR AMENDMENT (generalpm, binding all apps): alert must POST TO aro:mars (aro-tagged), NOT private DM to PM, so Elazar sees the in-room notification; PM runs the ENTIRE fly-solo troubleshoot IN aro:mars (coder/db/audit dispatch aro-tagged+visible), not private DMs; verify Elazar is a member of aro:mars. Rail target change DM-pm-mars→post-aro:mars is infra (nw-venus/coder-venus). Dedup/throttle still applies (no aro flood).
• wi-cli-venus commented · 2026-06-21
  bs-mqn8ogqjpze CLOSED (fleet-binding). Final deltas to Mars scope: (1) C RECIPIENT = WORK email, env-driven (pensanta.com), NOT gmail — my earlier 'elazarpimentel@gmail.com' was wrong; the email-outbox.ts firePersistentFailureAlert gmail hardcode is a SEPARATE low-pri cleanup WI, do not propagate it. (2) C delivery = DIRECT in-process SMTP (DB-independent, in-memory module-scope throttle 1/route/10min, never-throw), confirmed NOT outbox — a DB-down 500 can't INSERT into a DB-resident queue. B (log-error client-reported, DB-up) → outbox enqueue STAYS. (3) LEG-2 aro requirements for Mars: (a) daemon target DM→aro:mars = nw-venus infra; (b) fan-out RE-ENABLED on aro:mars = hub aro_fanout_allow config, owner pm-llmmsgsrv/llmmsg-srv; (c) Elazar ∈ aro:mars VERIFIED ✓ (aro_list: members coder/db/elazar/pm). (4) PM runs the ENTIRE incident troubleshoot IN aro:mars (coder/db/audit dispatch aro-tagged+visible), routine non-incident dispatch stays DM. Close-proof: genuine serverError → alert VISIBLE in aro:mars + PM visibly debugs there + email lands to work address.
• wi-cli-venus commented · 2026-06-21
  STATE (fleet standard RATIFIED+frozen, bs-mqn8vka8iky CLOSED). B FINISHED + delivery-verified: v2.16.80, audit PASS:8d345ba, outbox row SENT 3.4s, 0 send-failed — inline-SMTP-in-after() ESOCKET (10/10) → outbox enqueue+pg_cron drain. WI stays OPEN for C. C (leg-1 instant server-error email, Mars NET-NEW) GREENLIT + building by coder-mars, corrected shape confirmed: instrumentation.ts onRequestError → DIRECT in-process SMTP (NOT outbox), in-memory module-scope throttle 1/route/10min, never-throw, ERROR_NOTIFY_ENABLED default-ON, recipient=WORK email env-driven (pensanta.com), daily digest stays; coordinating Pluto's proven sendEmailCore-immediate helper; ETA ~30-40min; PTD incl genuine serverError round-trip (DM-to-aro + email). A/EVO-12 (MARS-155, server-side 404 capture + own-route→ERROR w/ referer-path≠requested-path discriminator + bot-probe→WARN carve-out) dispatched to audit-ca as 2nd coder, parallel (A=log-404 vs C=instrumentation, no collision). LEG-2: Elazar∈aro:mars✓; nw-venus restarted Mars listener on keepalive 60cf5e6 (clean 03:50:18 UTC); APPLOG_ARO=aro:mars daemon flip gated on (1) pm-llmmsgsrv re-enabling hub fan-out on aro:mars (requested) + (2) the applog-listen.ts APPLOG_ARO code change. Drainer-gate: Mars */5 pg_cron drainer VERIFIED ✓ so the outbox fallback branch is safe.
• wi-cli-venus assigned · 4w ago
  coder03-mars-cc
• wi-cli-venus decision · 4w ago
  Assigned coder03-mars-cc by pmmaster-evolutiva-cc under Elazar's standing GO. Load, measured 2026-09-12 01:58Z: mars carries 82 UNASSIGNED rows while coder-mars-cc holds 52, coder02-mars-cc 6 and coder03-mars-cc 1. This row was unassigned and it restores a BROKEN path - Mars errors no longer reach Elazar by email at all - which is the clearest case of 'obvious and would fix something broken' in the mandate. Scope is the bs-mqn8g0rhso5 fleet decision's arms B and C only: B is the ESOCKET operator-email hotfix (after -> emailOutbox enqueue), C is the instant-error-email onRequestError -> enqueue direct-to-Elazar, throttled. Do not widen it. The throttle is load-bearing and needs a stated bound, not a TODO - an unthrottled instant-error email is how one broken route becomes a mailbox flood, and the failure would arrive as Elazar's inbox rather than as an alarm. Acceptance must force a red arm: show the path FAILING to deliver before the change, not merely delivering after it, per the standard db-enamel-cc set on enamel_060. Coordinate with coder-mars-cc before pushing - it is BUSY on MARS-724 and only one agent runs gitpush.sh per task.
• wi-cli-venus note · 4w ago
  LANE BOUNDARY WITH THE OTHER LIVE ERROR-EMAIL ROW, set by coder-mars-cc (PM seat) 2026-09-12. MARS-161 (coder03-mars-cc) and MARS-724 (coder-mars-cc) are both in flight and both end in an operator email. They do NOT collide on capture and they DO share one surface. Read this before writing either. CAPTURE IS CLEANLY SPLIT, and the split is structural rather than a convention we have to remember: MARS-161 leg C owns onRequestError in instrumentation.ts, which is the THROWN-error path. MARS-724 owns the server-side 404 capture path and is FORBIDDEN from instrumentation.ts - notFound() renders the boundary WITHOUT throwing, so onRequestError never sees a 404 and wiring it there would produce a handler that can never fire. Neither row can do the others job even by accident. THE SHARED SURFACE IS THE ENQUEUE PLUS THROTTLE PRIMITIVE, and this is where two lanes would silently build two of everything: both rows terminate in an emailOutbox enqueue addressed to the operator, throttled. Two independently-written throttles over one mailbox is not additive - they do not see each others counters, so the effective rate is the sum, and whichever lands second looks correct in isolation while the pair spams. DECISION: MARS-161 OWNS THE PRIMITIVE. MARS-724 CONSUMES IT AND BUILDS NO SECOND ONE. Leg B/C establishes the operator-email enqueue and its throttle; the 404 capture path calls that. If MARS-724 reaches the email leg before MARS-161 has landed the primitive, MARS-724 STOPS AT THE CAPTURE AND THE DM and says so on the row rather than writing a private enqueue to finish - a duplicate throttle is worse than an unfinished row, because the duplicate ships looking done. CONSEQUENCE FOR MARS-724 DEFINITION OF DONE, stated so it is not discovered late: done requires a wrong-URL round trip where the DM AND the email both land, so MARS-724 cannot close before MARS-161 lands its enqueue. That is a real dependency, not a preference. It does not block starting - every plan-independent leg (middleware pathname header, not-found.tsx server-side logging, the catch-all route handler for API/server 404s) proceeds now. File-level: nobody edits instrumentation.ts except MARS-161. Nobody edits not-found.tsx, middleware or the 404 capture path except MARS-724. Shared mailer/emailOutbox helpers change under MARS-161 only; MARS-724 raises a request rather than editing them.
• wi-cli-venus note · 4w ago
  Fleet-wide STOP (Elazar direct, relayed pmmaster 2026-09-11 23:56). Not done, not started - was about to read row for scope B/C. Next action on resume: read this row's full spec + bs-mqn8g0rhso5 fleet decision, then implement leg C (onRequestError->throttled emailOutbox enqueue) per coder-mars-cc's split note (I own instrumentation.ts + the shared throttle primitive; MARS-724 depends on it for its own close).
• wi-cli-venus descriptionChanged · 4w ago
  was: (empty)
• wi-cli-venus note · 4w ago
  The 2026-09-12 note stating implementation was not started is superseded: Leg C landed in 6d0d25f. The remaining work is operational enablement, exercised production round trip, and assignment rescope.