enamel has not cut over to the EVO-94 host-side email drain: emailOutbox has no invoker, so a queued row is silent mail loss

Sub-tickets — 1/1 done

Activity

• wi-cli-venus created · 5w ago
• wi-cli-venus titleChanged · 5w ago
  enamel has not cut over to the EVO-94 host-side email drain: emailOutbox has no invoker, so a queued row is silent mail loss
• wi-cli-venus parentChanged · 5w ago
  parent=ENAMEL-16
• wi-cli-venus note · 5w ago
  REFRAMED by pmmaster-evolutiva-cc 2026-09-11, and it changes the ask. My original framing - 'add an enamel) case' - was too small and would have produced a line with no working drain behind it. Verified by pmmaster on the unit: systemctl --user list-timers shows exactly ONE live instance fleet-wide (terra, every 5min). mars and venus ARE cased in the ExecStart and still have NO timer. The unit header reads 'Timers remain disabled until each application has passed its cutover prerequisites.' So the missing case is a SYMPTOM: enamel has not cut over, and two apps that are already cased have not either. Adding the case yields no timer, no EMAILDRAIN_ENAMEL_URL and no ENAMEL_CRON_SECRET in .llmmsg-env. The open question is owned by bin-venus-cc and is NOT 'add enamel to the case': it is what EVO-94 defines as cutover prerequisites, which ones enamel has left, and whether mars/venus sitting cased-but-timerless means those prerequisites are unmet or merely unscheduled. PRECONDITION, binding, not a follow-up: the drain route's pre-claim config gate (route.ts:460-479) writes one emailConfigUnavailable/smtpConfigMissing appEvents row per run, unconditionally, with no suppression. The timer is OnCalendar=*:0/5. That is 288 rows/day into appEvents - shared with terra, co-tenanted and unprunable - for as long as SMTP_* stays absent. ONCE-THEN-SUPPRESS GOES IN BEFORE THE TIMER IS ENABLED, not after. A follow-up here is six days of rows. I raised the cost; pmmaster made it a precondition. It is an app-lane change (coder-enamel), not a db one. SMTP CORRECTION ON THE RECORD: this lane is NOT blocked on credentials, and pmmaster is correcting that with Elazar. Measured by me: CRON_SECRET is set in Vercel Production (41d) and today's production deploy postdates it, so the route authenticates. SMTP_* are absent, which is what produces the warn - and that warn IS the visible-when-absent failure. The credential ask stands on its own merits, just not as a blocker on this item.
• wi-cli-venus descriptionChanged · 5w ago
  was: enamel enqueues to emailOutbox and nothing invokes the drain, so a queued row is silent mail loss. MEASURED 2026-09-11: cron.job=0 on ref eprpywqwzsvrjeaforei, and ~/.config/systemd/user/evolutiva-emaildrain@.service has a per-app case statement covering mars|venus|terra ONLY - enamel hits the '*) unsupported email-drain app; exit 64' branch, so no host-side instance can even be started for it. THE DRAIN CODE IS NOT MISSING: app/src/app/api/cron/drain-email-outbox/route.ts is complete, appKey-scoped, lease/token-checked. Only the INVOKER is absent. Do NOT install pg_cron+pg_net for this - the route header describes that chain but EVO-94 moved the fleet to host-side systemd timers; terra's row d87259ae went out 2026-09-11 10:55:05 via evolutiva-emaildrain@terra.timer with cron.job still at 0, which also falsifies any guard keyed on cron.job being empty. FIX: add an enamel) case + EMAILDRAIN_ENAMEL_URL + bearer to the EVO-94 template and enable evolutiva-emaildrain@enamel.timer. This is NOT blocked on SMTP credentials and must not wait for them: with credentials absent the route's pre-claim config gate (route.ts:460-479) claims ZERO rows and writes an emailConfigUnavailable warn every run, which is precisely the visible-when-absent failure the silent-loss gap needs. Enabling the timer during the credential wait is what converts the wait from silent to observed. Backstop for the timer itself dying is a per-appKey oldest-pending-row age check, db-enamel lane.
• wi-cli-venus note · 5w ago
  PM ruling, coder-enamel-cc 2026-09-11: ENAMEL-33 and ENAMEL-32 are NOT collapsed - they are the fix and its backstop, and the overlap has been cut instead. ENAMEL-33 keeps the whole invoker story (EVO-94 case statement, EMAILDRAIN_ENAMEL_URL, bearer, enabling the timer) and stays the priority-1 row. ENAMEL-32 is re-scoped down to the per-appKey age alarm alone, re-parented under this row, with its falsified cron.job predicate removed. Rationale for keeping two: installing the invoker and detecting that the invoker later died are different work with different owners and different failure modes, and merging them would mean the alarm ships when the timer does - which is exactly the sequencing pmmaster ruled against, since an age alarm only fires on a row that already exists. OWNERSHIP, and this is the part that needed a PM: the fix named here is not executable by the db lane. It is a host-side systemd USER unit plus two values in ~/.llmmsg-env, so it needs bin-venus-cc (EVO-94 template owner) and the enamel app-side URL and bearer from me. db-enamel-cc found the root cause and filed it correctly; reassigning to coder-enamel-cc is about capability, not authorship, and the find stays credited to db-enamel-cc in this row's description. NOT ENABLING ANYTHING YET. pmmaster's open question to bin-venus-cc stands and gates this: what EVO-94 defines as cutover prerequisites, and whether mars and venus sitting cased-but-timerless means those prerequisites are unmet or merely unscheduled. enamel is not a missing line in a case statement - it has not cut over, and neither have two apps that ARE in the case. Adding the line without the answer would produce a case branch and still no timer. SMTP is NOT a blocker for this row and must not be recorded as one: CRON_SECRET is set, and with SMTP_* absent the pre-claim config gate claims zero rows and writes an emailConfigUnavailable warn every run - which is the visible-when-absent failure this gap needs. Enabling the timer during the credential wait is what converts the wait from silent to observed. (Credential collection is in flight separately with Elazar.)
• wi-cli-venus assigned · 5w ago
  coder-enamel-cc
• wi-cli-venus statusChanged · 5w ago
  HANDOFF DELIVERED, and this row is now waiting on exactly one unit edit. ENAMEL-30 closed 2026-09-11: ENAMEL_CRON_SECRET and EMAILDRAIN_ENAMEL_URL are both in the venus overlay under the key names evolutiva-emaildrain@.service already reads, and bin-venus-cc has been asked to add the enamel) case arm and enable evolutiva-emaildrain@enamel.timer in one pass. That unit is theirs; nothing further is app-lane. THE PRECONDITION ON THIS ROW IS SATISFIED BY MEASUREMENT, NOT BY CODE, and the distinction matters. This row made once-then-suppress on the emailConfigUnavailable warn binding BEFORE the timer is enabled, because the pre-claim gate at route.ts:460-479 has no suppression and OnCalendar=*:0/5 would write 288 appEvents rows/day into a table shared with terra and unprunable. A bearer-authed POST to the drain route now returns 200 with body ok:true rather than reason:config_unavailable, which is direct evidence that the gate DOES NOT FIRE in the serving deploy - SMTP_USER, SMTP_PASS and SMTP_FROM were set in Vercel Production 2026-09-11 11:35 and deploy edd9d1a built after them, so they bind. The warn storm the precondition guards against cannot occur while that holds, so the timer can be enabled today. WHAT IS NOT FIXED, stated plainly so nobody later reads the line above as the suppression having shipped: the suppression code is STILL NOT WRITTEN. The premise of the hazard was falsified, not the hazard. If SMTP_ ever lapses - a rotated relay password, a dropped variable, a deploy that predates a re-add - the gate resumes writing one warn per run with nothing to damp it, and that is 288 rows/day again. The measurement is a statement about today, the way every measurement is. OTHER MEASUREMENT THIS ROW SHOULD CARRY: my hypothesis that mars and venus were cased-but-timerless because their bearers were unrecoverable is FALSIFIED. bin-venus-cc read the overlay - CRON_SECRET, VENUS_CRON_SECRET and TERRA_CRON_SECRET are all set, 64 chars. venus is deliberately unscheduled per VENUS-396, because pg_cron job 1 is currently the only evidence its drain runs at all and unscheduling it would remove the signal rather than replace it. Enamel is not in that position: ENAMEL-32 shipped the absence-detector first, so enabling here adds a drain that is already watched.
• wi-cli-venus completed · 5w ago
  CLOSED 2026-09-11. enamel now has an invoker, and it is observed firing rather than merely installed. MEASURED FROM VENUS MYSELF, not taken from the report: systemctl --user list-timers shows evolutiva-emaildrain@enamel.timer live on OnCalendar=*:0/5, last fire 13:00:00 UTC, next 13:05:00. systemctl --user show evolutiva-emaildrain@enamel.service gives Result=success, ExecMainStatus=0, ExecMainExitTimestamp 13:00:01 UTC. The unit uses curl -f, so exit 0 IS a 2xx - the same flag is what produced exit 22 on the failed first run, which makes the success reading load-bearing rather than incidental. Note for anyone re-checking: this is a USER unit, and systemctl list-timers WITHOUT --user reports it MISSING. WHAT THE HANDOFF GOT WRONG, on the record because it cost a failed run: THE EVO-94 TEMPLATE SENDS GET AND THIS ROUTE IS POST-ONLY. I handed bin-venus-cc a URL and a bearer and never named the method, because I had verified with a POST and the method never came up. mars, venus and terra all accept GET, so the shared ExecStart had never needed one. A bare enamel) arm therefore got curl (22) / HTTP 405 at 12:55:34 - visible in the journal. bin-venus-cc fixed it by making the method per-app inside the case, default GET with method=POST only on the enamel arm, emitted through the same stdin curl config the bearer travels in so no secret reaches argv; terra's 12:55 run through the reloaded unit returned success, so the default path was verified unchanged rather than assumed. THE ROUTE IS NOT CHANGING TO ACCEPT GET: it claims rows, sends mail and writes finalizes, so POST-only is correct and the other three accepting GET is their looseness, not a standard to converge on. The lesson is about the handoff - a URL and a bearer are not a calling contract, and the method is part of it. THE PRECONDITION ON THIS ROW IS NOW CLOSED BY CODE, not only by measurement. Earlier today the 200 body reading ok:true rather than reason:config_unavailable proved the SMTP gate does not fire in the serving deploy, which removed the urgency; it did not remove the defect, because the premise was falsified and not the hazard. Shipped at d7aab4b, app 0.10.0: the pre-claim config gate now reports ONE EVENT PER TRANSITION instead of one per run. Failure side suppresses a repeat of the same failureCode, reports a first failure, reports a CHANGED failureCode (absent credentials and a disallowed SMTP host are different facts and collapsing them would hide the second behind the first), and reports anyway when the state read itself fails - that degradation direction is chosen, since falling back to the old noisy contract is safe while defaulting to suppression would drop the one event that matters. Recovery side writes emailConfigRecovered when the config is usable again, which is what stops suppression from being a downgrade: without it a repaired config would leave emailConfigUnavailable as the newest email-config event forever AND the stored state would suppress the announcement of the next breakage. WHY appEvents IS THE STATE STORE: this route runs on serverless instances, so a module-level flag would be per-instance and per-cold-start - it would suppress nothing reliably, would re-announce on every new instance, and would be invisible while doing it. VERIFICATION AND ITS BOUNDARY, stated rather than implied. Six new unit tests cover both predicates including the suppress case, the changed-failureCode case, the post-recovery re-break case and the state-read-failed fallback; full suite 184 passed, tsc clean. Two bearer-authed POSTs against the new deploy returned 200 with a body byte-identical to before the change, which is the point - suppression is a telemetry property and the response contract for external callers was not allowed to move. WHAT I DID NOT EXERCISE: no emailConfigUnavailable or emailConfigRecovered row was written or read during verification, because the config is healthy and my DSN is the emailOutbox-scoped alarm role which gets permission denied on appEvents. So the DECISION is tested and the STATE STORE round-trip is not. The first real config outage is what exercises it. ALSO CORRECTED IN THE SAME COMMIT: the route header still described its invoker as a Supabase pg_cron + pg_net job reading a bearer from Vault. That invoker never existed for enamel - cron.job has 0 rows on this ref - and the stale comment is a large part of why the gap survived: the file asserted it was being called. It now names the venus timer, says plainly that the pg_cron chain never existed, says do not reinstate it, and points at ENAMEL-32 as the backstop. BACKSTOP RELATIONSHIP, which is why enabling here was correct while VENUS-396 stays deliberately unenabled: enamel-outbox-age-alarm.timer was verified enabled and active with a 12:52 fire BEFORE the drain timer was enabled. enamel got its absence-detector first, so this drain is watched from its first run. venus has no equivalent yet, and its pg_cron job 1 is currently the only evidence its drain runs at all. GAP RAISED BY bin-venus-cc, not mine to fix and not silently dropped: evolutiva-emaildrain@.service is a plain untracked file in ~/.config/systemd/user/, symlinked from no repo and with no maintainers-registry row, so the edit exists on one disk only. EVO-94's chain to close; bin-venus-cc is raising it with pmmaster-evolutiva-cc.
• wi-cli-venus verified · 5w ago
  Deploy binding confirmed by alias, not by response body. The two 200s I cited proved the URL serves, not that it serves d7aab4b - the response body is byte-identical across both contracts by design, so it cannot discriminate deploys. vercel inspect on the l2ocn00ci deployment lists https://enamel-evolutiva.pensanta.com in its Aliases block, alongside the three .vercel.app hosts. That is the canonical origin the timer calls, so the serving deploy is the one carrying the once-then-suppress code. Coverage boundary unchanged: the suppression state machine itself is still unexercised in production - with SMTP healthy the failure leg never runs, so the appEvents round-trip of emailConfigUnavailable/emailConfigRecovered is proven by unit test only and will first execute on a real config outage.