Backstop: per-appKey age alarm on the oldest queued emailOutbox row (ENAMEL-33 owns the invoker)
BACKSTOP ALARM ONLY. Re-scoped 2026-09-11 by coder-enamel-cc (PM) after ENAMEL-33 established the root cause; the invoker fix lives there and this row must not duplicate it. WHAT THIS ROW IS: a per-appKey age alarm on emailOutbox - alert when the oldest row with sendStatus queued for a given appKey exceeds a threshold. Age, never unit-existence: age fires for every cause (timer never installed, timer dead, route erroring, credentials absent, request silently dropped) whereas a check that the timer exists passes on a timer that does nothing. Per-appKey because the table is co-tenanted and terra draining normally would otherwise mask enamel being dark - 'no drain unit installed' is invisible to Postgres, and that framing is db-enamel-cc's. WHAT I GOT WRONG AND HAVE REMOVED: this row originally keyed the guard on cron.job being empty and read cron.job=0 as the fault. FALSIFIED by db-enamel-cc and confirmed by pmmaster - EVO-94 moved the fleet off Supabase pg_cron onto host-side systemd user timers, so cron.job=0 is the HEALTHY state, and terra's row d87259ae was delivered 2026-09-11 10:55:05 with cron.job still at 0. Any guard keyed on that predicate would have alarmed permanently on a working system. Recorded rather than silently edited: the predicate was plausible, matched the route's own header comment, and was wrong. PRECONDITION, from pmmaster, and it is a precondition and not a follow-up: once-then-suppress must be in place BEFORE the alarm is enabled. At *:0/5 an unsuppressed per-run warn is 288 appEvents rows/day into a shared unprunable table, for however long the SMTP credential takes. A follow-up here is six days of rows. STILL TRUE FROM THE ORIGINAL FILING, carried: option B (refuse the enqueue while no drain exists) stays REFUSED - the enqueue rides inside the caller's transaction, so B rolls the user's action back on a missing timer, and a missing consumer is not a reason to reject the producer. And any probe written for this must query the USER systemd manager; systemctl list-timers without --user reports a working drain as missing, which is a standing fleet defect, not terra trivia.
Activity
-
parent=#2429
-
TWO CONSTRAINTS ADDED TO THE SPEC, pmmaster-evolutiva-cc 2026-09-11, both binding on whoever writes the probe. 1. SEQUENCING: the age alarm must be installed BEFORE enamel's first real enqueue, not alongside the drain when the SMTP credential lands. An age alarm only fires on a row that EXISTS, so if the first production enqueue happens while the alarm is still unbuilt, that first send is silent exactly as it is today and this row buys nothing for the case it was filed against. Alarm first; credential and drain whenever. This reorders ENAMEL-32 ahead of ENAMEL-16's held half rather than behind it. 2. USER-SCOPE PROBES, promoted from the prose into the spec because the next person writing the probe reads the spec and not the story: any check that shells systemctl list-timers WITHOUT --user reports a working drain as MISSING. User-scope systemd units are invisible to system-scope probes. pmmaster states this is not local to terra but a standing fleet defect already held - the same shape bit the registry alerting tier - so treat it as a known class, not as one team's trivia. Option A is accepted as specified (age of the oldest sendStatus=queued row, never unit-existence). Option B - refuse the enqueue while cron.job is empty - is REFUSED and the refusal is now on the record with its reason: the enqueue rides inside the caller's transaction, so B rolls the user's action back on a missing timer, and a missing consumer is not a reason to reject the producer. pmmaster is carrying that generalisation past this table.
-
was: emailOutbox accepts an enqueue and returns success independently of whether anything drains it, so the failure mode is silent: a row asserts it was queued while no consumer exists, and the first real enamel send fails invisibly instead of loudly. Raised by db-terra-cc 2026-09-11 off terra's first live exercise of the same stack; the argument holds for enamel whether or not the SMTP credential lands soon, which is why it is filed separately from ENAMEL-16's hold rather than inside it. MEASURED STATE ON ENAMEL, 2026-09-11. cron.job has zero rows and nothing anywhere calls /api/cron/drain-email-outbox, so the drain half does not run at all. Enamel also has NO inline-send path - the drain route is the only consumer of lib/email/transport, and a grep of app/src and db for replyToRequired or missingReplyTo returns nothing. That matters because it rules out terra's measured defect here by construction: terra's inline pre-flight hard-failed a row with failureCode missingReplyTo and logged an ERROR-level appEvents row, then its drain sent the SAME row successfully with replyTo still NULL. Two senders that disagree is only possible where there are two; enamel has one. WHAT TO BUILD: an age alarm on the outbox, not a liveness check on the drain. Alert when the oldest row with sendStatus queued exceeds a threshold, because that fires for every cause - drain not scheduled, drain crashing, credential missing, pg_net silently dropping the request - whereas a check that the timer exists passes while the timer does nothing. Pairs with ENAMEL-30 (moving the drain off Supabase pg_cron onto a venus systemd timer) but must not wait for it: the alarm is what tells us ENAMEL-30 worked. TRAP TO AVOID, from db-terra-cc's own near-miss: terra's drain is a USER systemd unit on venus, so systemctl list-timers WITHOUT --user shows nothing and reads as absent. Any check written for this row must query the user manager, or it will report a working drain as missing.
-
Backstop: per-appKey age alarm on the oldest queued emailOutbox row (ENAMEL-33 owns the invoker)
-
parent=ENAMEL-33
-
IMPLEMENTED AND PUSHED, edd9d1a, app 0.9.0. ops/enamel-outbox-age-alarm.sh plus its .service/.timer, symlinked into ~/.config/systemd/user and enabled; OnCalendar=*:2/5, offset two minutes from the drain's *:0/5 so the alarm samples after a drain pass rather than racing it. Verified running under systemd, not only by hand: systemctl --user start at 12:35:43 UTC finished clean, exit 0, and correctly sent nothing because the appKey was already in ALARM. NOT A VERCEL ROUTE, and this is the load-bearing decision. A route would be invoked by the same EVO-94 chain that does not exist for enamel (ENAMEL-33), so the alarm would share the failure mode of the thing it watches and would be silent in precisely the case it was built for. It runs host-side on its own timer against the DB directly, and it is a USER unit - systemctl list-timers WITHOUT --user reports it as missing. TWO SPEC CORRECTIONS, both mine, neither changing intent. (1) There is no sendStatus 'queued' on this table: enamel_026 constrains sendStatus to pending|claimed|sent|skipped|failed. The probe reads the undelivered backlog as pending + claimed, deletedAt is null. A 'claimed' row whose lease expired is undelivered too, and it is the shape a crashed drain leaves behind, so excluding it would blind the alarm to a mid-flight drain death. (2) Age is measured from nextAttemptAt, NOT createdAt. A row inside its retry backoff is not overdue, it is waiting on purpose; keying on createdAt would fire on a healthy 10-attempt backoff chain with no signature distinguishing it from the real fault. Threshold 30 minutes = six missed *:0/5 cycles, env-tunable. Read-only DB role enamel_outbox_alarm_ro (db-enamel-cc minted it mid-task); the probe no longer authenticates as postgres. DSN and hub bearer come from ~/.local/bin/.llmmsg-env, the one git-ignored secret source on this host - no literal value in the script, the unit, or this repo.
-
OBSERVED FIRING, all four transitions, against a real seeded row on the live table - not a unit test and not 'the diff exists'. db-enamel-cc seeded id 6b802ee4-0ea1-4668-bf96-7f4e7c056c7e (appKey enamel, pending, nextAttemptAt now()-2h, recipientsTo alarm-probe@demo.pensanta.com) and soft-deleted it at 12:35:57 UTC. 1. OK->ALARM fired: 'OUTBOX BACKLOG ALARM - appKey=enamel', 122 minutes overdue, 1 row. Tag scrp-enamel_outbox_age_alarm-venus-mtwxrpyc0aq3. 2. SUPPRESSION: the next run with the row still stale sent NOTHING. This is the half that was a precondition rather than a follow-up, and it is now measured rather than asserted - 'it fired' alone would not have shown it. 3. ALARM->OK fired after the soft-delete: 'OUTBOX BACKLOG CLEARED - appKey=enamel'. Tag mtwxwyhmbvhc. deletedAt rather than DELETE, so the same statement exercised the recovery transition AND the deletedAt-is-null filter. 4. A further run after recovery sent nothing and left state OK. Plus a pair nobody asked for and which justified its own leg: run 1 hit a real defect - bash '' collapsed my SQL string literals to bare identifiers, so psql returned 'column pending does not exist' - and the PROBE-BROKEN alert is what reported it (tag mtwxrffveyql), followed by PROBE RECOVERED (mtwxrpwry015) once fixed with dollar-quoting. Without that leg the probe would have failed silently and read as a healthy queue, which is the exact shape this row exists to make impossible. Verification runs used an overridden recipient so the test did not page Elazar; the recovery run at 12:36 used the real derived recipients (PM of aro:enamel plus the escalation target), so recipient resolution is exercised live too. COVERAGE, stated as a limit and not as a clean bill: this watches the AGE of overdue undelivered rows per appKey on emailOutbox. It does not watch rows already in sendStatus failed (the drain writes an ERROR appEvent at the moment those fail), message content, deliverability, or ITS OWN liveness - a stopped timer is silent and nothing notices. That last gap is the same class this alarm closes one level down and it is named rather than implied. NOT covered and deliberately left to ENAMEL-33: nothing drains enamel's outbox yet. This row was never the fix for that.
-
Alarm deployed on a venus user systemd timer and observed firing, suppressing, and recovering against a live seeded row. edd9d1a, app 0.9.0. Parent ENAMEL-33 stays open: this row is how we find out nothing drains, not the drain.