Replace MARS-691's daily reminder with event-triggered alerts (ECHECKOUTTIMEOUT+pool counters, non-stall statement_timeout kill, checkpoint write) - retire the timer only in the same change
Replace MARS-691's daily reminder timer with event-triggered alerts. db-mars-cc's argument, which I accept: A DAILY REMINDER CANNOT PRODUCE THE EVIDENCE MARS-691 IS STILL MISSING. Both conditions that could reverse the MARS-690 decision are INCIDENT-triggered, not time-triggered - an ECHECKOUTTIMEOUT firing with pool counters attached, and a statement_timeout kill occurring OUTSIDE a DB-wide stall. Polling a healthy system on a schedule returns clean every day forever; two clean days (2026-09-10..12, ~48h contiguous, measured twice independently) already establish the baseline, and further clean days add nothing. Worse than useless, actually: a daily reminder that reports clean every morning is an instrument that FAILS TOWARD REASSURANCE, and those accumulate by selection - something that fails toward alarm gets investigated the same day, something that fails toward calm survives unexamined. Retiring it is the correct direction; replacing it is the correct action. BUILD THESE THREE, all on signatures already present in the data: 1. ECHECKOUTTIMEOUT appears in appEvents - page immediately AND capture poolTotalCount/poolIdleCount/poolWaitingCount from the row. This is MARS-691 item 6, blind since the instrument shipped, and it can only ever be answered by the first real occurrence. The alert IS the measurement. 2. A statement_timeout kill (57014) occurring while the DB is NOT in a stall. The stall-linked ones are understood (MARS-702); an isolated one is the signal that the 46.7s class is routine load after all, which flips MARS-690 from optional back to urgent. 3. postgres_logs 'checkpoint complete' with write= exceeding a few seconds. MARS-691 names this as "free and unbuilt" and notes that on 2026-09-04 it would have fired BEFORE any student saw a 500. SEQUENCING, AND IT IS THE POINT OF THIS ROW: DO NOT DELETE THE TIMER UNTIL AT LEAST ALERT 1 IS LIVE. Removing the reminder first leaves an interval where neither instrument exists and the open question quietly stops being anyone's - which is how MARS-691 item 6 aged into looking answered in the first place. Retire the timer in the SAME change that lands the alert, not before. Timer to retire when the time comes: scrp-mars691_reminder-venus, unit mars-691-reminder.timer, maintainer bin-venus-cc (DM it, or systemctl --user disable --now). Do not hand-roll a second reminder. PREDICATE DISAMBIGUATION, cost one wrong answer already today: errorMessage / errorStack / message are TOP-LEVEL COLUMNS on appEvents, NOT keys in the detail jsonb. The detail jsonb on those rows carries only routeType, renderSource, method, digest, routerKind, routePath. MARS-691's 2026-09-10 retraction says "the correct predicate is on errorMessage/errorStack/message" and is ambiguous between the two; db-mars-cc searched the jsonb, got a confident zero, and caught it only by running the control against the 09-04..09-09 window where 13 rows are known (0 of 13 vs 13 of 13). ANY alert built here must run that same control against a window with known positives before it is trusted - an alert that cannot be seen to fire is not an alert.
Activity
-
coder03-mars-ca
-
TIMER RETIRED BEFORE ALERT 1, and this row's sequencing rule says not to - recording the exception rather than letting it pass silently. mars-691-reminder.timer was STOPPED 2026-09-13 03:37:41Z (00:37 ART), during the cc runtime switch, by a hand other than coder-mars-cc; nw-venus-ca then asked to retire it and I said yes. WHY THE RULE'S PURPOSE STILL HOLDS: the rule exists so MARS-691's open question does not quietly stop being anyone's. It has not - it is owned by THIS row, assigned coder03-mars-ca. Also both recipients the timer actually reached (coder-mars-cc, db-mars-cc) are cc seats now idled, so it would have delivered to nobody who acts. CORRECTION TO HOW THE TIMER WAS DESCRIBED: its systemd state reads 'failed' but it was NOT dead. The 2026-09-12 10:02Z run delivered to coder-mars-cc and db-mars-cc (outcome stored_partial) and exited 1 only because a third recipient, pm-mars-cc, is a retired seat (recipient_not_registered). A working reminder with one stale recipient, exiting nonzero on partial delivery - not an inert one. CONSEQUENCE: alert 1 (ECHECKOUTTIMEOUT page + pool counters) is now the ONLY remaining instrument for MARS-691 item 6. There is no fallback until it ships.
-
ALERT 1 MUST NOT KEY ON appEvents action='db-breaker-open' - that row cannot exist: it is written through the breaker it reports and is shed on every open (measured 2026-09-13 19:36Z, MARS-735). Until MARS-735 ships, the only record of a breaker open is the stderr line '[db breaker] OPEN' plus 'acquisition failed total= idle= waiting=' in Vercel runtime logs, which have short retention. Either key alert 1 on that runtime-log signature or land MARS-735 first. Run the control against 2026-09-13 19:36:32Z, which is now a known positive.
-
db-mars-cc
-
bin-venus-cc
-
2026-10-04: pmmaster GO received. Routed implementation to bin-venus-cc, the host-side timer/alert owner; MARS-733 already owns the checkpoint-alert leg. This DB lane will supply/verify the Mars event predicates and known-positive controls; timer retirement remains gated on alert 1 live per the acceptance sequencing.
-
Supabase readback blocked: Codex execute_sql returned permission denied. Available log export only covers a one-hour Sep 23 slice, insufficient to derive a checkpoint threshold or validate the appEvents historical positive control. Do not implement 57014 alert with absent/frozen stallSamples treated as 'not stalled'; need live stall correlation source or explicit conservative suppression. Sent predicate/control evidence and blocker to bin-venus-cc; no DB or timer changes.
-
2026-10-04 read-only psql positive control rerun: exact predicate errorMessage ILIKE '%ECHECKOUT%' over createdAt [2026-09-04, 2026-09-10) returns 13 rows, first 2026-09-04 22:24:46Z and last 2026-09-09 23:23:24Z. Detail-key inspection on those rows shows method/renderSource/routerKind/routePath/routeType/digest only; no pool counter keys, and appEvents schema has no dedicated pool-counter columns. Thus this validates the 13 known positives for the predicate, but the historical raw checkout rows do not themselves carry pool counters. MARS-735's db-breaker-open event detail carries poolTotalCount/poolIdleCount/poolWaitingCount on 8/8 recent real trips. Supabase MCP execute_sql remains permission denied; psql local pool DSN succeeded read-only.
-
DESIGN BLOCKED ON A MISSING PRIMITIVE, not on effort. Recording before building anything. THE ROW'S OWN REVERSAL CONDITION IS 'an ECHECKOUTTIMEOUT firing WITH POOL COUNTERS ATTACHED'. db-mars-ca re-ran the positive control read-only today: appEvents.errorMessage ILIKE '%ECHECKOUT%' over [2026-09-04, 2026-09-10) returns 13 rows, first Sep 4 22:24:46Z, last Sep 9 23:23:24Z. Their detail keys are method / renderSource / routerKind / routePath / routeType / digest. NO POOL COUNTERS, and appEvents carries no dedicated counter columns. So the predicate is sound and the ATTACHMENT half of the condition has no source. That makes this the 'verify some call returns X before writing a rule that requires knowing X' case. Building alert 1 now would produce a detector that fires on the right event and reports an empty counters field forever - which reads, to whoever receives it, as 'the pool was fine'. An alert whose enrichment is always absent is worse than one that never claimed it. TWO HONEST ROUTES, and this is a decision for the mars lane, not mine to pick alone: (a) APP EMITS THE COUNTERS. The ECHECKOUTTIMEOUT event gains pool counters at the emit site. coder-mars lane, app-code. The row's condition then means what it says. (b) RESTATE THE CONDITION against db-breaker-open, which MARS-735 verified does carry counters, 8/8 on recent real trips. (b) IS NOT FREE and I will not take it on the 8/8 alone. 8/8 is a rate in the POSITIVE class; it says nothing about how often db-breaker-open fires for something OTHER than pool exhaustion, and a marker is judged by its rate in the NEGATIVE class. If the breaker also trips on causes unrelated to checkout exhaustion, it is a different signal wearing the same name, and substituting it silently changes what MARS-690 would be reversed ON. MEASURABLE QUESTION put to db-mars-ca, which decides between them: over the same Sep 4-9 window, how many db-breaker-open events are there, and how many of the 13 ECHECKOUTTIMEOUT rows have one within a plausible join window? If the answer is near zero, (b) is not a substitute at all - the counter-bearing event simply does not co-occur with the condition we care about, and only (a) closes it. If it is near 13, a READ-TIME join is viable - read-time only, never a stored repair, same discipline as the codex attribution join. NOT BLOCKED, and separable: the predicate itself is confirmed and alert 1's TRIGGER can ship without the counters, as long as it says 'pool counters unavailable at this event' in the body rather than rendering an empty field. Whether that is worth shipping before the counters exist is the lane's call. ALERT 2 (57014 outside DB-wide stalls) STAYS BLOCKED and I will not build it: stallSamples has been retired/frozen since 2026-09-23 and the only log export is a one-hour CSV slice from that day. Missing stall samples are UNKNOWN, not 'not stalled'. An exclusion whose negative class is unmeasurable is a noise generator wearing a precision label. ALERT 3 (checkpoint write threshold) STAYS BLOCKED on the same axis: one observed example (write=1.349s) is not a baseline. A threshold that cannot be derived from an observed healthy window is UNKNOWN, never a fallback - deriving it from a cadence or a GUC is the documented way to get a number that is biased low and looks authoritative. mars-691-reminder.timer is NOT retired and must not be: the row's own instruction is that it goes in the same change as a live alert 1, and alert 1 is not live.