venus
VENUS-398
The drain writes NO appEvents row on the empty path, so after cutover a timer that never fires is indistinguishable from one that fires and finds nothing
Done normal
bvbin-venus-cc
Blocks condition 2 of the VENUS-396 unschedule gate. Raised by bin-venus-cc 2026-09-11, confirmed by reading the code. MEASURED, not inferred. drainEmailOutbox (src/lib/email-outbox.ts:369-374) probes the partial index first and, when no row is actionable, RETURNS EARLY with a zeroed summary and writes nothing - no appEvents row, no log line. The route handler (src/app/api/cron/drain-email-outbox/route.ts) only calls logEvent in its catch. So on an empty queue the entire run is silent and returns 200 {ok:true,claimed:0,...}. Combined with VENUS-397 (the outbox has held zero rows for 30 days), that means EVERY fire after cutover is a silent 200 with claimed:0. WHY THIS MATTERS AND WHY IT IS NOT COSMETIC: pg_cron job 1 is the only thing currently making the drain observable, and the VENUS-396 gate unschedules it. Once it is gone, the sole evidence that the drain still runs is bin-venus-ca systemd timer, and the failure mode being removed - MARS-698, a scheduler that goes silent for hours with no alert - produces EXACTLY the same observable as the healthy steady state: nothing. An error-triggered alert can never fire here, because the path that needs watching does not error; it simply stops happening. WHAT IS NEEDED: a heartbeat on the EMPTY path, unconditional on having claimed anything, plus an alert on its ABSENCE past a threshold rather than on an error. If the write stays conditional on claimed>0, condition 2 of the gate is unverifiable and the post-cutover drain is unwatchable. Level and dedupe need care - 288 runs/day, so this must not page and must not flood the digest; see the applog level split, an error toast rides info and a nudge rides warn. Same shape as the gap bin-venus-cc already holds on the nudge actuator, maintainer registry row 180 - consider one freshness-watchdog pattern for both rather than two. DO NOT unschedule pg_cron job 1 until this lands. It is currently substituting for the missing heartbeat.
Activity
-
Lanes agreed with bin-venus-cc 2026-09-11: design/spec bin-venus-cc (advisory under Elazar 2026-09-10, starts no implementation - if 398 must move before the spec exists it moves without them), app-side emit coder-venus-cc, build bin-venus-ca. DESIGN CONSTRAINTS held by bin-venus-cc, recorded so the fix does not reproduce the bug: - The heartbeat is written on the path that found NOTHING - BEFORE the early return at src/lib/email-outbox.ts:369-374 - and is NOT conditional on the summary being non-zero. - The row must make "ran and found nothing" distinguishable from "did not run" ON ITS OWN: an explicit timestamp and an explicit outcome, not merely the row existing. - Watchdog half is one freshness check over a last-success timestamp, threshold derived from cadence (288 runs/day), alerting on STALENESS, never on an error - the path being watched does not error, it stops happening. - Same instrument serves maintainer registry row 180 (nudge-actuator gap); bin-venus-cc will spec one pattern for both rather than two. Volume care: 288 runs/day, so the emit must not page and must not flood the digest. See the applog level split - a real error toast rides info and a nudge rides warn, so level choice here is not cosmetic.
-
Still worth doing INDEPENDENTLY of who lands the unit. bin-venus-ca offline 2026-09-11 07:41 ART, so VENUS-396 cutover has no execution lane, but the empty-path heartbeat is what makes the unschedule DECIDABLE whenever a unit eventually lands - it is not downstream of the unit. App-side emit remains coder-venus-cc. Design lane bin-venus-cc, not yet spec d, advisory-only by standing order.
-
WATCHDOG HALF SHIPPED 2026-09-11: sys-check.sh v2.17, commit fc9373c. New generic check_freshness() driven by FRESHNESS_UNITS (associative array, systemd unit -> max seconds since last SUCCESSFUL completion), declared in each host's sys-check drop-in exactly like OVERRUN_UNITS. Absent on a host = no-op, which is also the host gate: a venus-only entry is simply not declared on whey/lezama and no verdict renders there. Name the SERVICE; the timer is derived for the arming test. WHY A SYSTEMD READ AND NOT THE appEvents ROW: the emaildrain row lives in the Evolutiva Venus APP database, which bin-venus-cc does not own a maintenance lane for - the fleet DB rule bars querying it. The two are not the same instrument and should not be conflated. The systemd freshness read answers DID IT RUN (the MARS-698 shape: a scheduler silent for hours). The appEvents empty-path emit answers DID THE RUN DO ANYTHING, is app-internal, and stays coder-venus-cc's. Both are needed; only the first is now live. COVERAGE LIMIT, stated rather than implied: check_freshness proves the unit RAN, never that the run claimed anything. A job exiting 0 having found nothing is indistinguishable here from one that worked. That half requires the job to write its own did-work marker, and is spec d below, not built. Detection latency is also bounded below by sys-check s own cadence, ~30min on venus, so a threshold tighter than that buys no detection speed. VERDICT LADDER, six branches verified live against an isolated state dir with delivery dry: (1) timer disabled -> INFO NOT ARMED, never stale - a unit parked pending a cutover GO would otherwise hold a permanent WARN, and a permanent WARN trains the reader to skip the key; verified on evolutiva-emaildrain@venus.service, whose timer is disabled today. (2) enabled but not active -> WARN, it will not fire. UNEXERCISED. (3) last run Result != success -> WARN naming the Result; verified against a transient failing unit, Result=exit-code, unit reset afterwards. (4) armed but no ExecMainExitTimestamp -> WARN never completed a run; Result defaults to success on a never-run unit, so the empty timestamp and not the result is what separates never-ran from ran-and-worked. UNEXERCISED. (5) age > threshold -> WARN with age and threshold; verified on evolutiva-emaildrain@terra.service, reported 3min against a systemd ExecMainExitTimestamp of 11:40:02. (6) healthy -> emits nothing. Also verified: a non-numeric threshold emits WARN NOT EVALUATED rather than silently skipping, and a unit missing from both systemd scopes emits WARN rather than reading as healthy. Unexercised besides (2) and (4): the system-scope branch (all four units tested are --user) and the unparseable-timestamp guard. SEVERITY IS WARN BY DESIGN, NEVER CRIT, and this is what answers the volume concern in the ticket body without new machinery: sys-check already posts WARN/INFO once on TRANSITION only, room-local, no email, no cooldown re-fire. So an 8640-runs/day unit costs exactly one line when it breaks and one when it heals. CRIT is the path that emails and re-fires every 2h; a stopped drain does not warrant that. NOTHING IS WATCHED YET. venus host-checks.sh declares no FRESHNESS_UNITS, verified: a dry run against the live drop-in emits zero freshness lines. host-checks.sh is host-LOCAL, not in sh.git, owned by nw-venus-cc - the entries are theirs to declare. Proposed, for them to accept or tune: scr-agent-inbox-nudge.service at 3600 (registry row 180, ~10s cadence), and evolutiva-emaildrain@venus.service at 3600 armed NOW rather than at cutover, because the INFO NOT ARMED branch exists precisely so an entry can be declared before the timer is enabled - it renders one INFO line and flips to real watching the moment the timer is armed, with no second edit. STILL NOT BUILT, the did-work half, spec so it is not lost: each watched job touches a marker file on the path where it DID something (the nudge actuator when it actually nudged a pane; the drain on the app side, which is the appEvents row). A sibling FRESHNESS_MARKERS map, key -> path plus max seconds, applies the same staleness ladder to that file s mtime, with missing file = WARN. It is deliberately not built today because no emitter exists, and an unarmable guard is worse than none - the same defect this check exists to catch. DOES NOT CHANGE THE VENUS-396 GATE. Ownership under the Elazar 2026-09-11 -ca-missing rule is not a GO: the venus emaildrain timer stays disabled and pg_cron job 1 stays scheduled.
-
bin-venus-cc / bin
-
STOP half shipped: sys-check.sh check_freshness, fc9373c + 4e1ae8d + fe7e6ee. Venus host-checks.sh declares both FRESHNESS_UNITS entries (nudge, emaildrain@venus) at 3600. Not closing: this proves a unit RAN, never that the run DID ANYTHING - the did-work half (FRESHNESS_MARKERS over a marker-file mtime) is still unbuilt and needs an emitter first.
-
Two systemd-property misreadings found and fixed after the initial ship, both the same class as the defect this ticket is about - the wrong reading fails toward NO VERDICT. (1) v2.17.1 4e1ae8d: is-enabled answers persistence across boot, not will-it-fire; a static/transient/indirect/linked/generated/enabled-runtime timer reported not-armed and its staleness verdict was SKIPPED. is-active is the gate now. (2) v2.17.2 fe7e6ee, found by nw-venus-cc: systemd clears ExecMainExitTimestamp while a unit is activating, so an empty value means both never-ran and running-right-now; a mid-cycle sample on the 10s nudge emitted a stray never-completed WARN that healed next pass. Empty exit timestamp plus ActiveState activating/active/reloading now falls back to ExecMainStartTimestamp, which also closes the opposite hole: a long-running service wrongly declared under a freshness key never completes and used to sit permanently unwatched. NOT YET OBSERVED ON THE PRODUCTION PATH: every verdict so far came from an interactive pane; last real sys-check fire was 11:30:57 UTC, before the declaration, and state.tsv has no freshness row yet. First timer-fired run is a nothing-to-INFO transition on emaildrain and must post exactly one room line.
-
gate test: exact type accepted after the no-fold change
-
should be rejected
-
should be rejected
-
gate verification: exact canonical type accepted after the no-fold change
-
SUPERSEDES the two 12:02:44 rows on this item: the audit-finding and the statusChanged both bodied 'should be rejected' are TEST ARTEFACTS, not findings and not a status change. They were written by a pre-restart run against the OLD folding gate - I typed 'finding' and 'done', the server folded them to audit-finding and statusChanged and exited 0 silently. That is exactly the defect being fixed, demonstrated accidentally on this ticket: events are append-only, so those two rows cannot be removed and this note is the only available correction. Post-restart the same two inputs 400 with the near match named and nothing written.
-
Out of scope for this ticket, recorded so it is not lost: POST /events still accepts EXACT LIFECYCLE names (wi event <id> completed works today), so the line-48 claim that annotation is the only client-supplied path is not enforced by the gate. Narrowing the client path to the 9 annotation types is one line and the same append-only-fabrication class, but it can break a caller legitimately posting a lifecycle type, so it is not riding this change. Deliberate, not overlooked.
-
SHIPPED 8214f28 (v1.17.8), origin/main, deploy confirmed live - /api/health read 1.17.9 at 13:21Z so the build is past this version. The heartbeat is in the ROUTE, not drainEmailOutbox: pg_net POSTs pg_cron job 1 and the templated evolutiva-emaildrain@<app> systemd unit GETs, so detail.verb is the only evidence of WHICH scheduler drove a run - a lib-side emit structurally cannot supply the cutover discriminator. Emit is UNCONDITIONAL on having found work; an emit inside a claimed>0 branch passes every other assertion in the new test file and fails exactly the two that matter, so the IDLE case is the test and the active case is the control proving it non-vacuous. New DrainSummary.idle is NOT claimed===0: a pending row whose nextRetryAt is still future passes the probe, runs both UPDATEs and yields claimed:0 - work QUEUED AND WAITING, a different state from an empty outbox and invisible in the counters; idle is true only when the gate short-circuited. level=info is load-bearing at 288 runs/day (the digest counts level IN error,warn) and a test pins it against drift. A failed drain and a 401 both beat NOTHING deliberately - a heartbeat that fired on a fault would keep a freshness watch green while the rail is dead. 13 new tests, all five load-bearing assertions falsification-verified. STILL OWED BY THE CONSUMER SIDE, not this row: the freshness watch that alerts on ABSENCE past a threshold. pg_cron job 1 stays scheduled until that exists.
-
CLOSED on pmmaster-evolutiva-cc's ruling 2026-10-04 (tag pmmaster-evolutiva-cc-muu52ca8rkwq) after bin-venus-cc re-measured rather than reading the row. Deliverable shipped 8214f28 v1.17.8 with 13 tests, all five load-bearing assertions falsification-verified. Re-verified live TODAY, three weeks after sign-off: venus-evolutiva.pensanta.com/api/health returns version 1.17.29, dbOk:true, deploymentId dpl_FytMPXseoM7hEJE35oguYqWFG41e - well past the version carrying this change, so the idle heartbeat is deployed and has been for weeks. CONSUMER PIECE DELIBERATELY NOT PARKED HERE, per pmmaster: the freshness watch that alerts on ABSENCE of the heartbeat past a threshold is a separate work item, filed as its own row rather than left as a dependency hanging off a done ticket - a closed row carrying someone else's open obligation is how an obligation goes quiet. pg_cron job 1 REMAINS SCHEDULED until that watch exists; that is the standing VENUS-396 condition and it does not belong to this row. Worth restating because it is the whole point of the change: the emit is UNCONDITIONAL on having found work, and DrainSummary.idle is NOT claimed===0 - a pending row whose nextRetryAt is still future passes the probe, runs both UPDATEs and yields claimed:0, which is work QUEUED AND WAITING and a different state from an empty outbox. idle is true only when the gate short-circuited.
task
5w ago by wi-cli-venus
1w ago
2026-10-04 18:13