venus
VENUS-378
applog log plane has no external liveness prober - a clean Stopped is invisible to OnFailure and hid a 16.8-day relay outage
Backlog normal
unassigned
MEASURED 2026-08-18 from the venus user journal (persistent to 2026-05-05, dense, no retention gap). THE OUTAGE: applog-listen@enamel Stopped 2026-07-30 10:55:44, next Started 2026-08-16 06:05:22. ~16.8 days down. No alert fired and nobody noticed. ROOT CAUSE OF THE BLINDNESS (not of the stop): OnFailure= fires only on a FAILED result. A clean `Stopped` fires nothing - whether deliberate or the collateral of a restart sequence that never came back. So applog-failnotify-* is structurally incapable of seeing this outage mode. Every existing check is a UNIT-STATE check made BY the same systemd instance that would have to be working for the unit to be up. See the standing note that OnFailure fires per RESTART, not on terminal failure - the is-failed gate silences the rail 100%. WHY A UNIT-STATE CHECK CANNOT CLOSE THIS: the question is "did the relay keep relaying", and unit state answers "does systemd believe it is loaded". Those diverge exactly when it matters. Needs an EXTERNAL prober - something off the venus user manager that asserts freshness of the plane (e.g. last-relayed-row age per app, or a heartbeat row) and alerts on staleness, on the model of the VENUS-367 telemetry canary: answer 503 on a real broken write rather than inferring health from a process being alive. SCOPE NOTE: this is about the LOG PLANE (applog-listen@ x6 + applog-pull@ timers), which currently carries error+fatal -> PM DM for ayudarg/enamel/mars/pluto/terra/venus. It is the reason I told aro:coders NOT to make a venus user-scope timer a delivery dependency. SECONDARY, bounded, NOT the same defect: 19 OOM kills across applog units all on 2026-08-09 (67 OOM lines host-wide that morning) - a one-day host memory incident; Restart=always recovered them. Not chronic, do not conflate. CURRENT STATE (so nobody reads this as an open outage): linger=yes, all 6 enabled, active/running since 2026-08-17 15:27:39, NRestarts=0. DELIBERATELY NOT CLAIMED: per-unit availability percentages. My Stopped->Started pairing misaligns on restart sequences that log consecutive Starteds and manufactured a phantom 29h ongoing gap for units demonstrably running. The enamel gap above is from a direct read of consecutive journal lines, not from that script. Any future measurement here should pair on explicit ActiveEnterTimestamp/InactiveEnterTimestamp or an external freshness signal, not on log-line pairing.
Activity
-
CONSTRAINT ON ANY FIX - the detection/delivery circularity, from coder-mars-cc 2026-08-18. Recording it here because it is the thing most likely to be discovered AFTER the prober is built. Mars already has an external prober on the right side of the dependency: cron_watchdog_check() runs in Mars own Supabase, reads appEvents, and pages on the ABSENCE of a recent row for a named action - positive evidence that work happened, not liveness of a runner. Being in a DB on a different host is what lets it observe a venus scheduler silence, which no venus-side check can do about itself. THE CIRCULARITY: that watchdog does not page directly. It WRITES A WARN appEvents ROW, and the thing that turns that row into a DM is the applog rail - i.e. exactly the units this WI is about. So if the venus rail is down, Mars detects the outage and cannot report it. Their honest phrasing: "detected locally, delivered over the thing that failed". CONSEQUENCE FOR THIS WI, stated as a requirement: an external prober whose ALERT PATH runs through the monitored component is not external. Whatever gets built here must have a delivery path that does not depend on applog-listen being up - otherwise it reproduces the enamel failure one layer up and looks like coverage. Neither Mars watchdog nor a venus-side prober is sufficient alone, and there is no third path today. ALSO relevant to scope, from mars: cron_watchdog_check currently covers 2 actions (ux-pain-digest, error-digest-ran) of 5 HTTP-firing crons, and nothing on the email outbox. Mars sequencing is extend-watchdog -> prove it pages on a deliberately skipped run -> THEN move the schedule. Same order applies here: prove the prober fires on an induced outage before anything depends on it. A prober that has only ever stayed quiet is a warning that has only ever not fired.
-
TIMESTAMP CORRECTION + a reported second defect RULED OUT. Both from coder-enamel-cc re-deriving the gap independently rather than taking my number. CORRECTED: enamel resume was 2026-08-16T05:53:28Z, NOT the 06:05:22Z I first quoted. I had picked the Started after a SECOND stop/start pair at 06:05:10/06:05:22, so the relay was actually back 12 minutes earlier. Authoritative gap: Stopped 2026-07-30T10:55:44Z -> Started 2026-08-16T05:53:28Z = 16.79 days. Gap length and every conclusion unaffected. REPORTED AND RULED OUT: 48 min before the clean Stop, applog-listen@enamel produced four `Failed with result exit-code` (10:07:21/23, :28/:30, :35/:37, :42/:42) - the start-limit path, which DOES enter failed state - and journalctl for applog-failnotify-listen@enamel over all of 2026-07-30 returns "-- No entries --". That reads as the failnotify rail not firing on the one occasion it had a firable event. IT IS NOT. The rail did not exist yet: - earliest journal entry for ANY applog-failnotify-* unit, all time: 2026-08-16T09:44:49Z - applog-failnotify-listen@enamel.service: ZERO entries ever - unit files: applog-failnotify-listen@.service mtime 2026-08-16 09:52, applog-listen@.service 2026-08-16 09:42 The OnFailure rail was built 17 days AFTER the 07-30 failures. "No entries" is the expected reading, not a silent failure to emit. Corroborated by the text itself: the earliest failnotify description (08-16 09:44) read "fires only via its OnFailure= (terminal Failed state after StartLimitBurst exhausts; transient crashes do NOT reach here, by design)" - the OPPOSITE of the current text, which I corrected later the same day after measuring 14 firings for restart counter 14. Both the rail and its documented semantics postdate the incident. SO: ONE hole, not two. The clean-Stopped blindness stands and is what this WI exists to close. The FAILED path is not additionally broken - it was merely absent then and is present now. Do not scope work against the second defect. NOTE ON METHOD: absence of a log line was consistent with two very different stories - "the alarm failed" and "the alarm was not built". Only the unit-file mtimes separated them, and those are visible only from the venus lane. An empty journal is not evidence until you have established the writer existed.
-
VENUS LEG BUILT AND DEPLOYED — 6a80323, v1.14.0 live (dpl_3GYCU3t5TuUCaft12vGrzFgPJiA2). NOT yet coverage; the induced-outage gate is still open, see the bottom of this note. SHAPE. The signal is FRESHNESS, not liveness, and it already existed: applog-listen writes a periodic applogListenerHeartbeat row into the app own appEvents (scripts/applog-listen.ts section C). It was simply never read by anything. So the build was a reader, not a new signal. - POST /api/cron/applog-watchdog (bearer CRON_SECRET). Re-measures staleness itself and owns the alert + suppression decision. The pg_cron function is ONLY a trigger and posts an empty body — nothing from the caller is trusted. Same division as fn_drainEmailOutboxGuard -> /api/cron/drain-email-outbox. - src/lib/applog-watchdog.ts — classifyHeartbeat() extracted so the decision is testable without server-only, and so the route cannot keep a second copy of the comparison. A test asserts the route delegates to it. - db/migrations/pending/venus73-applog-heartbeat-watchdog.sql — PENDING, db-venus-cc applies. Hourly pg_cron, Vault bearer, pg_net POST. I hold no DDL authority. THE CIRCULARITY CONSTRAINT IS SATISFIED BY CONSTRUCTION. Alert path is Supabase pg_cron -> pg_net -> Vercel -> SMTP. No leg touches the venus user manager or applog-listen. A test strips comments and asserts the route code contains no llmmsg/9703 reference, so a later well-meaning "also DM the PM" cannot silently re-introduce the mars failure mode. BEAT PERIOD CHANGED 24h -> 6h. APPLOG_HEARTBEAT_MS=21600000 in ~/.config/applog/venus.env, relay restarted 00:51:29Z, confirmed landed by reading the row back: detail.intervalMs 21600000. This was required, not cosmetic — with a 24h beat any workable threshold gives 26-48h detection latency. At 6h the 13h threshold is ~2.2 beats: one missed beat (restart, reconnect, Supabase blip) does not page, two consecutive do. Cost is 4 info rows/day against a 90d prune, ~360 rows steady-state on an 8479-row table. IF THE BEAT PERIOD CHANGES AGAIN, APPLOG_WATCHDOG_STALE_MS MUST CHANGE WITH IT. ABSENT IS A THIRD STATE. total=0 means the writer was never provisioned, not that nothing is wrong. Folding it into fresh would report an all-clear from a population that structurally cannot contain a stale beat. It alerts with its own wording. MEASURED IN PROD JUST NOW, v1.14.0: - no bearer -> 401; wrong bearer -> 401 - correct Vault bearer -> {"ok":true,"coverage":"fresh","ageMs":296267,"beatMs":21600000,"staleMs":46800000} — a real ageMs matching the 00:51:34Z beat, so the query leg is exercised against real rows, not a constant. - 9 tests, falsification-verified: flipping > to >= fails 1, making absent return fresh fails 2. Full suite 705 pass, lint 0 errors. WHAT IS **NOT** PROVEN, stated plainly so nobody reads this as closed: 1. THE STALE BRANCH HAS NEVER EXECUTED IN PROD. Only the fresh path has. The stale path is unit-tested at the real 16.79-day magnitude and its transport (sendOutboundEmail) is the same one error-digest uses in production daily, but THIS route has not sent a mail. A prober that has only ever stayed quiet is a warning that has only ever not fired. Inducing it needs either a 13h relay stop or a temporary threshold change, both of which mail Elazar a deliberately false alert — worth doing, worth announcing first, NOT worth doing silently. 2. venus73 is unapplied, so nothing calls the route on a schedule yet. Until db-venus-cc applies it, this is a route nobody invokes. COVERAGE GAP FOUND WHILE BUILDING, and it is the worst-placed one possible: APPLOG_HEARTBEAT_DSN is set for venus/mars/pluto/ayudarg and UNSET for enamel and terra, so the heartbeat writer is dormant on both. enamel is the app that was down 16.79 days and it STILL has no freshness signal today. Those two share the co-tenanted DB and need an INSERT-only applog_writer role provisioned there — db-enamel-cc / db-terra-cc work, not mine. Routing via pm-venus-cc. Until that lands, this WI covers venus only; the pattern is portable per-app with no code change.
-
venus leg shipped 6a80323 v1.14.0; open: venus73 apply (db-venus-cc), induced-outage proof, enamel/terra heartbeat DSN
-
venus73 AMENDED per db-venus-cc review, pushed 031bb71 (v1.14.1), awaiting their apply. They reviewed the draft and ruled the local 13h short-circuit out. It is the right call and the reasoning is the transferable part: a constant duplicated across the SQL and the route with no test tying them together does not drift symmetrically. Too LOW costs a wasted empty-body POST. Too HIGH suppresses the exact call the migration exists to make — silently, with every surface still green. An hourly empty-body POST is far too cheap to buy that asymmetry with. fn_applogHeartbeatWatchdogGuard() is now an unconditional hourly poke: read the Vault bearer, WARNING+RETURN if absent, net.http_post an empty body. Zero occurrences of "13 hours" in the SQL. APPLOG_WATCHDOG_STALE_MS in the route is the single home for the threshold; the route already reads the heartbeat rows itself, so it lost nothing. The absent-vs-stale distinction moved to a comment at the POST, documenting route behaviour instead of gating anything in SQL. Their structural review passed everything else as drafted: SECDEF with pg_temp pinned last, Vault-missing fail-safe, and NULL-not-an-early-return. Unchanged from the previous note: the stale branch has still never executed in prod, and nothing calls the route on a schedule until venus73 is applied.
-
STATE AS OF 2026-08-22: trigger leg LIVE, coverage leg NOT PROVEN. Shipped 6a80323 (external liveness prober), 031bb71, 6507655 (venus73 applied by db-venus-cc, verified against the live catalog: cron.job jobid 15 venus-applog-heartbeat-watchdog, 15 * * * *, active true, prosecdef true, search_path pins pg_temp LAST, and prosrc LIKE %13 hours% is FALSE confirming the amended body not the first draft), 0c18290 (the heartbeat INSERT omitted appKey, which would have 23502d on every beat on a co-tenanted DB while writeHeartbeats swallowed it to a console.error - caught by pm-enamel-cc BEFORE enamel DSN activation; statement is now BUILT not branched, extracted to scripts/lib/heartbeat-insert.ts, 8 tests). WHAT IS PROVEN: the question gets ASKED hourly at :15. WHAT IS NOT: the routes-stale branch has NEVER executed in prod - only the fresh path has - so no real outage has ever produced mail. Proving it needs an INDUCED outage, which mails Elazar a deliberately false alert; that is an external/irreversible action, filed with Elazar as eq#47 by pm-venus-cc, GO still outstanding. This WI does not close on the cron row existing. RELATED, DELIBERATELY SEQUENCED BEHIND THIS: VENUS-380 - /api/cron/applog-watchdog queries venus own pool with FROM appEvents WHERE action = $1 and NO appKey predicate, so it answers only is-venus-beat-fresh; nothing enumerates which apps SHOULD have a rail. Writers now exist on all six apps (enamel 2026-08-18, terra TERRA-81 2026-08-19, both row-verified) so enamel/terra-DSN-unprovisioned is a dead premise. Residual scope there is an expected-apps registry plus a reader that iterates it, NOT a new detector.
-
Parked per pmmaster serialize ruling 2026-09-11 11:02 ART (delegated Elazar GO): four rows in inProgress is four things almost done and nothing shipped. VENUS-372 is the single lane because it is the only one of the four that changes what a USER can do; 378 is telemetry-observability and does not fix a broken user path. No work lost - state is on the row.
-
unassigned
-
Unassigned and moved to backlog by pmmaster-evolutiva-cc 2026-09-12. The assigned agent did not exist: coder-venus-cc, db-venus-cc, coder-pluto-cc and db-pluto-cc are all absent from the hub roster and have no tmux pane, and venus and pluto currently have no staffed coder or db lane at all. A todo or inProgress row against an absent assignee reads as staffed-and-underway and is neither, so the status was a lie about this row rather than a claim about the work. The work itself is untouched and the priority is unchanged. Cause of the cluster, and it is a fleet-wide pattern rather than a bookkeeping slip: five auto-succession call sites plus resolvePm can leave a row pointing at an agent that has been deregistered, and nothing reconciles an assignee against the roster afterwards. Filed context: 178 rows across these four names, of which 23 carried a non-backlog status. Restaffing venus and pluto is Elazars decision, not a reassignment I can make.
task
2026-08-18 by wi-cli-venus
4w ago