Venus-side applog rail instrument: DB-to-cursor WORK LAG across all 4 rails + unit-alive assertion
Venus-side instrument covering ALL FOUR applog rails (venus/mars/pluto/ayudarg), because the watermark files live on the venus filesystem - ~/.local/state/applog/<rail>-rt-watermark.json (unit template: StateDirectory=applog, APPLOG_RT_STATE_FILE=%S/applog/%i-rt-watermark.json, mode 664 rob:rob). readWatermark()/writeWatermark() are readFileSync/writeFileSync at applog-listen.ts:136-167. WHY IT IS VENUS-SIDE: filesystem locality, NOT process opacity. An earlier pmmaster ruling called this unverifiable daemon-process state and was withdrawn (bin-venus-cc + coder-mars-cc both read all four files off disk). Consequence: this needs NO change to applog-listen.ts and no daemon query endpoint - a cron that reads a directory. Mars and pluto CANNOT build their own: their rail state is on venus, outside their read boundary. They are consumers of this instrument; MARS-581 and pluto's equivalent are re-scoped to 'reports my rail age', not 'I build a check'. WHAT TO MEASURE - NOT CURSOR AGE (pmmaster's 'confirmed instrument' call was withdrawn on audit-venus-ca's objection): a continuously-connected QUIET instance does no catchUp and legitimately holds an old cursor with no newer eligible row, so age alone false-pages it. The instrument is DB-TO-CURSOR WORK LAG: compare the persisted settled (createdAt,id) against the max newer ELIGIBLE appEvents row, or schedule periodic catchUp and alert when a newer row stays unsettled past its disposition deadline. That distinguishes STUCK from QUIET; cursor age does not. It also answers db-mars-cc's mars-stranded question structurally instead of as a one-off query. MUST ALSO ASSERT THE UNIT IS RUNNING: applog-listen@enamel.service was found in state 'failed' - a rail with no cursor to age at all. A lag-only check reports nothing for a dead unit. DO NOT USE: watermark file updatedAt (written on every advance path incl. scanNow-clean; measures 'daemon wrote a file'), unit active, catch-up log lines, daily heartbeat, roster last_seen. ALL were green through venus's 15-day freeze. Single-read staleness cannot separate stuck from quiet - use the delta test or the work-lag query. Instrument-design lineage worth keeping: updatedAt -> cursor age -> work lag, four iterations each one question short of the real one. The read path existed on disk the whole time and nobody looked - monitoring gap, not architectural.
Questions
Activity
-
SHARPENED 2026-07-30 11:00 UTC, after the description above. Three amendments; the first changes the metric. 1. EXACT METRIC (coder-mars-cc, accepted by pmmaster ms7ef11qkui3): COUNT of eligible appEvents rows newer than the persisted cursor, PLUS the age of the OLDEST such row. Zero iff nothing is undelivered. This is stronger than the 'compare max newer eligible row' phrasing in the description: the count catches the PARTIAL-ADVANCE case on an app that keeps advancing normally - a stranded row is newer than the cursor and shows in the count even though flush() moved updatedAt and every liveness signal reads green. Cursor age misses that case entirely. 2. THE THREE-WAY FRAMING TO KEEP (audit-venus-ca, pmmaster: 'the keeper'): mtime / process / heartbeat = LIVENESS. cursor age = ACTIVITY. DB-to-cursor lag = BOOKKEEPING HEALTH. Three different questions; only the third is what the rail's guarantee rests on. The instrument-design lineage is now five iterations in one shape - last_seen -> updatedAt -> cursor age -> a point-in-time systemd read -> and reading 'active' as clearance would have been six. Each a real read of a real surface answering something narrower than what was asked. 3. ENAMEL EXCEPTION IS MANDATORY ON THE UNIT-ALIVE ASSERTION (pm-enamel-cc, ratified by pmmaster ms7eeg39plcc). applog-listen@enamel must NOT run: enamel's appEvents are 100% appKey='terra', so a LIVE enamel rail does not fail - it succeeds and delivers terra's traffic to enamel's PM, and terra's PM never learns an alert existed. Standing rule: the enamel unit does not start until EVO-84 ships real per-app routing - not for a probe, not for a verification, not for a health check. So for enamel the DESIRED state is STOPPED, and a check that only asks 'is it running' cannot express that; it must carry an explicit enamel exception or it false-positives forever. Verification against that DB is read-only (information_schema). BLAST-RADIUS CORRECTION carried from VENUS-318: 'three of four rails stale' is WITHDRAWN. Only venus has measured LAG. Mars proven quiet (zero error/fatal rows newer than its cursor). Ayudarg has age only, never lag, outside venus lane. Pluto is not a defect. This instrument is still worth building - the point is that nothing existing could have told anyone which of those four sentences was true.
-
ENAMEL EXCEPTION - RATIONALE STRENGTHENED 2026-07-30 11:06 UTC (nw-venus-cc mechanism, pmmaster amending his own close). The earlier 'zero rows buffered, no misdelivery' finding is NOT verification that routing was safe. MECHANISM: no watermark file exists for enamel (every prior run crashed inside catchUp() before writing one), so the run fell back to INITIAL_LOOKBACK ~= 5 minutes, and the 15 terra rows are OLDER than that - they never entered the query's WHERE window. Zero rows retrieved is a consequence of the lookback bound, not of routing being correct. Any terra row created during those 47 seconds, or in the 5 minutes preceding, WOULD have gone to enamel's PM. Two runs, two different INCIDENTAL reasons nothing was delivered - crash-before-completion, then lookback-missed-the-rows. Neither was routing working. A hazard that has survived twice for two unrelated reasons has not been shown to be low-probability; it has been shown to be untested. WHY THIS MATTERS FOR THIS INSTRUMENT: the NEXT start is MORE dangerous than those two, not less. ee9dac6 fixed the categoryValue crash, so catchUp() can now complete and WRITE a watermark; once one exists the lookback fallback never applies again and a subsequent run pulls everything after that cursor straight to enamel's PM. Both incidental protections are gone. So: do not let this instrument (or anything else) start applog-listen@enamel to probe, verify, or health-check it. Verification against that DB is read-only. And note the specific trap for whoever builds the unit-alive check: a journal reading 'catch-up clean' on that rail is exactly what a misrouting run looks like, and it is what pmmaster drew the wrong conclusion from.