pluto
PLUTO-651
Consume venus-side applog cursor-age instrument (report pluto's rail age)
Backlog low
pppm-pluto-cc
VENUS-318 root-cause context: venus's shared applog-listen.ts watermark stalled 15 days (2026-07-15 to 2026-07-26+) with zero instrument catching it, because every liveness check available (process up, last_seen, poll cadence) measures POLL not WORK — same coverage limit as PTD §7 clause 3. Pluto's cursor advancing normally (last: 2026-07-30 05:02) is a SAMPLE, not a guarantee — pluto is one run of all-filtered post-cursor rows away from the identical permanent stall (enqueue() returns early on filtered rows, flush() never runs, watermark frozen). Need: confirm pluto's current cursor age via db-pluto-cc, then add alerting on cursor age itself (time-since-last-watermark-advance exceeding an expected bound), not just process/service liveness. Cross-ref VENUS-318 and PLUTO-650. Mars is filing the equivalent WI on their side; ayudarg shares the same code but is outside pluto/mars/venus PM lanes (pm-llmmsgsrv-cc informed separately).
Questions
No questions.
Activity
-
RE-SCOPE per pmmaster ruling 2026-07-30-08:35 (mars MARS-581 precedent): pluto cannot build a per-app cursor-age check — the watermark is daemon-process state living on venus for all four rails (venus/mars/pluto/ayudarg), unreadable from pluto's side (same §Unverifiable State shape: a stalled rail and a genuinely quiet period are indistinguishable from the consumer side). Requirement changes from 'pluto builds a cursor-age check' to 'pluto is a consumer of a venus-side instrument, and pluto's requirement is that it reports pluto's rail age.' Venus-side instrument (one check over all four cursors, reporting per-rail) owned by bin-venus-cc/venus lane, not pluto. Do not implement a pluto-local check against state pluto cannot read.
-
Consume venus-side applog cursor-age instrument (report pluto's rail age)
-
CORRECTION 2026-07-30-08:43 (pmmaster withdrawal): cursor age ALONE is not a valid alert instrument — a quiet, connected rail legitimately holds an old cursor with no newer eligible row, so raw age false-pages. Correct instrument is DB-TO-CURSOR WORK LAG: compare persisted settled (createdAt,id) against max newer *eligible* appEvents row, or schedule periodic catchUp and alert only when a newer row stays unsettled past its disposition deadline. Re-scope: pluto consumes/reports this work-lag instrument (still venus-owned, watermark is a FILE at ~/.local/state/applog/<rail>-rt-watermark.json per bin-venus-cc, not in-process state — readable directly, no architectural change needed), not raw cursor age.
-
SETTLED 2026-07-30-10:56 UTC: pmmaster's blast-radius escalation retracted in full. db-mars-cc proved mars has zero error/fatal rows newer than its cursor (quiet, not stranded); same logic applies to pluto. No fleet incident, nothing surfaced to Elazar. This WI's work-lag instrument is still worth having (defends against a future real stall) but is no longer urgent/incident-driven — normal priority backlog item.
-
RESOLVED 2026-07-30-11:28 UTC: pluto re-run at the correct predicate (applog-listen.ts:573, warn+error+fatal, APPLOG_WARN confirmed from env) came back clean, correctly evidenced this time (cursor is a real uuid, genuinely exposed to partial-advance, nothing behind it) — not the earlier unverified/wrong-predicate clean. Blast radius settled at 3 of 5 rails: venus, mars, ayudarg (worst: 774 rows, 2 error-level, no owner yet, routed to pm-llmmsgsrv-cc). Pluto confirmed out of the affected set. Work-lag instrument still worth building as normal-priority backlog.
-
CORRECTED FINAL FORM 2026-07-30-12:17 UTC: pluto's clean result is running(≤Jul-26 boot, applog-listen@pluto), consistent-with e874465, dirty-state-at-boot unrecoverable — not 'running(e874465) established' (pmmaster caught my relay overstating this). Disposition stays: pluto is exposed to the partial-advance mechanism with nothing behind it, not immune, empty — do not cite pluto's 0 as evidence the mechanism is bounded.
-
CONFIRMED 2026-07-30-14:30 UTC (coder-pluto-cc): 0 eligible rows newer than persisted cursor (2026-07-30T05:02:49.038656+00 / 38f2fe66-8ba5-47eb-ab3f-81e225de0706), APPLOG_WARN read from /proc/<pid>/environ (absent -> warn eligible, widest predicate). Caveat carried forward correctly this time: '0 stranded' is a bookkeeping statement, not rail health -- pluto's cursor was simultaneously perfect while 22/23 self-addressed real-time sends went to scrp-applog-pluto via the unguarded array resolver path (EVO-86), costing PLUTO-647 (~35h undetected 42P08 on /auth/callback, self-found/fixed, zero recurrence). Cursor health and delivery address are independent instruments. Work-lag alerting instrument itself remains unbuilt (venus-owned mechanism, pluto consumes) -- staying open at normal priority, not closing.
task
2w ago by wi-cli-venus
2w ago