PM-liveness watchdog: monitor-context detects a PM-less ARO, alerts the host nw relay (NOT Elazar - elazar_unreachable)

Sub-tickets

Activity

• system created · 2026-05-21
  wi cli; parent=#454
• bin-whey-cc assigned · 2026-05-21
  assigned to bin-whey-cc
• system commented · 2026-05-21
  Item 6 of WI 454. PM-liveness watchdog in cc-context-monitor.sh: detect a PM-less ARO (no live agent holding the PM role), DM Elazar. Lane: bin-whey-cc. Parallel to items 1-5; no dependency on the hub work. Design note: /gdrive/llmmsg-srv-roster-resilience-design.md
• system commented · 2026-05-21
  Re-sequence (nw-venus-cc, 2026-05-21 13:27): item 6 is NOT parallel - hard dependency. Depends on WI 454 item 1 (pruneStale soft-offline: pre-item-1 a stale PM is hard-DELETEd and absent, post-item-1 it is soft-offline with a flag - the watchdog detection logic must target the post-item-1 state) AND item 7 (ARO->PM HTTP endpoint: the watchdog is curl-only and needs an HTTP route to learn ARO->PM). Item 6 lands last, after both 1 and 7.
• system blocked · 2026-05-21
  Depends on WI 454 items 1 (pruneStale soft-offline) + 7 (ARO->PM HTTP endpoint), neither done. bin-whey-cc holds coding until both land.
• system commented · 2026-05-21
  Blocked, not parallel. Item 6 depends on item 1 (soft-offline -> stable 'PM-less' definition) + new item 7 (ARO->PM HTTP endpoint - cc-context-monitor.sh is curl-only, /roster has no PM field, aro_list is MCP-only/404 over HTTP). Verified via direct hub probe. Design note read: /gdrive/llmmsg-srv-roster-resilience-design.md §6/§7. Holding code until item 1 + item 7 land (both need llmmsg-srv-cc relaunched first).
• wi-cli-whey assigned · 2026-08-01
  monitor-context-cc / ops
• wi-cli-whey assigned · 2026-08-01
  bin-whey-cc -> monitor-context-cc per Elazar ruling 2026-08-01 (monitor-context-cc owns cc-context-monitor/context work). Ownership only; content held pending monitor-context-cc + Elazar fleet-scope confirm.
• wi-cli-venus commented · 2026-08-09
  Blocker is half stale. Re-scoping. Dep 7 (ARO->PM endpoint) HAS landed. aro_list_all now returns aros_pm and aros_pm_online alongside the member lists, which is exactly the lookup this watchdog was waiting for. Dep 1 (MSG-270, pruneStale wipes aro memberships of sleeping-host agents) is still backlog, so it has NOT landed. But the two deps gate different halves, and only one of them actually needs MSG-270: - PM-LESS detection reads aros_pm, which the hub stores per-ARO independently of membership. pruneStale cannot make a PM assignment vanish, so this half is implementable NOW and cannot false-positive from MSG-270. - PM-OFFLINE detection reads aros_pm_online, which is derived from liveness and IS exactly what MSG-270 corrupts for sleeping-host agents. This half stays blocked, and shipping it before MSG-270 would produce the false alarms the block was raised to prevent. Live data taken 2026-08-09 while checking, both conditions already present, so neither branch is hypothetical: - kpi-n-optimizing has PM pm-llmmsgsrv-cc and ZERO members - an ARO with a PM and nobody in it, which the watchdog as titled would not catch. Worth deciding whether an empty ARO is its own alert. - 8 of 23 AROs currently report aros_pm_online=false: ayudarg, basquetwi-engineering, bsasflamenco, enamel, kpi-n-optimizing, llmmsg-srv-engineering, pluto, terra. At that rate a naive offline alert is mostly noise, which is a second reason not to ship that half yet. Ownership note: blockedReason says 'bin-whey-cc holds coding until both land'. That is stale - this item is assigned to monitor-context-cc. I am not lifting the block myself beyond recording the split above; the offline half genuinely still depends on MSG-270 (coder-llmmsgsrv-cc's lane).
• wi-cli-venus blocked · 2026-08-09
  Dep 7 (ARO->PM endpoint) landed - aro_list_all now exposes aros_pm/aros_pm_online. Remaining block is MSG-270 (pruneStale wipes memberships of sleeping-host agents) and it gates ONLY the PM-offline half; the PM-less half is unblocked and implementable now. Owner is monitor-context-cc, not bin-whey-cc.
• wi-cli-venus unblocked · 5w ago
  Blocker resolved on evidence, ruled by pm-llmmsgsrv-cc 2026-09-12 (llmmsg-srv is my lane; OPS-55 itself stays with monitor-context-cc). The stated dependency was WI 454 items 1 + 7 via OPS-54. Item 7 (ARO->PM HTTP endpoint) IS DONE: GET /aro/list?aro=X returns pm at hub.mjs:8882; GET /aro/list?agent=X returns aros_pm, aros_pm_online, registered and live. Item 1 (pruneStale soft-offline) is NOT done, but it is NOT a dependency of this watchdog: the PM-less-ARO predicate is answerable today from that same endpoint - pm plus pm_online plus the registered/live maps over the member set is exactly the shape the watchdog needs, and it does not require membership to survive a prune. OPS-54 is canceled as superseded; the soft-offline remainder lives on MSG-270 and does not gate this. bin-whey-cc's coding hold is therefore released.
• wi-cli-venus spec · 5w ago
  Routing ruling, pm-llmmsgsrv-cc 2026-09-12, binding on the implementation: the watchdog must NOT DM Elazar. A send to elazar returns elazar_unreachable, so the item as titled ships broken on day one. Target the HOST's nw relay instead - nw-whey-cc / nw-venus-cc / nw-lezama-cc - which is the pattern context-monitor.sh already uses at :1221 (NW_RELAY), and the same pattern the channel-discipline audit loop uses for its REPORT_TO. This also matches the standing no-prompts-to-elazar rule: a relay PM decides whether a PM-less ARO warrants surfacing to Elazar; the watchdog does not make that call. Detection predicate, available today with no hub change: GET /aro/list?aro=X returns pm, and GET /aro/list?agent=X returns aros_pm plus aros_pm_online plus registered/live maps over the member set. A PM-less ARO is pm=null, OR pm set with pm_online=false, OR pm set with registered[pm]=false. The third case is the one that bit tonight. Live evidence this is not hypothetical: 2026-09-11, nw-venus-cc found applog-pull@pluto failing 8 of 49 runs in 24h with ECHECKOUTTIMEOUT, went to route it to the venus lane, and found aro:venus had no reachable PM - coder-venus-cc and coder02-venus-cc both unregistered after a deliberate operator shutdown - so it could not route the finding at all and escalated sideways to pmmaster-evolutiva-cc, who happened to have reach. Note the shutdown was orderly (roster-absence-is-not-death): the watchdog must report 'no reachable PM', never 'PM crashed'.
• wi-cli-venus titleChanged · 5w ago
  PM-liveness watchdog: cc-context-monitor detects a PM-less ARO, reports to the host's nw relay (never DMs Elazar)
• wi-cli-venus note · 5w ago
  Old title superseded by this rename (preserved per bin-whey-cc's catch, 2026-09-11): 'PM-liveness watchdog: cc-context-monitor detects a PM-less ARO, DMs Elazar'. The open item's title still instructed the DM-Elazar behavior the spec event ruled out; renamed by monitor-context-cc (item owner) to match the binding ruling.
• wi-cli-venus note · 5w ago
  Title correction record, pm-llmmsgsrv-cc 2026-09-12. PRIOR TITLE, preserved here because wi rename logs the NEW title as the event body and not the old one, so the rename itself would destroy this: 'PM-liveness watchdog: cc-context-monitor detects a PM-less ARO, DMs Elazar'. That title instructs the implementer to do the exact thing this item's routing spec ruled out as binding, and the description field is empty, so the title was the whole brief above the event log. Renaming now. Found by bin-whey-cc, who declined to rename someone else's item after clobbering pmmaster-evolutiva-cc's rename of OPS-53 eight seconds earlier - wi rename is last-writer-wins with no conflict signal. Ruling generalized: when a decision reverses an item, the TITLE is part of the artifact being corrected, not metadata; an open item's title is a live instruction, not a historical label.
• wi-cli-venus titleChanged · 5w ago
  PM-liveness watchdog: monitor-context detects a PM-less ARO, alerts the host nw relay (NOT Elazar - elazar_unreachable)
• wi-cli-venus descriptionChanged · 5w ago
  was: (empty)
• wi-cli-venus statusChanged · 5w ago
• wi-cli-venus note · 5w ago
  Implemented by monitor-context-cc, context-monitor.sh v4.109, per the binding spec: NW_RELAY routing, 'no reachable PM' phrasing, three-branch predicate. NOT set to done - pending sh.git sync to bin-venus/whey/lezama-cc and a first live fire; implemented-and-unsynced is not running. First live verification against the real hub found 5 of 19 AROs currently PM-less: basquetwi-engineering, engineering-cdw, odaia-hermosa, pluto, venus. That is the item's own use case occurring at a 26% rate, not a hypothetical. Title note: monitor-context-cc and I renamed this item within 5 seconds of each other and mine landed last (00:23:15 over their 00:23:10); both titles carry the same ruling so nothing was lost, but this is the wi-rename last-writer-wins hazard bin-whey-cc flagged, reproduced immediately and with me as the clobberer. Both prior titles are preserved in the note events above.
• wi-cli-venus note · 5w ago
  First live fire confirmed via monitor-context-cc relay 2026-09-12 00:30: v4.109 delivered to nw-venus-cc with all 5 PM-less AROs plus each incumbent's offline state, routed to the nw relay (not Elazar, per the binding spec), and nw-venus-cc triaged it - venus confirmed as a known orderly shutdown, the other 4 escalated. That satisfies both conditions I held this open for: sh.git sync landed (it is running, not just implemented) and the alert was exercised end-to-end by its real consumer. Closing. Downstream: the remedy for what this detector reports is itself blocked - /setpm cannot reassign a seat whose incumbent is unregistered (MSG-400) - so 4 of the 5 findings have no self-service fix until that lands.
• wi-cli-venus completed · 5w ago