hub restart strands shim SSE: roster sweeper prunes agents that didn't auto-reconnect

Sub-tickets

Questions

Activity

• system created · 2026-05-11
  wi cli
• llmmsg-srv-cc assigned · 2026-05-11
  assigned to llmmsg-srv-cc
• llmmsg-srv-cc assigned · 2026-05-11
  assigned to llmmsg-srv-cc
• system note · 2026-05-11
  Today 2026-05-11 hub restart at ~20:20 ART. 4 interactive CC sessions on whey (pm-mars-cc, pm-pluto-cc, whey-nw-cc, llmmsg-srv-cc) had MCP shims alive but did not auto-reconnect SSE. Shim has reconnect logic (hub/llmmsg-srv-mcp.mjs:595 connectToHub with 5s retry on error + 60s watchdog), but in practice all 4 shims stayed in a stranded state until manually re-poked (forceReconnectHub via register tool, then HTTP /register on their behalf). Roster sweeper (hub.mjs:1231 ROSTER_STALE_MS=10min) then pruned them ~10min after restart, making them unaddressable for DMs. Need to: (a) verify why shim reconnect didn't fire — possibly clientReq.on('error') not reached, or watchdog interval got cleared but timer didn't fire, (b) consider not running roster sweeper for at least 1× SHIM_WATCHDOG_S (60s) after hub uptime <2min so cold-start grace exists, (c) consider hub broadcasting 'hub-restart' system event so connected SSE drains and shims re-register cleanly. Elazar visibility on hub restart should not require terminal CC user activity to recover.
• system evidence · 2026-05-18
  2026-05-18 dropout wave from hub restart at 05:00. Failure modes: (A) zombie socket — writes buffered, last_seen_at stays fresh, but no offline-alert fires (PM case); (B) shim no-reconnect — process alive, tunnel up, SSE not re-established (ayudarg auditor/db). Recovery: MCP register() tool call triggers forceReconnectHub() — correct recovery path. Asymmetric alert: team agents got EPIPE-style failures → swept → alerted; PM zombie kept last_seen_at fresh → not swept. Fix needed: hub-side write-fail eviction + hub-restart broadcast + sweeper offline-alert for channels-dead but last_seen_at-fresh case.
• system completed · 2026-05-18
  fixed in commit 42ba619: shim reconnect-guard bug + hub channel-orphan sweep. Shim fix prevents stranding when initial /connect fails (hub restart window). Hub fix evicts agents whose bootstrap hook refreshes last_seen_at via /register without /connect.