llmmsg-srv
MSG-413
Claude push-then-mark-read: feature-gated at-most-once delivery, measure the missed-message rate
Done normal
unassigned
Elazar CLI directive 2026-09-20, with his explicit acceptance of the tradeoff. coder-llmmsgsrv-ca audited and BLOCKED this as a production default; Elazar's decision is to ship it feature-gated and measure, and he relaxed the auditor's stricter instrumentation demand: do NOT require a mandatory channel_ack tool or block the feature on delivery guarantees. Lightweight counters where cheap. The change: in llmmsg-srv-mcp.mjs, a message delivered INLINE via the channel push is not buffered in pendingForModel and its id is treated as drained, so it is marked read at push time and renders once instead of twice. Prototyped and measured as PROBE-6 on a test-only copy at /home/rob/agents/test-llmmsgsrv/shimtest/ (prod byte-unchanged): agent got the body inline at turn start, never called read_unread, its send result carried NO inbox dump, text appeared exactly once, cursor advanced to 128354 with no drain. Diff: /data-venus/llmmsg-srv/diffs/2026-09-20-probe4-inline-ack-TESTONLY.diff Why it is at-most-once: await mcp.notification() proves a transport write, not model receipt. Per the channels reference a properly loaded session queues events reliably; the loss case is registration/policy failure - channel not loaded or policy-blocked - which the shim cannot observe from a notification return value. That state was NEVER forced in any probe, so the rate is unknown. This row exists to find it. Instrument (pragmatic version): log every inline-acked id + tag to a file, then correlate against the receiving session's own jsonl for that tag inside a <channel> block. Missed = logged-but-absent. Do NOT define missed as 'no reply' - an action-only message can be processed without a reply, which would inflate the rate. Report registration/policy failures separately from loaded-channel no-receipt. Gating: test-only/feature-gated (env flag, default off) until enough observations establish a rate. Not a production default while the BLOCK stands. Scope note: this is the CLAUDE side only. The Codex side is MSG-412 and needs none of this - it acks after a durable codex queue persist, which the auditor accepted. Findings: /data-venus/llmmsg-srv/findings/2026-09-20-codex-push-and-inline-body.md section 3 and section 6.
Activity
-
Shipped 2026-09-20 as commit 724431b, hub 2.61.0, hub/llmmsg-srv-mcp.mjs only. Root cause confirmed at source: llmmsg-srv-mcp.mjs:784 pushed [~N-token from <sender> buffered] with no body, and since OPS-228 no instruction either; the body reached the model only through drainBuffered() inside wrap(), which fires on an llmmsg tool call. A reply that called no tool never saw the message. Observed live the same day when coder-llmmsgsrv-ca characterised a message it had never read.\n\nImplemented, per Elazar's explicit tradeoff relayed by coder-llmmsgsrv-ca (no delivery guarantee required, measurement must not block deployment): full body inline with no default cap; a message whose notification resolves is NOT buffered and its id advances lastDrainedId so the next poll acks the hub cursor; ack is the highest CONTIGUOUS success walking ascending ids, so a later success cannot ack past an earlier failure (coder-llmmsgsrv-ca's correction); a notification that throws falls back to the old buffer-and-drain path and breaks the ack chain. A 50-entry pushedRecently ring holds bodies delivered by push alone and is rendered ONLY by read_unread, so recovery from a dropped notification costs one explicit call and the normal path never pays twice. Counters push_full_total/push_full_acked/push_full_recovered plus a per-push stderr JSON record {evt,agent,id,tag,from,chars,pushed_at,acked} are exposed through the version tool for JSONL correlation. LLMMSG_PUSH_FULL=0 restores 2.60.0 behaviour byte-for-byte; LLMMSG_NOTIFY_INLINE_MAX_CHARS still sets an explicit cap.\n\nExercised on a fake-hub harness (copied to /data-venus/llmmsg-srv/findings/fakehub.mjs + run-probe.sh + msg413-shim.diff): 4000-char body pushed inline, has_unread 0, hub ack advanced to the message id with no drain, read_unread re-rendered it once from the ring; with LLMMSG_PUSH_FULL=0 the placeholder went out, the body buffered, and the ack stayed at 0 until the drain. NOT exercised: notification-throws path, over-cap path, PENDING_CAP/STUBBED_CAP eviction, a real Claude Code client, and a multi-message batch with a mid-batch failure - that last one is where the contiguous-ack logic lives and it has no test.\n\nRollout: venus runs the shim from /opt so venus sessions pick it up on their next spawn. whey and lezama need scripts/install-mcp-shim-runtime.sh --expected-commit 724431b via bin-whey-cc/bin-lezama-cc, requested. Running sessions keep their old shim until Elazar relaunches them. Audit request sent to audit-general-ca.
-
2.61.0 (724431b) was BLOCKED by both audit lanes within minutes, independently, on the same defect. Fixed in afb3927, hub 2.61.1.\n\nThe defect: the push loop kept pushing after a notification failed. A message pushed AFTER a gap was in no list at all - not buffered (it had been pushed), not ackable (a lower id was unresolved, and the hub cursor is a high-water mark), and skipped by the next poll's since because lastDeliveredId had moved past it. Result: the hub cursor sat behind it permanently and a shim respawn redelivered it indefinitely. audit-general-ca's repro: prior cursor 99, rows 100/101, 100 throws and 101 succeeds; after read_unread drains 100 the cursor reaches 100 and 101 is never reconsidered.\n\nFix: stop pushing at the first failure. That message and every remaining message in the batch are buffered unpushed, so the pushed prefix IS the ackable prefix and every message is in exactly one place. State machine extracted as createPushAckChain() in hub/poll-state.mjs. push_full_acked now counts only messages that actually advanced the acknowledged prefix. The 2.61.0 recovery ring (pushedRecently, rendered only by read_unread) was REMOVED on coder-llmmsgsrv-ca's contract argument: the directive is push-then-mark-read with no guarantee, and re-serving already-read mail changes that tool's contract.\n\nNot changed, flagged instead: coder-llmmsgsrv-ca asked for an immediate ack after the batch rather than on the next poll tick. An ack call is itself a GET /unread, so discarding the messages it returns would lose mail. The ack still rides the next tick and the code now documents that a process exiting in between redelivers, instead of claiming push-time ack.\n\nCoverage added: scripts/test-mcp-push-ack.mjs - success-success, failure-success, success-failure-success, a five-message batch with a mid-batch gap, kill-switch/over-cap, ack-never-moves-backward, NaN-safe construction. Forced red against 724431b behaviour before accepting it green (mutating the loop to push past the gap fails the success-failure-success case). .gitpush-pre.sh was EMPTY; it now runs node --check on hub/llmmsg-srv-mcp.mjs and hub/hub.mjs plus both test files, so this class of regression cannot ship silently again.\n\nStill not exercised: the real notification transport (a throw is modelled, not induced), PENDING_CAP eviction, a live Claude Code client. Redeploy of 2.61.1 requested from bin-whey-cc and bin-lezama-cc; venus serves it from /opt on next session spawn.
-
Two more rounds after afb3927, both from re-review, both real.\n\n23eda29 (2.61.2) - coder-llmmsgsrv-ca: LLMMSG_PUSH_FULL=0 was NOT the byte-for-byte revert I claimed. With push-delivery off nothing is ever delivered, so the first row broke the ack chain and every later row took the stop-at-gap branch and was buffered WITHOUT calling mcp.notification() - 2.60.0 sent a notification for every row. A revert flag has to restore the old behaviour, not merely stop the new one. Fixed by scoping the gap rule to push-delivery: if (PUSH_FULL && ackChain.broken). Over-cap with push-delivery ON still stops at the first failure, deliberately, documented at the site. Telemetry corrected in the same commit: push_full_acked -> push_full_delivered, and the stderr record says ack_pending:true not acked:true, because at log time only the local high-water has moved and the hub ack rides the next poll.\n\n62d3d48 (2.61.3) - bin-whey-cc: test-mcp-push-ack.mjs is a green line, not a control, for the defect it claims to guard. It asserts a MODEL of the loop plus createPushAckChain, and that helper was introduced BY the fix, so the test cannot be run against 724431b at all. Mutating the model proves the assertions bite; it does not prove the shipped loop is wired to them. Added scripts/test-mcp-push-batch-live.mjs: spawns the REAL shim process against a fake hub, delivers ids 100/101/102 with 101 over an explicit inline cap (the one failure mode inducible without breaking the MCP transport), and asserts 100 pushed and acked, 101 sent as a placeholder, 102 never notified, 101 and 102 both reaching the model through the drain, 100 not rendered twice, and the ack never passing the gap. Forced RED against the real shipped loop: copy the shim, replace the guard with if (false), run with LLMMSG_TEST_SHIM pointed at the copy - fails on notified == [t-100,t-101,t-102], the original defect exactly. Both test files plus node --check on both hub files are now in .gitpush-pre.sh, which was empty before this work; the gate costs about 11s.\n\nLesson worth keeping beyond this row: a regression test written against a helper that the fix introduced can never be run against the bug. The control has to drive the shipped path.
-
SHIPPED at 62d3d48, hub 2.61.3. coder-llmmsgsrv-ca verdict: PASS, no further code finding, HEAD verified equal to origin/main.\n\nFour commits: 724431b (2.61.0, full-body push) -> afb3927 (2.61.1, stop pushing at the first failed notification; recovery ring removed) -> 23eda29 (2.61.2, LLMMSG_PUSH_FULL=0 restored to a true revert; telemetry renamed push_full_delivered / ack_pending) -> 62d3d48 (2.61.3, live real-shim/fake-hub control + gate).\n\nWhat is now true: a Claude Code session receives the full message body in the channel notification at turn start, so a reply that calls no llmmsg tool is no longer answering from sender-and-size metadata. A delivered message is not buffered and its id advances the local ack high-water, which rides the next poll query. Push stops at the first failure so the pushed prefix is exactly the ackable prefix. LLMMSG_PUSH_FULL=0 is a real revert to 2.60.0 behaviour.\n\nCoverage: scripts/test-mcp-push-ack.mjs (state machine + loop model, incl. notification-attempt assertions) and scripts/test-mcp-push-batch-live.mjs (spawns the real shim against a fake hub; forced red against the shipped loop via LLMMSG_TEST_SHIM). Both plus node --check on hub/llmmsg-srv-mcp.mjs and hub/hub.mjs are in .gitpush-pre.sh, which was empty before this work.\n\nNOT exercised, stated rather than hidden: real Claude Code channel retention (a resolved mcp.notification proves handoff, not that the client kept it - this is the accepted at-most-once risk, Elazar's explicit call), a transport throw from mcp.notification (modelled, not induced), and PENDING_CAP eviction.\n\nOpen rollout, tracked outside this row: venus serves 2.61.3 from /opt and picks it up on each session's next spawn; lezama is on 2.61.3; whey's install of 62d3d48 was requested and not yet confirmed. Every running session keeps its booted shim until Elazar relaunches the pane, so no agent is exercising this yet.
task
3w ago by wi-cli-venus
3w ago
2026-09-20 05:58