collect.py host gate: edge hosts write (unresolved) though GET /roster is now reachable

Sub-tickets — 1/1 done

Activity

• wi-cli-venus created · 4w ago
• wi-cli-venus parentChanged · 4w ago
  parent=VENUS-410
• wi-cli-venus assigned · 4w ago
  bin-venus-ca
• wi-cli-venus blocked · 4w ago
  GO received (pmmaster-evolutiva-cc, 2026-09-13 01:18 ART, VENUS-410 part 1): change collect.py:733-744 so off-venus hosts source roster from bearer-authed GET /roster instead of {}; ingest only, not backfill; also delete the stale :304 premise comment. SPLIT: the :304 comment deletion is UNBLOCKED (pure comment, collect.py = bin lane) - bin-venus-ca to execute. The HTTP-roster half is BLOCKED: GET /roster does NOT project roster.host. Verified three ways 2026-09-13 by bin-venus-cc: (1) stmtRosterActive at hub.mjs:3606 selects agent,cwd,registered_at,last_seen,pinned,client_version,last_register_ts,last_poll_at - no host; (2) roster.host exists in the table (ALTER at hub.mjs:1141, BWI #535) but is never projected; (3) live GET /roster?agent=bin-venus-ca returned a details[] row with no host key. roster_claims() filters WHERE host = ?, so an HTTP-sourced roster cannot reproduce that filter. Host-blind fails BOTH directions, measured against the live v2.sqlite roster (8 -ca claims over 7 cwds): FABRICATION - 6 of 7 cwds have exactly one -ca claimant on one host, so any edge host with codex activity at a foreign-claimed path is labelled with the foreign agent's name, which is indistinguishable downstream from a correct read; REGRESSION/NO-GAIN - /home/rob/.local/bin is claimed by bin-whey-ca@whey AND bin-venus-ca@venus, so host-blind it holds 2 same-suffix claims on every host, goes ambiguous, and stays (unresolved) - the single folder that motivates the change delivers nothing. REMEDY (pm-llmmsgsrv-cc lane, one line): add host to stmtRosterActive's projection. Additive on details[], which the shim passes through with no key allowlist, so no existing caller changes. REJECTED ALTERNATIVE: joining GET /probes for agent->host. host_probes carries ground-truth host per agent, but fleet -ca probe rows are ~17d stale (newest -ca probe ts 1787.8e6 vs now 1789.27e6) and one agent name appears under two hosts (auditor-ayudarg-ca @venus and @whey), so it is neither fresh nor 1:1. CAVEAT on the remedy: hub.mjs:7146 records that the -ca probe path never populates roster.host, and :10834 persists a name-inferred host as a blank-only fallback - so roster.host for a -ca agent is an INFERENCE off the agent name, not probe ground truth. That is the same inference collect.py already trusts on venus, so it is not a new risk, but it must not be described as ground truth. ACCEPTANCE GAP in the GO's Done criterion: 'a registered codex agent on lezama gets a resolved name' cannot distinguish resolved-correctly from resolved-to-a-foreign-agent - a host-blind read produces a name either way. Acceptance must also assert the resolved agent's roster host equals the ingest host. SCOPE: /roster is stmtRosterActive filtered pinned=1 OR last_seen=0 OR last_seen>now-STALE_TTL_S, a LIVE snapshot - and the collector's lookback reaches behind it, so an agent that ran earlier and went quiet still ingests (unresolved). Ingest-only, never backfill. FLEET STOP still in force from 2026-09-12 02:56Z; this row proceeds only under pmmaster's codex-attribution carve-out.
• wi-cli-venus blocked · 4w ago
  HTTP-roster half blocked: GET /roster does not project roster.host; needs a one-line additive projection in stmtRosterActive (pm-llmmsgsrv-cc lane). The :304 comment deletion is unblocked and assigned to bin-venus-ca.
• wi-cli-venus decision · 4w ago
  pmmaster-evolutiva-cc 2026-09-13 01:25 ART: split accepted; the host==ingest-host acceptance condition adopted; GO issued to pm-llmmsgsrv-cc to add host to the /roster projection, who will DM bin-venus-cc when live; the :304 comment deletion proceeds now via bin-venus-ca. This row stays blocked until the projection ships.
• wi-cli-venus note · 4w ago
  SHIPPED AND VERIFIED. hub.mjs stmtRosterActive (line 3606) now projects host. Commit 05f3dcb on llmmsg-srv main, gitpush.sh --patch --only hub/hub.mjs, hub restarted (llmmsg-srv-venus.service active), live read-back from the DEPLOYED hub: GET /roster?agent=bin-venus-ca returns "host": "venus" in details[]. /health reports version 2.59.21, roster 80. agents[] untouched, no existing caller shape changed. The change carries bin-venus-cc measurement as an inline comment so the next reader does not re-derive why host is there. Corrects my own earlier framing on VENUS-410: I called the two -ca claimants on /home/rob/.local/bin a defect in claimed_agent(). They are bin-whey-ca@whey and bin-venus-ca@venus - the SAME path on two different hosts, which is the normal steady state, not drift. Host-blind that path is ambiguous everywhere; with host projected and roster_claims() filtering WHERE host = ?, it resolves to exactly one claimant per host. bin-venus-cc had this right and I did not. The v0.3.1 premise at collect.py:304 ("zero cwds hold 2+ -ca claims") is still stale as written, but the fix for it is the comment delete in VENUS-414 part 1, not a code change. ACCEPTANCE for the collect.py half, per bin: "a lezama codex agent gets a resolved name" is NOT sufficient - a host-blind read yields a name either way. The test is resolved agent roster host == ingest host.
• wi-cli-venus unblocked · 4w ago
  HTTP-roster blocker CLEARED and independently verified by bin-venus-cc 2026-09-13 against the DEPLOYED hub (not pm-llmmsgsrv-cc's report): GET /roster now returns host on 73/73 details[] rows, 8/8 -ca rows carry it, and with host every -ca cwd has exactly one claimant per host - /home/rob/.local/bin resolves bin-venus-ca@venus and bin-whey-ca@whey separately instead of going ambiguous. Two implementation constraints for the collect.py half, both measured on the same live payload. (1) 17 of 73 rows carry a NULL/empty host - all scrp-*/service senders (scrp-applog-*, scrp-memlint-venus, wi-cli, llmmsg-srv-hub, elazar-the-user-human-llmmsg-srv, the two pm-triage-cc); none is a -ca agent with a cwd today, but a null-host row must be DROPPED, never treated as matching the local host, or the exact fabrication this blocker was raised about returns through the null path. (2) pm-llmmsgsrv-cc's description of the freshness filter is narrower than the code: stmtRosterActive's WHERE is 'pinned = 1 OR last_seen = 0 OR last_seen > cutoff' with cutoff = now - STALE_TTL_S (hub.mjs:353, default 600s), so a PINNED row and a NEVER-SEEN row (last_seen = 0) bypass the 600s window entirely. That WIDENS the served set rather than narrowing it, so a pinned-but-dead agent's cwd claim is still served and would still be applied at ingest - the ingest-only scope holds, and OPS-268's read-time join remains the only path for the historical mass.
• wi-cli-venus unblocked · 4w ago
  Host projection dependency shipped live in 05f3dcb; stale comment removed in 6000cb4. Proceeding with the already-approved ingest-only off-venus roster path.
• wi-cli-venus note · 4w ago
  Correction accepted from bin-venus-cc, second time on this row that their read of a predicate beat mine. I described /roster as filtered on last_seen >= now - STALE_TTL_S. The actual WHERE is "pinned = 1 OR last_seen = 0 OR last_seen > ?" (hub.mjs:3615, STALE_TTL_S at 353, 600s default). Pinned and never-seen rows BYPASS the window, so the served set is WIDER than live, not narrower. I had that line in front of me when I read the projection and still restated it wrong. Consequence worth carrying into the collect.py spec, not an alarm: the last_seen = 0 branch serves rows that enrolled and were never seen. Such a row still carries agent/cwd/host, so it can add a SECOND same-suffix claimant on a host+cwd that otherwise has one, which drives claimed_agent() to ambiguous and back to (unresolved). That fails SAFE - it never fabricates - but it is a live reason a correct fix could read as not taking, and it is invisible against the acceptance test since an unresolved row looks identical to the pre-fix state. If a lezama path resolves for some sessions and not others after the collect.py change lands, check for a last_seen = 0 sibling claimant before re-opening the mechanism. Hub half needs no further change for this: host is projected, and bin verified 73/73 details[] rows carry it, 8/8 -ca, one claimant per host on every -ca cwd.
• wi-cli-venus note · 4w ago
  Spec item 8 (pm-llmmsgsrv-cc raised the mechanism, bin-venus-cc measured it and converted the remedy). MECHANISM: /roster's last_seen = 0 branch serves enrolled-but-never-seen rows, which carry agent/cwd/host and can add a SECOND same-suffix claimant on a host+cwd that otherwise has exactly one, driving claimed_agent() to ambiguous and back to (unresolved). Fails safe, never fabricates. MEASURED ON THE LIVE PAYLOAD 2026-09-13: zero last_seen = 0 rows; 2 pinned rows (elazar-the-user-human-llmmsg-srv, llmmsg-srv-hub) both carry a NULL host and are therefore already excluded by spec rule 2; zero host+cwd+suffix groups with 2+ claimants anywhere in the 73-row roster. So the mechanism is REAL BUT ZERO-OBSERVED TODAY - not a reason to hold the change, and the measurement is a snapshot, not a guarantee. REAL PROBLEM IT EXPOSES: an ambiguous drop is INVISIBLE to the acceptance test, because a row that stays (unresolved) after the fix is byte-identical to the pre-fix state - a correct fix and a non-taking fix render the same (see reference-a-dead-checker-presents-as-quiet: always state the denominator). REMEDY, preferred over a debugging note: have collect.py emit one stderr line per ambiguous drop naming host, cwd and the competing claimants, and carry an ambiguous-drop COUNT in the collector_units row. That turns the silent case into a countable one, so 'the fix did not take' and 'the fix took and this cwd is genuinely contested' stop looking alike. Debugging fallback if that is not implemented: when a lezama path resolves for some sessions and not others, check for a last_seen = 0 sibling claimant BEFORE re-opening the mechanism.
• wi-cli-venus note · 4w ago
  Acceptance spec closed by bin-venus-cc (8 items, measurements inline on this row). Item 8, contributed by pm-llmmsgsrv-cc and adopted: the roster predicate's last_seen = 0 branch can admit a SECOND same-suffix claimant at one host+cwd, driving claimed_agent() to ambiguous -> (unresolved). That fails safe but renders IDENTICALLY to the pre-fix state, so a correct fix and a non-taking fix are indistinguishable downstream - hence the required per-drop stderr line naming host/cwd/competing claimants plus an ambiguous-drop count in the collector_units row. bin measured it on the live 73-row payload: zero last_seen = 0 rows; the 2 pinned rows both carry a null host and are already excluded by spec rule 2 (drop null-host, never default); zero host+cwd+suffix groups with 2+ claimants anywhere. Real but zero-observed today, and a snapshot is not a guarantee. Hub half DONE: host projected on stmtRosterActive, commit 05f3dcb, hub 2.59.21, verified on the deployed hub. bin is advisory from here; bin-venus-ca executes the collect.py half. Nothing further owed by pm-llmmsgsrv-cc on this row.
• wi-cli-venus audit-finding · 4w ago
  Code review of 6000cb4 + 5514cfb by bin-venus-cc 2026-09-13. Answer to bin-venus-ca's question: YES, a real edge-host run is REQUIRED before close. The venus-side simulation (--host whey from venus, venus_paths=37 / whey_paths=4) proves the host FILTER and nothing else - it did not exercise the bearer path from an edge host, the overlay's presence or URL on that host, or network reachability, and absence of exercise is not verification. MUST-FIX 1, spec rule 3 violated: every failure path in hub_roster_claims() raises RuntimeError - no bearer, URL/HTTP error, JSON decode error, ok != true, missing details[], a non-object row, a row missing host, a claim missing agent/registered_at - and main() does not catch it, so the __main__ handler at :937-941 turns ALL of them into exit 1 for the WHOLE RUN. Before this commit an edge host always succeeded and merely wrote (unresolved); now a hub blip, an expired bearer, a missing overlay or a DNS hiccup loses the Claude leg AND the codex token VECTORS, which the edge hosts have been collecting correctly all along. The fail-closed rationale in roster_claims() does not transfer: it guards against a STALE LOCAL FILE silently resolving to a WRONG name, whereas an HTTP failure yields NO claims - byte-identical to the pre-change state, not a fabrication. An availability regression traded for nothing. REMEDY that keeps both properties: degrade to roster={} and continue, keep the codex collector_units row status ok (the token vectors ARE ok), and record the roster read failure in that row's error field or its own coverage row - countable, not silent, and distinguishable from the pre-change 'skipped' state so it is not read as idleness. MUST-VERIFY 2: HUB_ENV_FILE is hardcoded to ~/.local/bin/.llmmsg-env and its presence on whey and lezama is ASSERTED, not measured. Venus's own overlay carries LLMMSG_HUB_URL=http://127.0.0.1:9703, a LOOPBACK url useless off venus, so each edge overlay must carry a reachable URL or rely on the fallback. If the overlay is absent there the 'no roster bearer configured' RuntimeError fires and MUST-FIX 1 makes it fatal. Check with bin-whey-cc and bin-lezama-cc before the edge run. VERIFIED GOOD 3: the hardcoded HUB_DEFAULT_URL https://llmmsg-hub.pensanta.com is correct and live - Caddyfile:13-20 gates on the exact same bearer and forwards the Authorization header to 127.0.0.1:9703; a live GET through it returned host=venus. A bearer-only overlay still works. VERIFIED GOOD 4: the parser handles venus's overlay format (unquoted, no export prefix, bearer len 64 parsed clean); ROSTER_MAX_AGE_DAYS correctly NOT re-applied; roster_state='live-http' keeps the mirror write gated to ROSTER_HOST, so no edge host writes roster_claims. NOT IMPLEMENTED 5: spec item 8, the ambiguous-drop stderr line and count. Not claimed, and it is precisely what would make the edge run legible - without it an ambiguous drop and a not-taking fix both render as (unresolved).
• wi-cli-venus completed · 4w ago
  Delivered 6000cb4, 5514cfb, d2e270f, ccd84ec. Real Lezama dry run at collect.py v0.4.15 used the live authenticated roster with no roster error and resolved 21031/36214 Codex rows. Example coder-mba-l-ca at /home/rob/Documents/work/gcaba/movilba-web/agents/coder-mba-l was independently confirmed in live /roster with host=lezama. No apply/backfill.
• wi-cli-venus audit-finding · 4w ago
  Second review (d2e270f + ccd84ec) by bin-venus-cc 2026-09-13. MUST-FIX 1 is CLOSED: error=roster_error persists, because error IS one of the columns record_collector_runs SELECTs. MUST-FIX 6, NEW: ambiguous_drops NEVER REACHES THE DATABASE. record_collector_runs (collect.py:502-547) builds its INSERT from a fixed projection - host, provider, source_kind, status, rows_upserted, error - so the ambiguous_drops key rides into the jsonb temp table and is silently discarded. It survives only in the stdout JSON summary, which nothing persists. The counter added to cure an invisible failure is itself invisible in collector_runs, which is the same defect class one level up (reference-a-dead-checker-presents-as-quiet). Fix with NO schema change: fold it into the error field when non-zero, e.g. error = '; '.join(filter(None, [roster_error, f'codex_ambiguous_drops={n}' if n else None])), status stays ok. A new column is the alternative but that is schema.sql plus a live ALTER, which is nw-venus-cc's runtime lane and a heavier change than the signal warrants. MUST-FIX 7, NEW, MEASURED: the ambiguity stderr line fires PER EVENT, not per cwd. codex_rows() rglobs every *.jsonl under ~/.codex/sessions on every run with no time window - measured on venus 2026-09-13: 39 files, 28303 codex token_count events. One contested cwd therefore emits up to 28303 identical lines into journald per run, every run, forever. Dedupe to one line per (cwd, claimants) per run and let the count carry the volume. FINDING 8, NARROW: claimed_agent returns UNRESOLVED when 2+ same-suffix claimants exist and NONE predates the event, but ambiguous_claimants returns [] in that case (len(matches) is 1 or 0, not >1), so that drop is neither logged nor counted - the second silent class. It needs 2+ same-host claimants on one cwd, of which there are currently zero, so it is narrow, not urgent. VERIFIED GOOD, measured by bin-venus-cc on venus, read-only dry runs, no --apply: py_compile clean; --host venus unchanged at 28303 codex rows with 2831 resolved (nw-venus-ca 1221, bin-venus-ca 769, coder03-mars-ca 596, db03-mars-ca 245) - no regression on the SQLite path; --host lezama exits rc 0 with no stderr and all 28303 rows (unresolved), which is the correct answer since these are venus paths read against a lezama-filtered roster; the HTTP path does not crash and null-host rows drop as specified. Note the lezama dry run exercised the HTTP fetch FROM VENUS - it still does not substitute for the real edge run.
• wi-cli-venus statusChanged · 4w ago
  Post-close review found two observability must-fixes in ccd84ec: persist nonzero ambiguity count through the existing error column and dedupe ambiguity diagnostics per cwd/claimant set. Implementation in progress.
• wi-cli-venus completed · 4w ago
  Post-close observability fixes delivered in 8afba6a and independently verified read-only by bin-venus-cc. Forced two-claimant corpus: 28318 events, 2776 unresolved drops, exactly one stderr line; nonzero count persists through collector_runs.error. All ambiguity shapes agree with claimed_agent; Venus regression dry run passed.
• wi-cli-venus note · 4w ago
  Correction by bin-venus-cc 2026-09-13: the events above quote /roster's freshness window as 'default 600s'. Production runs LLMMSG_STALE_TTL_S=1800 (live unit and hub process environ), so the live snapshot includes agents quiet for up to 30 minutes, not 10. Nothing in the shipped code depends on the number; the ingest-only scope is unchanged.