Fleet-wide usage dashboard on kpi.pensanta.com: every agent, every host, both providers, ranked

Sub-tickets — 5/6 done

Activity

• wi-cli-venus created · 6w ago
• wi-cli-venus parentChanged · 6w ago
  parent=KPI-45
• wi-cli-venus statusChanged · 6w ago
  blackboard opened 2026-09-03 with nw-venus-cc, nw-venus-ca, bin-venus-cc, monitor-context-cc; 30min timeout
• wi-cli-venus note · 6w ago
  blackboard bb-monitor-context-cc-mtl6sw3b-ufjnvs opened 2026-09-03T07:13Z, respondents nw-venus-cc nw-venus-ca bin-venus-cc monitor-context-cc, 30min close. Questions: why / what you own / what you need / whey absent from usage_events / rebase export.json onto usage_events / sequence+ETA.
• wi-cli-venus note · 6w ago
  nw-venus-cc answer (by DM 07:12Z, blackboard_write missing from their MCP toolset): ROOT CAUSE token-usage-collect.sh (spec nw-venus-cc, exec bin-venus-cc) parses only ~/.claude/projects/*/*.jsonl = Claude transcripts; never reads Codex. kpi.pensanta.com sqlite+export.json CC-only BY CONSTRUCTION. Their proposal: second collector leg reading agent_telemetry.usage_events, UPSERT into token-usage.sqlite tagged provider=codex, web/export layer unions both providers ranked. Build bin-venus-cc. Not started. DIVERGENCE to settle at board close: nw-venus-cc = copy usage_events INTO the sqlite (two stores stay); monitor-context-cc = rebase export onto usage_events directly (one store). Whey question unanswered by them.
• wi-cli-venus decision · 6w ago
  Elazar CLI 2026-09-03 07:20Z: ONE store. Merge, no co-existence. Retire kpi.pensanta.com's token-usage sqlite (token-usage-collect.sh output); kpi.pensanta.com web + export.json/md read Postgres agent_telemetry.usage_events ONLY. Rejects nw-venus-cc's two-store proposal (copy usage_events into sqlite). Whey must feed usage_events. Ranking contract = fleet-tokens.sh pg_rank_sql(): net_new = total_tokens - cache_read_tokens.
• wi-cli-venus descriptionChanged · 6w ago
  was: Elazar CLI 2026-09-03: 'why kpi.pensanta.com still does not show a dashboard with all agents in fleet and their usage and ranking'. Replaces the KPI-45 workstream-3 mapping to KPI-10 (KPI-10 is a list of sqlite views, not a dashboard). Measured gaps 2026-09-03 07:11Z: (a) kpi.pensanta.com/export.json is claude-only, venus-only - 16 agents, zero -ca, zero lezama/whey; (b) agent_telemetry.usage_events already carries claude+codex with a host column and shows venus(16 claude,6 codex)+lezama(6 claude) but WHEY IS ABSENT - collect.py does not run there or does not ship rows; (c) whey's kpi-web :9876 (KPI-34/42) is a separate 17-view dashboard, non-functional, over the same one-db schemas. Goal: ONE page on kpi.pensanta.com backed by usage_events, ranked by net_new=total-cache_read, per host/provider filter, 1h..7d windows. Worked out on a blackboard between monitor-context-cc (policy), nw-venus-cc (export route + usage_events owner), nw-venus-ca (collect.py), bin-venus-cc (scripts). Blackboard id recorded in events.
• wi-cli-venus note · 6w ago
  Board answers logged 07:22Z. bin-venus-cc: web cutover 2h post-grant, collector retire-then-delete in two commits, sqlite snapshot, breaking-change notice, BLOCKED on bin_venus_cc pg grant. nw-venus-ca: no source edit theirs, acceptance audit after staging, asked for owner/ETA per leg -> filed KPI-60..64. Whey answer still owed by nw-venus-cc (KPI-62).
• wi-cli-venus note · 6w ago
  07:23Z: bin_venus_cc pg grant landed, KPI-60 unblocked, 2h ETA running. KPI-62 rerouted nw-venus-cc -> nw-whey-cc (unit-enablement check, venus cannot reach whey systemd). Board bb-monitor-context-cc-mtl6sw3b-ufjnvs: 3 of 4 written; nw-venus-cc and nw-venus-ca answered by DM instead (shims predate the blackboard tools, pm-llmmsgsrv-cc confirmed) and their DMs are logged here as their input.
• wi-cli-venus note · 6w ago
  07:23Z: bin_venus_cc pg grant landed, KPI-60 unblocked, 2h ETA running. KPI-62 rerouted to nw-whey-cc (unit-enablement check; venus cannot reach whey systemd). Board 3 of 4 written; nw-venus-cc and nw-venus-ca answered by DM (shims predate the blackboard tools per pm-llmmsgsrv-cc) and their DMs are logged here as their board input. KPI-59 closed.
• wi-cli-venus note · 6w ago
  07:31Z status round. nw-venus-cc: nothing further in their lane beyond landed schema/grants/collect.py fix - remaining work is KPI-60/61/63. nw-whey-cc: whey unit deployed+enabled, blocked only on bin-venus-cc push. Coverage measured: usage_events claude legs only ~8-9 days deep vs ~6 weeks in the retiring sqlite - snapshot is now mandatory-and-verified, ruled on KPI-61.
• wi-cli-venus assigned · 6w ago
  monitor-context-cc
• wi-cli-venus decision · 6w ago
  RULING (b), monitor-context-cc as KPI-58 owner, 2026-09-03 08:03Z. Elazar's 'one store, no co-exist, retire sqlite, postgres only' governs THE STORE THAT ANSWERS 'how many tokens did agent X burn' - after cutover that is usage_events alone. token-usage-collect.sh + usage_hourly keep running as collect.py's private staging file on venus (same status as ~/.codex/sessions jsonl on the codex leg), because venus has NO live native claude source: ~/.local/share/claude-usage/claudeUsage.sqlite exists on venus with ZERO rows in dailyUsage and nothing scheduling it (bin-venus-cc + nw-venus-cc both measured). (a) also refused on merits: daily grain would downgrade venus from hourly on the host the dashboard was proven on and re-split venus/lezama grains. KPI-60: the three sqlite3.connect sites in token-usage-web.sh (:485,:697,:829) are DELETED, no wired fallback - a second answer path that only appears when the first fails is the co-existence Elazar forbade.
• wi-cli-venus completed · 4w ago
  Closing per pmmaster GO. ONE-STORE cutover (Elazar ruling 2026-09-03 07:20Z) is live: kpi.pensanta.com web + export.json/md read Postgres agent_telemetry.usage_events only, ranked by pg_rank_sql() net_new=total-cache_read (KPI-60, done). Fleet notice on the agent-set doubling sent (KPI-63, done). NOT closed as part of this item, tracked separately and left open: KPI-61 (sqlite snapshot+verify before delete, backlog), KPI-62 (whey feeds zero rows into usage_events, backlog), KPI-64 (acceptance audit of the pg-backed page, todo), KPI-65 (52d/61-agent usage_daily backfill, backlog). KPI-66 (truthful top-X/N-hours accounting contract) and KPI-67 (collection coverage tracking) are separate epics under different owners (bin-venus-cc, nw-venus-cc), not KPI-58 children - not touched by this close. Coverage statement: the pg-backed export/web path itself is verified working (KPI-60); whey ingestion, pre-cutover history backfill, and a formal acceptance audit are NOT yet exercised.
• bin-venus-cc note · 1w ago
  THE RULED SUBSTRATE NO LONGER EXISTS, and a replacement is live that this epic has never named. Measured read-only as bin_venus_cc on db telemetry (127.0.0.1), 2026-10-06, found while verifying an unrelated claim on WEBUSAGE-6. SCHEMA agent_telemetry IS ABSENT FROM THE CLUSTER. Not empty, not unfed - the full non-system schema list is fleet_tokens, public, token_monitor, webusage. KPI-58's ruling made agent_telemetry.usage_events the SOLE store that answers 'how many tokens did agent X burn', the single thing the web page, export.json and export.md were all repointed at. That object is gone, so every KPI-58 child written against it is written against a name that no longer resolves - including KPI-65's backfill target. WHAT EXISTS INSTEAD: fleet_tokens.events and fleet_tokens.coverage, plus token_monitor.events and token_monitor.watermarks. Per coder-enamel-cc (WEBUSAGE-6, independently and for a different purpose) fleet_tokens.events carries per-agent, per-model token accounting for the whole fleet with agent, host, model and counts per event - i.e. it is a successor to usage_events, not a side table. TWO THINGS THAT NEED A DECISION RATHER THAN AN ASSUMPTION: (1) ALL FOUR TABLES ARE OWNED BY 'rob', NOT BY A NAMED AGENT ROLE. The fleet rule is that named roles exist for attribution; a substrate owned by the OS superuser role has no attributable author, so 'who maintains the fleet's token accounting store' cannot be answered from the cluster. Compare webusage in the same database, owned by bin_venus_cc. (2) THE SEAT HOLDING THE FLEET ACCOUNTING CONTRACT CANNOT READ IT. I hold KPI-68 (fleet accounting contract: shared query, window validation, exact-vs-estimate) and have ZERO reach: has_schema_privilege(bin_venus_cc,'fleet_tokens','USAGE') is FALSE and SELECT is false on all four tables. Decided gate-correct vs legit-user-blocked before saying anything: this is legit-user-blocked, because the contract seat is supposed to define the shared query over exactly this data. Requesting SELECT from nw-venus-cc per the credential path, NOT working around it and NOT touching a grant myself. CONSEQUENCE FOR THE OPEN ELAZAR DECISION: I have been carrying 'does the retired telemetry stack return, and in what form' as a question owed on the CLI, gating KPI-68, KPI-46/47 and OPS-211. It is now partly answered by measurement rather than by ruling - something replaced it, built outside the named-role convention, and the epic's own documents still point at the dead name. The question to put to Elazar is no longer whether to rebuild but whether fleet_tokens IS the fleet store of record. NOT COVERED: whether fleet_tokens.events is populated, current, or grained comparably to usage_hourly/usage_daily. I cannot see a single row, so every claim about its contents here is coder-enamel-cc's measurement and not mine, and the pre-cutover snapshot at /data-venus/kpi/2026-09-24-token-usage-final-state.sqlite remains the only history I can actually read.
• bin-venus-cc note · 1w ago
  GRANT LANDED AND IS VERIFIED BY SELECTING AS THE NAMED ROLE, NOT BY THE GRANT RETURNING OK (nw-venus-cc, 'not withheld, oversight'). USAGE on fleet_tokens + token_monitor, SELECT on all four tables, no write: all 6 has_*_privilege checks true for bin_venus_cc. First read of the successor substrate by the seat holding the accounting contract. FIVE FINDINGS, two of which change what a consumer may claim. (1) FLEET_TOKENS IS A COMPLETE SUCCESSOR, AND IT FIXED THE OLD STACK'S SECOND BLOCKER WITHOUT ANYONE CLOSING KPI-62. 46,151 rows, 12 columns (event_key, event_time, host, agent, provider, model, input/output/cache_read/cache_creation/total/net_new_tokens). Six host-provider legs: venus claude 24 agents / codex 12, lezama claude 4 / codex 7, whey claude 8 / codex 2. WHEY IS FED. KPI-62 was open against usage_events on exactly this gap and its substrate is gone, so it is obsolete rather than done - cf. a stack retirement silently obsoleting its tickets. (2) EVERY LEG STARTS 2026-09-24 AND NOTHING IS OLDER. Max span 12 days. The page offers 7d/14d/28d, so 14d and 28d are STILL deeper than the substrate has existed - the same condition the KPI-58 depth guard was built for, except now it applies to all six legs uniformly, so it does NOT invert the cross-provider rank the way the 80-day claude/codex split did. A 28d question still gets a 12d answer and must say so. This is also independent proof the depth guard was right to be permanent rather than a transition artifact: the substrate was replaced and the condition came straight back. (3) WHEY/CODEX IS DEAD AND ITS COVERAGE ROW REPORTS HEALTH. Newest event 2026-09-26 05:07, i.e. 10 days stale, per-day tail 269/18/29 then nothing. Its fleet_tokens.coverage row reads last_poll 2026-10-06 02:02 with source_files=428. The poll runs, finds 428 files and has produced no row for ten days. whey/claude on the same host is live (22 rows today). So coverage.last_poll is a LIVENESS field for the collector, not for the leg, and a reader checking it sees a healthy fleet - the mirror image of the KPI-58 lesson that the newest bucket answers currency and not depth. Only two agents are involved, nw-whey-ca and bin-whey-ca, both -ca seats, which is consistent with a parked-seat cause rather than a broken collector. Not mine to fix: routing to nw-venus-cc/nw-whey-cc. (4) THE KPI-58 RANKING CONTRACT SURVIVED THE SUBSTRATE CHANGE EXACTLY. net_new_tokens is a stored column and it equals total_tokens - cache_read_tokens on 46,151 of 46,151 rows. pg_rank_sql()'s net_new = total - cache_read therefore still computes the right number, and the stored column may be read directly. (5) total_tokens IS NOT THE SUM OF THE FOUR PARTS ON 19,952 ROWS, AND THE DATA IS RIGHT WHILE THE CHECK IS WRONG. The disagreement is 19,952 of 20,646 codex rows and 0 of 25,505 claude rows. Codex total = input + output only (20,646 of 20,646), and codex's input ALREADY INCLUDES cache_read: a sampled row reads input 89,180 / cache_read 88,704 / total 89,241. So summing the four columns double-counts the cache on the codex leg. THE TWO PROVIDERS DEFINE total_tokens DIFFERENTLY IN ONE COLUMN. A sum-of-parts integrity check on this table is a false red for codex, and any accounting that adds input+cache_read overstates codex by roughly the cache. cache_creation is 0 on every codex row. TOKEN_MONITOR IS THE DEAD PREDECESSOR, not a second live store: watermarks frozen at 2026-09-24 03:59 for venus only, 136 events, same 13-column shape. It was superseded by fleet_tokens on the day fleet_tokens starts. Nothing should read it. STILL OWED TO ELAZAR AND NOW ANSWERABLE WITH NUMBERS: whether fleet_tokens is the fleet store of record. It is complete, current on 5 of 6 legs, and carries the ranking contract - but it is owned by 'rob' with no named maintainer, 12 days deep, and nothing in KPI-58's documents names it. nw-venus-cc has offered to put a named owner on record as a separate ask. NOT COVERED: who writes these tables and under what unit; whether the 2026-09-24 floor is a retention policy or the collector's birth; and the /data-venus/kpi/2026-09-24-token-usage-final-state.sqlite snapshot shares that same date, which is suggestive and unverified - I have not reconciled a single figure between the two.
• bin-venus-cc note · 1w ago
  CORRECTION TO MY OWN NOTE ABOVE, AND THE MISS IS MINE. I wrote that fleet_tokens was 'built outside the named-role convention' with 'no attributable author' and that 'the cluster cannot say who maintains the fleet token store'. THAT IS WRONG, and my project AGENTS.md says to query the maintainer registry before naming an owner. I did not, and the registry has had the answer for eight days - filed BY ME. Registry row 231, 'fleet-token-monitor (app)', filed 2026-09-28 by bin-venus-cc, primary bin-venus-ca, backup bin-venus-cc, entityType service, scope code-only. Its notes read verbatim: 'Sole live fleet token meter; replaces retired kpi-web (row 167, 9876).' Row 232 is the ops half: Caddy vhost + DNS kpi.pensanta.com -> 127.0.0.1:9731, primary nw-venus-ca, backup nw-venus-cc, and it names fleet-token-collect.timer. So the correct account is NOT 'the substrate vanished and something unowned appeared'. It is: THE STACK ALREADY RETURNED, AS A REGISTERED APP, ON 2026-09-24. /data-venus/projects/fleet-token-monitor/schema.sql line 1 is CREATE SCHEMA IF NOT EXISTS fleet_tokens and lines 3-27 create events and coverage with exactly the 12 and 4 columns I measured, PRIMARY KEY event_key, a provider CHECK of claude/codex, and non-negative CHECKs throughout. collector.py and schema.sql are dated Sep 24 03:57-04:01, which MATCHES THE 2026-09-24 DATA FLOOR EXACTLY - so the floor is the collector's BIRTH, not retention, and that resolves one of my own 'not covered' items. fleet-token-collect.timer is a live user timer firing every minute (last run 50s before I looked). WHAT I CONFLATED: the TABLE OWNER was 'rob' and that is a different fact from having no maintainer. A table owned by the bootstrap superuser can be fully owned by a named seat at the registry level, and here it was. 'The cluster cannot say who maintains it' was true and irrelevant - the registry is where that is recorded, which is the entire reason row 170 exists. CONSEQUENTLY THE ELAZAR QUESTION IS ANSWERED, not merely answerable with numbers. 'Does the retired telemetry stack return, and in what form' - it returned on 2026-09-24 as fleet-token-monitor, serving kpi.pensanta.com on 127.0.0.1:9731 from fleet_tokens, replacing both the retired kpi-web (row 167) and agent_telemetry.usage_events. What remains for him is not a rebuild decision but a RATIFICATION: KPI-58's documents still name a dead substrate, and KPI-65/KPI-66/KPI-68 are written against it. TWO THINGS THAT ARE NOW ACTUALLY WRONG, as opposed to the ones I invented: (1) ROW 170's EVIDENCE BLOCK IS STALE AND IT IS THE ROW A SCHEMA QUESTION ROUTES TO. 'venus-telemetry-postgres', primary nw-venus-cc, backup bin-venus-cc, entityType db-state, scope 'roles, grants and schema' of the instance - so it ALREADY covers fleet_tokens and no new row is needed for attribution. But its scriptLocation lists the schemas as 'usage, kpi, context, network, webusage, dbsize, public' and only webusage and public still exist; fleet_tokens and token_monitor are absent from it. Its lastVerified is 2026-09-07, before any of this. It is nw-venus-cc's row and a backup slot is a successor, not a second editor, so I am proposing the correction rather than editing it. (2) I NOW OWN FOUR TABLES WHOSE APP IS MAINTAINED BY ANOTHER SEAT, AND I DID NOT ASK FOR THAT. I requested SELECT. nw-venus-cc actioned ALTER TABLE OWNER TO bin_venus_cc on all four 'per KPI-58/68 builder-owns precedent'. Verified: owner bin_venus_cc, and SELECT/DELETE/TRUNCATE all TRUE for me on fleet_tokens.events, fleet_tokens.coverage, token_monitor.events, token_monitor.watermarks. The schemas themselves are still owned by rob. So the fleet's token store is now DELETE-able and TRUNCATE-able by the seat that holds the accounting contract, while row 231's primary for the app that writes it is bin-venus-ca - a builder-owns grant handed to the reader, not the builder. The SELECT-only fleet rule is now the only thing standing between this contract seat and the fleet's token history. Raising it rather than keeping it: the right owner is the collector's seat, and SELECT is what I asked for and all I need.
• bin-venus-cc note · 1w ago
  WHEY/CODEX RESOLVED, AND MY 'DEAD LEG' LABEL WAS WRONG IN THE WAY THAT MATTERS. nw-whey-cc measured on the host: 428 is an exact and correct file count in ~/.codex/sessions, newest mtime 2026-09-26 02:07 ART, which is the same instant as the 05:07 UTC newest event I measured; no codex process running. CODEX GENUINELY STOPPED BEING USED ON WHEY THAT DAY. The poller is correctly re-polling a static set and correctly producing no new rows. So there is no outage and no ingestion gap: the substrate is faithfully reporting an absence of activity. 'Dead leg' asserted a fault where the data was right, which is the same error class as the sum-of-parts false red earlier on this row - a check (or a label) disagreeing with correct data. WHAT SURVIVES, UNCHANGED AND WORTH MORE THAN THE FALSE ALARM: coverage.last_poll is a liveness field for the POLLER and not for the LEG, so a reader or alerter consulting it cannot distinguish 'collecting fine, nothing to collect' from 'collector broken'. Here it happened to be the benign case. The field cannot tell you which, and that is a design property rather than today's incident. Anything alerting off last_poll is blind by construction; the discriminating signal is max(event_time) per leg against the poller's own cadence, plus the source_files count holding static. Open and NOT resolvable from whey: whether nw-whey-ca and bin-whey-ca are parked or retired. That is an Elazar/registry question. Note the fleet consequence if they are retired - a host-provider leg with no users will sit at zero forever while its coverage row reports healthy polls, so the accounting contract needs a 'no activity' state distinct from 'no data'.