EPIC: agents-efficiency-KPI - per-agent context/token efficiency report Elazar actually sees (Elazar 2026-07-02: 'never seen a report on that'). Inputs: usage polls + context samples (venus PG). Deliverable: recurring per-agent report (tokens/turn, cache-read overhead, context growth, compact frequency) surfaced to Elazar. Coordinate: bin-whey-cc (one-db data), coder-llmmsgsrv-cc (kpi views/report script)

Sub-tickets — 0/2 done

Activity

• wi-cli-venus created · 2026-07-02
• wi-cli-venus commented · 2026-07-02
  PM calls 2026-07-02: (1) lezama EXCLUDED from efficiency-KPI v1 (no cc-context-monitor by design, no PG egress; 200k maxTokens cap is its budget mechanism) - exclusion stated in report so it reads as deliberate. (2) kpi-web dead-view panels: repoint-or-retire folds into this epic's spec, bin-whey-cc owns kpi-web. (3) whey telemetry-retention.timer pruning retired tables: harmless, left as-is.
• wi-cli-venus note · 5w ago
  ELAZAR GO relayed by pmmaster-evolutiva-cc 2026-09-11 20:22 ART, P1, mine. This is the one Elazar actually asked for - backlog since 2026-07-02 carrying his own quote, 'never seen a report on that'. Per-agent efficiency report: tokens/turn, cache-read overhead, context growth, compact frequency. DELIVER A FIRST VERSION OFF WHAT EXISTS TODAY - explicitly do NOT wait on bin-venus-cc's model/effort capture; that capture improves a later version. Coordinate bin-whey-cc for one-db data, coder-llmmsgsrv-cc for the views. Sizing context measured today: 97.7% of all tokens billed fleet-wide over 24h is cache_read, ~166k re-read per turn across ~10,100 turns - so the report's headline metric is turns and standing-context size, NOT message payload.
• wi-cli-venus note · 5w ago
  DATA SOURCES ANSWERED by bin-whey-cc 2026-09-11 20:25 ART, read from schema + collector source in sh.git ~/.local/bin/one-db/ (whey has no PG reach, so row counts remain unconfirmed on venus). Two schemas, and my four questions split across BOTH: agent_telemetry (usage_events, per agent per hour) and context (samples, per agent per session per sample, written by context-monitor on all three hosts). 1. TURNS - exists but is a trap on the obvious column. usage_events.response_count is a REAL turn count only on source_kind claude_sqlite_hourly (collect.py:217). On codex_token_count it is the literal 1 per event (collect.py:398), so summing it counts records, not turns. A claude_rows_daily path emits response_count=0 and its own docstring calls that an ABSENT field, not a measured zero (collect.py:140) - must not be allowed to average in. BETTER SOURCE for this report: context.samples.cum_turns + cum_tool_calls, cumulative per session, fleet-wide. Turns-per-window is a DELTA on cum_turns, not a SUM. 2. CACHE_READ VS NET-NEW - exists fully, the only question with no caveat. usage_events has input/output/cache_write/cache_read/reasoning/total, all NOT NULL, keyed agent_name + hour_bucket + host. Net-new = total_tokens - cache_read_tokens, which is what fleet-tokens.sh already ranks on. context.samples carries the same split cumulatively as cum_new_tokens / cum_cache_read if per-session is wanted. 3. CONTEXT SIZE PER TURN - exists, NOT where I would have looked. usage_events.context_used_tokens and context_window_tokens are effectively empty for claude: the collector populates context_window_tokens only on the codex branch (collect.py:391) and bin-whey-cc found NO assignment to context_used_tokens on any branch. Real source is context.samples: ctx_tokens and cap_tokens per sample with over_cap, idle_min, model, session_id. That is a DIRECT measurement of standing context, not a proxy - it is the series the headline needs. 4. COMPACT EVENTS - no event source exists; no compact ledger anywhere in sh.git (ops.compact_ledger appears only as an unbuilt plan in bin-whey-cc's memory). What DOES exist and is already wired: context.v_kpi_compact_7d (schema.sql:397), served by kpi-web/server.mjs:163, which its own comment calls the compact-frequency proxy - gaps-and-islands collapsing each maximal run of consecutive over_cap=1 samples in a session into ONE episode, per agent, 7d, no new ingest. STATED LIMIT I MUST CARRY INTO THE REPORT: it counts OVER-CAP EPISODES, not compacts. An episode that drained without a compact counts; a compact taken below the cap does not. Do NOT relabel it 'compacts' in a report Elazar reads. FOUR SKEWS TO GUARD IN ANY PER-AGENT RANKING: - WHEY AGENTS READ AS ZERO. usage_events has always held zero rows for whey/claude and zero for lezama/codex; expected_coverage records this as DECLARED-expected, not a gap. But a per-agent table built on usage_events silently OMITS every whey agent, and an omission reads as innocence. Must be shown as no-coverage, never as a good score - same failure shape as the empty-sentinel rule. - CODEX ATTRIBUTION IS DEAD FLEET-WIDE. roster_claims holds only live rows and prunes what it stops seeing, and the attribution join is unbounded, so an agent leaving retroactively un-attributes everything it ever burned. Codex per-agent numbers are not rankable. - QUOTE coverage.unattributed_net_new, NEVER unattributed_total - the latter is inflated ~14x by cache reads (92,796,944 published vs a true 6,674,704 at last check). - KEY GRAIN ON source_kind, NEVER provider. Not interchangeable; collector_runs is unique on (run_key, host, provider, source_kind) for that reason. OPEN, RAISED BY bin-whey-cc, NOT YET CHECKED: usage_events.source_kind has CHECK (source_kind IN ('claude_sqlite_hourly','codex_token_count')) but collect.py:163 emits 'claude_sqlite_daily'. Either the live constraint differs from the file in sh.git, or that path's rows are being REJECTED. One SELECT DISTINCT source_kind settles it and it must happen before v1 is built on top. BLOCKER ON THAT CHECK: no /home/rob/.config/postgres/pm-llmmsgsrv-cc.env exists, so I hold no PG role. Per the fleet credential rule I am not reusing another agent's env. Provisioning requested from nw-venus-cc 2026-09-11. Not blocking the report DESIGN, which proceeds off the above.
• wi-cli-venus note · 5w ago
  SOURCE_KIND CHECK RESOLVED by nw-venus-cc 2026-09-11 20:28 ART - NOT a defect on venus, and my framing of it was wrong in a way worth recording. FINDINGS: SELECT source_kind,count(*) FROM agent_telemetry.usage_events GROUP BY source_kind = codex_token_count 84728, claude_sqlite_hourly 2073, ZERO claude_sqlite_daily. Live CHECK constraint matches sh.git exactly, verified via pg_get_constraintdef - no constraint drift. MY FRAMING WAS WRONG: I asked whether rows were being 'silently rejected'. They could not have been. collect.py's apply() runs the whole batch as ONE transaction via psql -v ON_ERROR_STOP=1 plus subprocess check=True, so a CHECK violation aborts the ENTIRE transaction - no partial insert, no collector_runs row either (same tx) - raises CalledProcessError, non-zero exit, systemd unit goes FAILURE. That is LOUD at the systemd/journald layer. It only looked silent because I was reasoning from a PG-only read, where a failed transaction leaves nothing to see. Venus journal over 14d shows zero CHECK-violation failures and the unit is Result=success. LESSON, same class as the empty-sentinel rule: absence of rows in the destination is not evidence of the write path's behaviour - the error surface for an aborted transaction is the SUPERVISOR, not the database. ACTUAL ROOT CAUSE of the zero rows, and it is benign: claude_rows_daily()'s source_kind='claude_sqlite_daily' path (collect.py:158-163) only fires when a host's local sqlite has dailyUsage but NOT usage_hourly (collect_claude_with_retry:227-231). Venus has usage_hourly, so the branch is dead on venus. Nothing is being lost here. NOT EXERCISED, carry into v1's coverage statement: whether whey or lezama ever hit that dailyUsage-only branch is UNVERIFIED. nw-venus-cc cannot check - host-bound DB rule, no filesystem reach. If row counts for those hosts matter, ask nw-whey-cc / nw-lezama-cc whether their local claude-usage sqlite has ever been dailyUsage-only, or have them grep journald for the same CalledProcessError signature. Note this compounds the KNOWN whey-reads-as-zero skew already recorded on this WI - do not let the two be mistaken for one finding. CREDENTIAL UNBLOCKED: ~/.config/postgres/pm-llmmsgsrv-cc.env provisioned (600), role pm_llmmsgsrv_cc, SELECT-only, live-tested before handoff - USAGE+SELECT on agent_telemetry and context (all tables/views), write denied and confirmed by a no-op DELETE returning permission denied. Standard PGHOST/PGUSER/PGPASSWORD/PGDATABASE shape. SAVETOKE-33 is no longer blocked on data access.
• wi-cli-venus note · 5w ago
  WHEY HALF ANSWERED by bin-whey-cc 2026-09-11 20:36 ART, and the mechanism is a THIRD thing - neither the constraint rejection I suspected nor the dailyUsage-only branch nw-venus-cc identified. Three findings now, and they must not be collapsed into one. WHAT WHEY ACTUALLY DOES: whey's local Claude sqlite IS dailyUsage-only (~/.local/share/claude-usage/claudeUsage.sqlite, tables dailyUsage/usagePolls/usageAlerts, no usage_hourly), so on the table test alone it would take claude_rows_daily(). It never reaches the test. The unit runs collect.py with NO --claude-db, so it uses the argparse default ~/.local/share/token-usage/token-usage.sqlite, WHICH DOES NOT EXIST ON WHEY. The open raises OperationalError before any table inspection and lands in skipped_absent (collect.py:723-730), a branch that HARDCODES source_kind='claude_sqlite_hourly'. So whey has never emitted claude_sqlite_daily and never will while that default stands. My zero count is therefore correct FLEET-WIDE, not merely on venus. THE FINDING THAT MATTERS FOR THIS REPORT, and it is worse than a missing-rows bug: whey's claude telemetry has never been collected at all, and expected_coverage DECLARES whey/claude zero rows as expected. A coverage declaration saying 'expected zero' is exactly what made a misconfiguration invisible - the system records the symptom as intended behaviour. This is the same class as the empty-sentinel rule and as the whey-reads-as-zero skew already on this WI, now with a confirmed cause rather than a declared one. CONSEQUENCE FOR SAVETOKE-33 v1, binding on the design: whey agents must render as NO COVERAGE, never as a zero or a good score. A per-agent efficiency ranking that shows a whey agent at the top would be reporting a broken collector as excellence. Doubly confirmed now - the host has no live source AND the collector points at the wrong file. SECOND-ORDER, do not 'fix' by repointing: claudeUsage.sqlite has mtime 2026-06-06, frozen three months. Pointing --claude-db at it would backfill a DEAD file, producing plausible-looking history that stopped in June. The honest state is no-source, and the report must say so rather than acquire numbers. PROVENANCE OF THE BUG: collect.py's own header (lines 117-122) already documents this exact path confusion - naming claudeUsage.sqlite as present on whey and token-usage.sqlite as absent there - and the argparse default at :694 still points at the absent one. The comment was written and the default was never corrected. Classic case of documentation recording a hazard that the code then walks into. MY DISCRIMINATOR CONFIRMED, negative result: bin-whey-cc grepped whey's journal, exactly one CalledProcessError in 30d (2026-09-09 23:09:36, user-scope agent-telemetry-collect.service) and it is 'psql: could not translate host name venus.pensanta.com' at 102s uptime - the collector racing DNS at boot, exit 2. Not a CHECK violation. Unit Result=success since. So 'an aborted transaction surfaces at the supervisor, not the database' holds, and whey shows zero instances of that failure mode. Incidental: a collector racing DNS at boot is its own small defect in bin-whey-cc's lane, flagged to them, not tracked here.
• wi-cli-venus note · 5w ago
  v1 DELIVERED: /data-venus/projects/savetoke/reports/2026-09-11-agent-efficiency-v1.md - 47 agents, 7d window, per-agent netnew/cache%/turns/nn-per-turn/ctx p50-p95/pct-over-cap, from usage_events + context.samples. SUPERSEDES the 2026-07-02 exclusion of lezama: that call cited 'no cc-context-monitor by design, no PG egress'. First half holds (context.samples has NEVER held a lezama row). Second half is false - lezama claude usage_events are present and current, 5 agents/24h, and lezama holds the #1 and #2 net-new burners fleet-wide (coder02-mba-l-cc 18.76M, db02-mba-l-cc 13.45M). Lezama is rankable on TOKENS, not on context. Exclusion is now per-metric, not per-host. WHEY renders NO COVERAGE per the collector defect (bin-whey-cc): unit runs collect.py with no --claude-db, hits the absent argparse default, lands in skipped_absent which hardcodes source_kind=claude_sqlite_hourly; whey claude telemetry never collected, and expected_coverage DECLARES that zero as expected. Do NOT fix by repointing --claude-db: whey claudeUsage.sqlite mtime 2026-06-06, frozen 3 months. HEADLINE FINDING: over-cap is bimodal and tracks CAP SIZE not token volume - bin-venus-cc 9.58M netnew at 3.6% over a 170k cap vs coder02-mars-cc 5.32M at 96.6% over a 160k cap. That column measures the COMPACTION ACTUATOR, not the agent: a p50 standing context of 281,463 against a declared 160,000 cap means auto-compact is not firing at threshold. Reported as an actuator defect list, not an efficiency score. monitor-context-cc (the actuator) is itself at 16.3%. CAVEAT ON cap_tokens, unverified: samples report 170000 for pm-llmmsgsrv-cc uniformly across 3 model values while this host settings.json says AUTO_COMPACT_WINDOW 250000 and my identity memory says 160000 - three sources, no agreement, so pct_over is COMPARATIVE only until someone reads context-monitor.sh. Flagged to monitor-context-cc. NOT COVERED in v1: model/effort split (waits on bin-venus-cc capture, per the GO); true compact events (nothing in either table records one - v_kpi_compact_7d counts EPISODES and is named misleadingly).
• wi-cli-venus audit-finding · 5w ago
  DEFECT in context.v_kpi_compact_7d (schema.sql:397, served kpi-web/server.mjs:163, owner bin-whey-cc): the view does not filter sample_flag. A zero_read sample is a FAILED READ sentinel - ctx_tokens min=avg=max=0 across all 372 in 24h, vs 60,011..558,454 on real samples - so it carries over_cap=0 unconditionally and a failed read inside a sustained over-cap run SPLITS one gaps-and-islands episode into two. Measured 7d: view reports 196 episodes, 190 with zero_read excluded = 6 split artifacts, 3.1% inflation. Exposure is real: of 37 agent/session pairs in the window, 21 carry zero_read samples and ALL 21 also carry over-cap samples. 24h rate 372/2147 = 17.3%. FIX is one clause in the s CTE: AND coalesce(sample_flag,'') <> 'zero_read'. SECOND artifact in the same table, for monitor-context-cc not bin-whey-cc: 4 agents (audit-pluto-cc 492 samples/5d, audit-enamel-cc 231, coder02-pluto-cc 39, db02-pluto-cc 39) emit samples in which ctx_tokens has exactly ONE distinct value and cum_turns never moves - the monitor re-emits a stale reading of a dead session rather than recording it stopped. The frozen-sample set and the zero-turn-delta set are IDENTICAL, which is what identifies the artifact. Their sample counts read as coverage while carrying no information. Verified separately and clean: cum_turns NEVER regresses within an (agent,session_id) pair over 7d, zero violations, so the max-min turn delta itself is sound.
• wi-cli-venus note · 5w ago
  v1 re-ranked and republished. The tier framing in the first two drafts was wrong twice: (1) %over_cap / ROLE_CAP is a per-role COST budget, not a context ceiling; (2) the NUDGE_AT 190k / COMPACT_AT 220k / QUARANTINE_AT 250k constants are in context-monitor.sh, which SAMPLES and alerts every 15min and does not actuate. The sole actuator is tick.py (context-venus-tick.service, 30s), whose scaled_tiers(window) scales per agent: 170k/200k at a 250k window, 294k/320k at 400k. monitor-context-cc confirmed 294k/320k against event_log (mars seats compacted 8-9x over 7d). Re-ranked against each agent's own tier, the max share above the compact tier anywhere in the fleet is 4.0%; the 'five agents above the line' finding is WITHDRAWN in full. cap_tokens/%over_cap columns removed from the per-agent table. Fleet runs two windows (31 on 250k, 6 on 400k) and the 400k seats are invisible to both settings.json and the model label - window_tokens or the live statusline is the only read. Open with monitor-context-cc: coder02-mars-cc peak 558454 = 140% of a 400k window (same shape on bin-venus-cc and on 9 of my own samples), likely sample integrity, not established.
• wi-cli-venus note · 5w ago
  ANOMALY RESOLVED + report corrected again. monitor-context-cc confirmed the 558,454 over-window peak is the FROZEN-SAMPLE artifact of finding 5, not a real context and not sample-write corruption: 3 agents / 8 instances, signature = identical ctx_tokens across consecutive polls with cum_turns frozen, then a real drop. That collapses two findings into one defect with two symptoms that look unrelated - an impossibly large number in the context column and a zero in the turn column - because a frozen sample keeps ASSERTING a stale context size, not merely failing to record new work. Report updated: finding 3 no longer says unexplained, finding 5 carries the wider blast radius, the finding-2 peak column is captioned show-for-shape-only (6 rows exceed their own window; the two share columns are unaffected and the 4.0% ceiling holds since a frozen row sits at one fixed value and can only push a share up), and the coverage statement now says the frozen samples are IDENTIFIED not root-caused, with an unquantified stale-row share whose direction is known and magnitude is not. Root cause stays in monitor-context-cc's lane, untracked here. ALSO added this pass, from review: a 'win' column (each agent's own window from context.samples.window_tokens; '-' = no context coverage) to the 47-row per-agent table, because p50/p95 ctx were sitting side by side across two window classes and invited exactly the comparison finding 3 declares invalid - a 231,624 p50 is 58% of a 400k seat and 93% of a 250k one. Legend now says read win before comparing two rows. Table verified uniform at 12 fields x 48 rows after the awk edit. wi note of 23:59:50 verified present by direct read, not inferred from silence.
• wi-cli-venus decision · 5w ago
  ONEDB-20 (bin-whey-cc, p1) UNBLOCKED and cleared to patch context.v_latest. monitor-context-cc answered the gating question: v_over_cap is consumed at the KPI/report tier ONLY. tick.py (sole compactor, OPS-24) reads monitor-context.sqlite context_samples/agent_state directly with no pg; context-monitor.sh's over-cap alert reads its own in-memory samples list and writes OUT to pg context.samples but never reads the view back. So a session whose newest sample is a failed read dropping off v_over_cap is a real reporting defect but silences nothing in the actuator path, and bin-whey-cc's gauge-that-warns-by-disappearing objection to patching v_latest does not apply. Relayed to bin-whey-cc. STILL OPEN, different owner: sh.git 91eb3e6 (one-db/schema.sql, v_kpi_compact_7d + AND ctx_tokens <> 0) needs a superuser re-run on venus PG - bin-venus-cc's, idempotent, and the KPI serves inflated episode counts until it lands.
• wi-cli-venus decision · 5w ago
  v1 correction: the 4.0% fleet ceiling above the compact tier is WITHDRAWN. The finding-2 denominators are contaminated by the finding-5 frozen re-emissions, and my published bias direction (a frozen row can only push a share UP) is wrong: a frozen value below the tier dilutes the share. Deduplicating consecutive identical (ctx_tokens,cum_turns) runs shrinks denominators 2x-34x (coder-apiimages-cc 672 samples/15 distinct; audit-pluto-cc 492/1) and moves the ceiling to 20.0%. Neither raw nor dedup is a share of time and the sampler cannot produce one; the only real compaction counts in the report are monitor-context-cc's event_log figures (mars 294k/320k, 8-9 compacts each 7d). Report updated in place: finding 2 table now carries raw+dedup columns, peak ctx dropped, the 55-70% median band withdrawn (its upper edge was coder02-mars-cc's frozen 281,463), and the per-agent p50/p95/n legend now states they are raw and NOT de-weighted. Verdict that survives: the actuator fires correctly, so context pressure is not this fleet's token cost - cache read at 97.7% is.
• wi-cli-venus decision · 5w ago
  Finding 2 re-reported in COUNTS, not shares. The 20.0% dedup ceiling published in the prior pass is WITHDRAWN: it was 1 observation out of 5 (coder02-enamel-cc), the smallest non-zero value that series can express; coder03-terra-cc's 16.7% was 1/6. Dedup removed the re-emission contamination but shrank denominators into single digits, where a percentage is a quantization artifact. Table now prints integer counts for all 22 agents with >=1 observation at or above their own compact tier (the prior table was also truncated at 15 rows and omitted bin-venus-cc, coder-mars-cc, coder-pluto-cc, nw-venus-cc, coder-llmmsgsrv-cc, db-mars-cc, coder-apiimages-cc). Fleet ceiling = 9 observations (db-terra-cc, of 60 deduped readings, 7d) and is deliberately NOT converted to a rate. Ranking inverts under counts: db-terra-cc 9/60, bin-venus-cc 7/110, pm-llmmsgsrv-cc 4/120 carry the evidence; a 1/5 row carries none. Added an explicit coverage statement: actuator firing confirmed against event_log for the mars 400k seats ONLY (coder-mars-cc, coder02-mars-cc, db-mars-cc); the 250k seats topping the table are unchecked, so 'the actuator cleared these contexts' is an inference for them. That check is one query on monitor-context-cc's surface, queued for v2. Report republished in place at /data-venus/projects/savetoke/reports/2026-09-11-agent-efficiency-v1.md.
• wi-cli-venus statusChanged · 4w ago
• wi-cli-venus note · 4w ago
  v1 delivered; the unanswered half is now tracked as SAVETOKE-35 (v2) rather than carried in a PM's head: the event_log coverage check for the 250k seats, the standing ban on re-deriving any share from context.samples, the still-absent model/effort capture, whey's NO-COVERAGE rendering, and the recurring-delivery leg that is the epic's actual goal and has not been started.