KPI-58 retire the WEB/EXPORT read of token-usage.sqlite + verified snapshot (collector stays)

Sub-tickets

Activity

• wi-cli-venus created · 6w ago
• wi-cli-venus parentChanged · 6w ago
  parent=KPI-58
• wi-cli-venus note · 6w ago
  COUPLING with KPI-62: collect.py's claude leg reads token-usage-collect.sh's usage_hourly schema. Retiring the collector leg is fine, but confirm collect.py no longer depends on that table before the delete commit, or venus's own claude ingestion into usage_events stops and the pg-only dashboard goes blind on the host it was proven on.
• wi-cli-venus decision · 6w ago
  MEASURED, changes KPI-61 from formality to critical: nw-venus-cc 07:31Z reports usage_events backward coverage claude/venus 2026-08-25->now (536 rows), claude/lezama 2026-08-26->now (283 rows), codex/venus 2026-06-06->now (24263 rows). The CLAUDE legs are only ~8-9 days deep. token-usage-collect.sh's sqlite has run since 2026-07-14, so ~6 WEEKS of claude history exists ONLY in that sqlite. RULING (monitor-context-cc, KPI-58 policy): the VACUUM INTO snapshot to /data-venus/kpi/2026-09-03-token-usage-pre-pg-cutover.sqlite is MANDATORY and must be verified (row count + date range read back) BEFORE the collector is disabled, and the sqlite file must not be trashed until the snapshot is confirmed readable. Never plain cp - WAL sidecar.
• wi-cli-venus descriptionChanged · 6w ago
  was: Elazar ruling: no co-existence. Step 1 same commit as web cutover: disable the collector timer, script exits non-zero with reason if invoked. Step 2: VACUUM INTO snapshot to /data-venus/kpi/2026-09-03-token-usage-pre-pg-cutover.sqlite (only pre-cutover history; usage_events backward coverage unverified). Step 3, separate commit after 1 day on pg: delete script + sqlite (trash). Do not delete artifact and evidence in the same change.
• wi-cli-venus decision · 6w ago
  Snapshot stays mandatory under the reduced scope, but its reason changes: it is no longer a deletion safeguard (nothing is deleted), it is the citable pre-cutover baseline. bin-venus-cc's earlier correction stands and sharpens it - the six weeks of claude history is in usage_daily (925 rows, 2026-07-14 to 2026-09-03, 52 days, 61 agents, 2.04e10 tokens), a table collect.py has NO leg for, while usage_hourly is only 321 rows from 2026-08-31 and usage_events claude/venus already reaches DEEPER (2026-08-25). Backfill of usage_daily ruled IN SCOPE as a separate item under KPI-58, not here.
• wi-cli-venus descriptionChanged · 6w ago
  was: SCOPE REDUCED by the (b) ruling 2026-09-03 08:03Z. KPI-61 is now: (1) retire the WEB/EXPORT read of token-usage.sqlite - achieved when KPI-60 lands and the three sqlite3.connect sites are deleted; (2) take a VACUUM INTO snapshot to /data-venus/kpi/2026-09-03-token-usage-pre-pg-cutover.sqlite BEFORE the cutover and read it back verifying row count plus min/max date range (never plain cp - WAL sidecar). NOT in scope any more: disabling token-usage-collect.sh, deleting the sqlite, the two-commit retire-then-delete shape. The collector keeps running as collect.py's staging source on venus. The snapshot is kept because it is the only citable copy of the pre-cutover numbers if the pg figures are disputed.
• wi-cli-venus titleChanged · 6w ago
  KPI-58 retire the WEB/EXPORT read of token-usage.sqlite + verified snapshot (collector stays)
• bin-venus-cc completed · 1w ago
  DONE, and the row's own ruling had already been violated before I reached it. THE SNAPSHOT WAS NEVER TAKEN AND THE SOURCE WAS TRASHED ANYWAY. KPI-58's ruling was explicit: the sqlite must not be trashed until the snapshot was confirmed readable, because ~6 weeks of claude history (2026-07-14 onward) existed ONLY there while usage_events reached back only to 2026-08-25. On 2026-09-24 03:08 the entire stack went to trash - collector, sqlite, web surface, units, and agent-telemetry/collect.py - and /data-venus/kpi/ did not exist until today. The only reason this row is closeable is that fleet policy is 'trash, never rm': the file was still in ~/.local/share/Trash/files/, integrity_check ok, with no orphaned -wal sidecar (its source directory no longer exists, so there was nowhere for one to be stranded). (1) SNAPSHOT TAKEN. VACUUM INTO, never cp. /data-venus/kpi/2026-09-24-token-usage-final-state.sqlite (462,848 bytes) plus a provenance sidecar .md beside it - an unexplained database on a shared mount is what a later cleanup deletes. Second copy at ~/gdrive/backup/kpi/, because /data-venus is NOT covered by venus-state-backup.sh and a 'preserved' copy on the same disk as the original preserves nothing. NAMED FOR WHAT IT IS, DELIBERATELY DEVIATING FROM THE ROW'S PATH. The row says 2026-09-03-token-usage-pre-pg-cutover.sqlite. That name would assert a pre-cutover state this copy does not hold - under ruling (b) the collector kept running for three more weeks, exactly as intended, so the file is the 09-24 FINAL state. The 09-03 baseline is still obtainable as a subset (WHERE date <= '2026-09-03' -> 928 rows / 52 days / 61 agents / 20,521,987,574 tokens), and that is a derived figure rather than a snapshot, which is why the file is not named for it. Flagging the deviation rather than silently picking either option. (2) VERIFIED BY READING THE SNAPSHOT BACK, not by exit 0. integrity_check ok; usage_daily 1295 rows, 2026-07-14 -> 2026-09-24, 73 days, 76 agents, 28,018,661,242 tokens; usage_hourly 381 rows, 2026-09-20T19 -> 2026-09-24T02. All identical to the source. total_tokens equals input+output+cache_creation+cache_read on all 1295 rows (0 disagreements), so the column and the components agree and either may be cited. AND CROSS-CHECKED AGAINST THIS ROW'S OWN 2026-09-03 MEASUREMENT, which is the part that makes the copy trustworthy rather than merely readable. The row recorded 925 rows / 52 days / 61 agents / 2.04e10 tokens for the <=09-03 window. This copy gives 928 / 52 / 61 / 2.0522e10. Days, agents and tokens agree; the +3 rows are the rest of 09-03, still being written after the 08:02Z measurement, and the PK is (date, project_path) so a project path first active later that day adds a row. A figure nobody could have derived from this file agreeing with it is a real control. Note the token figure only reconciles WITH cache tokens included - input+output alone is 8.8e9, 2.3x off - so a future reader citing 2.04e10 must include cache_creation+cache_read. (3) NOTHING READS IT, FOR AN ANSWER OR OTHERWISE. Swept with grep -r --binary-files=text (never bare grep - it is ugrep here and skips gitignored AND binary files at rc=0) for token-usage.sqlite, share/token-usage and usage_hourly across sh.git and all user systemd units: ZERO real references. The only sh.git hits for 'token-usage' are prose in comments (orphan-sweeper.sh, journal-digest.sh, git-history-deleted.sh) describing the retired collector's history. No unit names it. KPI-62's coupling warning - that retiring the collector would blind venus's claude ingestion into usage_events - is moot in both directions: collect.py is itself trashed (2026-09-15). WHAT I DID NOT COVER. - The gdrive copy is verified by an IMMEDIATE read-back through the rclone VFS, which can be served from local cache. That proves the write path, not durable remote residency. A cold re-read later would settle it; I did not force one. - I did not re-verify the Postgres side. Whether usage_events/usage_daily in pg now hold the pre-08-25 history is KPI-65's backfill, not this row, and this snapshot is precisely the artifact that backfill would read from. - The trashed source is LEFT IN PLACE as evidence. It is not a backup - trash has no retention guarantee - and the two verified copies are what this row delivers. - This row is another orphan of the 09-24 retirement: its steps (1) and (2) were genuinely still owed, while step (3) was satisfied by the retirement itself rather than by the KPI-60 cutover it was written against.