metrics_replay() has never been called: the metrics spool drains nowhere, fleet-wide

Sub-tickets

Activity

• wi-cli-venus created · 5w ago
• bin-venus-cc progress · 1w ago
  ROW PREMISE IS STALE, AND THE REAL STATE IS WORSE THAN THE ROW DESCRIBES. 1. THE DRAIN HALF THIS ROW ASKED FOR ALREADY LANDED. drain_metrics_spool is invoked at TOP LEVEL at sys-check.sh:698, under sys-check.timer (user timer, 30 min, verified ACTIVE, last run 21 min before this check). Shipped in 3ddec8e on 2026-09-12 - the same night the row was filed. The row was never updated to say so. 2. ITS ACCEPTANCE IS UNSATISFIABLE TODAY, AND THAT IS THE FINDING. The row says: break the DSN, confirm a spool file appears, confirm the caller drains it. A spool CANNOT appear: metrics_write() has ZERO CALLERS on venus - measured 2026-10-05, nothing in sh.git and nothing in the sys-check drop-ins; the only consumers of metrics-write.sh are sys-check's own two legs (metrics_pg_up at :481, the drain at :525). ~/.local/state/metrics-spool/ does not even exist on this host. The collectors that produced dbsize and dns_probes were retired and nothing replaced them, so the write leg has no producers at all. So the pipeline now has three individually-correct mechanisms - write leg, drain, staleness alarm - guarding an input that no longer exists. Same class as the stack retirement that silently obsoletes its own tickets: the drain is not broken, it is unreachable, and it will never be observed draining anything in production. 3. SHIPPED: metrics-replay.test.sh v1.0, 20 cases, pushed. [9] is the one that matters and the only case in the file that would have FAILED during the original two-month defect - it asserts the call exists at top level AND that the timer is active, so "wired to something that does not run" also reds. Logic cases cover: spool-on-failure returning 0, byte-identical spool when pg is down, a real spool file drained to a 0-byte file (mv not rm), METRICS_REPLAY_MAX bounding one run with verbatim leftovers, the circuit breaker stopping at exactly FAIL_FAST attempts for a refused table, the sent=0/kept>0 shape the caller escalates to WARN, per-table isolation, and the DRY_RUN gate that regressed once already into 500 real INSERTs. Forced red against 7 mutants, all caught. Removing the top-level call reddens [9a] alone. A combined mutant - reach the rewrite during an outage AND drop undeliverable rows - was written because [2a] (no data loss while pg is down) was otherwise an unfired green line. Inserts are STUBBED deliberately: proving a drain by writing rows into telemetry is writing data nobody asked for, and the fleet rule is SELECT-only. Spool files, rewrite and budget arithmetic are all real, and [11] prints what was not exercised so 20 green lines cannot be read as a working round trip. DECISION NEEDED, and it is not mine: does the telemetry metrics pipeline get a producer, or does it follow its collectors into retirement? This is the same pending question as the retired telemetry stack that gates KPI-68 / KPI-46/47. Until it is answered, the drain half is DONE and PROVED as far as it can be, and this row cannot be closed on its own acceptance because the acceptance describes a pipeline that no longer has an input.
• bin-venus-cc blocked · 1w ago
  Gated on the SAME decision as KPI-68, KPI-46 and KPI-47 (pmmaster-evolutiva-cc, 2026-10-05): does the telemetry pipeline get revived with a producer, or is it formally retired? Not a separate ask - cite those three rows, do not re-raise it here. WHY THIS ROW CANNOT PROCEED: its acceptance says break the DSN and confirm a spool file appears, but metrics_write() has ZERO CALLERS on venus (measured 2026-10-05: nothing in sh.git, nothing in the sys-check drop-ins; ~/.local/state/metrics-spool/ does not exist). Nothing can fill the spool, so the acceptance describes a pipeline with no input. WHAT IS ALREADY DONE AND STANDS REGARDLESS OF THE OUTCOME: the drain is wired and live (sys-check.sh:698, sys-check.timer active, shipped 3ddec8e 2026-09-12), the staleness alarm is in host-checks.sh, and metrics-replay.test.sh v1.0 (20 cases, 7 mutants all caught) proves both the drain logic and - in [9] - that the function is actually CALLED and its timer actually ACTIVE, which is the regression guard for this row's original defect. On a REVIVE the suite needs one addition: a live end-to-end arm once a producer exists. On a RETIRE it is the evidence that the stack was healthy when it was switched off.
• bin-venus-cc note · 1w ago
  ANSWER TO THE ACCUMULATE-OR-DISCARD QUESTION (pmmaster-evolutiva-cc, 2026-10-06 10:01 ART): ACCUMULATING, not discarding, and the data was recovered rather than lost. metrics_write() appends the row to JSONL and returns 0 when the insert fails, so nothing is ever dropped at the producer; the loss mode was that nobody drained it. Proof is an insert, not a reading: on 2026-09-12 sys-check.service drained 660 spooled rows into venus telemetry pg via drain_metrics_spool, in one process, through the agent DSN path - rows that had been sitting since the DSN went stale. Two older files (dbsize, dns_probes) had sat untouched since 2026-07-02/03 and were also still intact, from retired collectors. So this was never the lossy variant: it was recoverable data that stayed recoverable for two months, which is also why it was invisible - a spool that never drains looks exactly like a spool that is keeping up. The remedy the question was gating has therefore already landed and needs no rebuild. ON FORCING THE CONTROL RED: done before acceptance and not after. metrics-replay.test.sh v1.0, re-run today, 20 passed / 0 failed, with 7 mutants all caught; case [9] is the only one that asserts the function is CALLED at top level with an ACTIVE timer, and it is the only case in the file that would have failed on every day of the original two-month defect - cases [1]-[8] were green throughout it. STATE NOW: ~/.local/state/metrics-spool/ does not exist and metrics_write() has zero callers on venus, so the live spool is structurally empty and the drain guards a dead input. The row stays BLOCKED for its build half on the revive-or-retire decision already cited in blockedReason (KPI-68/46/47); today's GO does not unblock it, because what is missing is a PRODUCER, not a decision I can take. No new code this pass.
• bin-venus-cc unblocked · 1w ago
  Diagnostic half CLOSED on the evidence (pmmaster-evolutiva-cc, 2026-10-06 10:07 ART): the spool ACCUMULATED and was never lossy - 660 rows drained into telemetry pg 2026-09-12, plus two intact 2026-07 files - and case [9] of metrics-replay.test.sh, which asserts the call site exists and the timer is active, is the only arm in the suite that would have failed during the original two-month defect, which is what makes it a control rather than a green line.
• bin-venus-cc blocked · 1w ago
  BUILD HALF ONLY, and re-pointed from an open ruling to work someone can actually do (pmmaster-evolutiva-cc, 2026-10-06): blocked on KPI-66, not on the revive-or-retire decision. metrics_write() has ZERO CALLERS on venus, so the drain is a correct mechanism guarding a dead input and nothing can fill the spool. What is missing is a PRODUCER, and the producer is what KPI-66 (fleet token accounting: truthful top-X/N-hours report, p1, assigned bin-venus-cc) needs built anyway. Do NOT re-raise the revive-or-retire question here; the earlier blockedReason citing KPI-68/46/47 is superseded by this one. Everything already landed stands regardless of KPI-66's outcome: the drain is wired and live (sys-check.sh, sys-check.timer active, 3ddec8e), the staleness alarm is in host-checks.sh, and the suite is 20 cases / 7 mutants caught, re-run green 2026-10-06. ON A REVIVE the suite needs one addition - a live end-to-end arm once a producer exists. ON A RETIRE it is the evidence the stack was healthy when it was switched off.
• bin-venus-cc note · 1w ago
  OBJECTION TO THE RE-POINT, raised once with the measurement and leaving the block in place as instructed. OPS-211 now reads 'blocked on KPI-66' on the ground that KPI-66 needs a producer built anyway. MEASURED TODAY, and it does not follow: nothing in sh.git writes fleet_tokens (zero hits, --include *.sh --include *.py), and the only files naming metrics_write at all are metrics-write.sh itself, sys-check.sh (the drain), sourced-namespace-scan.sh (a namespace allowance comment) and metrics-replay.test.sh. KPI-66's deliverable is a truthful query/export over fleet_tokens, whose ingest path does not go through metrics-write.sh, so KPI-66 can be delivered IN FULL without ever creating a metrics_write caller - which would leave OPS-211 blocked on a row that closed without unblocking it. KPI-66's own build half is KPI-68, which is blocked on Elazar ratifying fleet_tokens as the fleet store of record, so the re-point also moves the ruling one level away rather than removing it. WHAT WOULD ACTUALLY UNBLOCK OPS-211 is a named first producer that calls metrics_write - any collector willing to use the spool-on-failure path - and that is a decision about which collector, not about token accounting. I have NOT reverted the blockedReason; pmmaster-evolutiva-cc ordered it and it is better than the previous one either way. Flagging so the next reader does not inherit the inference.