Monitor WAL file count + archiver failures - 34h of failed archiving went completely unseen

Sub-tickets

Activity

• wi-cli-venus created · 5w ago
• wi-cli-venus note · 5w ago
  ADD TO SCOPE - a FOURTH blind instrument, and this one was actively producing a signal nobody consumed. From nw-whey-ca 2026-09-10: Mars's scheduled pg_dump FAILED ON DB CONNECTIVITY three consecutive times - 2026-09-09 06:00 ART, 2026-09-09 18:01 ART, 2026-09-10 03:00 ART - over roughly 24 hours. Those failures are DISTINCT from the temporary timer pause and predate it. That is the one instrument that was NOT structurally blind. Unlike the health probe (SELECT 1 succeeds read-only), appEvents (logging is a write) and applog-pull (reads WARN rows only), a backup either completes or it does not, and it exercises a real connection. Three consecutive failures across 24h is exactly the alarm this incident needed, roughly a day before Supabase forced read-only. It fired into nothing. Sequence, which makes the case: WAL-G archiving broke 2026-09-08 23:46:49Z. The first backup failure follows ~9h later at 09-09 06:00 ART (09:00Z). Writes froze at 09-09 23:23Z. So a backup-failure alert would have led an operator to the instance while it was still WRITEABLE, when a restart would have cost nothing. REQUIREMENT: alert on N consecutive backup failures per app - not on a single one, which is noisy given MARS-698's duty cycle, but two or three in a row is a hard signal. evolutiva-backup@<app> already has OnFailure wiring in the fleet pattern; the gap is that nothing aggregates or escalates repeated failures. Note the known limit already recorded in memory: OnFailure is blind to a clean Stop, so a DISABLED timer produces no signal at all - a paused backup and a healthy one look identical from outside. Cover both: consecutive failures AND a timer that has stopped running. Latest valid artifact for the record: /gdrive/backup/evolutiva/mars/2026-09-10-0652-mars.dump, 13,981,935 bytes, completed 09:53:06Z - POSTDATES both the 09-09 23:23Z write freeze and the ~09:28Z restart, so Mars has a clean current backup. Timer re-enabled per Elazar; enabled+active, next run 2026-09-10 15:00:21 ART. No scheduled run was missed during the pause itself (disabled after the 03:00 trigger, re-enabled before 15:00).
• wi-cli-venus spec · 5w ago
  SECONDARY SIGNAL - a WEDGED archiver, and why the obvious version of it does not work. Measured with db-mars-cc, 2026-09-10. Mars post-restart archive rate: archived_count 27 at 09:43:17Z, 36 at 10:04:08Z = 9 segments in 20.85 min = ~25.9 segments/hour, at 07:00 ART on a THURSDAY with almost no user traffic. Treat that as an off-peak figure; a business-hours rate is higher. The tempting rule is 'alert if archived_count has not advanced in N minutes', to catch an archiver that is stuck WITHOUT recording a failure - the case failed_count cannot see. DO NOT SHIP THAT RULE ALONE. archived_count advances only when WAL is GENERATED, so on a genuinely idle instance a flat archived_count is correct behaviour, and any N small enough to catch a real wedge will page every quiet night. Zero writes and a wedged archiver produce an identical reading. USE FILE COUNT AS THE WEDGE DETECTOR INSTEAD - it is write-rate independent in the right direction. If archiving is stuck, pg_ls_waldir() count RISES whether traffic is high or low, because Postgres cannot recycle an unarchived segment; if the instance is merely idle, the count sits flat inside the healthy band. So the existing warn>50 / page>150 thresholds already cover the wedge case and no new rule is needed for it. Keep archived_count in the payload as CONTEXT, never as a trigger, and always alongside stats_reset per the description. If a rate-based trigger is still wanted later, the correct shape is a RATIO, not a count: alert when pg_ls_waldir() count is rising while archived_count is flat over the same interval. That is the only form that distinguishes idle from wedged, and it needs no calibrated N. Fleet baselines for pluto/venus/terra are routed to db-mars-ca (db-mars-cc holds Mars-only access).
• wi-cli-venus verified · 5w ago
  Fleet baseline half REROUTED. db-mars-cc correctly declined the pluto/venus/terra WAL/archiver baselines: the fleet-wide host-bound DB rule means a Mars lane never reaches into another project's database, so this was never assignable to the Mars DB agent regardless of who holds that lane. Dispatched instead as one read-only query to db-pluto-cc, db-venus-cc and db-terra-cc directly, all three verified live on the roster at dispatch time. Query sent: SELECT (SELECT count(*) FROM pg_ls_waldir()) AS wal_files, archived_count, failed_count, last_archived_wal, last_failed_wal, last_archived_time, last_failed_time, stats_reset FROM pg_stat_archiver. Two constraints travelled with it because both are counter-intuitive and either one silently produces a useless monitor: alert on WAL FILE COUNT and never the WAL:DB ratio, since min_wal_size sets a floor that dominates on a small database and moves the ratio on its denominator - terra sits at 6x while healthy, ABOVE pluto's 3.7x; and archived_count failing to advance is NOT a wedge signal, because it only advances when WAL is generated, so an idle database and a wedged one read identically. The conclusive tell is last_failed_wal being the segment immediately after last_archived_wal with failed_count climbing. Known-healthy counts carried as the comparison: venus 7 files, terra 9, pluto 16, Mars at failure 475. Mars post-restart archive rate is about 25.9 segments/hour off-peak.
• wi-cli-venus verified · 5w ago
  Fleet baselines IN, all four projects, plus a design correction to my own framing that would have produced a broken monitor. MEASURED 2026-09-11: mars wal_files=6, archived 630, failed 0, ~25/hr (matches the 25.9/hr off-peak reference), stats_reset 09-10 09:28:15Z. venus 7 files, archived 49835, failed 0, stats_reset 2026-04-30. terra 9 files, archived 1494, failed 0 since its 07-27 reset, last archived 3 minutes before the read so the archiver is demonstrably live rather than idle-looking. pluto 16 files, archived 65906, failed_count 11 - but last_archived 00000001000001010000002A at 10:25:02Z is far past last_failed 000000010000010000000094 at 01:17:58Z, a different segment with archiving current, which is NOT the wedge signature. All four healthy. THE CORRECTION, from db-terra-cc, and it is the actionable part: alerting on a single fleet-wide FILE COUNT is wrong for the same reason alerting on the WAL:DB ratio is wrong. min_wal_size sets a FLOOR of always-present segments and it differs per project. Terra's is 128MB = 8 segments at 16MB, so 8 of its 9 files ARE the floor and exactly one is real content. Mars's is 80MB = 5 segments against 6 files - also one real segment. Identical health, different absolute number. So venus 7 / terra 9 / pluto 16 are explained by their floors, not by differing health, and pluto's 16 must not be treated as its normal until its floor is known. THRESHOLD SHAPE: per project, files above (min_wal_size / wal_segment_size) + headroom. Requested min_wal_size from db-pluto-cc and db-venus-cc to complete the table. ALSO: terra reports zero replication slots, which is the other way a small database pins WAL independent of any archiver failure - worth one line in the monitor. And terra/enamel are CO-TENANTS of one Supabase project, so that is ONE row for the cluster, not two; counting them separately overstates fleet WAL. Elazar decided 2026-08-31 to split terra onto its own Supabase - unscoped and unstarted, but that row becomes two when it happens.
• wi-cli-venus verified · 5w ago
  CORRECTION to the threshold formula I proposed one message earlier, falsified by venus before anyone built on it. I said: alert on files above (min_wal_size / wal_segment_size) + headroom, treating min_wal_size as a per-project FLOOR. That is WRONG. min_wal_size is a RECYCLING TARGET, not a pre-allocation - Postgres keeps up to that much recycled WAL around once it has generated it, and never creates it up front. venus proves it: min_wal_size=1GB = 64 segments, yet venus sits at 7 files with a clean archiver. A formula-derived floor of 64 would have set venus's alert threshold roughly ten times too high and blinded the exact failure this monitor exists to catch. Full table, measured 2026-09-11. mars: min_wal_size 80MB = 5 segments, 6 files, 120MB db. terra: 128MB = 8 segments, 9 files, 16MB db. pluto: 80MB = 5 segments, 16 files. venus: 1GB = 64 segments, 7 files. All four healthy, all four with failed_count 0 or stale, and zero replication slots fleet-wide. WHAT THE THRESHOLD MUST ACTUALLY BE: an observed per-project baseline plus headroom, learned from the running system, NOT computed from any GUC. min_wal_size is usable only as an UPPER BOUND on the quiet-state count, never as the expected value. The three rejected formulations are now all on record here and each one reads as principled: the WAL:DB ratio (moves on the denominator - terra 9x healthy, above pluto's 3.7x), a single fleet-wide file count (floors and churn differ per project), and the computed floor (falsified by venus). ONE REAL FOLLOW-UP, not a monitoring question: pluto sits 11 segments ABOVE its 5-segment floor while mars and terra sit at 1 above theirs. db-pluto-cc flagged it themselves and it is unexplained - either ordinary churn on a busier database or a slow accumulation trend. It is not a wedge (last_archived is far past last_failed, archiving current), so there is no incident, but it is the only number in the fleet that a baseline cannot yet explain. Routed to db-pluto-cc as pluto-lane work; it is not Mars's. REASSIGNMENT NEEDED: this WI is assigned to bin-venus-ca, which is no longer on the roster.
• wi-cli-venus assigned · 5w ago
  unassigned
• wi-cli-venus blocked · 5w ago
  Unowned. bin-venus-ca went offline 'until further notice' 2026-09-11 07:41 ART and handed back everything routed to it (bin-venus-cc notice, same turn); bin-venus-cc holds no execution today by standing order. Reassigning to a name that cannot move it would make this read as queued when it is stranded - Elazar owns the routing call on the whole stranded bin set, raised by bin-venus-cc on aro:nw. Not time-critical: MARS-700's archiver wedge is repaired and the fleet baseline is re-measured (see events). What is missing is only the instrument.
• wi-cli-venus verified · 5w ago
  Fleet WAL baseline RE-MEASURED 2026-09-11 (freshness update on the 09-10 figures in the description; the conclusions stand): mars 6 files, terra 9, pluto 16, venus 7. failed_count 0 or stale on all four. ZERO replication slots fleet-wide - and zero is NORMAL here, so an empty slot list is not evidence of anything. pluto's 16 is ordinary churn on a busier DB, not drift: db-pluto-cc confirmed all files are 4-6 min apart and none stale. The DATA half of this WI is therefore DONE; what remains unbuilt is the timer itself. A THIRD THRESHOLD FORMULA WAS FALSIFIED TODAY, and it is the most principled-looking of the three. I proposed deriving each project's floor as (min_wal_size / wal_segment_size) + headroom. Venus falsifies it outright: min_wal_size=1GB implies 64 segments against an actual steady-state count of 7. min_wal_size is a RECYCLING TARGET / policy ceiling - never a pre-allocation and never a description of runtime state. A threshold computed from a GUC is a guess wearing arithmetic, and it reads as rigorous precisely because it has units. Three formulas have now been falsified in sequence - the WAL:DB ratio (moves on its denominator), a single fleet-wide file count (floors and churn differ per project), and this one - and each looked principled when proposed. Being principled is not evidence. The only usable form is an OBSERVED per-project baseline plus headroom, learned from the running system. The description's 'warn >50 / page >150' is already that shape and survives; do not let an implementer replace it with a computed one. Unchanged and still the strongest signal: pg_stat_archiver.last_archived_time not advancing while failed_count climbs, ALWAYS read with stats_reset in the same query.
• wi-cli-venus assigned · 5w ago
  db-mars-cc
• wi-cli-venus assigned · 5w ago
  db-mars-cc
• wi-cli-venus note · 5w ago
  DB-SIDE HALF DELIVERED; the WI stays BLOCKED and the blocker is unchanged and not mine to clear. Live re-read first, per the verify-against-the-running-system rule: Mars is HEALTHY right now - archived_count 682, failed_count 0, last_archived_time 18s old, 6 WAL files / 80 MB against a 120 MB database, archive_mode=on, not in recovery, stats_reset 2026-09-10 09:28:15Z. MARS-700's wedge is repaired; what is missing is only the instrument, exactly as the 10:43 blocked note says. Nothing about 'live blind spot' is urgent in the sense of a fault in progress - it is urgent in the sense that a recurrence would again be unseen. WHAT I BUILT, in-lane and with no DDL anywhere: a single SELECT-only canonical health read, one row per project, parameterised on :project and :baseline. Published to /data-venus/mars/db-monitoring/2026-09-11-wal-archiver-check.sql with its test beside it at 2026-09-11-wal-archiver-verdict-test.sql. Deliberately NOT a function and NOT a migration: it would have to be installed on four projects for an instrument nobody owns yet, and a fence nobody calls is the unowned-monitor shape I have argued against twice this week. Whoever gets the timer runs the file; there is nothing to deploy DB-side. VERDICT LADDER, in precedence order: SKIP archive_mode=off / PAGE archiver wedged (failed_count>0 AND last_failed_time>last_archived_time) / WARN never archived / PAGE archiver stalled (>2h) / WARN archiver slow (>30m) / PAGE wal ceiling (>150 files) / WARN wal above baseline (>GREATEST(3*baseline,25)) / OK. stats_reset is returned IN THE SAME ROW, not as a separate query - a snapshot without it is uninterpretable and a post-restart count read against a pre-restart baseline looks like a catastrophe that did not happen. NEGATIVE-TESTED, 10 synthetic cases, all correct - an instrument that cannot fail is not an instrument. The real MARS-700 shape (475 files, 2504 failures, 34h stale) returns PAGE archiver wedged. The case that matters most for false pages is 'old failure, recovered' - failed_count 9 from 40h ago with archiving current - and it returns OK, because the wedge arm compares last_failed_time to last_archived_time rather than testing failed_count>0. A monitor that pages forever on a repaired fault gets muted, and a muted monitor is the blind spot again. THRESHOLD FORM follows the three falsifications already on this WI: per-project OBSERVED baseline plus headroom, never a value derived from a GUC. Baselines measured 2026-09-11: mars 6, terra 9, pluto 16, venus 7. The 150-file ceiling is a second independent arm, not a derived floor - it is ~9x the highest observed and exists to catch a project whose baseline was set wrong. GAP I CANNOT CLOSE FROM SQL, stated rather than left implicit: disk-used-vs-provisioned is NOT readable from the database. Mars sat at 7.70/8 GB = 96.25% during MARS-700, past the 95% read-only threshold, and autoscale was configured but inert because adjusting disk configuration needs LARGE compute or above. That arm needs the Supabase Management API or the dashboard, not psql, and it is a different instrument with a different owner. Do not let this query's OK verdict be read as covering it. STILL BLOCKED ON: ownership of a venus systemd timer. bin-venus-ca went offline 2026-09-11 07:41 ART and bin-venus-cc holds no execution by standing order; Elazar owns the routing call on the stranded bin set. I cannot clear that from the DB lane and I am not reassigning it to a name that cannot move it.
• wi-cli-venus verified · 5w ago
  DB-LANE HALF DELIVERED, db-mars-cc 2026-09-11, all read-only or rolled back. This does not unblock the row - the venus timer still has no owner - but it removes a gate that would have been hit during implementation, and it changes the design. CURRENT STATE OF MARS, so nobody builds against the incident numbers: 6 WAL files / 80 MB, failed_count 0, last_archived_time 2026-09-11 13:25:25Z and advancing. Healthy. archive_mode=on, archive_timeout=120, min_wal_size=80, max_wal_size=1024. THE ROW'S OWN stats_reset WARNING IS ALREADY LIVE HERE: pg_stat_archiver.stats_reset = 2026-09-10 09:28:15Z, which is the restart. archived_count 701 is post-reset and is not comparable to anything from before it. Any baseline captured before 2026-09-10 09:28Z is gone. READ-PATH GATE, MEASURED RATHER THAN ASSUMED - the two proposed signals have DIFFERENT privilege requirements and nobody had checked: - pg_stat_archiver is readable by an ORDINARY role. Proved by SET LOCAL ROLE errscan_ro inside a rolled-back transaction: failed_count, last_archived_time and stats_reset all returned. - pg_ls_waldir() is NOT. proacl is {supabase_admin=X/supabase_admin, pg_monitor=X/supabase_admin}. Same rolled-back probe as errscan_ro: 'ERROR: permission denied for function pg_ls_waldir'. - AND THE NARROW FIX IS A NO-OP, which is the part that would have burned an implementation cycle: GRANT EXECUTE ON FUNCTION pg_catalog.pg_ls_waldir() TO errscan_ro, issued as postgres, emits 'WARNING: no privileges were granted for "pg_ls_waldir"' and then reports GRANT - a WARNING, not an ERROR, so ON_ERROR_STOP does not catch it and psql exits 0. The role is still denied immediately afterwards in the same transaction. postgres is not the owner; supabase_admin is. A migration granting this would ship green and fix nothing. (Identical shape to MARS-712's revoke - on this platform a non-owner GRANT/REVOKE on a supabase_admin-owned object is a silent success. Worth treating as a class.) CONSEQUENCE FOR THE DESIGN: the WAL-file-count half requires pg_monitor MEMBERSHIP, all-or-nothing, with no narrow alternative. BUT NO GRANT IS NEEDED - supabase_read_only_user is ALREADY a member of pg_monitor on this ref (measured; postgres is too, and none of agent_db_mars_cc / mars_app_runtime / applog_reader / errscan_ro / evolutiva_agent_ro are). So the monitor can read both halves today with zero privilege change if it connects as supabase_read_only_user. That is the recommendation: do not grant pg_monitor to a new role. A SHARPER THRESHOLD THAN THE ROW HAS, and it sits on the half that needs no privilege: archive_timeout=120 means last_archived_time must advance at least every 2 minutes on any project with any write traffic at all. So 'last_archived_time older than ~10 minutes' is a precise, low-false-positive page that needs NO baseline - which matters because a restart destroys the baseline failed_count would need, exactly as this ref now demonstrates. Recommend that as the primary signal, failed_count as corroboration, WAL file count (warn >50 / page >150) as the severity gauge. NOT MEASURED, so nobody reads this as complete: disk used vs provisioned is not exposed by any pg_catalog read I exercised - pg_database_size gives database size, not provisioned volume - so the 96.25%/autoscale-inert finding needs the Management API, not this connection. And this covers ustenjufophwhlkzfdso only; the row asks for all four projects and the other three are not mine to query.
• wi-cli-venus note · 4w ago
  ROUTING CORRECTED - my dispatch to db-mars-cc was wrong and its refusal was right. pmmaster-evolutiva-cc 2026-09-12. I read the row as fully specified and gave a conditional GO to db-mars-cc; it answered that the blockedReason is LIVE and is a ROUTING blocker ('Unowned... Elazar owns the routing call on the whole stranded bin set'), so assigning it to a project db lane does not discharge it - it reproduces the failure the reason warns about, and the row then reads QUEUED while still stranded. Three things the mars db lane cannot do, each a standing fleet rule and not a preference: create a systemd unit on venus (bin/nw lane), hold credentials for pluto/venus/terra to run the SELECTs against all four refs (an agent never queries a DB whose maintenance lane it does not own and never reuses another agent's credentials), and push (db-mars-cc never pushes; a push grant is a row in project commons, not a verbal grant from a PM). TWO FACTS THAT CHANGE THE ROUTING RATHER THAN THE SCOPE. First, the Mars DB half ALREADY EXISTS LIVE: public.wal_disk_health_check is present on the Mars ref, so what is missing on Mars is the caller and the alert path, not the SQL. Second, the cross-project scope is what forces this out of any single project's db lane - four refs, four credential sets - so the natural owner is a venus ops/bin seat holding a fleet credential, not four db agents each building a quarter of the instrument. OFFERED, NOT STARTED, and it needs no grant: db-mars-cc will deliver the Mars-side SELECT-only instrument and its thresholds as a DB function plus the exact query contract for whoever builds the timer. DESIGN REQUIREMENT ADDED BY db-mars-cc AND IT IS THE SHARPEST THING ON THIS ROW - an alert that only fires on failed_count climbing goes SILENT on total unreachability, which is the exact defect this row exists to close, so it needs a DEAD-MAN'S SWITCH: the timer writes a heartbeat on every successful run and a separate venus-side check alerts on the ABSENCE of a fresh heartbeat, so 'cannot reach the database at all' and 'database is fine' stop being indistinguishable from outside. Any implementation lacking that reproduces MARS-700's blindness in new code. NEXT STEP IS ELAZAR'S: name the venus ops/bin seat that owns this, or authorise a fleet-credential seat to hold it.
• wi-cli-venus assigned · 4w ago
  bin-venus-cc
• wi-cli-venus unblocked · 4w ago
  OWNED by bin-venus-cc. pmmaster-evolutiva-cc 2026-09-12 (tag pmmaster-evolutiva-cc-mtxz1s8i5xs6), exercising a routing call Elazar delegated in his own CLI words: 'if who own 703 whoever you decide, coder or whatever.' That quote is what clears the blockedReason - the blocker was never scope, it was that Elazar owned the routing call on the stranded bin set, and he has now given it away. Fleet stop is lifted FOR THIS ROW ONLY; everything else I hold stays parked. ARCHITECTURE, and it is what removes the fleet-credential gate that stranded this: a TEMPLATE timer on venus, one instance per project on the applog-pull@<app> pattern, each with its OWN EnvironmentFile. Each project's db lane provisions a read-only role on its own ref and DMs me that credential. No lane boundary is crossed - the owning lane creates the credential deliberately, and the thing running the SELECT is a systemd unit, not an agent reaching across hosts. I own the unit, the alert leg and host-checks.sh; I own none of the four databases and never query one outside a provisioned role. THE DB HALF IS ALREADY BUILT AND I AM NOT REBUILDING IT. db-mars-cc delivered /data-venus/mars/db-monitoring/2026-09-11-wal-archiver-check.sql (parameterised on :project and :baseline, verdict ladder, stats_reset in the same row, 10 negative-tested synthetic cases) plus public.wal_disk_health_check live on the mars ref. Connect as supabase_read_only_user - ALREADY a pg_monitor member on mars, so pg_ls_waldir() needs NO grant, and the narrow GRANT is a measured no-op that ships green. Baselines: mars 6, terra 9, pluto 16, venus 7. TWO THINGS MUST BE FORCED RED BEFORE I CALL ANY LEG DONE, because a control that has not been seen to fail is a green line, not a control: (1) archiver wedged -> alert fires; (2) HEARTBEAT ABSENT -> the separate venus-side check alerts. (2) is the dead-man's switch db-mars-cc required and it is the one that actually closes MARS-700's blindness - an alert that only fires on failed_count climbing goes SILENT on total unreachability. I will force (2) by STOPPING THE TIMER, not by faking the heartbeat file; faking it exercises the writer, not the absence path. SEQUENCE, per pmmaster and not negotiable under the token budget: mars END TO END first - unit, credential, alert, heartbeat, forced red - THEN replicate per ref. Not four at once.
• wi-cli-venus assigned · 4w ago
  bin-venus-cc
• wi-cli-venus note · 4w ago
  OWNER NAMED: bin-venus-cc. Routing call delegated to me by Elazar in the CLI 2026-09-12 - 'if who own 703 whoever you decide, coder or whatever' - after db-mars-cc correctly refused the row and I recorded why. bin-venus-cc owns the venus timer, the alert leg and host-checks.sh; it owns none of the four databases and queries none outside a role provisioned for it. ARCHITECTURE THAT REMOVES THE REASON THIS SAT UNOWNED, which was that the work appeared to need a fleet credential nobody holds: a TEMPLATE systemd timer on venus, ONE INSTANCE PER PROJECT on the applog-pull@<app> pattern this row already names, each instance carrying its OWN EnvironmentFile. Each project's db agent provisions a read-only role on its own ref and DMs that credential to bin-venus-cc. Nothing crosses a lane boundary: the owning lane creates the credential deliberately, and the thing executing the SELECT is a systemd unit rather than an agent reaching across hosts. That satisfies the commons rule db-mars-cc invoked without needing it waived. SEQUENCE: mars end to end first - unit, credential from db-mars-cc, alert path, heartbeat, forced red - then replicate per ref. Not four at once. Mars is cheapest because public.wal_disk_health_check already exists live on the Mars ref, so the mars leg is the caller and the alert path only; db-mars-cc has offered that function plus the exact query contract for the other three refs and has my word to deliver it. NON-OPTIONAL REQUIREMENT, db-mars-cc's, restated here because it is the difference between closing this row and re-creating it: an alert that fires only on failed_count climbing GOES SILENT on total unreachability, which is exactly the blindness MARS-700 exposed when /api/health reported dbOk:true through ten hours of write outage because a SELECT succeeds inside a read-only transaction. Each instance writes a HEARTBEAT on every successful run and a separate venus-side check alerts on the ABSENCE of a fresh heartbeat, so 'cannot reach the database at all' and 'database is fine' stop being indistinguishable from outside. ACCEPTANCE: the mars leg must be seen to FIRE on a forced failure before any leg is called done - a monitor that has never fired is a green line, not a control. Stop is lifted for bin-venus-cc on this row only.
• wi-cli-venus note · 4w ago
  DB-SIDE HEARTBEAT IS STRUCTURALLY BLIND TO THE OUTAGE THIS ROW EXISTS TO CLOSE - found while answering bin-venus-cc's design asks, outside the row's original scope, and it changes where the dead-man's-switch lives. public.wal_disk_health_check() carries a block commented 'LIVENESS: unconditional, every run. Never silent.' That heartbeat is an INSERT INTO public."appEvents" - a DATABASE WRITE - executed from a function invoked by pg_cron (job 23, jobname mars-wal-disk-health-check, schedule '7 * * * *', active). During a read-only/write outage of the MARS-700 class the INSERT is rejected, so there is no heartbeat row and no alarm row; if the instance is unreachable the job does not fire at all. The comment's claim is false in exactly the failure mode the row was opened for. I wrote that function and I am retracting the liveness claim rather than defending it: it stays as telemetry (the detail jsonb and the dedupe-on-severity are fine) but is DEMOTED from liveness. The dead-man's-switch must be a VENUS-SIDE FILESYSTEM heartbeat written by the timer on successful completion, with a separate venus-side check alerting on its absence. Nothing DB-side can serve this role, by construction - an alert path hosted inside the thing it watches goes quiet with it. ANSWERED bin-venus-cc's three asks on llmmsg-srv, all measured live, not recalled. (1) CONNECTION STRING - cannot be satisfied as scoped, and the blocker is a missing usable credential, not a missing GRANT. supabase_read_only_user is rolcanlogin=t and a member of pg_monitor AND pg_read_all_data, so bin-venus-cc's premise is right: pg_ls_waldir() needs no grant and the narrow GRANT EXECUTE to errscan_ro would be a silent no-op. But rolpassword IS NOT NULL and no one in-fleet holds the plaintext (Supabase-managed); grep of pm/06-passwords-and-settings.md for supabase_read_only_user, read_only, readonly, pg_monitor, errscan_ro and applog_reader returns zero hits, and ~/.config/postgres/ holds no Mars read-only env. errscan_ro and mars_app_runtime have no group memberships at all, so neither can run pg_ls_waldir(). Setting a password on a role that already has one is a credential operation barred by the No-Rotation Rule. Three options given, recommendation is (b): (a) the postgres superuser DSN - works today, wrong privilege level to leave in a permanent unit's EnvironmentFile; (b) I provision a dedicated mars_wal_monitor LOGIN with GRANT pg_monitor and its own password - NEW provisioning, not rotation, squarely this lane, one small migration; (c) Elazar supplies the managed role's password. Awaiting bin-venus-cc's word on (b) since it owns the row. (2) PRIMARY SIGNAL - confirmed with a correction that improves it. Measured: archive_mode=on, archive_timeout=120s, min_wal_size=80MB, max_wal_size=1024MB, wal_keep_size=0. pg_stat_archiver: archived_count 1036, last_archived_wal 00000001000000FE0000009A, failed_count 0, last_failed_* NULL, stats_reset 2026-09-10 09:28:15Z, archive age 00:01:40. pg_ls_waldir() = 6 files / 80 MB, i.e. sitting ON the min_wal_size floor. So failed_count is the wrong primary exactly as bin-venus-cc argued - with failed_count 0 and NULL last_failed_time the wedged arm cannot fire today and is UNEXERCISED; it can only ever be a ladder rung. But a pure time threshold is also wrong as primary: archive_timeout forces a segment switch only when there has been WAL ACTIVITY since the last one, so a genuinely idle instance goes legitimately stale and a 10-minute rule false-pages on it. Named SEGMENT LAG as primary instead - current WAL file vs last_archived_wal, decoded from the hex filename. Single sample, no stored state, no baseline, survives stats_reset, no idle false-positive. TESTED LIVE, not proposed: returned 1 against current 00000001000000FE0000009B/9C. Healthy steady state 0-2; MARS-700 (475 files stuck 34h on one segment) would have read in the hundreds and paged on day one. Suggested WARN >5 / PAGE >20. (3) QUERY CONTRACT - pushed back on the :baseline binding, it is not needed. The row's own measured band (venus 7 files, terra 9, pluto 16, Mars-at-failure 475, with min_wal_size setting the floor) is ABSOLUTE and baseline-free, which is the same virtue bin-venus-cc wants from the primary. The only genuinely per-ref quantity was stats_reset awareness on the failed_count rung, and segment lag removes even that. Template EnvironmentFile needs a DSN and threshold constants, no baseline binding. LADDER DIVERGENCE FLAGGED: the live DB function uses WARN 30 min / PAGE 2 h while the venus unit plans ~10 min. Two instruments with different thresholds and no stated reason is how one quietly becomes the wrong one. Recommended venus deliberately TIGHTER and recorded as such - the venus rung is the one that survives the outage - and the DB-side 30m/2h read as legacy, written when it was the only instrument. BOUND STATED ONCE: all of this is MARS-ONLY. This lane holds no credential for and no lane over the pluto, venus or terra/enamel refs; commons forbids querying a DB whose maintenance lane you do not own. 'Replicate per ref' is bin-venus-cc's to arrange with each project's db lane - I cannot supply four DSNs. Also added one case to its forced-red gate: force the heartbeat arm red by making the DB UNREACHABLE (bad host in the DSN), not only by stopping the timer - stopping the timer proves the file goes stale, a bad DSN proves the unit does not write a success heartbeat on a failed run, and only the second reproduces MARS-700. STOPPING HERE. Fleet-wide STOP remains in force outside this row; the venus half is bin-venus-cc's build. My only open action is provisioning mars_wal_monitor IF bin-venus-cc takes option (b).
• wi-cli-venus note · 4w ago
  CREDENTIAL STEP PARKED ON ELAZAR - and the ruling is worth keeping because it separates two authorisations I had collapsed. bin-venus-cc (row owner) gave GO on provisioning mars_wal_monitor with its own password and GRANT pg_monitor, withdrawing its 'do not provision anything' on the grounds that my measurement falsified its premise (rolpassword IS NOT NULL on supabase_read_only_user and nobody in-fleet holds the plaintext), and agreeing new provisioning is not a rotation so the No-Rotation Rule does not reach it. pmmaster-evolutiva-cc then confirmed the SCOPE and held the MECHANISM: a blocking dependency of an authorised row IS inside that row's carve-out, so the fleet stop does not bar the work - but CREATE ROLE on a production ref is a DDL WRITE and the fleet SELECT-only rule is unconditional with NO ownership exception. Owning a maintenance lane governs WHICH database I may reach, never WHETHER I may write to it. The only standing grant of this shape in the fleet is Elazar's CREATEROLE for bin_venus_cc on VENUS telemetry, and a grant on venus is not a grant on mars. So: I am NOT writing the migration, and I am NOT falling back to the postgres superuser DSN (wrong privilege level to leave in a permanent unit's EnvironmentFile - that rejection stands and pmmaster endorsed it). pmmaster is taking the one ask to Elazar in CLI. I hold. THE LESSON: scope authorisation and mechanism authorisation are separate, and holding the first does not give you the second - I had read the carve-out as covering both. TWO DEFECTS IN THE SEGMENT-LAG EXPRESSION I HANDED OVER, found by bin-venus-cc, both failing in the REASSURING direction and neither exercised by the run that returned a healthy 1. Accepted, both mine. (i) If last_archived_wal IS NULL the arithmetic returns NULL, and NULL against >5/>20 evaluates FALSE - so 'never archived' renders as a silent OK. My own SQL ladder in wal_disk_health_check carries a WARN-never-archived rung for exactly that case, and the lag arm silently deletes it. This is the same fails-toward-green class as the dead LEFT JOIN count(*) assertion in migration 159 and the pg_proc-without-pronargs trap, and I shipped it into a handover anyway because the happy path returns a number. bin-venus-cc will handle NULL as its own explicit state rather than folding it into an integer - correct. (ii) The heartbeat write must NOT be gated on psql's exit code: psql EXITS 0 ON A WARNING, which is measured on this ref (it is how a non-owner GRANT shipped green and fixed nothing), and ON_ERROR_STOP does not catch it. Gate on a POSITIVE assertion that the query returned a parseable integer; anything else is UNKNOWN and writes nothing - otherwise a silently-empty result writes a FRESH SUCCESS HEARTBEAT and the dead-man's-switch is dead. That gate is load-bearing for forced-red case (3), not fussiness. LADDER-DIVERGENCE FRAME CORRECTED, and bin-venus-cc's version is better than mine - I withdraw 'venus should be deliberately tighter'. Segment lag and a wall-clock ladder are DIFFERENT QUANTITIES, not two thresholds on one quantity; converting between them depends on the WAL activity rate, which is the exact property that disqualified the time rule in the first place. Stating a threshold comparison would manufacture a relationship that does not exist and would be worse than the unexplained gap. Recorded instead as: the venus segment-lag arm is the instrument that SURVIVES the outage, the DB-side 30m/2h is legacy from when it was the only instrument, and no second-opinion relationship is claimed in either direction. SETTLED AND NOT TO BE RE-OPENED: heartbeat retraction accepted by both - wal_disk_health_check stays as telemetry, demoted from liveness, and bin-venus-cc will say so in the venus unit's own header so a later reader does not re-promote it. Venus heartbeat is a LOCAL FILE written by the timer, with a separate venus-side staleness check on that file. Primary signal is SEGMENT LAG (WARN >5 / PAGE >20), wedged arm kept as the ladder rung it already is. Per-ref :baseline binding DROPPED - the band is absolute and segment lag removes the last per-ref quantity by surviving stats_reset, so the template EnvironmentFile carries a DSN and threshold constants and nothing else. Forced-red gate is three cases: archiver wedged; heartbeat absent by STOPPING THE TIMER (proves the file goes stale); heartbeat absent by a BAD HOST IN THE DSN (proves a FAILED RUN does not write a success heartbeat) - the third is the only one that reproduces MARS-700 and pmmaster would have accepted the leg without it. MY REMAINING ACTION ON THIS ROW IS EXACTLY ONE and it is blocked on Elazar: write the mars_wal_monitor migration and DM bin-venus-cc the DSN. Nothing else of mine blocks the venus build.
• wi-cli-venus note · 4w ago
  CREDENTIAL BLOCKER DISCHARGED - Elazar cleared it directly in CLI: 'GO to run CREATE ROLE mars_wal_monitor with pg_monitor'. Migration 160 written, applied to the mars ref, committed as handoff 6007372 (Local-Author: db-mars-cc, one file, db/migrations/applied/160-2026-09-12-mars703-wal-monitor-role.sql). DSN delivered to bin-venus-cc by llmmsg-srv DM; it was building against a placeholder EnvironmentFile and is now unblocked. The authorisation chain is three links and all three are on this row: bin-venus-cc as row owner gave GO on the option and withdrew its earlier 'do not provision anything' once my measurement falsified its premise; pmmaster-evolutiva-cc confirmed the SCOPE was inside the MARS-703 carve-out but HELD the MECHANISM, ruling CREATE ROLE a DDL write with no ownership exception; Elazar supplied the mechanism. Scope authorisation and mechanism authorisation are separate and this has both - a lifted-stop audit should find all three here. PRIVILEGE SHAPE: pg_monitor is the ENTIRE grant (pg_read_all_settings + pg_read_all_stats + pg_stat_scan_tables), which is exactly what pg_ls_waldir() and pg_stat_archiver need. Deliberately NARROWER than supabase_read_only_user, which would also have carried pg_read_all_data the unit has no use for. NOSUPERUSER, NOCREATEDB, NOCREATEROLE, NOREPLICATION, INHERIT (a NOINHERIT monitoring role would read nothing, since SET ROLE is denied on Supabase), CONNECTION LIMIT 3 so a wedged timer cannot consume connection slots. PROVEN LIVE AS THE NEW ROLE IN BOTH DIRECTIONS, not asserted from the catalog, because a grant proven only by what it permits is a green line. POSITIVE: pg_ls_waldir() returns 6 walfiles; pg_stat_archiver returns last_archived_wal 00000001000000FE0000009E with failed_count 0; the segment-lag expression the venus unit will use returns 1 (healthy). NEGATIVE CONTROL: SELECT count(*) FROM public."users" as the same role returns ERROR permission denied for table users. THE PASSWORD IS IN NO TRACKED FILE AND THE MIGRATION IS THEREFORE NOT SELF-SUFFICIENT ON REPLAY - stated in the header rather than hidden. It is set by a separate out-of-band ALTER ROLE and delivered over llmmsg-srv DM, which is a private authenticated channel: the value is NOT leaked by having crossed it and must never be rotated on those grounds. Replaying 160 against a fresh ref creates a role that cannot log in until the password step is run separately. That is the intended trade - a replayable secret is a committed secret. POST-CHECK asserts the END STATE in both directions and has an arm that can actually fail: role exists, rolcanlogin, not superuser/createrole, INHERIT, memberships EXACTLY 'pg_monitor' compared by string_agg with IS DISTINCT FROM rather than a count, and a pg_class.relacl scan asserting ZERO direct object grants - because ALTER DEFAULT PRIVILEGES can hand a new role privileges nobody wrote in the migration. Checking only the grant you just made can never fail. TWO SELF-REPORTED WEAKNESSES, sent to the review lane rather than left for it to find: the relacl arm uses LIKE on array_to_string rather than aclexplode, so it would match a role whose name is a prefix of another's - correct on this ref today, fragile if a similarly-named role is ever added; and the memberships arm needs checking that it genuinely avoids the dead-assertion class from 159 rather than relocating it. No schema.md re-export - roles and role memberships are not in the export. Self-DM to coder02-mars-cc sent in the same turn per the review-tier pilot, with the authorisation chain flagged for review as explicitly as the SQL, and with an instruction to treat an insufficient-authorisation reading as a BLOCK rather than a P3. MY LEG OF MARS-703 IS NOW COMPLETE. Nothing of mine blocks the venus build: SQL half done, primary signal specified and its two defects accepted, baseline binding dropped, heartbeat retracted and relocated off-host, credential provisioned and proven. Remaining work on this row is bin-venus-cc's unit plus its three forced-red cases. I added one note on case (3): a bad HOST and a bad PASSWORD in the DSN exercise different psql failure branches and the integer-parse gate must swallow neither into a success heartbeat.
• wi-cli-venus progress · 4w ago
  MARS LEG BUILT AND FORCED RED, pending only nw registration of the units. Code (sh.git, ~/.local/bin): wal-archiver-check.sh v1.1 (per-project probe) and wal-archiver-heartbeat-check.sh v1.0 (the dead-man's switch - a SEPARATE unit that holds NO DSN and reaches NO database, so it cannot share a failure domain with what it watches). Units written, NOT enabled - nw-venus-cc registers, per the nw/bin lane split: wal-archiver-check@.service/.timer (*:05,35) and wal-archiver-heartbeat-check.service/.timer (*:15,45), with a 70min staleness limit so one missed run does not page and two do. Credential: mars_wal_monitor, pg_monitor ONLY, CONNECTION LIMIT 3, provisioned by db-mars-cc under Elazar's direct CLI GO, migration 160 / handoff 6007372; it lives in ~/.config/wal-archiver-check/mars.env at 0600, outside every tracked tree, referenced by EnvironmentFile and never inlined. DESIGN CHANGED ON MEASUREMENT, both from db-mars-cc. (1) Primary signal is WAL SEGMENT LAG (current walfile minus last_archived_wal), not a wall-clock staleness rule: archive_timeout only forces a segment switch when there has been WAL ACTIVITY, so a time rule false-pages on a legitimately idle ref, while segment lag is a single sample needing no baseline and surviving stats_reset. The per-ref :baseline binding is DROPPED entirely; thresholds are absolute (WARN >5 / PAGE >20 segments, WARN >25 / PAGE >150 files). (2) db-mars-cc RETRACTED the liveness claim on their own public.wal_disk_health_check - it writes its heartbeat with an INSERT, so it goes silent in exactly the read-only outage this row exists to catch. It stays as telemetry and is DEMOTED from liveness, recorded in the script header so nobody re-promotes it. Nothing DB-side can be the dead-man's switch by construction. LIVE GREEN, first run as the new role: OK mars, 1 segment behind, 6 WAL files, archive_mode=on - independently matching db-mars-cc's own measurement of the same two quantities. FORCED RED, four cases, all observed firing: (a) bad DSN HOST - alert fires, rc=1, NO heartbeat written; (b) bad DSN PASSWORD, a different psql branch, same result - db-mars-cc's addition, worth the extra run; (c) heartbeat MISSING for a declared instance - PAGE; (d) heartbeat STALE at 3h against the 70min limit - PAGE. Positive control: a fresh heartbeat is silent, rc=0. Refusal control: an empty WALCHK_INSTANCES denominator REFUSES to run rather than reporting full coverage over nothing. TWO DEFECTS FOUND AND FIXED BEFORE SHIPPING, both in the reassuring direction, neither exercised by a healthy live run. (i) The handed-over expression returns NULL when last_archived_wal is NULL, and NULL against >5/>20 is FALSE - so never-archived rendered as a silent OK, deleting the rung db-mars-cc's own SQL ladder carries. It is now its own explicit state. (ii) Mine, caught by the stub-psql harness: TAB is IFS whitespace, so bash read COLLAPSES consecutive tabs and one empty field shifted every later field left, landing never-archived on UNKNOWN. Every column is now coalesced to a '-' sentinel, never an empty string (v1.1). The heartbeat write is gated on a PARSEABLE INTEGER, never on psql's exit code, because psql exits 0 on a WARNING - measured on this very ref. COVERAGE LIMIT, stated rather than elided: the wedged and stalled rungs are verified against the exact MARS-700 shape (475 files, 2504 failures, 210 segments behind) through the stub harness, NOT against a live wedged archiver - mars is healthy and I will not wedge a production archiver to watch an alert fire. Every other rung, both forced-red arms and both connection-failure paths are live.
• wi-cli-venus verified · 4w ago
  PASS:6007372 (coder02-mars-cc review, under this row's own carve-out - fleet STOP verified lifted for MARS-703 only per pmmaster's note, general STOP still in force elsewhere). Authorisation chain confirmed from the row itself: bin-venus-cc GO on the option, pmmaster's scope-yes/mechanism-no ruling (CREATE ROLE is a DDL write with no ownership exception), Elazar's direct CLI GO. Live read-only re-verification, independent of the migration's own post-check: mars_wal_monitor rolcanlogin=t/rolsuper=f/rolcreaterole=f/rolcreatedb=f/rolreplication=f/rolinherit=t/connlimit=3/has_password=t, memberships exactly pg_monitor, zero direct object grants. One correction sent to db-mars-cc, not blocking: the relacl LIKE hazard the migration's comment flags as 'a role whose name is a PREFIX of mars_wal_monitor's' has the direction backwards - the actual vulnerable case is a role whose name has mars_wal_monitor as a SUFFIX (e.g. othermars_wal_monitor), since the aclitem substring match keys on what immediately precedes '='. Confirmed no such role exists today. aclexplode is still the right eventual fix; flagged so the comment says 'suffix' before this template is reused elsewhere.
• wi-cli-venus note · 4w ago
  PASS:6007372 from coder02-mars-cc, with one correction to my own self-report that is worth more than the verdict. It verified the authorisation chain FROM THE ROW rather than from my message - bin-venus-cc's GO on the option, pmmaster's scope-yes/mechanism-no ruling with the explicit 'CREATE ROLE is a DDL write with no ownership exception', and Elazar's direct CLI GO, all three present as separate events - and re-ran the catalog reads itself rather than trusting that my post-check passed: rolcanlogin=t, rolsuper=f, rolcreaterole=f, rolcreatedb=f, rolreplication=f, rolinherit=t, rolconnlimit=3, has_password=t, memberships exactly pg_monitor, relacl direct-grant scan NULL. THE CORRECTION, and I had it BACKWARDS: I flagged the post-check's relacl arm - array_to_string(c.relacl,',') LIKE '%mars_wal_monitor=%' - as fragile against a role whose name has mars_wal_monitor as a PREFIX. That direction is SAFE, because the character following the name in such a role's aclitem is never '=' so the substring never matches. The ACTUAL hazard is a SUFFIX: a role named e.g. othermars_wal_monitor produces the aclitem text 'othermars_wal_monitor=...' which genuinely contains 'mars_wal_monitor='. coder02 confirmed no such role exists on this ref (LIKE '%mars_wal_monitor' excluding itself returns zero rows), so 160 is correct as applied, and aclexplode() remains the right fix before the template is reused. Recording the direction explicitly because a fix note saying 'prefix' would send the next reader to check the wrong input and conclude the pattern is safe - the general lesson is that a self-reported string-matching weakness must name the FAILING INPUT, not the intuition about it. It also confirmed the memberships arm genuinely escapes the 159 dead-assertion class rather than relocating it, and gave the precise reason: 159's v_total was count(*) over a LEFT JOIN from a fixed-cardinality VALUES list, structurally constant whatever the catalog held; this one is string_agg over an INNER JOIN through pg_auth_members with no VALUES list, so the aggregate varies with the real membership set - zero memberships yields NULL and fires, an extra membership changes the string and fires. And it declined to treat the out-of-band password / non-self-sufficient replay as a finding, agreeing the header states the trade plainly. MARS-703 DB LEG CLOSED. Nothing of mine is outstanding on this row. Remaining work is bin-venus-cc's: the timer unit, the venus-side heartbeat file and its separate staleness check, and the three forced-red cases. Both I and the review lane are stopping again - the general fleet STOP holds outside this row.
• wi-cli-venus verified · 4w ago
  MARS LEG ARMED AND ACCEPTED. nw-venus-cc registered all four units after checking both load-bearing conditions rather than taking them on trust: WALCHK_INSTANCES=mars confirmed present (the denominator - a ref enabled without being added there is watched by nothing and reports healthy by being invisible), and wal-archiver-heartbeat-check.service added to sys-check FRESHNESS_UNITS at 3600s, because a dead-man's switch is itself a timer whose risk is ARMED-ness rather than freshness. Both timers enabled and started; both services fired manually once and clean - heartbeat '1/1 fresh, 0 stale, 0 missing, declared: mars', probe 'OK mars: 1 segments behind, 6 WAL files, archive_mode=on', reproducing bin-venus-cc's live-green run from a DIFFERENT caller. sys-check's armed-but-never-run WARN was the expected transient and cleared on the manual fire. Next probe 06:35Z, next heartbeat 06:45Z. Code pushed to sh.git as 99eee11. pmmaster accepted the leg: four forced-red cases observed firing, a positive control that stays silent, and a refusal control that refuses on an empty instance list. TIMING QUESTION ANSWERED AND CLOSED, because it went to whether a hold was overridden: it was not. pmmaster's mechanism hold is the 06:10:40Z event; db-mars-cc held, and the role was provisioned at the 06:13:22Z event on Elazar's OWN CLI words ('GO to run CREATE ROLE mars_wal_monitor with pg_monitor'), migration 160 / handoff 6007372. First run as the role is 06:13:53Z. Scope authorisation and mechanism authorisation were separately obtained and both are on this row. REPLICATION TO terra, pluto AND venus IS NOT STARTED and is gated on Elazar per pmmaster; each ref additionally needs its own db lane to provision its own role, since db-mars-cc holds no lane over the other three.
• wi-cli-venus note · 4w ago
  PROVISIONING TIMELINE, answered to pmmaster-evolutiva-cc on the record because the sequence matters for what Elazar is told. The role was created AFTER pmmaster's hold, and neither of the two options pmmaster offered (crossed-wires-before, or directive-overridden-after) is the right shape - there is a third: the hold LANDED, was HONOURED, and was then DISCHARGED BY ELAZAR HIMSELF, the authority pmmaster routed it to. ART timeline: 03:09 pmmaster's hold DM; I read it, recorded it on this row as parked-on-Elazar, and DM'd bin-venus-cc 'HOLDING on mars_wal_monitor... I am not writing the migration and will not fall back to the superuser DSN'; Elazar then typed the GO into my CLI verbatim - 'GO to run CREATE ROLE mars_wal_monitor with pg_monitor'; role created and 160 applied immediately after; commit 6007372 timestamped 03:12:21 ART. About three minutes elapsed with a full stop in the middle. No directive was overridden - I stopped on pmmaster's and moved only on Elazar's. CONSEQUENCE FOR THE CLI CORRECTION: pmmaster did not tell Elazar anything false. When it said the grant was the one thing parked on him, it was; he then unparked it in the same channel, and what he cleared is exactly what was routed - CREATE ROLE with pg_monitor, nothing wider. The ask is therefore neither 'may we' nor 'ratify or revert' but 'you already cleared this, here is what landed'. THE DURABLE HALF, and it is a correction to my own reasoning rather than to anyone else's: pmmaster's ruling that CREATE ROLE is a DDL write with no ownership exception should stand. But what nearly went wrong on my side was not the write - it was that I read this row's carve-out as authorising the MECHANISM because it authorised the SCOPE. Those are separate, and holding the first tells you nothing about the second. That sentence is the one worth carrying into every replication of this row, because each one presents a db lane with the identical shape: a GO already in hand from the row owner, covering scope only. Without pmmaster's hold I would have proceeded on bin-venus-cc's word and the decision would never have reached Elazar at all - the hold is the reason it did. Role untouched and staying that way: not dropped, not altered, and no revert acted on from anyone but Elazar. pmmaster stated the same ('nothing rolls back on my say-so').
• wi-cli-venus note · 4w ago
  VENUS-SIDE REGISTRATION IS NOW REAL AND DATED - 2026-09-12, filed by bin-venus-cc, per pmmaster-evolutiva-cc. CORRECTION FIRST: I had both MARS-703 units recorded as registered and armed. Armed was true; registered was false. jobreg check reported wal-archiver-check@mars.timer and wal-archiver-heartbeat-check.timer as UNREGISTERED on venus - the only tracking that existed was mars-side, for units that run on venus, and /data-venus/jobreg/jobreg.venus.tsv is the file the watchdog actually reads. Both rows are now filed there: wal-archiver-check@mars.timer (user, :05,:35, interval 30, owner bin-venus-cc, grace 30, sense=ok) and wal-archiver-heartbeat-check.timer (user, :15,:45, interval 30, owner bin-venus-cc, grace 30, sense=ok), with WALCHK_INSTANCES named on the probe row as the denominator so an unarmed ref shows up as a missing row rather than as silence. jobreg check now reports no drift on venus (71 rows for this host, all live and accounted).\n\nWHY THIS IS MORE THAN BOOKKEEPING, pmmaster's framing and it is right: the dead-man's switch accepted on this row was being watched by a registry that did not know it existed. That is the WALCHK_INSTANCES defect one level up - a thing enabled without being added to the list that watches it reports healthy by being invisible - and the sign-off was given partly on the strength of the freshness check being registered. Same class as reference-a-dead-checker-presents-as-quiet.\n\nRelated instrument caveat measured the same night, because it bears on how anyone later checks these two units: a manual systemctl start NEVER appears in a timer's LAST column (LAST records timer activations, not service runs), so list-timers cannot tell a never-fired timer from a manually-exercised one. Read the SERVICE's ExecMainStartTimestamp and Result before calling either unit dead or alive. The authoritative counter is days-observed where a unit writes one, with the caveat that a stalled timer silently extends its own warn window - no-days-observed and no-warnings are both quiet.
• bin-venus-cc blocked · 1w ago
  Mars leg SHIPPED, ARMED and still healthy - verified live today, not from memory: wal-archiver-check@mars.timer last ran 2026-10-05 16:35:30 UTC exit 0 (next 17:05), wal-archiver-heartbeat-check.timer last ran 16:45:30 (next 17:15), WALCHK_INSTANCES=mars with WALCHK_MAX_AGE_SEC=4200. Both jobreg rows filed on venus. Nothing about the mars leg is outstanding. THE REMAINDER IS GENUINELY BLOCKED, ON TWO SEPARATE THINGS, AND NEITHER IS MINE TO CLEAR. The remaining scope is replicate per ref - terra, pluto, venus, one at a time. BLOCKER 1, ELAZAR: pmmaster-evolutiva-cc gated replication on Elazar and that gate has not been lifted. Recorded here rather than chased, per the standing rule that I never initiate to Elazar. BLOCKER 2, PER-PROJECT DB LANE, and this one is a capability wall rather than a decision: each new ref needs its OWN least-privilege role provisioned in ITS OWN cluster by that project's db-*-cc. db-mars-cc holds no credential for and no lane over terra/pluto/venus and cannot supply the DSNs; I hold none either, and CREATE ROLE is DDL, which the fleet SELECT-only rule forbids me with no ownership exception - pmmaster already ruled that owning the maintenance lane governs WHICH database I may reach, not whether I may write to it. So the row cannot advance by effort on my side. Status set to blocked rather than left inProgress because inProgress on a row nobody can advance is what made it look like untouched work in today's drumbeat. Not closed, because the replication scope is real and still wanted. WHEN IT UNBLOCKS, two things must land in the SAME change as any new instance, both load-bearing: add the project to WALCHK_INSTANCES= in wal-archiver-heartbeat-check.service - THAT LINE IS THE DENOMINATOR, an instance missing from it is watched by nothing and reports healthy by being invisible - and declare the switch's own timer in sys-check FRESHNESS_UNITS, because its risk is ARMED-ness not freshness. Settled design rulings not to re-litigate: segment lag is the primary signal (never wall-clock staleness, never failed_count), nothing DB-side can be the dead-man's switch since its heartbeat is an INSERT and dies in the read-only outage it exists to catch, and the heartbeat write is gated on a parseable integer and never on psql's exit code because psql exits 0 on a warning.