Disk used-vs-provisioned alarm: unreachable from SQL, and autoscale reports configured while inert below LARGE compute

Sub-tickets

Activity

• wi-cli-venus created · 5w ago
• wi-cli-venus note · 5w ago
  MEASURED THE READ PATHS, and the important finding is that the OBVIOUS numerator is itself a dead instrument. Read-only, 2026-09-11 ~15:05Z, mars ref ustenjufophwhlkzfdso. 1. pg_database_size across all 3 databases = 135 MB (141,095,089 bytes). During MARS-700 this ref sat at 7.70/8 GB = 96.25%, past Supabase's 95% read-only threshold. That is a 57x gap. AN ALARM BUILT ON pg_database_size WOULD HAVE READ ABOUT 1.7% OF 8 GB AT THE EXACT MOMENT THE DATABASE WENT READ-ONLY. This is the same defect class as the row's title - an instrument that reports healthy through the outage it exists to catch - and it is the one someone would reach for first. 2. The component that actually moves IS readable: pg_ls_waldir() = 80 MB over 6 files right now, and pg_stat_archiver shows archived 729 / failed 0 / last_archived 2026-09-11 15:01:37Z. So the archiver is healthy today. MARS-700's 7.7 GB was overwhelmingly unarchived WAL, not table data, which is consistent with the read-only-from-WAL-archiver-failure mechanism already on record. USABLE NUMERATOR = pg_database_size sum PLUS pg_ls_waldir sum. Today that is ~215 MB of 8 GB, ~2.7%. Either component alone is misleading: data-only missed the real outage entirely, and WAL-only would miss a genuine data-growth fill. DENOMINATOR IS NOT READABLE AND I VERIFIED THAT RATHER THAN ASSUMING IT. The Supabase MCP exposes no compute, disk or provisioning surface at all - it is scoped to database, edge functions, storage config and advisors (enumerated, not guessed). The Management API needs SUPABASE_ACCESS_TOKEN, which is absent from my lane: no env block in agents/coder-mars/.claude/settings.local.json, not in pm/06-passwords-and-settings.md, not in the environment. So provisioned size and compute tier are dashboard-only from here. THAT MAKES THE DENOMINATOR A TEXTBOOK CASE FOR THE UNVERIFIABLE-STATE RULE, and mars/shared/md/mars-unverifiable-state.md is a file I own. Provisioned size is a CONSTANT per ref until someone resizes, so it does not need a live read - it needs a recorded value with last-verified, an explicitly named absence of a read path, and an owner. The alarm then compares the SQL-readable numerator against the recorded constant and fires well below 95 percent. The autoscale arm does NOT reduce to a constant the same way: configured-but-inert depends on the compute tier, it reports as configured, and it is exactly the trap this row names. It stays dashboard-only until someone has a management token. WHAT IS LEFT, stated plainly rather than implied: (a) record provisioned=8GB + compute tier per ref in the unverifiable-state file, which needs Elazar or anyone with dashboard access to confirm the figure for mars, venus, pluto and the shared terra/enamel ref eprpywqwzsvrjeaforei - I have the mars number only from MARS-700's incident report, which is a citation and not a read; (b) wire the numerator query into a scheduled check. Neither is blocked on a decision, (a) is blocked on one dashboard read.
• wi-cli-venus note · 5w ago
  ITEM (a) IS NOT BLOCKED - IT WAS MISFRAMED, AND THE CORRECTION CHANGES THE DESIGN OF (b). I said the 8 GB was 'a citation from MARS-700, not a read'. It is weaker than that: NOBODY EVER READ IT. MARS-700's numerator halves are real SQL reads by db-mars-cc - pg_ls_waldir 7,568 MB / 474 files, pg_database_size 131 MB - and their sum is 7,699 MB. 7699/8000 = 96.2 percent, which reproduces the '96.25%' quoted throughout that incident EXACTLY. The percentage was computed FROM an assumed 8 GB volume; it is not evidence OF one, and no reader is named anywhere on the row. Per evolutiva-commons.md The Never-True Class, the tell is the ABSENCE of a date, not a stale one - this premise carries none because there was never a read to stamp. The only thing actually established about the volume is that it is larger than 1 GB, since 7,568 MB of WAL cannot sit on the free tier's 1 GB - which falsifies pm/06 line 26 'Plan: Free' without supplying a replacement. RECORDED, so (a) is DONE for mars as far as this lane can take it: a full entry is now in mars/shared/md/mars-unverifiable-state.md (my file) - no read path stated explicitly with the instrument bounds, the derived-not-measured provenance above, last verified NEVER, and the close route named (a dated dashboard read of provisioned size AND compute tier per ref). Per rule 5 a stale entry is trusted less than no entry, and per rule 6 the absence of a read path is named rather than implied. What remains for venus, pluto and terra/enamel is one dashboard read each and is NOT a Mars deliverable - it goes to Elazar as a single ask, not as a blocker on this row. THE DESIGN CONSEQUENCE, which is the real output of this measurement: THE PRIMARY ARM MUST NOT DEPEND ON THE DENOMINATOR AT ALL. db-terra-cc established during MARS-700 that the WAL:DB RATIO is the wrong instrument - min_wal_size sets a floor, so on a small database the floor dominates and the ratio moves on the denominator; healthy terra reads 6x, HIGHER than healthy venus 2.3x and pluto 3.7x. The signal is ABSOLUTE WAL BYTES AND FILE COUNT. Fleet band measured 2026-09-10: venus 96 MB / 7 files, pluto 240 MB / 16 files, terra 96 MB / 9 files, Mars in incident 7,568 MB / 474 files. An absolute-threshold alarm over that band needs no provisioned size, takes no unverifiable input, and would have fired roughly a day before writes froze. SPEC FOR db-mars-cc - pg_cron job plus plpgsql function, DB lane, not mine to author. Mirror practicas_integrity_check (migration 033 section 5): error-tier appEvents insert riding applog_notify_error_trg, dedupe window so an unfixed condition does not re-page every run. 1. NUMERATOR, two independent components, both already measured read-only this session: sum(pg_database_size(oid)) over pg_database = 135 MB, and sum(size) from pg_ls_waldir() = 80 MB / 6 files at 15:05Z. Baseline total ~215 MB. Record BOTH separately in the appEvents detail, never only the total - data-only missed MARS-700 entirely and WAL-only would miss a genuine data-growth fill. 2. PRIMARY ARM, no denominator: absolute WAL bytes and WAL file count against the fleet band above. Pick the thresholds from that band rather than from a percentage - 474 files against a 7-16 file healthy band is the shape that was diagnostic, and it was diagnostic a full day before the volume filled. 3. SECONDARY ARM, advisory only and explicitly marked as such: percent against a provisioned constant read from ONE named location, so that landing a real dashboard read later changes a VALUE and not the function. Point it at the unverifiable-state entry's figure and make the appEvents detail print numerator AND denominator on every fire, so a reader can see what was compared. 4. LIVENESS: the check writes something EVERY run, info-tier, carrying the two numbers. The existing cron_watchdog_check is silent when healthy, which cannot distinguish 'nothing to do' from 'not looking' - do not copy that half of the pattern. 5. NO EXECUTE GRANT TO mars_app_runtime. pg_ls_waldir is pg_monitor-gated and pg_cron runs in-process as postgres, so no SECURITY DEFINER is needed and the runtime role has no business calling it. 6. FIRST QUESTION BEFORE WRITING ANYTHING: is /data-venus/mars/db-monitoring/2026-09-11-wal-archiver-check.sql already SCHEDULED, or is it still a hand-run file? If it is scheduled, this numerator belongs in that SAME function as a second verdict rather than a new cron row - the WI description says the archiver read does not cover disk, and one function returning both verdicts closes that gap without a second job. If it is not scheduled, scheduling it is the larger half of this row and should land first. 7. Portable recipe for venus/pluto/terra-enamel, but mark the schema-contingent steps (appEvents table name and the notify trigger differ per app). FAN-OUT IS A FOLLOW-ON after the mars instance is proven live, not part of this row. DEFINITION OF DONE ON MY SIDE: db-mars-cc's handoff commit published under my next scoped push, then a cron.job row plus the first heartbeat appEvents row read back. The migration file existing is not done.
• wi-cli-venus note · 5w ago
  PUSHED 68221e6 - item (a) recorded, mars half done from this lane. Doc-only push so THE VERSION INSTRUMENT IS INERT for this deploy and I am not reporting a version match: the minted version stays 2.79.53, and git log origin/main..HEAD was EMPTY before the push, so nothing else rode the branch and deploymentId is the only usable signal. review-depth.sh 68221e6 returns DEPTH=deep CHECK=2a REASON=known-deep path shared/md/mars-unverifiable-state.md SAMPLED=no - I ran it rather than inferring shallow from doc-only, which is the exact error I made on MARS-659. Cold brief owed to coder02-mars-cc and HELD, not sent: that lane is stood down until 17:00 UTC. MY SPEC POINT 1 IS WRONG AND db-mars-cc IS RIGHT - RECORDING THE CORRECTION RATHER THAN QUIETLY DROPPING IT. I said mirror practicas_integrity_check: error-tier appEvents insert riding applog_notify_error_trg. That builds the alarm out of the one instrument that CANNOT WORK IN THE INCIDENT IT EXISTS FOR. Logging to appEvents is a database write, and MARS-700's failure mode was Supabase forcing default_transaction_read_only=on, which refuses every write. The rows describing the outage are rejected BY the outage. That is MARS-703's founding finding and I reproduced the defect it names while citing the row that names it. BUT FOLLOWING THE CORRECTION THROUGH BREAKS ITS REPLACEMENT TOO, and this is the part that decides the architecture. The proposed escape is net.http_get on the same pass, egressing before the verdict can be lost. pg_net does not send inline - it ENQUEUES into net.http_request_queue and a background worker drains it. That enqueue is itself a database write. UNLOGGED skips WAL but still writes heap pages, and forced read-only refuses writes by transaction attribute regardless of durability. So under the terminal condition pg_net fails the same way appEvents does. STATED BOUND: this is reasoned from the mechanism, NOT measured - nobody has exercised pg_net against a read-only Mars and I am not proposing we manufacture one. THEREFORE, and this is the design decision: NO IN-DATABASE ALARM CAN FIRE FROM INSIDE A READ-ONLY DATABASE, so the in-DB job is an EARLY-WARNING arm and must be labelled as one on its own row and in its own comment. It is not weakened by this - absolute WAL FILE COUNT gave roughly a DAY of lead time in MARS-700 (474 files against a 7-16 healthy band, climbing from 09-08 23:46Z while writes did not freeze until 09-09 23:23Z), so an arm that fires at 3-5x the band fires with the database still fully writable and still able to page. What it must never be sold as is coverage of the outage itself. Per evolutiva-commons.md A BROKEN COMPONENT IS NOT A CONTROL: an alarm that is silent because the database is read-only is indistinguishable at the receiver from an alarm that is silent because disk is fine, and only one of those is a guarantee. The arm that survives the outage is necessarily EXTERNAL - the venus timer db-mars-cc names in its point 3, polling from outside the database - and that is the same instrument that gives venus/pluto/terra-enamel coverage, which Mars's own pg_cron structurally cannot. A green Mars alarm is not fleet coverage; both halves go on MARS-703, not here. DECISIONS ANSWERED, so db-mars-cc is not blocked on me: (i) GO on the migration, single migration, same shape as 149, whenever cap allows - I publish the handoff commit. (ii) YES to bytes - the verdict carries WAL BYTES AND FILE COUNT, both arms, because the file's current thresholds are count-only and the incident moved both; the data is already selected and being discarded. (iii) Egress via net.http_get to the hub is accepted as the best available in-DB route, with the bound above written into the migration comment rather than left implicit. (iv) DEPENDENCY RECORDED: MARS-712 measured that postgres's EXECUTE on net.* comes only from PUBLIC and the live ask is to revoke it. If that revoke ships without a paired GRANT TO postgres it takes out email drain, weekly KPI, reminders, the watchdog AND this alarm. That is an argument FOR the paired grant, not against pg_net, and it belongs on MARS-712 as a named consumer before the revoke lands. (v) The psql script cannot be scheduled as written - :'project' and :baseline are client-side interpolation pg_cron does not have - so turning it into a DB object with a baseline argument IS the work of this row, not a wrapper. Accepted. (vi) Privilege fact, measured by db-mars-cc on MARS-703 and worth keeping: pg_ls_waldir() has proacl {supabase_admin=X, pg_monitor=X}; pg_cron runs as postgres which IS in pg_monitor, so cron reaches it, while mars_app_runtime and every agent read-only role is denied - and GRANT EXECUTE as postgres is a NO-OP that emits a WARNING, reports GRANT and exits 0 while leaving the role denied. So SECURITY DEFINER owned by postgres is mandatory the moment anything but cron calls it, and no GRANT substitutes for it. A grant that reports success and changes nothing is the same class as everything else on this row.
• wi-cli-venus commented · 5w ago
  DB LANE STATE, db-mars-cc 2026-09-11 - GO received from coder-mars-cc, migration NOT started, deliberately. pmmaster's cap projection puts exhaustion at ~16:15-16:30Z and it is 16:2xZ now; coder-mars-cc's decision 1 says explicitly not to half-author under cap pressure, and a partial DDL file sitting in applied/ is exactly the intended-vs-verified confusion this lane was pulled up on earlier today. Everything needed to write it in one pass is recorded below so nothing is re-derived. BLOCKING QUESTION ANSWERED: /data-venus/mars/db-monitoring/2026-09-11-wal-archiver-check.sql is NOT SCHEDULED. Mars has 14 pg_cron jobs and ZERO match wal|archiv|waldir|disk - it is a hand-run psql artifact written 12:35 today. So scheduling is the bigger half and lands first, and it cannot be scheduled as written: :'project' and :baseline are psql client-side interpolation, which pg_cron does not have. Converting it to a DB object with the baseline as an argument IS the work of this row. PRIVILEGE FACTS THE MIGRATION MUST BE BUILT ON (measured on MARS-703, not inherited): pg_ls_waldir() has proacl {supabase_admin=X/supabase_admin, pg_monitor=X/supabase_admin}. A pg_cron job runs as postgres, which IS in pg_monitor, so it works from cron. It is DENIED to mars_app_runtime, agent_db_mars_cc, applog_reader, errscan_ro and evolutiva_agent_ro - proven by SET LOCAL ROLE in a rolled-back transaction. GRANT EXECUTE ON FUNCTION pg_catalog.pg_ls_waldir() TO <role> issued as postgres is a NO-OP: WARNING 'no privileges were granted', then reports GRANT, exits 0, role still denied in the next statement of the same transaction. postgres is not the owner. So the function must be SECURITY DEFINER owned by postgres the moment anything other than cron calls it, and no GRANT substitutes for that. THE ARCHITECTURE POINT, settled with coder-mars-cc and it must be IN the migration comment rather than implicit: no in-database alarm can fire from inside a read-only database. appEvents cannot record this class of event because logging is a write. coder-mars-cc then followed that through to pg_net and it breaks the replacement too - pg_net does not send inline, it ENQUEUES into net.http_request_queue and a worker drains it, and that enqueue is a database write. I VERIFIED THE DURABILITY HALF RATHER THAN LEAVING IT REASONED: both net.http_request_queue and net._http_response are UNLOGGED (relpersistence='u'), so they skip WAL - but forced read-only refuses by TRANSACTION ATTRIBUTE regardless of durability, so the conclusion stands and the unlogged status does not rescue it. Nobody has exercised pg_net against a read-only Mars and nobody should manufacture one. CONSEQUENCE, to be stated in the file: the in-DB job is an EARLY-WARNING arm, not coverage of the outage itself. It is not a demotion - 474 files against a 7-16 fleet band was climbing from 09-08 23:46Z while writes did not freeze until 09-09 23:23Z, so an arm at 3-5x the band fires with the database fully writable and able to page. What it must never be sold as is outage coverage: at the receiver, a silent alarm because read-only and a silent alarm because disk is fine are indistinguishable. The surviving arm is the external venus timer, which is also the only thing that can cover the other three refs - both halves belong on MARS-703, not here. AGREED SPEC DELTAS: verdict carries WAL BYTES **and** file count (the shipped script selects bytes and discards them; the incident moved both). Primary arm carries NO denominator - pg_database_size read 131 MB through a full volume, and the 8 GB figure was inferred, never read. net.http_get accepted as the best in-DB egress with the bound above written in. NAMED DEPENDENCY ON MARS-712: postgres's EXECUTE on net.* comes ONLY from PUBLIC. If the revoke ships without the paired GRANT EXECUTE ON net.http_post/net.http_get TO postgres, it takes out email drain, weekly KPI, reminders, the watchdog AND this alarm. This alarm is to be listed as a named consumer on 712 before the revoke lands.
• wi-cli-venus note · 5w ago
  PTD 68221e6, self-verified, Class-S equivalent - and the version half is deliberately NOT counted. Live /api/health returns version 2.79.53, remote package.json on main reads 2.79.53, and THAT MATCH IS INERT: the push was markdown-only so no version was minted, and a frozen number matching itself is not evidence a deploy landed. Established by identity instead: Vercel deployment dpl_494e6LmAqR3SR8EGYv5C5D34s6ty is state=READY target=production and its githubCommitMessage is this commit's; live /api/health serves that same deploymentId; git ls-remote origin main = 68221e6. So prod is serving this commit, proven by the deployment's own commit identity rather than by a number that could not have changed. Coverage statement rather than a clean verdict: this establishes that the doc change is published. It exercises no code path, because there is none in the diff - one markdown file, 15 insertions, zero deletions. Nothing about the disk alarm is live; item (b) has not started. STATE OF THE ROW: (a) DONE for mars from this lane, with the correction that it was never merely a citation - nobody read it. (a) for venus/pluto/terra-enamel is one dashboard read each, goes to Elazar as a single ask, and is NOT a blocker here. (b) is specced, corrected, and GO given to db-mars-cc; it is holding on cap, not on a decision, and the honest scope is larger than 'wire a query into cron' - the psql script cannot be scheduled as written, so converting it into a DB object with a baseline argument IS the work.
• wi-cli-venus note · 5w ago
  DB LANE STATE, db-mars-cc 2026-09-11 - GO received from coder-mars-cc, migration NOT started, deliberately. pmmaster's cap projection puts exhaustion at ~16:15-16:30Z and it is past that; coder-mars-cc's decision 1 says explicitly not to half-author under cap pressure, and a partial DDL file in applied/ is exactly the intended-vs-verified confusion this lane was pulled up on earlier today. Everything needed to write it in ONE pass is recorded here so nothing is re-derived. BLOCKING QUESTION ANSWERED: /data-venus/mars/db-monitoring/2026-09-11-wal-archiver-check.sql is NOT SCHEDULED. Mars has 14 pg_cron jobs and ZERO match wal|archiv|waldir|disk - it is a hand-run psql artifact written 12:35 today. Scheduling is therefore the bigger half and lands first, and it cannot be scheduled as written: :'project' and :baseline are psql CLIENT-SIDE interpolation, which pg_cron does not have. Converting it to a DB object with the baseline as an argument IS the work of this row. PRIVILEGE FACTS THE MIGRATION MUST BE BUILT ON (measured on MARS-703, not inherited): pg_ls_waldir() has proacl {supabase_admin=X/supabase_admin, pg_monitor=X/supabase_admin}. A pg_cron job runs as postgres, which IS in pg_monitor, so it works from cron. It is DENIED to mars_app_runtime, agent_db_mars_cc, applog_reader, errscan_ro and evolutiva_agent_ro - proven by SET LOCAL ROLE inside a rolled-back transaction. GRANT EXECUTE ON FUNCTION pg_catalog.pg_ls_waldir() TO <role> issued as postgres is a NO-OP: WARNING 'no privileges were granted', then reports GRANT, exits 0, role still denied in the very next statement of the same transaction, because postgres is not the owner. So the function must be SECURITY DEFINER owned by postgres the moment anything but cron calls it, and no GRANT substitutes for that. THE ARCHITECTURE POINT, settled with coder-mars-cc, and it must be IN the migration comment rather than implicit: NO IN-DATABASE ALARM CAN FIRE FROM INSIDE A READ-ONLY DATABASE. appEvents cannot record this class of event because logging is a write. coder-mars-cc followed that through to pg_net and it breaks the replacement too - pg_net does not send inline, it ENQUEUES into net.http_request_queue and a worker drains it, and that enqueue is a database write. I VERIFIED THE DURABILITY HALF RATHER THAN LEAVING IT REASONED: net.http_request_queue and net._http_response are both UNLOGGED (relpersistence='u'), so they skip WAL - but forced read-only refuses by TRANSACTION ATTRIBUTE regardless of durability, so the conclusion stands and unlogged does not rescue it. Nobody has exercised pg_net against a read-only Mars and nobody should manufacture one. CONSEQUENCE, to be stated in the file: the in-DB job is an EARLY-WARNING arm, NOT coverage of the outage itself. Not a demotion - 474 files against a 7-16 fleet band was climbing from 09-08 23:46Z while writes did not freeze until 09-09 23:23Z, so an arm at 3-5x the band fires with the database fully writable and able to page. What it must never be sold as is outage coverage: at the receiver, a silent alarm because read-only and a silent alarm because disk is fine are INDISTINGUISHABLE. The surviving arm is the external venus timer, which is also the only thing that can cover the other three refs - both halves belong on MARS-703, not here. AGREED SPEC DELTAS: verdict carries WAL BYTES **and** file count (the shipped script selects bytes and then discards them; the incident moved both). Primary arm carries NO denominator - pg_database_size read 131 MB through a full volume, and the 8 GB figure was inferred, never read. net.http_get accepted as the best in-DB egress with the bound above written in. NAMED DEPENDENCY ON MARS-712: postgres's EXECUTE on net.* comes ONLY from PUBLIC. If that revoke ships without the paired GRANT TO postgres it takes out email drain, weekly KPI, reminders, the watchdog AND this alarm. Recorded as a named consumer on 712.
• wi-cli-venus note · 5w ago
  TWO CORRECTIONS TO MY OWN NOTES ABOVE, both narrowing overclaims I made this hour. Neither changes the design. 1. I wrote 'NO in-database alarm can fire from inside a read-only database'. That absolute is too strong and MARS-700's own record contradicts it: db-mars-cc measured that BEGIN; SET LOCAL default_transaction_read_only=off; ... is ACCEPTED from inside, so a postgres-owned function COULD override the GUC and enqueue while space remains. The correct statement carries the qualifier: no in-database alarm can fire WITHOUT OVERRIDING THE GUC, and overriding it is the documented emergency-data-extraction path that MARS-700 explicitly ruled out as a fix, because every write it enables generates more unarchivable WAL and deepens the outage. It is fragile as an alarm besides - pg_net's worker does its own queue-DELETE and response-INSERT in separate transactions, which fail under the GUC, so the request either never sends or resends every tick. CONCLUSION UNCHANGED: in-DB arm is early-warning, the surviving arm is external. But an absolute gets re-inherited and this one would have been false. 2. I wrote 'nobody ever read it' about the 8 GB. What I actually established is that NO READER IS NAMED anywhere on MARS-700 and that the 96.25 percent is reproducible as arithmetic over the two SQL reads. Elazar was in the dashboard during that incident - he restarted the project and read the autoscale requirement - so he may well have seen the disk figure without it being written down. The repo entry at 68221e6 says 'no reader is named', which is the claim the evidence supports; my WI and DM wording overshot it. CONSEQUENCE FOR THE ASK: the dashboard read request to Elazar must include MARS, not only venus/pluto/terra-enamel. That is how this entry closes - one read, four refs, provisioned size AND compute tier each.
• wi-cli-venus note · 5w ago
  MIGRATION 150 APPLIED AND VERIFIED LIVE, db-mars-cc 2026-09-11. Handoff commit e529df3 (Local-Author db-mars-cc), UNPUBLISHED - migration plus regenerated db/schema.md and db/reference-data.tsv. coder-mars-cc: ready for your next scoped push. SPEC POINT 6 ANSWERED FIRST BECAUSE IT DECIDED THE SHAPE: the /data-venus script is NOT scheduled (14 pg_cron jobs, zero match wal|archiv|waldir|disk), so scheduling was the larger half and it lands here as ONE function carrying BOTH verdicts, not a second cron row. It could not have been scheduled as written regardless - :'project' and :baseline are psql CLIENT-SIDE interpolation, which pg_cron does not have. LIVE STATE, read back after apply: cron.job 23 'mars-wal-disk-health-check', schedule '7 * * * *', username postgres, active=t. First heartbeat at 16:37:43Z - 'wal 6 files / 80 MB, data 135 MB, total 215 MB (2.62% of a DERIVED 8 GB, advisory) - OK'. public.wal_disk_health_check() is SECURITY INVOKER, search_path pinned to 'pg_catalog, pg_temp' with every public reference schema-qualified, acl {postgres=X,service_role=X}, no PUBLIC. All three MARS-707 fences exit 0. EVERY SPEC POINT, and where it went. (1) BOTH numerator components recorded separately in the detail on every row, never only the total. (2) Primary arm is absolute WAL bytes AND file count against the fleet band, no denominator; thresholds files >50 warn / >150 page and bytes >1 GB warn / >3 GB page, with the derivation argued in the header rather than asserted - the byte figures come from max_wal_size=1024 MB, so exceeding it means WAL is not being recycled, which IS the failure. (3) Percentage is advisory-only, printed on every fire with numerator AND denominator visible, and the 8 GB lives in ONE named constant pointing at mars-unverifiable-state.md so a real dashboard read later changes a VALUE and not the function. (4) Liveness: info-tier heartbeat every run, explicitly departing from cron_watchdog_check's silent-when-healthy half. (5) No EXECUTE grant to mars_app_runtime, no SECURITY DEFINER - pg_cron runs as postgres which is in pg_monitor. (7) Portable recipe: NOT attempted here, correctly a follow-on after the Mars instance is proven live; the schema-contingent parts are the appEvents table name and the notify trigger. THE BOUND IS IN THE FILE RATHER THAN IMPLIED, per your correction, and I verified the half you reasoned: net.http_request_queue and net._http_response are both UNLOGGED (relpersistence='u'), so they skip WAL - but forced read-only refuses by TRANSACTION ATTRIBUTE regardless of durability, so unlogged does not rescue it and your conclusion stands. The function therefore carries NO egress at all. The header states plainly that this is an EARLY-WARNING arm, that a silent alarm because read-only and a silent alarm because disk is fine are indistinguishable at the receiver, and that the surviving arm is the external venus timer on MARS-703. THE MIGRATION PROVES ITSELF: pre-flight asserts postgres is in pg_monitor AND that pg_ls_waldir() is actually callable, so a future loss of that membership fails at apply instead of silently every hour; the post-check invokes the function for real, reads the heartbeat back, asserts both numerator components are present, and asserts exactly one cron.job row. Your definition of done - a cron row plus a first heartbeat read back - is performed by the migration, not claimed by it. DRY-RUN CATCH KEPT IN THE FILE: the 8 GB constant written as 8 * 1024 * 1024 * 1024 raised 'integer out of range' - the multiplication evaluates in int4 BEFORE assignment to the bigint variable. Now 8::bigint with the reason in the comment. STILL OPEN ON THIS ROW AND NOT MINE: (a) the provisioned-size and compute-tier dashboard read for mars, venus, pluto and eprpywqwzsvrjeaforei, which is one ask to Elazar; and the autoscale configured-but-inert arm, which does not reduce to a constant the way provisioned size does and stays dashboard-only until someone holds a management token.
• wi-cli-venus note · 5w ago
  REVIEW OF MIGRATION 150's THREE UNSPECIFIED DECISIONS (db-mars-cc asked to be attacked on these rather than on the parts already ruled). Two accepted outright, one has a real problem that is not a bug and does not block. (b) DEDUPE ON (action, level) RATHER THAN action ALONE - ACCEPTED, and it is better than the pattern I told them to mirror. practicas_integrity_check dedupes on action, so under that rule a WARN escalating to PAGE inside the 55-minute window is SWALLOWED by the warn that preceded it: the alarm goes quiet at the exact moment the condition gets worse. I specified parity without noticing that. Do not revert it to parity; if anything practicas_integrity_check should adopt this, and that is a separate row. (c) THE MIGRATION PROVES ITSELF - ACCEPTED, and it is the strongest part. Pre-flight asserts postgres is in pg_monitor AND that pg_ls_waldir() is genuinely callable, so a future loss of that membership fails AT APPLY instead of silently every hour; post-check invokes the function for real, reads the heartbeat back, asserts both numerator components are present and asserts exactly one cron.job row. That performs my definition of done rather than claiming it, and makes a re-apply self-verifying. This is the answer to the inert-arm class the row is about - an alarm that is never exercised is indistinguishable from one that works. (a) BYTE THRESHOLDS >1 GB WARN / >3 GB PAGE - ACCEPTED AS NUMBERS, but the stated rationale for having BOTH arms does not survive arithmetic, and the row should say so rather than carry an overstated claim. MARS-700 measured 7,568 MB across 474 files = 15.97 MB per file, i.e. the standard 16 MB segment. So MARS-703's file thresholds convert directly: 50 files = 800 MB, 150 files = 2.4 GB. Against 1 GB warn and 3 GB page, THE TWO ARMS ARE THE SAME INSTRUMENT EXPRESSED IN TWO UNITS. They will fire within one segment of each other, and neither can catch a case the other misses while segment size is fixed. That is not a defect - redundant expression of a threshold is cheap and survives a segment-size change, which is the one scenario where they DO diverge - but 'bytes AND file count' must not be recorded as two independent signals, because it reads as more coverage than exists. WHERE THE INDEPENDENT SIGNAL ACTUALLY IS, and this is the part worth fixing later: the genuinely orthogonal component is the DATA half, pg_database_size. It is the one that moves for a reason WAL cannot express - a real data-growth fill - and it is precisely the case I gave as the reason for a two-component numerator in the first place. Today it is 135 MB and folded into the total only. An arm on the data component alone, at a level derived from its own history rather than from max_wal_size, is what makes the numerator two signals instead of one. NOT filed as a change to 150: it needs a baseline nobody has, since Mars has never recorded data-size history, and inventing a threshold for it now would be the unsourced-number class this row keeps catching. Recording it as the named next increment. Also accepted without argument: >1 GB warn derived from max_wal_size=1024 MB is the right anchor even though max_wal_size is a SOFT target that Postgres routinely overshoots under load, because Mars's healthy band is 80-240 MB - so 1 GB is 4-12x healthy and the soft-overshoot noise sits far below it. And 3 GB gives real lead: at MARS-700's measured accumulation rate of roughly 220 MB/h, 3 GB to the 7.5 GB terminal figure is about 20 hours. CORRECTION db-mars-cc ISSUED AGAINST ITSELF, worth keeping because it changes when a re-export is owed: schema.md carries no function BODIES, which is why migration 148 produced an empty diff - but it does carry a function COUNT, which moved 68 to 69 here. So a migration that ADDS or DROPS a function does change the mirror; only one that REPLACES existing functions does not. The earlier 'function-only migrations need no re-export' is too broad. AND THE DEPENDENCY I RECORDED ON MARS-712 IS NARROWER THAN I WROTE IT: the shipped function carries NO pg_net egress at all, so it is not a consumer of that grant today. It stays listed there because the design reached for net.http_get first and the reason it was rejected belongs next to the revoke decision - but MARS-712 must not count this alarm among the live consumers a bare revoke would break.
• wi-cli-venus note · 5w ago
  MIGRATION 150 CORRECTED, RE-APPLIED AND RE-VERIFIED, db-mars-cc 2026-09-11. Second handoff commit 2d29d82 on top of e529df3; both UNPUBLISHED and both go out with coder-mars-cc's next push. Declared per the applied-but-unpublished rule: the file was corrected in place rather than patched in a follow-up, because it had not been published, and prod was re-verified against the final text. THE DEFECT (coder02-mars-cc, found after apply, and it inverted my own comment): c_dedupe was interval '55 minutes' while the job runs '7 * * * *' = every 60 minutes. The guard looks for an existing alarm row newer than now() - 55 minutes, so at 11:07 the 10:07 row is already 60 minutes old, fails the test, and a second alarm is written - hourly, forever. The comment claimed an unfixed WARN would not re-page every hour; it would have. A 55-minute window could only ever suppress a SECOND invocation inside the same hour - a manual run or a cron catch-up - which is not the case the dedupe exists for. Inherited from practicas_integrity_check, which runs hourly with a 55-minute window and therefore carries the SAME latent defect; worth a look on its own row, not fixed here. FIX: 6 hours, with the governing rule now stated at the declaration - THE DEDUPE WINDOW MUST EXCEED THE CRON INTERVAL OR IT IS DECORATIVE. A persistent unfixed condition surfaces 4x/day, and a WARN escalating to PAGE still gets through immediately because the dedupe keys on LEVEL as well as action. Both reviewers independently told me to KEEP that (action, level) divergence from practicas_integrity_check rather than revert to parity: deduping on action alone means a WARN escalating to PAGE inside the window is swallowed by the warn before it, so the alarm goes quiet exactly when the condition worsens. WHY THE MIGRATION'S OWN PROOF MISSED IT, now recorded in the file: a single invocation cannot exercise a dedupe. The post-check invokes the function for real and reads its heartbeat back, which is the right standard for everything else in the file and is structurally blind to this one. Found by review, not by proof - and that is the honest description of the coverage. TWO HEADER CORRECTIONS, no code change, both from review. (1) The byte thresholds were justified BY max_wal_size, which is a CHECKPOINT TRIGGER and not a ceiling - Postgres exceeds it legitimately under write bursts, behind archiving, or on an unconsumed replication slot. The numbers stand on the evidence actually held (1 GB is ~12x Mars's healthy 80 MB; 3 GB is ~40% of the 7,568 MB terminal figure, ~20h of lead at MARS-700's ~220 MB/h) and are now pinned to the MEASURED FLEET BAND. The hazard was forward-looking: a threshold justified by a GUC moves when someone tunes the GUC, and a later reader raising max_wal_size to 4 GB would mechanically lift the warn and delete the early warning. (2) The header presented bytes and file count as two independent signals. They are the SAME instrument in different units - MARS-700 measured 7,568 MB over 474 files = 15.97 MB, standard 16 MB segments, so 50 files is ~800 MB and 150 files ~2.4 GB against the 1 GB / 3 GB arms; they fire within one segment of each other. Both are kept (free, and they diverge if segment size changes) but the file no longer claims coverage it does not have. NAMED NEXT INCREMENT, not guessed at: the genuinely orthogonal component is the DATA half (pg_database_size), the one that moves for a reason WAL cannot express. It is recorded in the detail on every run but gates NO verdict, because a data-size threshold today would be an invented number with no baseline - the unsourced-premise class. RE-VERIFIED LIVE: prosrc carries interval '6 hours', no trace of the 55-minute window, no max_wal_size reference. cron.schedule upserts by jobname so the job was updated rather than duplicated - exactly one row, jobid 23, '7 * * * *', postgres, active. Pre-flight and post-check passed again, verdict OK at 6 files / 80 MB. All three MARS-707 fences exit 0.
• wi-cli-venus note · 5w ago
  PUBLISHED AND CONFIRMED WORKING - definition of done met. coder-mars-cc pushed 68221e6..0e8fef0 (v2.79.56); both my handoff commits for this row, e529df3 and 2d29d82, are live on origin. Verified AUTHORITATIVELY with git ls-remote origin main = 0e8fef038c63661ac96405581070eba097ddc936 and a merge-base ancestry check, NOT with git log origin/main..HEAD - that family reads the stale remote-tracking ref and is not a publication check, which is what mutually blocked all three Mars lanes for an hour today. LIVE STATE RE-CONFIRMED AFTER PUBLICATION: exactly 1 cron.job row named mars-wal-disk-health-check, 2 heartbeat rows, latest reading 'wal 6 files / 80 MB, data 135 MB, total 215 MB (2.62% of a DERIVED 8 GB, advisory) - OK'. The corrected 6-hour dedupe window is what is live; the 55-minute version never survived a scheduled fire. MARS-711 MY HALF IS COMPLETE. What remains on this row is not mine and not blocked on me: (a) the provisioned-size and compute-tier dashboard read for mars, venus, pluto and eprpywqwzsvrjeaforei - one ask to Elazar, and it is what turns the advisory percentage arm into a real one; (b) the autoscale configured-but-inert arm, which does not reduce to a constant the way provisioned size does and stays dashboard-only until someone holds a management token; (c) the portable recipe for the other three refs, correctly a follow-on now that the Mars instance is proven live - the schema-contingent parts are the appEvents table name and the notify trigger; (d) the named next increment from review, a data-size arm using pg_database_size, deliberately left ungated because a threshold today would be an invented number with no baseline. ONE THING I AM HANDING BACK RATHER THAN FIXING SILENTLY: public.practicas_integrity_check (migration 033 section 5) runs hourly with a 55-minute dedupe window and therefore carries the same defect this row's review found in 150 - the window can never suppress a consecutive run, so an unfixed condition re-pages every hour. That is another migration's object and belongs on its own row, not folded into this one.
• wi-cli-venus note · 5w ago
  SCOPE STATUS AT HANDOFF TO ELAZAR. Both work items this row named are DONE; what remains is not work, it is one read only Elazar can perform. (b) WIRE THE NUMERATOR INTO A SCHEDULED CHECK - DONE AND LIVE. db-mars-cc's migration 150, published under 0e8fef0 (e529df3 + 2d29d82). cron.job 23 'mars-wal-disk-health-check', '7 * * * *', username postgres, active. First heartbeat 16:37:43Z: wal 6 files / 80 MB, data 135 MB, total 215 MB, verdict OK. Function is SECURITY INVOKER, search_path pinned to pg_catalog+pg_temp with every public reference schema-qualified, acl {postgres=X,service_role=X}, no PUBLIC, not granted to mars_app_runtime. All three MARS-707 fences exit 0. The migration proves itself at apply rather than claiming it - pre-flight asserts postgres is in pg_monitor AND that pg_ls_waldir() is genuinely callable, post-check invokes the function for real and reads the heartbeat back. (a) RECORD PROVISIONED SIZE + COMPUTE TIER - DONE for mars as far as any agent lane can take it, published at 68221e6 in mars/shared/md/mars-unverifiable-state.md with 'Last verified: NEVER', the instrument bounds named explicitly, and the derived-not-measured provenance of the 8 GB figure proved by arithmetic. pm/06 line 26's falsified 'Plan: Free' annotated in place at 0e8fef0 rather than left standing as the uncorrected copy. WHAT IS ACTUALLY LEFT, and neither half is agent-actionable: (i) ONE DASHBOARD READ - provisioned disk size and compute tier for ustenjufophwhlkzfdso, venus, pluto and eprpywqwzsvrjeaforei. (ii) THE AUTOSCALE ARM, which does NOT reduce to a constant the way provisioned size does - configured-but-inert depends on the compute tier and reports as configured, which is the trap this row's title names. Both need the dashboard; the Management API route needs SUPABASE_ACCESS_TOKEN, absent from this lane's settings.local.json, pm/06 and the environment. SETTING THIS ROW TO BLOCKED WITH THE CONDITION STATED, deliberately NOT deferred: setting a row to deferred CLEARS blockedReason, so the waiting condition would survive only in prose and be invisible to every query - which is how a row ends up looking like work in progress that nobody owns. The alarm is LIVE and protecting Mars in the meantime; this row is not gating any protection. ONE BOUND THAT MUST TRAVEL WITH THE CLOSURE WHENEVER IT COMES: the shipped arm is EARLY-WARNING ONLY and says so in its own header. No in-database alarm can fire from inside a read-only database without overriding the GUC, and that override is the emergency-extraction path MARS-700 ruled out as a fix. At the receiver a silent alarm because read-only and a silent alarm because disk is fine are indistinguishable. The arm that survives the outage is external and lives on MARS-703, which also carries the other three refs - Mars's own pg_cron structurally cannot see them. A green Mars alarm is not fleet coverage.
• wi-cli-venus blocked · 5w ago
  BOTH WORK ITEMS DONE AND LIVE. The scheduled check is live (migration 150, cron.job 23, hourly, first heartbeat 16:37:43Z, verdict OK) and the unverifiable-state record is published at 68221e6. What remains is NOT agent-actionable and is not work: (i) ONE DASHBOARD READ by Elazar - provisioned disk size AND compute tier for ustenjufophwhlkzfdso (mars), venus, pluto and eprpywqwzsvrjeaforei (terra/enamel); (ii) the AUTOSCALE arm, which unlike provisioned size does not reduce to a constant - configured-but-inert depends on the compute tier and reports as configured, which is the trap in this row's title. No SQL or MCP read path exists for either: Supabase MCP exposes no compute/disk/provisioning surface at all, and the Management API needs SUPABASE_ACCESS_TOKEN, absent from this lane's settings.local.json, from pm/06 and from the environment. Blocked rather than deferred ON PURPOSE - deferred clears blockedReason, which would leave this condition visible only in prose and invisible to every query. Mars is not unprotected while this waits: the early-warning arm is live and would have fired roughly a day before MARS-700's writes froze.
• wi-cli-venus assigned · 4w ago
  coder03-mars-ca
• coder-mars-ca assigned · 2w ago
  coder-mars-ca / coder
• coder-mars-ca assigned · 6d ago
  coder-mars-cc