P0: Mars production database is in READ-ONLY mode - all writes failing

Sub-tickets

Activity

• wi-cli-venus created · 5w ago
• wi-cli-venus note · 5w ago
  Diagnosis blocked on connectivity, plus a strong lead and a risk. db-mars-ca 2026-09-10: direct DB host timed out at 10s; the production transaction pooler then failed after 15s with FATAL ECHECKOUTTIMEOUT; Supabase connector execute_sql is permission-denied. So pg_is_in_recovery, default_transaction_read_only and the latest row timestamps are all still UNCONFIRMED. Those connection failures are MARS-698, a separate concurrent problem (Supavisor's 15s budget, not our 10s connectionTimeoutMillis), and because it duty-cycles, one failure proves nothing - retry in bursts. QUOTA IS RULED OUT, WHICH CHANGES THE HYPOTHESIS. Elazar's dashboard snapshot reports Database Size 0 / 0.5 GB. Supabase's usual cause for this error is disk-quota exhaustion forcing a project read-only; that is not what is happening. Therefore the read-only state is CONFIGURED, not enforced. LEAD, timing-based, not asserted: MARS-694 cut the app runtime over to a low-privilege mars_app_runtime role and was ROLLED BACK, with heavy role/grant work through 09-09 - function grants, SECURITY DEFINER search_path pins, a revoke of schema-extensions USAGE. The estimated write-failure onset (~2026-09-09 16:25Z, from appEvents having no row since then) sits inside that window. A default_transaction_read_only=on left on a role by that rollback would produce exactly this signature: reads fine, SELECT 1 healthy so /api/health?db=1 reports dbOk:true, every UPDATE refused, sub-second. First query when a connection lands: SELECT r.rolname, d.datname, s.setconfig FROM pg_db_role_setting s LEFT JOIN pg_roles r ON r.oid=s.setrole LEFT JOIN pg_database d ON d.oid=s.setdatabase; Also check the ALTER DATABASE level, and read transaction_read_only over the :6543 POOLER path specifically - a Supavisor-side setting would not show on a direct session. Nothing to be fixed by an agent. Reverting a role setting is a write and is Elazar's call, mid-incident especially. RISK RAISED SEPARATELY: nw-whey-ca disabled evolutiva-backup@mars.timer today. Its own verification shows the timer was HEALTHY - a real twice-daily pg_dump, latest run succeeded at 13,981,935 bytes. Disabling the only durable recovery artifact for the one app in an unexplained write-failure state, while that failure may yet prove to be a bad config change, is backwards. I have asked nw-whey-ca for the reason and to hold as-is; re-enabling is Elazar's call. Escalated to him.
• wi-cli-venus note · 5w ago
  CONFIRMED by db-mars-cc 2026-09-10 09:06:45Z, and the compute-stall hypothesis is excluded. Both poolers answered in ~1s (direct :5432 still timed out at 12s, which is MARS-698 and separate): pg_is_in_recovery=f, default_transaction_read_only=ON, transaction_read_only=ON, pg_database_size=131,550,355 bytes. The database is reachable and healthy and is explicitly in READ-ONLY MODE at role/session level - not a replica, not recovery, not a stall. QUOTA IS EXCLUDED on two independent readings: dashboard reports 0/0.5 GB and the measured size is ~0.13 GB. Supabase's usual storage-cap read-only does not fit and must not be reported as the cause. ONSET CORRECTED, and I am correcting my own figure: I estimated ~2026-09-09 16:25Z from applog's 'no new warn events since' line. That was a WARN-FILTERED read, so it was a floor, not an onset - a general lesson for this WI, a filtered max is not a max. True figures: max(appEvents.createdAt)=2026-09-09 23:23:24Z, i.e. ~9.7h with no write of any kind; max(practicas.createdAt)=2026-09-08 22:07:05Z, which is student activity and not by itself evidence of failure. WHY NOTHING ALARMED FOR 9.7 HOURS, restated because it is the reusable finding: /api/health?db=1 probes SELECT 1, and a SELECT succeeds inside a read-only transaction, so the probe returns dbOk:true straight through a total write outage. appEvents cannot record it either - logging is a write. Both instruments are structurally blind to this class, the same way they are blind to MARS-698 but for a different reason. Mars currently has NO instrument that detects 'writes are refused'. TWO HYPOTHESES REMAIN and they need opposite responses; one query separates them - SELECT r.rolname, d.datname, s.setconfig FROM pg_db_role_setting s LEFT JOIN pg_roles r ON r.oid=s.setrole LEFT JOIN pg_database d ON d.oid=s.setdatabase: (a) a row attaches default_transaction_read_only=on to a role or the database -> WE did it, almost certainly residue of the MARS-694 mars_app_runtime cutover rollback whose role/grant work ran across 09-09; ours to revert, one line, in our control. (b) no such row -> platform-enforced, a Supabase dashboard/support matter only Elazar can act on. Requested from db-mars-cc, which holds a working connection. No agent reverts anything found - a revert is a write and Elazar's call.
• wi-cli-venus note · 5w ago
  HYPOTHESIS (a) RULED OUT - MARS-694 is not the cause. db-mars-cc read pg_db_role_setting (10 rows, on attempt 6; the pool is duty-cycling and most attempts still hit ECHECKOUTTIMEOUT/auth-timeout at 14s): - mars_app_runtime: {statement_timeout=10000} only - NO read-only setting - postgres: {search_path=...} only - NO read-only setting - database postgres: {app.settings.jwt_exp=3600} only - NO read-only setting - the ONLY default_transaction_read_only=on belongs to supabase_read_only_user, which is Supabase's own dedicated read-only role and is expected to carry it; the app does not connect as that role. :6543 diagnostic session current_user/session_user = postgres. Independently confirmed by db-mars-ca. Nothing was changed. So the read-only state does NOT come from role or database settings. It is being imposed from outside pg_db_role_setting - instance-level configuration, i.e. platform-enforced. OPEN GAP, and it is the difference between handing Elazar a state and handing him a cause: Supabase forces read-only when the INSTANCE DISK fills, and pg_database_size is NOT disk usage. 131 MB of database says nothing about WAL, temp files, or a stuck replication slot. A slot that stops advancing pins WAL indefinitely and fills the volume while the database itself stays tiny - which would explain all three open symptoms at once: disk-full read-only (this WI), Disk IO Budget depletion and pool pressure (MARS-698), and a dashboard reporting a small Database Size against a full volume. Requested from db-mars-cc, SELECT-only: pg_replication_slots with retained WAL via pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn); total WAL via pg_ls_waldir(); and the size of the net schema, since pg_net is UNLOGGED/beta and its bloat on this stack was previously noted and never explained. An inactive slot with a large retained figure, or a WAL total dwarfing the 131 MB database, is the answer. If all three come back small it is genuinely platform-side with no local explanation, and Elazar still takes a far better ticket to Supabase than 'the database went read-only'. Nothing to be dropped or reverted either way - a stuck slot is exactly the kind of object that looks safe to drop and is not.
• wi-cli-venus note · 5w ago
  ROOT CAUSE LOCATED - WAL is filling the volume. db-mars-cc, SELECT-only, clean read on attempt 2: - pg_replication_slots: 0 ROWS. No slot exists, active or inactive. Stuck-slot hypothesis RULED OUT. - net schema (pg_net): 6 objects, 7,432 kB. Not a bloat source. RULED OUT, and this also retires the long-standing unexplained-pg_net-bloat note for Mars. - WAL: 7,568 MB across 474 files, against pg_database_size = 131 MB. A ~57x discrepancy. 7.5 GB of WAL with NO replication slot holding it is the anomaly and it explains everything open: - MARS-700: Supabase forces a project read-only when the INSTANCE DISK fills. The disk is filling with WAL, not table data, which is exactly why the dashboard shows a small Database Size (131 MB / 0.5 GB) while the volume is full. Every read stays fine, every write is refused. - MARS-698: WAL that cannot be recycled means continuous write and checkpoint pressure, which matches the reported Disk IO Budget depletion and pool pressure, and matches MARS-688's checkpoint writing 32 KB with write=85.144s. It also explains why the stalls duty-cycle and self-recover, and why a DIRECT :5432 connection failed alongside pooled ones - the constraint is the instance's IO, not Supavisor. WHY WAL IS NOT BEING RECYCLED IS NOT DETERMINABLE FROM SQL HERE. With no slot pinning it, the remaining candidates are checkpointing falling behind or WAL ARCHIVING FAILING - a stuck archive_command piles WAL up indefinitely, and Supabase's PITR/backup machinery is a WAL archiver. That would also fit the 'mystery backup repeatedly downloading Mars' Elazar reported. checkpoint state and archive_command status are not queryable on this surface, so this is where agent diagnosis genuinely ends. FOR ELAZAR - this is a Supabase support ticket with hard evidence, not a vague report. Suggested content: project ustenjufophwhlkzfdso is in enforced read-only; pg_database_size is 131 MB but pg_ls_waldir() reports 7,568 MB across 474 WAL files; pg_replication_slots is EMPTY so nothing is holding WAL; writes have been refused since approximately 2026-09-09 23:23Z (last appEvents row); ask why WAL is not being recycled, whether WAL archiving/PITR is failing on this project, and to clear the read-only lock once resolved. Also ask whether the repeated full-database downloads observed against this project are theirs. NO AGENT ACTION REMAINS AND NONE SHOULD BE TAKEN. Nothing here is fixable by us: we cannot force a checkpoint, cannot touch archiving, and must not drop anything. db-mars-cc attempted no writes and stopped.
• wi-cli-venus note · 5w ago
  RESEARCH: the documented mechanism matches our signature, including the part that looked contradictory. Supabase archives WAL with WAL-G. The documented failure is explicit - if WAL-G backups are failing, WAL is NEVER cleaned up and accumulates with NO slot involved. Separately, Supabase CAPS slot WAL retention at roughly 4 GB, not user-configurable, and INVALIDATES a slot that exceeds the cap. Those two facts together resolve what looked like a contradiction in our own measurement. 'Runaway WAL' normally means a stuck replication slot, and we measured ZERO slots, which initially argued against the whole class. The cap explains the absence: a slot (Realtime, or an ETL/replication consumer) falls behind, blows past ~4 GB, gets invalidated and disappears - so by the time anyone looks, pg_replication_slots is empty - while the WAL it already pinned is never archived away because WAL-G is failing or behind. Disk fills. Project goes read-only. AN EMPTY SLOT LIST IS THEREFORE NOT EVIDENCE THAT A SLOT WAS NEVER THE PROBLEM; it is consistent with a slot having been the trigger and then being cleaned up. Do not quote 'no slots, so slots are ruled out' as a finding on this WI. It also predicts the two symptoms we could not otherwise place: the 'mystery backup repeatedly downloading Mars' that Elazar observed is what a failing archiver retrying looks like, and that retry loop is a credible source of the Disk IO Budget depletion behind MARS-698. Documented remedy per the same sources: ensure WAL-G backups are running - otherwise WAL is never cleaned up - then drop stale slots and vacuum, after which disk shrinks safely. NONE OF THAT IS REACHABLE BY US. WAL-G is Supabase-managed and we have no slot to drop. This stays a Supabase support matter for Elazar; the value of the research is that the ticket can now name WAL-G/archiving as the suspected failure rather than asking what happened. Sources: supabase.com/docs/guides/platform/database-size ; github.com/orgs/supabase/discussions/40546 ; supabase.com/docs/guides/database/replication ; mydbops.com/blog/postgresql-wal-file-retention ; medium.com/@jramcloud1 How to Clean Up WAL Files and Replication Slots in PostgreSQL. PREVENTIVE ACTION TAKEN: asked db-mars-ca, db-pluto-cc and db-venus-cc to run the same three SELECT-only checks (pg_ls_waldir total, pg_database_size, pg_replication_slots with retained WAL) against their OWN projects, since nothing in the fleet monitors WAL size and Pluto and Venus can reach this state the same way. Also asked whether their project has previously hit enforced read-only and what actually cleared it - Elazar recalls this being solved on another Evolutiva project, and a real precedent beats a reconstruction. Explicitly instructed: do not drop any slot (it may be Realtime's or a live ETL consumer, and dropping is irreversible), no writes, no dashboard changes.
• wi-cli-venus note · 5w ago
  FLEET WAL COMPARISON - venus reports, and it gives us the healthy baseline Mars lacked. VENUS (db-venus-cc, measured 2026-09-10): wal_total = 96 MB across 7 files, db_size = 41 MB, pg_replication_slots EMPTY. Ratio WAL:DB = ~2.3x. MARS: wal_total = 7,568 MB across 474 files, db_size = 131 MB, pg_replication_slots EMPTY. Ratio ~57x. So an empty slot list is NORMAL on this stack - venus has none either and is perfectly healthy. That kills any reading of Mars's zero-slot result as itself abnormal, and it isolates the anomaly to a single number: WAL VOLUME. 474 WAL files against 7 is the finding; 96 MB is what a working archiver leaves behind. Mars is retaining ~79x the WAL of a comparable Evolutiva app with a 3x larger... correction, with a 3x SMALLER database (41 MB vs 131 MB), so normalising by size makes the gap worse, not better. This also makes the case that Mars's WAL is not being archived away rather than merely being generated fast: venus runs the same platform, same extensions family and the same drain-cron pattern, and its WAL is recycled down to 7 files. ASK 1 ANSWERED, NEGATIVELY, and recorded as such: db-venus-cc reports NO precedent in venus memory or incident history for enforced read-only, full disk, or runaway WAL on any Evolutiva project, and explicitly declined to force a match. Elazar recalls this being solved somewhere before; venus is not it. Pluto and Mars lanes still to answer. Do not treat the recollection as confirmed until a lane produces the actual incident - a half-remembered precedent that turns out to be a different failure would send the Supabase ticket in the wrong direction. LANE MOVE: db-venus-cc hands its active lane to db-venus-ca (token constraint), assist-only from here. Route venus DB asks to db-venus-ca. That is the fourth lane handover today - bin-venus-cc->bin-venus-ca, coder-venus-cc->coder-venus-ca, db-mars-cc->db-mars-ca, db-venus-cc->db-venus-ca.
• wi-cli-venus note · 5w ago
  FLEET WAL BASELINE COMPLETE - Mars is the sole outlier by a wide margin. VENUS: WAL 96 MB / 7 files, DB 41 MB -> ratio ~2.3x, slots EMPTY PLUTO: WAL 240 MB / 16 files, DB 65 MB -> ratio ~3.7x, slots EMPTY MARS: WAL 7,568 MB / 474 files, DB 131 MB -> ratio ~57x, slots EMPTY Two conclusions the comparison licenses that a Mars-only reading could not: 1. An EMPTY slot list is NORMAL on this stack. All three Evolutiva projects have zero replication slots and two of them are perfectly healthy. Mars's zero-slot result is not itself abnormal and must not be reported as part of the anomaly. 2. The anomaly is a SINGLE number - WAL volume, 474 files against 7 and 16. The healthy ratio band on identical platform, extensions and drain-cron pattern is 2-4x; Mars is at 57x. That is the argument that Mars's WAL is NOT BEING ARCHIVED AWAY rather than merely being generated quickly: two sibling projects on the same platform recycle down to single- and double-digit file counts. ASK 1 ANSWERED NEGATIVELY BY BOTH PEERS. db-venus-cc: no precedent in venus history for enforced read-only, full disk or runaway WAL, and it explicitly declined to force a match. db-pluto-cc: same, no precedent. Elazar's recollection of this being solved before is NOT in pluto or venus. Do not treat it as confirmed until a lane produces the actual incident - a half-remembered precedent that turns out to be a different failure would aim the Supabase ticket wrongly. Asking terra/enamel next to finish the search. FINDING THAT NEEDS RE-READING, from pluto's memory - PLUTO-708. On 2026-08-22 BOTH Pluto and Mars received a Supabase 'Disk IO Budget depleting' email while idle, and it was closed as a TIER-UPSELL NOTICE, no-defect. For Pluto that verdict still looks right - its WAL is 240 MB and it is healthy today. FOR MARS IT MAY HAVE BEEN A GENUINE EARLY WARNING that was discarded because a sibling project got the same email at the same time. The reasoning at the time - 'both projects got it while idle, therefore it is marketing' - treated correlation across projects as evidence of noise, but a shared platform alert can be true for one project and noise for another. Mars is now measurably IO-depleted with 7.5 GB of unrecycled WAL, roughly three weeks later. Not proof the August alert was about this, and I am not reopening PLUTO-708, whose Pluto-side verdict stands. Recording it because it is a plausible onset marker for the Supabase ticket timeline and because the discard logic is the reusable lesson.
• wi-cli-venus note · 5w ago
  PRECEDENT SEARCH CLOSED - NEGATIVE ACROSS THE WHOLE FLEET. db-terra-cc grepped its full memory corpus and agents/db-terra/evidence for 'read-only transaction', pg_ls_waldir, replication slot, disk full, PITR and WAL: 10 hits, all one unrelated subject (WAL replay bypassing delete triggers on a physical restore) plus 'PITR is off, free plan'. No precedent. That is now three lanes - venus, pluto, terra/enamel - all reporting no prior enforced read-only, full disk or runaway WAL, and all three explicitly declining to force a match. Elazar's recollection of this having been solved before is NOT reproducible anywhere in Evolutiva's recorded history. It should not steer the Supabase ticket. INSTRUMENT CORRECTION, from db-terra-cc, and it corrects MY framing: the WAL:DB RATIO IS THE WRONG INSTRUMENT. min_wal_size sets a floor, so on a small database the floor dominates and the ratio moves on the denominator. Terra is 96 MB / 9 files against a 16 MB database = 6x, which is HIGHER than venus (2.3x) and pluto (3.7x) while being perfectly healthy. Quoting Mars at '57x' therefore overstates a number that is partly an artifact. THE SIGNAL IS ABSOLUTE WAL VOLUME AND FILE COUNT, not the ratio. Corrected fleet reading: venus 96 MB / 7 files, pluto 240 MB / 16 files, terra 96 MB / 9 files, MARS 7,568 MB / 474 FILES. 474 files against a 7-16 file band is the finding. Use that in the ticket, not the ratio. Also noted: terra's db_size is the postgres database only, not every DB on the instance, so it is not strictly comparable to a whole-instance figure. DIFFERENTIATOR HYPOTHESIS, offered by db-terra-cc as a hypothesis and NOT a finding: terra/enamel is FREE plan with PITR OFF and no daily backups (measured 2026-08-16, flagged for re-verification). PITR FORCES WAL RETENTION. If Mars has PITR or a paid backup tier enabled while venus/pluto/terra do not, that is the one variable the four projects do not share, and it would explain why only Mars accumulates WAL on an otherwise identical stack. COUNTER-EVIDENCE, and it is weak evidence: mars/pm/06-passwords-and-settings.md line 26 records 'Plan: Free'. That WEAKENS the PITR hypothesis, since PITR is a paid add-on. But treat that line with suspicion rather than as a refutation - it CARRIES NO DATE and no read path, which is precisely the never-true/unverifiable-state class: a plan value written once and never re-read is indistinguishable from one that was wrong from the start or changed since. Do not close the PITR hypothesis on the strength of an undated doc line. THE ONE CHECK THAT DISCRIMINATES, and only Elazar can run it - Supabase dashboard for project ustenjufophwhlkzfdso: (1) current PLAN, (2) whether PITR / point-in-time recovery is ENABLED and at what retention window, (3) the backup tier. If PITR is on, the cause is very likely WAL retained for the PITR window on an instance too small to hold it, and the remedy is a Supabase-side matter of retention or disk. If PITR is off and the plan really is Free, then WAL-G archiving is failing and that is squarely a Supabase bug to report.
• wi-cli-venus note · 5w ago
  CAUSE PROVEN, NOT INFERRED - it is (a), the WAL archiver. db-mars-cc, pg_stat_archiver: archived_count=63740, last_archived_wal=00000001000000F8000000CA, last_archived_time=2026-09-08 23:46:49Z failed_count=2504, last_failed_wal=00000001000000F8000000CB, last_failed_time=2026-09-10 09:17:49Z The failing segment ...CB is the VERY NEXT one after the last success ...CA. Archiving has been stuck on a single segment for ~34 hours across 2504 attempts and is still failing right now. Config confirms WAL would otherwise recycle normally: archive_mode=on, archive_timeout=2min, wal_keep_size=0, max_wal_size=1GB, min_wal_size=80MB, wal_level=logical. Nothing is retaining WAL by configuration and no slot exists - Postgres simply may not recycle a segment it has not archived. 474 files is precisely the WAL generated since 09-08 23:46:49Z. archive_command = /usr/bin/admin-mgr wal-push %p >> /var/log/wal-g/wal-push.log 2>&1 - Supabase-managed WAL-G. The 2>&1 means the real error is only in that log, which is not reachable from psql. (c) IS RULED OUT: a manual CHECKPOINT is NOT the lever. Checkpointing cannot recycle unarchived segments, so it would consume IO for nothing. CHAIN, now fully closed: WAL-G wal-push fails on segment ...CB (09-08 23:46Z) -> Postgres cannot recycle any WAL from that point -> 474 files / 7.5 GB accumulate -> volume fills -> Supabase forces read-only (~09-09 23:23Z, the last appEvents write, about 24h after archiving stopped) -> every write refused. The same unrecyclable-WAL pressure is the Disk IO Budget depletion behind MARS-698, and a WAL-G retry loop hammering the same segment 2504 times is the 'mystery backup repeatedly downloading Mars' Elazar observed. READ-ONLY OVERRIDE IS AVAILABLE AND MUST NOT BE USED AS THE FIX. db-mars-cc tested it non-destructively - BEGIN; SET LOCAL default_transaction_read_only=off; SELECT pg_current_wal_lsn(); ROLLBACK - all ACCEPTED, no error, nothing committed. So the block is a session-level GUC, overridable, exactly as Supabase's own troubleshooting doc describes (set session characteristics as transaction read write / set default_transaction_read_only='off'). BUT that documented remedy assumes the disk is full of DATA, which you then delete and VACUUM to reclaim. OUR DISK IS FULL OF WAL. Deleting rows and vacuuming reclaims nothing here, and every additional write GENERATES MORE UNARCHIVABLE WAL, making the volume fill faster. Using the override would restore writes briefly and deepen the outage. It is an emergency data-extraction path, not a repair. ALSO: mars/pm/06-passwords-and-settings.md line 26 'Plan: Free' is very likely STALE. Free-plan projects get 1 GB of disk; 7,568 MB of WAL cannot exist on a 1 GB volume. Whatever Mars is on, it is not the 1 GB free tier. Treat that line as unverified (undated, no read path) and re-read it from the dashboard. FIX THAT DOES NOT NEED A SUPABASE TICKET: restart the project from the Supabase dashboard. A restart restarts the admin-mgr/WAL-G archiver process, which is the standard self-service action for a wedged archiver. If it resumes, Postgres recycles the 474 backlogged files automatically, the volume drains, and read-only clears on its own - Supabase's docs state read-write is re-enabled automatically once usage falls below 95%. If a restart does NOT clear it, the next lever is temporarily increasing disk size, which both lifts read-only and gives the archiver room to catch up. Both are dashboard actions, no ticket, and both are Elazar's - no agent can restart or resize a project.
• wi-cli-venus note · 5w ago
  RESOLVED. Elazar restarted the project ~09:28:15Z (stats_reset timestamp) and the archiver unwedged. Recovery measured, not assumed: - 09:30:23Z (~2min post-restart): archived_count=0, last_archived_wal NULL, failed_count=0, WAL 7,584 MB / 475 files. Ambiguous - counters reset, nothing moving yet. - 09:35:09Z (~7min): archived_count=17 and climbing, last_archived_wal=00000001000000FA0000001E and ADVANCING, last_archived_time=09:35:03Z, failed_count=0, last_failed_wal NULL. WAL 4,304 MB / 270 files - 3.3 GB and 205 files recycled in five minutes. - App side, independently: the drain route returned {ok:true,claimed:0,sent:0,...} at 09:35:44Z and 09:35:55Z, i.e. its UPDATE completed. Writes restored after ~10h of total outage (freeze at 09-09 23:23Z). Total incident: WAL-G wal-push failed on segment 00000001000000F8000000CB from 2026-09-08 23:46:49Z, 2504 consecutive failures over ~34h; no WAL could be recycled; 7.5 GB accumulated on an 8 GB volume; disk reached 96.25%, over the 95% threshold; Supabase forced read-only ~24h after archiving broke; every write refused for ~10h until a project restart cleared the archiver. Confirmed NOT the cause, each measured rather than assumed: replication slots (zero, and zero is normal - all four Evolutiva projects have none), pg_net bloat (net schema 7,432 kB), MARS-694 role residue (pg_db_role_setting clean), storage quota (DB 131 MB of 500 MB), PITR retention (Elazar confirmed PITR DISABLED), incomplete checkpoints (archive failure, not checkpoint failure - CHECKPOINT was not the lever), and query I/O (a pg_stat_statements review in this window is invalid; see MARS-702). Fix required no Supabase ticket - Elazar declined one and was right: a dashboard restart cleared it. PREVENTION IS NOT DONE AND THIS WI SHOULD NOT BE READ AS CLOSING IT. Open: MARS-703 (WAL + archiver + backup-failure monitoring, fleet-wide) and the autoscaling gap - Mars's disk is set to autoscale at 90% but the dashboard states adjusting disk configuration requires LARGE compute or above, so it did not fire at 90% or 95%. The safety net is configured and inert. Elazar's decision.
• wi-cli-venus completed · 5w ago
  RESOLVED 2026-09-10 10:00Z. Cause: WAL-G wal-push failed on segment 00000001000000F8000000CB from 2026-09-08 23:46:49Z, 2504 consecutive failures. Postgres never recycles an unarchived segment, so WAL grew to 7,568 MB / 474 files on an 8 GB instance volume (96.25%), crossing Supabase's 95% forced-read-only threshold at ~2026-09-09 23:23Z (max(appEvents.createdAt)). pg_database_size was 131 MB of a 500 MB quota throughout - quota was never the cause and the dashboard number is not disk usage. Fix: Supabase dashboard project restart at 09:28:15Z (Elazar). No ticket needed, no rotation, no read-only GUC override (that remedy assumes the disk is full of DATA; deleting rows reclaims nothing when it is full of WAL and every extra write makes more). DB-side verified by db-mars-ca at 09:43:17Z: default_transaction_read_only=off, pg_stat_archiver archived_count=27 failed_count=0 since restart, WAL 96 MB / 7 files. APP WRITE PATH verified end-to-end by coder-mars-cc at 10:00:54Z: a probe-UA request to a nonexistent prod route produced appEvents row d28c5977-cc3a-4e8d-be2e-bdedf0cf6bcf (navigation/404). A real application write reached the database and committed. Note the zero appEvents rows between 09:28Z and 10:00Z are 06:28-07:00 ART low traffic, not a residual failure. Follow-ups already filed: MARS-703 (WAL + pg_stat_archiver + N-consecutive-backup-failure monitoring, fleet-wide - the scheduled pg_dump failed three times starting ~14h BEFORE writes froze and fired into nothing), MARS-701, MARS-702. Autoscaling was configured to grow at 90% and never fired at 90% or 95% because adjusting disk configuration requires LARGE compute or above - open with Elazar.
• wi-cli-venus verified · 5w ago
  ONSET CORRECTION, 2026-09-10 10:0xZ coder-mars-cc, measured after the close. The close body says writes froze at ~09-09 23:23Z from max(appEvents.createdAt). That figure is the LAST SURVIVING WRITE, not the onset. Hourly appEvents counts: 09-08 holds 12-31 rows in every hour through 22:00Z, then nothing. All of 09-09 holds NINE rows, never more than 2 in any hour, one-per-hour scattered. WAL-G's first wal-push failure was 09-08 23:46:49Z. So the user-visible degradation begins at the WAL-G failure, ~24 HOURS before Supabase forced read-only - the read-only flag was the end of a day-long decline, not the start of the incident. Caveat: appEvents is not a traffic meter, so this data alone does not exclude a quiet day; 09-06/09-07 were 471/597 rows, so weekday/weekend is not the driver. Consequence for MARS-703: an alert keyed on 'writes are refused' fires ~24h late. Key it on pg_stat_archiver.failed_count instead, which was already climbing at 09-08 23:46:49Z. General rule recorded: reconstruct an outage from volume-per-hour SHAPE, never from a max timestamp - a max is guaranteed to sit at the END of the degradation.