Supabase Disk IO Budget depleting on evolutiva-pluto - triage IO consumers before considering a compute upgrade

Sub-tickets

Activity

• wi-cli-venus created · 2026-08-18
• wi-cli-venus verified · 2026-08-18
  MEASUREMENT PASS 1 COMPLETE - no IO driver found in any instrument available to the agents. COVERAGE STATEMENT, not a clean bill. db-pluto-cc, read-only prod, three passes: pg_stat_statements by shared_blks_read shows no driver (top query 56 blocks); by temp_blks_written one outlier, fn_dbperf_statements_top50 (the dbPerfSnapshots collector, mig 033) at 63 calls / 141MB temp - top consumer but ~2.2MB per call, far too small to deplete a budget, explicitly NOT named as the cause; pg_cron 7-day totals all trivial (largest pluto-drain-email-outbox 2016 runs / 245s); seq scans high in count but every table 40KB-9.6MB so it is buffer-hit churn not physical IO; pg_database_size 61MB total. Window validated: stats_reset 2026-06-16, postmaster uptime since 2026-03-01, both predate the 08-15 alert. Checkpointer since 02-12: 48958 timed vs 174 requested (0.35%), ~14.7MB/day written. Autovacuum normal, no thrashing. pg_stat_wal since 02-12: 2141119 records, 265550 FPI (12.4%, the expected share for a 5-min checkpoint interval), 1.30GB total = ~7.5MB/day, wal_sync 270002 = ~1 fsync/minute, xact_commit ~65700/day = 0.76/sec. checkpoint_timeout 5min, full_page_writes on, max_wal_size 1GB nowhere near hit. Those are the numbers of an IDLE database. nw-whey-cc, whey host, read-only: evolutiva-backup@pluto.timer measured cadence equals documented, OnCalendar 00/8:00:00, 21 consecutive runs over 7 days ALL successful, zero failures or retries, 17-56s each. My 183MB/day premise for the dump was WRONG and wrong in our favour - it is pg_dump --format=custom compressed, scoped to public+auth only with 4 exclude-table-data, artefact 4.24MB, so ~12.7MB/day not 183MB. Growth monotonic +0.9% over 3 days, no anomaly. Backup ruled out with a measured number. CORRECTED REASONING WORTH KEEPING: I framed this in BYTES throughout; a Supabase Disk IO budget is consumed by IOPS. A 61MB database sits entirely in shared_buffers and does almost no read IO, but every COMMIT still forces a WAL fsync and an fsync is an IOP regardless of bytes. That reframing is why WAL and commit rate got measured at all - and they came back idle too, which kills the checkpoint/FPI hypothesis as well. NOT EXERCISED, the honest gaps: no live sub-minute sample (db-pluto-cc sandbox blocks shell sleep; sent them the pg_sleep-inside-the-statement workaround, still outstanding), so a burst hiding inside a 187-day average remains possible; no visibility into Supabase-side IO accounting for their own backups, PITR/WAL archiving, or the compute tier baseline. NEXT: Elazar has the daily/hourly Disk IO dashboard charts, the one instrument nobody here has. If the live delta is also flat, this stops being a database question and becomes a tier/account question - including the real possibility that the compute add-on baseline is small enough that ordinary background activity depletes it, which would make the upgrade correct rather than lazy. Do not upgrade before the charts.
• wi-cli-venus verified · 2026-08-18
  Live sample closes the burst-vs-averaging question for steady state. db-pluto-cc ran the pg_sleep-inside-the-statement workaround (60s tripped the MCP tool timeout, 20s worked - the DB was never the limit): 4016 bytes of WAL in 20 seconds, approx 200 bytes/sec, approx 17MB/day, same order as the 187-day average, and ZERO commits in the window. So the smoothed numbers were not hiding a burst behind a long denominator - the database really is quiet. Does not rule out a burst at another hour; one 20s window cannot see a nightly or hourly spike. FINAL AGENT-SIDE COVERAGE: no IO driver observed in blocks read, temp writes, cron runtimes, seq scans, checkpointer, autovacuum, WAL volume, commit rate, backup timer/artefact size, or a live 20s sample - all measured across a window containing the 08-15 alert date. NOT exercised: sub-minute sampling at other times of day, and Supabase-side IO accounting for their own backups, PITR/WAL archiving, or the compute tier baseline - none of which any agent instrument can reach. Nothing was changed at any point in this investigation. Both lanes have run out of visible angles; the remaining instrument is Elazars daily/hourly Disk IO dashboard charts. If those are flat too, this is a tier/account question and the compute upgrade becomes the correct answer rather than the lazy one - but not before the charts.
• wi-cli-venus verified · 2026-08-18
  Coverage extended to an axis the whole prior investigation was structurally blind to, and the candidate found there was then ruled out on magnitude. All measurement by db-pluto-cc, read-only, nothing changed or vacuumed. INSTRUMENT GAP, real: every number in the prior coverage statement is WAL-derived (7.5MB/day volume, 0.76 commits/sec, a 20s live sample at 4016 bytes and zero commits). pg_net's tables are UNLOGGED - verified relpersistence='u' on both net._http_response and net.http_request_queue from pg_class, NOT inferred from migration 081's header - so they generate no WAL by design and a whole class of write IO could not appear in any instrument used. CHURN, real: _http_response n_tup_ins=4894, n_tup_del=4892, 2 live rows in 5.49MB. AUTOVACUUM ASYMMETRY, real and unexplained: _http_response has ONE autovacuum ever, none since 2026-08-05; http_request_queue has 103 and is current at 40KB with 0 rows. Same table shape, same TTL-delete pattern, no reloptions override, nothing excludes the net schema. RULED OUT ANYWAY: 4894 total cycles is ~376 row-writes/day over the shortest defensible window, about 0.004/sec. The */5 cron ticks were dismissed at 306/day as orders of magnitude too small; this is the SAME ORDER. A mechanism being invisible does not make it large, and those are independent questions. Also cleared this pass: cron.job_run_details, 306 rows/day against a 288/day prediction, 976KB, actively pruned by its own 03:47 cron, no bloat. Cron catalog read from the installed state: 5 active jobs only. NOT EXERCISED: a live insert/delete-rate sample on _http_response specifically; sub-minute sampling at other hours; Supabase-side accounting for their own backups, PITR/WAL archiving, or the compute-tier baseline. STATUS UNCHANGED: open, no driver found. What survives is an accretion exhibit for the EVO-86 architecture argument - DB-resident HTTP leaves state nothing reclaims in tables no WAL-based instrument can see - at its true small size, not an IO finding.
• wi-cli-venus note · 2026-08-18
  REVISION - my ruling-out of the pg_net path was itself too strong, corrected by peers within the hour. Two errors in the kill. (1) I wrote 'nothing vacuums it' while holding n_dead_tup=0 in the same message. Zero dead tuples means autovacuum RAN and did its job; plain VACUUM does not return mid-relation free space to the filesystem, so pluto's 5.49MB is a HIGH-WATER MARK, not an accumulating leak. coder-mars-cc caught this on their ref and it is equally false on mine. Changes the remedy: neither autovacuum tuning nor moving the scheduler reclaims it, only VACUUM FULL or the table falling out of use. (2) coder-enamel-cc: n_tup_ins on an UNLOGGED table is a LOWER BOUND over an unknown window - unlogged tables truncate on unclean shutdown and pg_stat counters reset independently on stats reset or a Supabase-side restart. My '~376 row-writes/day, same order as the cron ticks' divides a lower-bound numerator by a window I never established. That is an arithmetic-SHAPED refutation, not an arithmetic one. CORRECTED STATUS of the pg_net path: NOT ruled out. UNQUANTIFIED, and unquantifiable with the instruments used so far. 'Ruled out' and 'not measurable by me' are different claims. NAMED NEXT INSTRUMENT: two reads of n_tup_ins on net._http_response separated by a measured interval, which makes the window explicit instead of assumed. Not yet taken by anyone. CROSS-REF CONFIRMATIONS, installed state from pg_class on three refs (pluto, mars, enamel+terra shared): relpersistence='u' on both net._http_response and net.http_request_queue everywhere. The premise is catalog-verified, not inherited from migration 081's header. The autovacuum asymmetry (_http_response stale, http_request_queue current) reproduces on mars at 2.7x pluto's volume - pg_net behaviour, not pluto-local config. Enamel/terra's never-scheduled ref reads 32KB/16KB at zero inserts, so pluto's 5.49MB is churn-attributable in full with no install overhead - though terra downgraded that from 'proven floor' to 'good estimate' for the same counter-durability reason. WRITE-UP WARNING from coder-enamel-cc, worth honouring: three refs are in three different states, so any sentence of the form 'pg_net tables are <X>' is wrong on at least one ref. Name the ref and the measurement time. WI STATUS UNCHANGED: open, no driver found, still waiting on Elazar's Supabase Disk IO charts, which remain the only instrument that can answer flat-vs-spiky.
• wi-cli-venus statusChanged · 2026-08-21
  Cause disproven, not fixed: cadence + reads exonerated (mars is measurably same posture, worse WAL, no reported issue). Cron reverted to */5 (no established benefit, real user-facing cost). checkpoint_timeout kept at 30min as a zero-cost hygiene change only, not credited as the fix. eq#46 filed to Elazar: does his other Supabase-account inbox (evolutivamars@gmail.com) get the same depletion email, or does nobody read it - decides whether this is a real per-project issue or a tier-upsell threshold every Free/nano project crosses.
• wi-cli-venus completed · 7w ago
  Not a defect - Elazar confirmed evolutiva-mars got the identical Disk IO Budget upsell email 2026-08-21. Both are idle Free-tier/smallest-compute projects near a threshold Supabase emails about; no real symptoms observed, pluto meter never above 48%. checkpoint_timeout kept at 30min as zero-cost hygiene only; cron reverted to */5.