Supabase saturation: instance-wide execution stalls (Oct-5 class) + PostgREST retry loop + Nano tier - root cause and fix plan (planning cost UNMEASURED: track_planning off)

Sub-tickets

Activity

• coder-mars-cc created · 2w ago
• coder-mars-cc statusChanged · 2w ago
  coder-mars-cc: findings doc §1-§7 at /data-venus/mars/ops/2026-09-28-supabase-saturation-findings.md. Dispatched: db-mars-cc §6.1 (bundle/view rewrite + PostgREST empty-schema fix + pg_stat_statements_reset), db-mars-ca §6.2 (five planning-heavy statements, split first; its Supabase MCP OAuth is expired - rerouted to psql via db-mars-cc creds), coder-mars-ca §6.3 (connection churn), support-evolutiva-cc §6.4 (fleet PostgREST loop), proxy-general-cc §6.5 dashboard reads + whey job inventory (§7b). Venus job audit §7: external jobs ~75 s/day vs app 3.15M ms/18d - not the load. Pending Elazar: compute tier after §6.1-6.2 land.
• db-mars-ca note · 2w ago
  db-mars-ca Venus live-unit recheck added to /data-venus/mars/ops/2026-09-28-supabase-saturation-findings.md §7c: Mars applog pull/listen, repage, dbperf, WAL check, and weekly UX relay paths documented; cadence overrides corrected. Venus DB backup is Venus-only. No jobs changed. Whey could not be independently reached; root cron/journals there remain unverified, with prior §7b evidence retained.
• support-evolutiva-cc note · 2w ago
  §6.4 fleet check - support-evolutiva-cc, measured 2026-09-28 03:30-03:45Z. PARTIAL: two of four projects measured, two blocked on credentials I do not hold. Report below is measurement plus one labelled inference; nothing here is a log count. REF MAP - five apps, FOUR Supabase projects. mars=ustenjufophwhlkzfdso, venus=fjnhfjwskixzsilxrkph, pluto=fdwjmzjwurbpkxersigg, terra AND enamel SHARE eprpywqwzsvrjeaforei. Any per-project fleet sweep is 4 runs, not 5, and a terra finding is automatically an enamel finding. TIER (max_connections / shared_buffers, read via SQL). venus 60 / 224MB. terra+enamel 60 / 224MB. Both identical to mars = Nano. pluto NOT measured. So of the three projects I could reach, all three are Nano - the tier is a fleet-wide default, not a mars-specific choice, and a mars-only tier bump leaves venus and terra/enamel on the same ceiling. POSTGREST LOOP - venus shows the mars signature. GET https://fjnhfjwskixzsilxrkph.supabase.co/rest/v1/zz_probe_nonexistent?select=1 with venus's own publishable key returns 503 PGRST002 'Could not query the database for the schema cache. Retrying.' - byte-identical in kind to mars. INFERENCE, NOT A MEASUREMENT: I am inferring the 2.6k/day crash-retry loop from the documented Data-API-disabled mechanism, because I cannot run the logs query on venus. Someone with the venus account PAT must confirm with select event_message, count(*) from logs where source='postgrest_logs' group by 1 before this is treated as a finding. If it confirms, venus is burning the same ~2637 connection cycles/day against the same 60-connection ceiling, and the fix (create schema pgrst_no_exposed_schemas; alter role authenticator set pgrst.db_schemas; notify pgrst) applies unchanged. pluto and terra/enamel NOT probed - no publishable/anon key available to me (see blocker). FOUR NEGATIVE RESULTS so no one repeats them: 1. A keyless probe cannot discriminate. All four refs return 401 'No API key found in request' for both /rest/v1/ and a table path - Kong answers before PostgREST, so a crash-looping PostgREST is invisible without a key. 2. /rest/v1/ ROOT with a publishable key returns 401 'Secret API key required', not PGRST002. The probe must use a TABLE path; a nonexistent table name is fine and preferable, because PGRST002 fires ahead of the 404. 3. pg_roles.rolconfig for authenticator is NOT a discriminator. I tried it as a way to detect Data-API-disabled over SQL alone and it fails: venus is Data-API-disabled by the REST probe, yet its authenticator rolconfig holds only session_preload_libraries=supautils,safeupdate | statement_timeout=8s | lock_timeout=8s and no pgrst.db_schemas at all. Supabase sets that at the PostgREST config layer, not as a role GUC. Same result on terra/enamel. Do not try to read the loop out of pg_roles. 4. Both non-mars projects carry statement_timeout=8s and lock_timeout=8s as authenticator role defaults - worth comparing against whatever mars has, since mars's timeouts are the symptom being chased. BLOCKER (why two projects are missing). The logs group-by needs per-project Supabase access; my PAT is scoped to the mars account (evolutivamars@gmail.com) only and does not even load until my session restarts. pluto/.env.local contains no DATABASE_URL and no NEXT_PUBLIC_SUPABASE_* (only SUPPORT_API_KEY + VERCEL_OIDC_TOKEN); terra has only .env.probe.local; enamel has no checkout here. My only non-venus DSN is ENAMEL_RO_DSN, which is how terra/enamel got measured. I deliberately did not read another agent's ~/.config/postgres/*.env. ASK, one line each, cheaper than giving me credentials: db-pluto-cc runs the two commands on fdwjmzjwurbpkxersigg; db-terra-cc or db-enamel-cc runs them on eprpywqwzsvrjeaforei (covers both apps); whoever holds the venus account PAT runs the logs query to convert the venus inference above into a finding. Commands: (a) select event_message, count(*) from logs where source='postgrest_logs' group by 1 - note source='postgrest_logs', FROM logs, not FROM postgrest_logs; (b) select current_setting('max_connections'), current_setting('shared_buffers').
• coder-mars-ca note · 2w ago
  §6.3 read-only findings, 2026-09-28 03:30Z. Live pg_postmaster_start_time=2026-09-10 09:28:15Z. pgbouncer.get_auth=163,301 calls (~9,200/day over 17.75d); pg_stat_database sessions=104,800 (~5,900/day), sessions_abandoned=7,605 (7.26%), sessions_fatal=5, sessions_killed=0. These are different populations: get_auth includes all Supavisor clients/roles; sessions measures PostgreSQL backend sessions. Neither is grouped by app role, Vercel container, or invocation. PostgreSQL defines abandoned as connection to client lost, not proof of app pool leak (https://www.postgresql.org/docs/17/monitoring-stats.html). src/lib/db.ts has one lazy global pool per process, max=6 on transaction-mode DATABASE_URL. src/lib/weekly-kpi.ts has a separate lazy global pool, max=10 on session-mode DATABASE_URL_DIRECT; only weekly KPI cron/admin paths use it. No per-request Pool creation or second pool on normal /practicas paths found. Both omit idleTimeoutMillis; installed pg-pool 3.14.0 and its docs specify default 10,000ms and min=0, so idle clients are discarded after 10s and later bursts reauthenticate (https://node-postgres.com/apis/pool). This is a plausible churn contributor, NOT measured attribution. Current pg_stat_activity shows 5 mars_app_runtime backends (3 active, 2 idle), not a container count. appEvents has no deployment/container identifier; sampled Vercel log JSON has deploymentId but no containerId/instanceId/pid. Deployment ID identifies release, not process. Thus actual container count and app-specific auth rate cannot be computed from current data. Next discriminating measurement: per-process pool-instance ID, pool connect/remove counters and lifetimes correlated with invocation/deployment, or Supavisor auth logs grouped by client role/source. No code or tuning change made; do not increase idle timeout or max based on these aggregates alone.
• coder-mars-ca proposal · 2w ago
  Elazar decision: enable log_min_duration_statement=2000 (ms) for Mars for a bounded one-week diagnostic window, after confirming Supabase supports the setting at the project/role level and checking log ingestion, cost, and privacy implications. Record baseline log rate and exact start/end; review 2s+ PostgreSQL statements during the next incident, correlate by timestamp/route without storing sensitive parameters, then restore prior value (-1) after 7 days. Current -1 provides no slow-statement evidence, and aggregate pg_stat_statements maxima cannot be pinned to the 2026-09-27 bursts. This is a proposal only; no configuration changed.
• coder-mars-ca note · 2w ago
  Independent read-only Venus host audit 2026-09-28 03:35Z for proxy-general-cc request: user crontab empty; visible system timer list has no Mars job. User timers and effective unit files confirm Mars applog-pull (:10/:40), repage-sweep (5 min), dbperf (07:35Z daily), WAL check (:05/:35), and weekly UX-pain relay (Mon 13:27Z), exactly as findings doc §7c. Also confirmed enabled/active applog-listen@mars.service (running since 09-22), persistent LISTEN/catch-up path already documented in §7c. Search of ~/.config for the Mars project ref found only applog, repage, dbperf, WAL per-project env files; no additional configured poller. /opt/soportepreguntas uses local 127.0.0.1 PostgreSQL, not Mars Supabase. venus-state-backup.sh archives local state (including mars-relay config) to gdrive/Whey outbox, not Mars DB; api-images-backup and Hostinger storage watcher use Hostinger storage, not Supabase. No net-new Mars DB job discovered. Whey not independently audited: ssh hostname whey does not resolve from Venus; findings §7b is separate agents’ evidence. Root cron/journals remain unverified. No services changed.
• coder-mars-ca note · 2w ago
  §6.3 closed as MEASURED, NO CODE CHANGE per coder-mars-cc decision (2026-09-28 03:35Z). Supavisor supavisor_logs ClientHandler events by user in the 24h window: mars_app_runtime 7,848 (transaction mode), applog_reader 1,119 (session mode), postgres 183, errscan_ro 105; app share of these named events = 84.8% (~5.45 app client connections/min). This is consistent with pg-pool default 10s idle eviction across warm Vercel functions; it does not measure container count or prove the exact eviction fraction. pgbouncer.get_auth mean cost ~9ms/auth, so auth churn is secondary. Supavisor transaction mode multiplexes server backends: keeping app-side client connections warmer would not pin a backend plan cache or solve the cold-planning bottleneck. No pool tuning, lifecycle telemetry, or code change authorized/needed for §6.3. This log split supersedes the earlier attribution gap in my read-only note.
• db-mars-cc note · 2w ago
  db-mars-cc §6.1: (a) PostgREST empty-schema fix applied live + committed (6367dcb) - CREATE SCHEMA pgrst_no_exposed_schemas + ALTER ROLE authenticator SET pgrst.db_schemas. (b) get_practica_detail_bundle/vPracticaHistorial rewrite BLOCKED: two rewrite attempts this session both failed the <100ms cold-planning target and one regressed execution to 5.9s (computed practicaId broke index pushdown in the conducto branch). Reverted vPracticaHistorial to the exact original definition, verified via EXPLAIN + grants/security_invoker diff - prod is back at baseline (~200ms planning / ~40-50ms exec), unchanged from before this session. The 14-self-join structure is not the dominant planning cost as hypothesized; a shared resolved_users CTE materializes correctly but planning 9 joins against it costs about the same as the original. Needs a different approach (denormalized/materialized read path, or per-branch typed functions) and more design time than one migration turn. pg_stat_statements_reset() withheld pending the bundle fix landing.
• db-mars-ca note · 2w ago
  §6.2 read-only EXPLAIN samples (2026-09-28 UTC; direct psql, explicit BEGIN READ ONLY; no changes). Planning / execution ms: comisiones json aggregate 6.9 / 11.3 (284 shared hits); DISTINCT programa UNION 18.2 / 10.2 (397 hits); get_layout_state(actor,false) 0.6 / 46.3 (199 hits); scoped alumnos roster 170.7 / 1336.0 (planning 599 buffers; execution 8007 hits); /practicas list 170.7 / 492.7 (15,581 hits). Scope/shape caveats: comisiones and programa SQL traced directly to current source; layout is exact function call shape with representative actor. List reused full page/CTE/aggregate shape and pageSize=100 but simplified authorization to membership-only (omitted postgrado + stored-SA scope arms). Alumnos reused scoped/roster/practice_stats shape with a representative docente role, but projected fewer output columns. These are current warm-cache samples, not historical burst attribution or exact app-parameter replays; no EXPLAIN sample establishes which statement timed out during past bursts. The plan/exec split is a first diagnostic, not a rewrite recommendation. No SQL/config/code writes performed.
• support-evolutiva-cc note · 2w ago
  §6.4 update - pluto CONFIRMED by db-pluto-cc 2026-09-28 00:47 ART, measured not inferred. Ref fdwjmzjwurbpkxersigg: max_connections=60, shared_buffers=224MB (Nano, same as mars/venus/terra-enamel). postgrest_logs over a 28h window: 2662x each of 'Config reloaded' / 'Successfully connected' / 'Connection Pool initialized' / 'Attempting to reconnect in 32s', and 2661x 'schema pg_pgrst_no_exposed_schemas does not exist' = ~2283 crash cycles/day. Same signature and same order of magnitude as mars's 2637/day. FLEET STATE, 3 of 4 projects: mars measured (2637/day), pluto measured (~2283/day), venus signature-matched by REST probe but count still uninferred, terra+enamel eprpywqwzsvrjeaforei still unmeasured - db-enamel-cc is parked under Elazar's 2026-09-23 order, db-terra-cc asked. So the PostgREST empty-schema fix in §6.1 is not a mars fix, it is a fleet fix on at least two and probably four projects, each leaking ~2.3-2.6k connection cycles/day against an identical 60-connection ceiling. INSTRUMENT TRAP from db-pluto-cc, worth more than the count: the logs endpoint REQUIRES explicit iso_timestamp_start / iso_timestamp_end. Without them it defaults to a 1-minute window and fails with 'Backend error' and zero matching rows. That is a range-required failure, NOT a syntax error, and it reads exactly like 'this project has no PostgREST loop'. Any earlier §6.4-style negative taken on the default window is void.
• support-evolutiva-cc note · 2w ago
  §6.4 - pluto FIXED, first fleet confirmation that §6.1 works outside mars. db-pluto-cc applied the 3 statements and closed PLUTO-754 2026-09-28 00:54 ART. Before 111 crash cycles/hr (~2662/day). After: schema cache loaded 0 relations, 0 postgrest_logs events over a 2-minute steady-state window. Loop stopped. Caveat on the after-figure, not a doubt about the fix: at a 32s retry interval a 2-minute window covers only ~4 cycles that did not happen. It proves the loop is not immediately re-entering; it is not yet a durable count. Asked db-pluto-cc for a 24h recount on the same query so the before/after is like-for-like against the 2662/28h baseline. 'Schema cache loaded 0 relations' is the expected success shape here - the target schema is deliberately empty, so 0 relations means PostgREST resolved it instead of dying on it, and is not a sign the fix pointed at the wrong schema. FLEET STATE: mars leaking (own lane), pluto fixed and verified short-window, venus open on VENUS-420 (reassigned db-venus-cc -> db-venus-ca, the cc seat forwards), terra+enamel unmeasured (db-enamel-cc parked, db-terra-cc asked). Two of four projects were confirmed by log count before any fix, so §6.1 is a fleet fix and pluto is now the reference that it lands cleanly on a second project.
• support-evolutiva-cc note · 2w ago
  §6.4 correction - mars is NOT leaking. My previous note's fleet line was stale. coder-mars-cc measured at 03:55Z: postgrest_logs 'Attempting to reconnect' ran 110/hr across 00Z-02Z and 0 since 03:40Z, 6367dcb having landed ~03:37Z. Mars fixed. FLEET STATE 2026-09-28 03:55Z: mars fixed (6367dcb), pluto fixed (PLUTO-754, 111/hr to 0, 24h recount scheduled), venus open (VENUS-420, db-venus-ca), terra+enamel unmeasured on ref eprpywqwzsvrjeaforei. Two of four projects were confirmed leaking by log count before any fix and both are now closed on the same three statements, so §6.1 generalises. Reference-hash check: the commit is 6367dcb. '6367cdb' appears in the 00:55 ART message and is a typo - git cat-file rejects it as not a valid object name. PLUTO-754 and VENUS-420 already carry the correct 6367dcb, so nothing downstream needs editing, but do not copy the transposed form into terra/enamel's row.
• db-mars-cc note · 2w ago
  §6.1(b) done: practicaHistorialEntries denormalized table + 4 AFTER INSERT triggers + vPracticaHistorial view swap (182a/b/c, 6367dcb+e23ed86, pushed by coder-mars-cc). 46,913-row backfill diff vs original view: 0 mismatches. Fresh-connection get_practica_detail_bundle on c604bbec: 12-13ms (4/5 psql samples), vs prior 195ms planning alone. pg_stat_statements_reset() run. Acceptance confirmed by coder-mars-cc.
• coder-mars-cc statusChanged · 2w ago
  coder-mars-cc: pushed db-mars-cc handoffs 6367dcb (PostgREST empty-schema fix; mars reconnect cycles 110/hr -> 0 since 03:40Z) and e23ed86 (practicaHistorialEntries flat table + 4 triggers + view swap, 46,913-row backfill diff 0 mismatches; bundle cold 12-13 ms vs 250-800 ms before). pg_stat_statements reset. Both DEPTH=deep, cold brief sent to audit-general-cc. Open: db-mars-ca exact-SQL plans for alumnos roster (1.3 s) + /practicas list (0.5 s); VENUS-420; tier decision (Elazar).
• support-evolutiva-cc note · 2w ago
  §6.4 - venus VERIFIED FIXED, 3 of 4 projects closed. db-venus-ca applied the 3 statements via migration venus81-postgrest-empty-schema.sql. Neither db-venus-ca nor I can read venus postgrest_logs, so I verified by effect instead, using venus's own publishable key against the same URL before and after: 503 PGRST002 'Could not query the database for the schema cache' at 03:45Z pre-fix, 404 PGRST205 'Could not find the table pgrst_no_exposed_schemas.zz_probe_nonexistent' at 04:00Z post-fix, 3 consecutive probes. PostgREST is answering from a loaded cache instead of dying and reconnecting every 32s. The REST probe is therefore a full pass/fail instrument for §6.1 on any project, not just a detector. Red state 503 PGRST002, green state 404 PGRST205 naming the target schema. It needs only the project's publishable key, no PAT, no log access - which is the credential that blocked this whole lane. Whoever does terra/enamel should use it rather than waiting on an account PAT. FLEET STATE 2026-09-28 04:00Z: mars fixed (6367dcb), pluto fixed (PLUTO-754, 111/hr to 0, 24h recount pending), venus fixed (VENUS-420, verified by effect, unblocked), terra+enamel ref eprpywqwzsvrjeaforei still unmeasured - db-enamel-cc parked under Elazar's 2026-09-23 order, db-terra-cc asked and not yet back. Three of four projects had the identical leak and the identical 60-connection Nano ceiling, so treat the 4th as leaking until probed, not as clean by default.
• support-evolutiva-cc note · 2w ago
  §6.4 COMPLETE - all four fleet Supabase projects measured. terra/enamel is the fourth and it is RED. db-terra-cc probed eprpywqwzsvrjeaforei 2026-09-28 01:01 ART: 3/3 consecutive GET /rest/v1/zz_probe_nonexistent with the publishable key returned 503 PGRST002. Tracked as TERRA-248 p1, assigned db-terra-cc, covering both apps since terra and enamel share the one project (db-enamel-cc is parked under Elazar's 2026-09-23 order). FINAL §6.4 ANSWER: the leak was fleet-wide and unanimous. 4 of 4 projects, every one on Nano with max_connections=60 and shared_buffers=224MB. Counted before any fix: mars ~2637 cycles/day, pluto ~2283/day over 28h (111/hr). venus and terra/enamel were identified by REST signature rather than log count, because no reachable agent holds those accounts' PATs. Closed so far: mars 6367dcb, pluto PLUTO-754 (111/hr to 0), venus VENUS-420 (verified red-to-green by probe). Open: TERRA-248. Two things this makes true beyond §6.4. First, the tier is a fleet default, not a per-app decision, so the compute-tier question in §6.5 is a fleet question - a mars-only bump leaves three projects on the same 60-connection ceiling with the same statement_timeout=8s and lock_timeout=8s authenticator defaults. Second, nobody needed the log credential in the end: the publishable-key REST probe answered both detection and verification. The PAT is still the only route to a cycle COUNT, which is worth having for the 'what did we save' number and worth nothing for pass/fail. Coverage statement: this covers the PostgREST empty-schema loop and the tier on all four projects. It does NOT cover any other log source on the three non-mars projects - no postgres, supavisor or auth log was read anywhere but mars, so no claim is made that those projects are otherwise healthy.
• coder-mars-cc statusChanged · 2w ago
  coder-mars-cc: audit PASS:6367dcb; BLOCK:e23ed86 MEDIUM (backfill/grants not executable in-file) fixed by db-mars-cc 057328e, pushed (e23ed86..057328e), re-briefed audit-general-cc. Hot path exercised as mars_app_runtime: changelog=47 rows on c604bbec. Open: audit verdict 057328e; db-mars-ca plans for alumnos roster + /practicas list; TERRA-248; tier decision (fleet-wide, Elazar).
• support-evolutiva-cc note · 2w ago
  §6.4 CLOSED - 4 of 4 fleet Supabase projects fixed. terra/enamel done by db-terra-cc (TERRA-248, commit bc54b71, pushed by me to origin/main since that lane has no push lane). Verified two ways: authenticator rolconfig on eprpywqwzsvrjeaforei gained pgrst.db_schemas=pgrst_no_exposed_schemas between 03:45Z and 04:06Z, and the REST probe went 3/3 404 PGRST205. Final tally, all four on Nano 60 conns / 224MB: mars 6367dcb, pluto PLUTO-754, venus VENUS-420, terra+enamel TERRA-248. Pre-fix counted rates where a PAT existed: mars ~2637 cycles/day, pluto ~2283/day. venus and terra/enamel were diagnosed and verified entirely on the publishable-key REST probe, no PAT involved. Remaining fleet exposure is the tier, not the loop. Four projects still share max_connections=60 and authenticator statement_timeout=8s / lock_timeout=8s, so §6.5's compute decision is a fleet decision. Coverage unchanged: this lane read the PostgREST empty-schema loop and the tier only, on four projects; no postgres, supavisor or auth log was read anywhere but mars, so nothing here supports a claim that any project is otherwise healthy.
• coder-mars-cc audited · 2w ago
  audit pass: commits 6367dcb, e23ed86, 057328e (db-mars-cc handoffs, MARS-769 §6.1a/b)
  payload

  {"scope":"commits 6367dcb, e23ed86, 057328e (db-mars-cc handoffs, MARS-769 §6.1a/b)","outcome":"pass","evidence":"audit-general-cc: PASS:6367dcb deep; PASS:057328e deep (backfill idempotent, asserts on 65 cols/row count/3 grants, 182c asserts security_invoker=true + zero anon/authenticated/PUBLIC grants; e23ed86 BLOCK lifted). Doc nit: 'grants=23' comment unexplained.","reportRef":"audit-general-cc-mukq4ppt6fjb"}

• coder-mars-cc statusChanged · 2w ago
  §6.4 closed by support-evolutiva-cc: TERRA-248 fixed at bc54b71 (eprpywqwzsvrjeaforei), rolconfig pgrst.db_schemas confirmed 04:06Z. PostgREST loop fixed on all 4 projects. Remaining: db-mars-ca EXPLAIN plans (alumnos roster, /practicas list); tier decision (Elazar).
• coder-mars-cc statusChanged · 2w ago
  Post-fix measurement NOT YET POSSIBLE, do not read the current idle numbers as the result. pg_stat_statements reset 03:57Z; 31min window at 01:15 ART (near-zero users) shows total exec 3.1s, top statement 502ms/6 calls, no statement over 111ms mean. The timeout-burst hour is 20:00Z (17:00 ART), ~16h out. Also: pgbouncer.get_auth runs at 369/hr now vs a 377/hr 18-day average pre-fix, so get_auth CANNOT see the PostgREST loop's removal - it counts pooler client auths, not backend spawns. Use supavisor_logs 'Backend authenticated' for the churn comparison, on the 20:00Z hour, before any tier decision. Tier stays undecided until that read exists.
• support-evolutiva-cc note · 2w ago
  ELAZAR DECISION, CLI 2026-09-28 (nqla 5a): the Supabase log credential request is DROPPED, not deferred. §6.4 was completed end to end without it - the publishable-key REST probe detected the loop on all four projects and verified red-to-green on two. The PAT would only have bought cycle counts. Nobody should carry this as an outstanding ask on Elazar. If a future incident genuinely needs per-event log correlation, it gets raised fresh against that incident with the specific reason.
• support-evolutiva-cc note · 2w ago
  ELAZAR DECISIONS, CLI 2026-09-28. (nqla 4) NO TIER BUMP - no paying. All four projects stay on Nano at max_connections=60 / shared_buffers=224MB, so §6.5 is settled: the capacity answer has to come from the code and query side, not from spend. The PostgREST loop fix already returned ~2.3-2.6k connection cycles/day per project at zero cost, and the §6.1-6.2 planning work is now the whole remaining lever. Do not re-raise a tier upgrade as a proposal; if Nano genuinely cannot hold, that is a finding to bring with numbers, not an option to re-offer. (nqla 7) log_min_duration_statement=2000 on mars APPROVED. Diagnostic only, reversible, no spend. Asked db-mars-cc to apply it. It is what makes the next slow-query incident legible in the moment instead of after, which is the exact gap that made MARS-752 and MARS-769 cost days of sampling.
• db-mars-cc note · 2w ago
  log_min_duration_statement=2000 applied (Elazar approval relayed via support-evolutiva-cc, 2026-09-28). Mgmt API config/database/postgres only exposes log_connections; ALTER SYSTEM permission-denied. Used ALTER ROLE ...SET (supautils) on postgres/mars_app_runtime/authenticator/anon/authenticated/service_role. Verified 2s on fresh direct connection; Supavisor pooled backends pick it up on cycle, not immediately.
• support-evolutiva-cc note · 2w ago
  log_min_duration_statement=2000 INDEPENDENTLY VERIFIED on mars, plus one divergence worth someone's attention. Verified: pg_db_role_setting carries log_min_duration_statement=2000 on all six roles db-mars-cc named - postgres, mars_app_runtime, authenticator, anon, authenticated, service_role. My own pooled session through Supavisor reports current_setting = '2s', so at least the backend serving me was established after the change. That is consistent with db-mars-cc's caveat rather than against it: a role GUC binds when the backend's session starts, so backends warm from before the change keep the old value until they cycle. DIVERGENCE, flagged not diagnosed. mars authenticator now reads session_preload_libraries=safeupdate. On the other two projects I read today it is 'supautils, safeupdate' - venus at 03:45Z and terra/enamel at 04:06Z, both before and after their own fixes. I have NO before-value for mars's authenticator, so I am not claiming anything was dropped tonight; mars may simply have always differed. But supautils is what enforces Supabase's privilege guardrails, and ALTER ROLE ... SET session_preload_libraries is not additive - any migration that set it to a bare 'safeupdate' would have silently removed supautils. Worth one look at whether mars is missing a guardrail the rest of the fleet has, because the failure mode is invisible until a privileged statement that should have been refused goes through. Also observed while verifying a support email: two emailOutbox rows from the 09:00 digest batch sat pending on 'Data command failed: 421-4.3.0 Temporary System Problem' and one recovered on attempt 2. Retry works, no incident. Recorded because a rising 421 rate would surface there first and nobody owns that surface.
• support-evolutiva-cc note · 2w ago
  Supautils divergence found while verifying log_min_duration_statement is CONFIRMED and is NOT part of the 6.1 fix. Split out to MARS-775, dashboard-only, blocked on Elazar. It does not reopen 6.4: all four projects are still fixed and verified. Second containment reason beyond the app-code read: authenticator is reachable only through PostgREST, and mars PostgREST now resolves the deliberately EMPTY pgrst_no_exposed_schemas, measured by probe as 404 PGRST205. So no table is reachable by that role regardless of what app code does.
• db-mars-ca note · 1w ago
  2026-10-04 post-fix measurement attempt (live DB at 16:50Z; peak log window requested Oct 3 19:00-21:00Z). Direct DB read via provisioned agent_db_mars_ca/evolutiva_agent_ro is SELECT-only and cannot see cluster-wide pg_stat_activity or other roles' pg_stat_statements. extensions.pg_stat_statements_info.stats_reset=2026-09-28 03:57:55Z. Queries matching scoped_candidates, vStudentAssignments, and get_practica_detail_bundle returned 0 rows to this role; this is visibility-limited, NOT evidence of zero app calls. Only visible related statement was SELECT FROM practica LIMIT (1 call, mean/max 0.5ms), not an app workload proxy. pg_stat_database (postgres DB, cumulative since reset) sessions=125519, abandoned=10997, shared hit=240091301/read=14148, temp bytes=64395305; pg_stat_io and pg_stat_wal are likewise cumulative and cannot establish peak-hour deltas. My Supabase logs route denied the Oct 3 query with permission error, so Supavisor peak-hour count unavailable from this seat. Asked db-mars-cc for a one-shot read-only Oct 3 19:00-21:00Z Supavisor Backend authenticated count by application role and same-window workload evidence. No DB/app changes. This is not a post-fix peak-hour result yet; do not infer improvement/regression from current role-limited cumulative counters.
• db-mars-ca note · 1w ago
  2026-10-04 follow-up: db-mars-cc measured Supavisor events via supabase-stdio query_logs, source=supavisor_logs, grouped hourly by log_attributes['user'], 2026-10-03T19:00-21:00Z. 19:00-20:00Z: mars_app_runtime=71, applog_reader=18, errscan_ro=2, postgres=1. 20:00-21:00Z: applog_reader=21, mars_app_runtime=8. AppEvents row counts from superuser for same hours: 11 and 1 respectively. These are measured complete counts, not visibility-zero. The 20:00Z window is low traffic by both counts; backend authentications do not map 1:1 to requests, so this does not demonstrate under-load performance. Previously recorded pre-fix 20:00Z peak count was 322 mars_app_runtime authentications; current 8 is lower, but traffic differs and no like-for-like performance or capacity conclusion follows. Direct read-only app-role DB stats remain visibility-limited/cumulative as noted above. No DB/app changes.
• db-mars-ca note · 1w ago
  2026-10-04 follow-up: db-mars-cc measured Supavisor Backend authenticated events via supabase-stdio query_logs, source=supavisor_logs, grouped hourly by log_attributes user, window 2026-10-03T19:00-21:00Z. 19:00-20:00Z: mars_app_runtime=71, applog_reader=18, errscan_ro=2, postgres=1. 20:00-21:00Z: applog_reader=21, mars_app_runtime=8. AppEvents row counts from superuser for same hours: 11 and 1 respectively. These are measured complete counts, not visibility-zero. The 20:00Z window is low traffic by both counts; backend authentications do not map 1:1 to requests, so this does not demonstrate under-load performance. Previously recorded pre-fix 20:00Z peak count was 322 mars_app_runtime authentications; current 8 is lower, but traffic differs and no like-for-like performance or capacity conclusion follows. Direct read-only app-role DB stats remain visibility-limited/cumulative as noted above. No DB/app changes.
• db-mars-ca note · 1w ago
  Correction: the immediately preceding follow-up note at 16:52:12Z was submitted with shell quoting corruption and is superseded by the complete measurement note at 16:52:22Z. Use only the later note for the Supavisor counts.
• db-mars-ca note · 1w ago
  2026-10-04 disposition per pmmaster: record the available partial low-traffic sample and move on; do not block this lane waiting for a peak-window read. Supavisor measured 8 mars_app_runtime Backend authenticated events in Oct 3 20:00-21:00Z with 1 appEvents row, and 71 in 19:00-20:00Z with 11 appEvents rows. This is not a peak-load performance comparison. Access-route investigation is handed back to PM/registration ownership; no additional log access request from this lane.
• support-evolutiva-cc note · 1w ago
  SEVERITY DATUM FROM A REAL BURST, 2026-10-05 11:50:09-11:51:09Z, offered because it bounds what the saturation costs at the small end rather than the heavy-query end. THE TIMEOUT KILLED A SINGLE-ROW INSERT WHILE A HEAVIER SELECT IN THE SAME HANDLER PASSED. In review-action.ts the post-commit block first runs a practicas JOIN users SELECT and then createNotification. The failure row carries detail.stage = "notification", and stage "notify-lookup" is the label the enclosing try would have used had the SELECT been the casualty - so the join SELECT completed and the one-row notification INSERT is what hit "canceling statement due to statement timeout". That is a different shape from MARS-736 CTE-aggregates-everything diagnosis: at 11:50 the instance could not complete a trivial write, so the cost is not confined to the known expensive readers and a fix that only optimises those queries will not cover this. LOAD CONTEXT, measured: 54 review events today and a sustained email cadence from 09:53:05Z to 11:52:59Z, one roughly every 40-60 seconds with no gap longer than about 30 minutes - two hours of continuous grading by one reviewer on Android mobile. The timeouts land at the TAIL of that, not at its start, which is consistent with accumulation rather than a single pathological request. Elazar also received two error-digest emails off this burst, 11:50:59.357808Z and 11:51:13.102413Z, subjects "Error 6af18122-22fa-4219-900a-bb957ef8d8f7" and "Error 92643931-1e85-493b-b13c-3379f0dbcecf".
• support-evolutiva-cc note · 1w ago
  CORRECTION TO MY OWN NOTE ABOVE - db-mars-ca narrowed it 08:56 ART and they are right. I wrote that the single-row INSERT timing out means "the cost is not confined to the known expensive readers and a fix that only optimises those queries will not cover this". That second clause claims more than the evidence carries. What the stage=notification row establishes is that the OBSERVED TIMEOUT SURFACE now includes a write; it does NOT establish a shared-stall cause, and it does not rule out that reader-query cost elsewhere on the instance is what consumed the headroom the INSERT then needed. Their wording, kept because it is the precise version: "it does not by itself establish the shared-stall cause or rule out reader-query cost elsewhere." So the datum stands as a surface observation and the inference about what a query-optimisation fix would or would not cover is withdrawn - a reader-cost fix freeing headroom could well prevent this INSERT from timing out, which is exactly the possibility my sentence foreclosed without evidence. The lane position on the rest, recorded so this row carries their verdict and not only my read: the post-commit isolation behaved as intended, no row repair and no DB change is warranted, and they agree the 11:51:17Z idle snapshot cannot characterize the burst.
• pmmaster-evolutiva-cc blocked · 1w ago
  Post-fix capacity/causal attribution unmeasurable: MARS-743 (timestamped DB/Supavisor+host telemetry) blocked on Supabase query_logs permission denied. Oct5 11:50Z showed 6 timeout rows post-fix; cannot yet claim resolution. Per coder-mars-ca report 2026-10-05 22:49.
• coder-mars-cc note · 1w ago
  coder-mars-cc 2026-10-06: (1) The recorded blocker (query_logs permission denied, via MARS-743) does NOT hold from the coder-mars-cc seat: query_logs returned supavisor/postgres/auth/edge logs for 2026-10-06 12:20-12:30Z. The Oct-5 11:49:30-11:52:00Z window is within the 24h API cap only until ~11:50Z today, so it is likely already out of reach; future windows are readable. (2) The 'CPU-bound planning' premise is UNMEASURED by pg_stat_statements: pg_stat_statements.track_planning=off, so total_plan_time sums to 0 since stats_reset 2026-09-28 03:57Z - an off instrument, not zero planning. Turning it on is a DB-lane config decision. (3) Today's trip attributes to SCRAM-first client close at Supavisor (see MARS-787 note), not upstream saturation.
• coder-mars-cc blocked · 1w ago
  Root cause still open. Remaining discriminators: per-attempt connect phase (MARS-799), planning cost (track_planning off - DB-lane decision to enable), host CPU/IO telemetry. Log read path is NOT the blocker: query_logs works from coder-mars-cc.
• coder-mars-cc note · 1w ago
  coder-mars-cc CORRECTION to my note above: the Oct-5 11:49:30-11:52:00Z window IS still readable (query_logs: 87 postgres_logs, 290 supavisor_logs rows). Finding from it, log_min_duration lines: during 11:50:09-11:50:56Z EVERY protocol phase of EVERY statement class was slow at once - PARSE of a one-line UPDATE emailOutbox 3638ms, parse of the appEvents INSERT 2276ms, parse of a lookup SELECT 6914ms, a primary-key 'select * from appEvents where id=...' 9130ms, a single-row emailOutbox INSERT 6382ms; then 6x 'canceling statement due to statement timeout' at 11:50:09.666 and 11:50:45.05-45.56. Parse does no planning and a PK lookup has no plan to be expensive, so this window is an INSTANCE-WIDE stall (CPU starvation or IO wait on the host), not cost concentrated in planning or in heavy readers. That narrows the root cause toward Nano-tier CPU/IO capacity, which is the one candidate on this row an app-side fix cannot address. Unmeasured: host CPU steal / IO wait for the window - Supabase exposes that only on the dashboard metrics, not in logs or SQL. Also in window: 11:50:46.796 'column durationMs does not exist' - a hand-written probe, not app code (no durationMs column exists).
• coder-mars-cc note · 1w ago
  coder-mars-cc RECONCILIATION, all 11 db-breaker-open trips vs supavisor_logs+postgres_logs, window -90s..+30s around each open (counts: slow=log_min_duration lines, to=statement-timeout cancels, sf=SCRAM-first timeout/abandon, sfin=SCRAM-final timeout). 09-16 18:49 slow0 to0 sf2 sfin1 | 09-21 23:05 slow0 to43 sf1 sfin2 | 09-23 15:13 slow2 to0 sf0 sfin2 | 09-23 19:22 slow10 to0 sf3 sfin1 | 09-24 14:12 ALL ZERO | 09-25 00:32 slow1 to22 sf3 sfin0 | 09-25 10:48 slow0 to0 sf0 sfin3 | 09-28 15:45 slow0 to0 sf2 sfin1 | 09-29 20:31+20:32 (one window, two trips) slow7 to0 sf0 sfin4 | 10-06 12:24 slow0 to0 sf2 sfin0. Baseline: a full day 2026-10-04 had ZERO SCRAM timeouts and ZERO statement timeouts (a Sunday, low traffic - a limit on the baseline). READING: 10 of 11 trips coincide with Supavisor SCRAM handshake timeouts, a signature absent at baseline. Only 2 (09-21, 09-25 00:32) also show DB-side statement-timeout bursts; only 09-23 19:22 and 09-29 show a meaningful slow-statement cluster. 09-24 shows nothing DB-side at all. And the Oct-5 11:50Z instance-wide stall produced NO breaker trip and NO SCRAM timeouts. So these are TWO failures sharing one symptom space, as pmmaster suspected: (A) handshake abandonment at Supavisor -> breaker trips (MARS-787/799's question); (B) instance-wide execution stall -> statement timeouts, which tripped the breaker only when it overlapped (A). Capacity explains (B), not the trips. Title corrected to stop asserting CPU-bound planning (track_planning=off, never measured).
• coder-mars-cc titleChanged · 1w ago
  Supabase saturation: instance-wide execution stalls (Oct-5 class) + PostgREST retry loop + Nano tier - root cause and fix plan (planning cost UNMEASURED: track_planning off)
• coder-mars-cc blocked · 1w ago
  Blocked on ONE Elazar ask, carried by pmmaster-evolutiva-cc: (a) Nano tier decision (cost) and (b) enabling pg_stat_statements.track_planning (Supabase project setting; mars has no write-capable db seat). Only unreachable measurement term: host CPU-steal / IO-wait, Supabase dashboard only.
• coder-mars-cc note · 1w ago
  coder-mars-cc weekday baseline added (pmmaster ask): 2026-09-30 Wed 5717 Supavisor client authentications, 2026-10-01 Thu 5608 - both ZERO SCRAM-first, ZERO SCRAM-final, ZERO statement timeouts. The Sunday 10-04 reference had 2284 authentications, also zero. So the handshake signature is absent under ~2.5x more connection traffic, not just absent for lack of connections.
• db-mars-ca note · 1w ago
  Oct-7 23:37:22-37Z event added as a separate same-window observation (full bounded appEvents census and exact SQL on MARS-743). Five generic unhandled-server-error rows, v2.85.22: /mi-cursada x3, /alumnos x1, /practicas/[id] x1; no fingerprint/sqlLength and zero separate db-statement-timeout appEvents. Supabase query_logs denied this seat for 23:37:15-23:37:40Z, so there is no Postgres-side lock/saturation attribution. This is cross-route temporal evidence only; the Oct-7 04:35 ART dbperf deltas are a different window and remain unlinked. No change to the dashboard/tier blocker.
• db-mars-ca note · 1w ago
  db-mars-ca batched snapshot/workload comparison (read-only, agent_db_mars_ca). stats_reset=2026-09-28 03:57:55.062625Z, so no reset between these snapshots. Exact snapshot-delta SQL: WITH snapshots AS (SELECT s."capturedAt",x->>'queryid' queryid,(x->>'calls')::bigint calls,(x->>'total_exec_ms')::numeric total_ms,(x->>'mean_exec_ms')::numeric cumulative_mean_ms,CASE WHEN x->>'snippet' ILIKE '%get_practica_detail_bundle%' THEN 'detail_bundle' WHEN x->>'snippet' ILIKE '%FROM "appEvents"%' THEN 'appEvents_reader' END stmt FROM public."dbperfSnapshots" s CROSS JOIN LATERAL jsonb_array_elements(s."statementsJson") x WHERE s.app='mars' AND s."capturedAt">=timestamptz '2026-10-06 00:00:00Z' AND (x->>'snippet' ILIKE '%get_practica_detail_bundle%' OR x->>'snippet' ILIKE '%FROM "appEvents"%')), deltas AS (SELECT *,lag(calls) OVER(PARTITION BY stmt ORDER BY "capturedAt") prior_calls,lag(total_ms) OVER(PARTITION BY stmt ORDER BY "capturedAt") prior_total_ms FROM snapshots) SELECT "capturedAt",stmt,calls-prior_calls call_delta,round(total_ms-prior_total_ms,2) total_ms_delta,round((total_ms-prior_total_ms)/nullif(calls-prior_calls,0),2) interval_mean_ms,cumulative_mean_ms FROM deltas WHERE prior_calls IS NOT NULL ORDER BY "capturedAt",stmt; Matched 24h windows: Oct6 07:35:53.285Z-Oct7 07:35:33.458Z; Oct7 07:35:33.458Z-Oct8 07:35:37.814Z. appEvents table inserts in those windows were respectively 995 and 2,215 (same count grouped by createdAt). Results: appEvents reader query +288 calls in each interval; total +123.239s then +134.542s; interval means 427.91ms then 467.16ms (+9.2%). Cumulative means 452.45ms and 453.90ms are not interval means. get_practica_detail_bundle +184 calls/+19.087s, interval mean 103.73ms in first window; +525 calls/+84.489s, interval mean 160.93ms in second (+55.1%). Its cumulative means were 119.76ms then 127.80ms. Therefore appEvents reader cadence is flat (~12/hour) with modest measured interval-cost rise; detail-bundle workload and interval mean both rose substantially. Workload/request mix (including differing practica complexity) is not controlled, so this does not establish a code regression or connect either query to the Oct7 23:37Z cluster. No new work item opened from this comparison.
• db-mars-ca note · 1w ago
  Oct-8 incident correlation correction: direct appEvents query shows 1 exact DB statement-timeout row at 11:21:10Z and 8 at 12:33:28-12:34:12Z (base /practicas route); neighboring client errors at 11:21:22Z and 12:34:06Z are not statement timeouts. Forwarded split 2+7 differs from exact DB-event split 1+8 (same total nine). No fingerprint/sqlLength is persisted. Source check: get_practica_detail_bundle is used on /practicas/[id] and /mi-cursada/[id], not the base /practicas list, which uses getScopedPracticas. Oct-8 07:35Z detail-bundle dbperf interval was prior 24h (525 calls, 160.93ms interval mean) and is not same-window evidence or direct attribution for the list-route burst. Query logs denied; attribution remains open.