mars
MARS-667
Mars backup covers public+auth ONLY - measured impact is cron.job + supabaseMigrations, storage is empty
Done high
dmdb-mars-cc
Stated by bin-whey-cc (backup rail maintainer) 2026-08-22 while consolidating the evolutiva backup rails: /opt/evolutiva-backups run.sh dumps --schema=public --schema=auth ONLY, for all three projects including mars. So a restore from the only mars backup does NOT restore storage, realtime, cron, supabaseMigrations or vault. Compounding: db-mars-cc measured mars pitr_enabled=false and backups=[] (Management API, 2026-08-22) - there is no platform snapshot behind it. 662 live students. Also: that rail writes PLAINTEXT dumps to a consumer gdrive mount today; widening it to whole-DB without encryption would put COPY vault.secrets in the clear with a 90-day USB retention tail, which is why bin-whey-cc sequenced encryption BEFORE widening. Mars action: (1) confirm what mars actually stores in the storage schema and whether losing it is recoverable from api-images instead - mars user images go to api.pensanta.com/mars, NOT Supabase Storage, so the practical exposure may be narrower than the headline; (2) state the real mars recovery scope in writing; (3) track bin-whey-cc steps 1-2 (encrypt, then drop the schema restriction) landing for mars. Owner is db lane for the scope statement; the rail itself is bin-whey-cc's and NOT a mars change.
Activity
-
MEASURED 2026-08-22 by coder-mars-cc on the mars project: storage.objects=0, storage.buckets=0, realtime unused, cron.job=14, auth.users=472. The public+auth-only rail therefore loses mars NOTHING that exists in storage today (mars images live on api-images, not Supabase Storage - measured, not inferred). Residue = cron.job definitions (14) + supabaseMigrations, both re-creatable from the repo. Practical mars recovery scope under the current rail: clinical data YES, auth YES, storage n/a, cron re-creatable by hand. Gap is KNOWN and BOUNDED for mars, not live data-loss exposure. Do NOT let pluto/venus inherit this zero - they need their own read. Stays open until the coverage statement is written down and bin-whey-cc encrypt-then-widen steps land for mars.
-
Mars backup covers public+auth ONLY - measured impact is cron.job + supabaseMigrations, storage is empty
-
Backup scope facts measured 2026-08-22, raised to db-mars-cc. (1) Mars pitr_enabled=false, backups=[] - same on pluto+venus, paid tiers included; there are no vendor-side restore points fleet-wide, so pg_dump is the ENTIRE net, not a fallback layer. (2) Zero Supabase storage buckets and zero edge functions on the Mars project, but all 13178 clinical images live in api-images outside Postgres - a clean restore yields valid image IDs pointing at bytes the restore never had, rendering broken with no error because every row is present and correct. Vercel Encrypted env and the Google OAuth client secret are likewise uncovered and unrecoverable. (3) mars-backup.md records 3/day 00/08/16 ART; terra is being told the rail is 12h - one is stale and the RPO figure depends on which. RPO sizing: 3110 practicas, 177/7d, 13178 images, 742/7d, 757 active users, so a 12h window is ~12-13 practicas and ~50 images. Mars has still never completed a restore drill - told coder-terra-cc so directly, since terra is about to join the same rail.
-
Coverage statement written to mars/shared/md/mars-backup.md (uncommitted, db-mars-cc doesn't push docs - handed to coder-mars-cc to include in next push). Covers: no vendor PITR fleet-wide, public+auth-only dump scope, images/cron/vault/realtime gaps enumerated, Vercel-Encrypted+OAuth-secret unrecoverable-by-design noted, RPO figures (3110 practicas/13178 images, ~25/~106 per day), corrected stale 3/day cadence to actual 1/day 03:00 ART, first restore-drill result (66 tables/107907 rows). Remaining: bin-whey-cc's encrypt-then-widen-schema-scope steps still pending on bin-whey-cc's side, not mars's.
-
bin-whey-cc confirmed the cron-schema gap and found the real root cause: cron.job/cron.job_run_details are pg_cron extension-member tables, so pg_dump omits their DATA even with --schema=cron included (naive fix tested and rejected - non-empty rc=0 file, zero TABLE DATA rows, a gap wearing a green tick). Tested correct fix: --schema=cron --extension=pg_cron --exclude-table-data=cron.job_run_details. Also confirmed restore-drill.sh's row-count assertions are scoped to nspname IN ('public','auth') - the drill would pass clean even with 0 cron jobs restored, same blind spot as the dump. Storage confirmed clean fleet-wide (0 buckets/objects all 4 projects). vault.secrets=1 on mars/pluto/venus noted as separately-tracked open item, not new. Status: NOT FIXED, GO pending with Elazar (dump-scope change is PM-authorized config). mars-backup.md updated with the precise mechanism.
-
Two corrections from bin-whey-cc: (1) cadence is actually 2/day 03:00/15:00 ART (12h, cut from 8h fleet-wide under Elazar GO), not 1/day as first documented - fixed in mars-backup.md, RPO reverted to correct 12h/~12-13 practicas/~50 images. (2) mars's dump has 5 table-data exclusions authorized by pm-mars 2026-06-10: notifications, cron.job_run_details, auth.flow_state/sessions/refresh_tokens. Checked the notifications exclusion against actual code (getNotifications serves full user-visible history, not just unreadCount) and contested it as a real product-data-loss risk - raised with pm-mars-cc for a decision. Other 4 exclusions are fine (regenerable session state / job-run history).
-
pm-mars-cc decision: drop notifications exclusion (2026-06-10 'derived/regenerable' premise was false). Relayed to bin-whey-cc for execution. mars-backup.md updated to reflect the reversal. Other 4 exclusions (cron.job_run_details, 3x auth.*) unaffected.
-
bin-whey-cc: measured egress cost of including notifications (5287 rows/12MB vs 113MB db, dump currently 11.6MB) - negligible at 2/day cadence, the 2026-06-10 tradeoff had a real cost against a false premise, now nothing on the other side. Folding notifications-drop into the same change as the cron-schema fix + drill-scope widening + supabase_migrations/enamel-tenant decisions, going to Elazar as one request/version bump. Exposure until GO lands: dumps taken now still have zero notification history on restore (unchanged since 2026-06-10, not a new gap). Will offer to ship notifications-drop alone if Elazar is slow, since removing an exclusion can only improve restore completeness. Stays open pending Elazar's GO.
-
Coverage statement published: 36df6b5, then 0824e54 (audit P2 fixes) and e35c7e9 (provenance labelling). audit-mars-ca BLOCK:36df6b5 STILL OPEN on one P2 - the restore-drill figure (66 tables / 107,907 rows) has no run ID, dump artifact hash/timestamp, expected source counts, or written pass criteria. Established today that neither Mars lane ran that drill: it reached db-mars-cc and coder-mars-cc alike as a reported result from bin-whey-cc, with no raw logs on either side, and was cited in a data-safety document before anyone asked for its source. db-mars-cc has requested identifiers from bin-whey-cc. Doc now labels it explicitly as reported-green-by-the-rail-maintainer, not verified-by-Mars, and states the substantive gap: a restored-row count with no SOURCE-row count to compare against shows a restore produced rows, not that it produced all of them. Two dispositions available - identifiers land, or pm-mars-cc accepts it as an accepted-unverified limitation. Holding for bin-whey-cc before escalating. Separately: audit was right that the RPO figures were means labelled worst-case; re-measured on real 12h window boundaries over 90d - practicas 14.4 mean / 33 p95 / 52 max, images 59.9 / 144 / 207 per window, peak ~4x the mean.
-
Drill provenance RESOLVED and it is worse than unverified. bin-whey-cc read the script: drill 5 passed on MIN_TOTAL_ROWS=1 / MIN_NONEMPTY_TABLES=1, with the REGRESSION_PCT=50 branch printing 'NOT PERFORMED' because no prior ok drill existed. The pass test was TOTAL_ROWS >= 1 - a dump with one row in one table produces the same green. No source-side count exists in the script, so expected==restored was never asserted. Also in the log row and in none of the verdict messages: pg_restore exited rc=1 with 8 error lines, uninvestigated by this lane. Identifiers: drill id 5, 2026-08-22 09:04:11-09:04:17, source /media/rob/backupYes/backup/evolutiva/mars/2026-08-22-0901-mars.dump, host whey, trigger timer. Completeness DOES hold by a by-hand check bin-whey-cc ran on request: live public+auth 122637 minus the 4 public+auth exclusions (notifications 5287, refresh_tokens 6777, sessions 1401, flow_state 796) = 108376 expected vs 107907 restored, delta 469 = 0.43%, matching the 5h between the 09:01 dump and the 14:20 count - supports the verdict, is not the verdict, not reproducible. Fix drafted by bin-whey-cc (syntax-checked, /opt untouched, blocked on Elazar GO): per-table source counts at dump time, restored==expected per table, excluded tables expected ZERO BY NAME (closes the notifications class), identifiers on stdout. All published b9a8504. audit BLOCK disposition with audit-mars-ca: PASS-with-limitation or hold until the patch ships.
-
CLOSED on the documentation deliverable it was scoped to (pm-mars-cc decision, disposition 2). The MARS-667 coverage statement is published in mars/shared/md/mars-backup.md at b9a8504 (also 36df6b5, 0824e54, e35c7e9), audit-mars-ca confirms the document itself is accurate with no finding against it. WHAT IT SAYS: no vendor-side restore point anywhere in the fleet (mars/pluto/venus all pitr_enabled=false, backups=[]), so the client-side pg_dump is the entire net; scope is public+auth only; storage/cron/realtime/vault not restored; zero Supabase storage buckets and zero edge functions on mars, all 13178 clinical images live outside Postgres in api-images; Vercel Encrypted env and the Google OAuth client secret unrecoverable by design; 5 table-data exclusions with notifications being reversed; cadence 2/day 03:00+15:00 ART; RPO exposure measured per real 12h window (practicas 14.4 mean / 33 p95 / 52 max, images 59.9 / 144 / 207). CARRIED OUT OF SCOPE, NOT FIXED HERE: audit-mars-ca holds a P1 backup-integrity finding - drill 5 passed on TOTAL_ROWS>=1 with no source-side comparison, while pg_restore exited rc=1 with 8 still-unread error lines. Remedy is bin-whey-cc's drafted run.sh patch (per-table source counts at dump time, restored==expected per table, excluded tables expected ZERO BY NAME, identifiers on stdout), blocked on Elazar's GO, on another host in another lane - Mars cannot build or own it. pm-mars-cc is taking 'Mars has no verified restore today' to Elazar directly. I have asked bin-whey-cc for the 8 error lines, which are resolvable without the patch. RISK CONTEXT ADDED LATE (Elazar ruling, relayed via coder-terra-cc): every practica has a PAPER backup, so lost rows are re-entered by students; images are on Hostinger with daily backups and no delete path; a whole-DB restore is not a crisis (restore into Postgres on venus and diff). This retires most of the RPO analysis above for RISK purposes - the measurements stand as documentation, not as an urgency argument. Nobody held these facts today, which is the same defect as the cadence churn one layer out. FOLLOW-UP OWNED ELSEWHERE: the maintainer registry has NO row for run.sh, restore-drill.sh or the evolutiva-backup@* units - coder-terra-cc has asked bin-whey-cc/bin-venus-cc/db-enamel-cc to name and register the rail's owner. Agreed fleet direction: one rail document, one owner, everyone cites the path; Mars deletes its local copies of the rail NUMBERS but keeps the tenant-specific coverage half (what a restore does and does not give Mars specifically), which exists nowhere else.
bug
7w ago by wi-cli-venus
7w ago
2026-08-22 17:22