basquetWi + New ticket
evolutiva EVO-98

Un rol de dump sin BYPASSRLS produce un backup cuyo auth.users restaura VACIO y no falla en el momento del dump: verificar el credencial de cada lane de backup

Done high bvbin-venus-cc

WHEY HALF MEASURED AND CLEAR, 2026-09-11, bin-whey-cc. VENUS HALF OUTSTANDING - reassigned to bin-venus-cc for that leg only. Nothing was changed and no credential was touched. THE RLS HAZARD IS NOT ARMED ON ANY WHEY-SIDE EVOLUTIVA LANE. All four DSNs authenticate as role postgres with rolsuper=f but ROLBYPASSRLS=TRUE, so the zero-rows-through-policy failure cannot occur. All four are on aws-1-sa-east-1.pooler.supabase.com:5432, the SESSION pooler - none on :6543, so that trap is not armed either. Artifact row counts against live auth.users, latest dump each: mars 514/514, pluto 400/400, venus 42/42, enamel 9/10. The three exact matches are the proof the dumps actually carry auth data. Enamel one-row gap is a TIMING artifact and not RLS: the dump ran 09:00 and the live count is now, and an RLS failure reads 0, never 9. A signup after 09:00 closes it outright - db-enamel-cc to confirm if it knows of one. METHOD ARTIFACT THAT MUST TRAVEL WITH THIS CHECK, because bin-whey-cc first pass hit it and nearly reported it: pg_restore -a -n auth -t users <file> WITHOUT -f - errors out and prints nothing, which reads as zero rows. That looks exactly like total data loss. Anyone repeating this check must pass -f - and must first confirm the TOC carries a TABLE DATA auth users entry. A check whose failure mode is indistinguishable from the defect it hunts is worse than no check. TWO CORRECTIONS TO THIS ROW OWN PROXY, from db-enamel-cc, both narrowing it: rolbypassrls is NECESSARY BUT NOT SUFFICIENT. A role can be effectively unfiltered by OWNING the tables instead - and auth.* here is owned by supabase_auth_admin, so a job authenticating as anything else needs the flag regardless. In the other direction, FORCE ROW LEVEL SECURITY applies policies to the OWNER too, so a lane that PASSES a rolbypassrls check can still dump zero rows if it was relying on ownership. Both collapse to the same instruction: the artifact row count is the only thing that closes it. WHAT REMAINS, AND IT IS THE ONE THAT MATTERS MOST. venus-db-backup.service is NOT on whey. venus-db-backup.sh v1.6 lives in sh.git there, but no whey unit schedules it - the lane runs on venus, and cross-host DB queries are barred, so bin-whey-cc could not measure its role or port. VENUS-350 established that this dump is venus prod ONLY restore path: pitr_enabled=false, backups=[]. Same three checks needed on venus: rolbypassrls on whatever role that DSN uses, port 5432 versus 6543, and an artifact row count for auth.users. MEASURE AND REPORT ONLY - VENUS-350 standing no-remediation directive holds and this row does not override it. ALSO NOT COVERED, stated rather than omitted: restore-drill.sh also calls pg_dump and was not checked - the backup lane was, the drill lane was not. Non-postgres lanes carry no RLS surface at all (backup-gdrive-to-hdd, whey-state-backup, venus-backup-pull, api-images-backup are file copies; the mariadb rail has no RLS concept). SEPARATE FINDING, deliberately NOT folded into this row: ~/.local/bin/backup-whey-to-ssd.sh v1.1.1 documents a pg_dump of the telemetry db, but backup-whey.service actually runs /opt/maintenance/bin/backup-whey-to-ssd.sh v2.0, which contains no pg_dump at all. Consequence today is nil - whey postgresql is masked, there is no local database to dump - but a tracked file advertises a capability that runs nowhere. Already escalated to Elazar by bin-whey-cc; if it needs a row it gets its own.

Sub-tickets

No sub-tickets.
+ Add sub-ticket

Activity

  • wi-cli-venus created · 5w ago
  • wi-cli-venus assigned · 5w ago
    bin-venus-cc
  • wi-cli-venus descriptionChanged · 5w ago
    was: Filed by pmmaster-evolutiva-cc 2026-09-11 under Elazar delegated GO. Raised by db-enamel-cc while scoping a dedicated dump role for enamel; generalised here because the defect is not enamel-specific and nobody has checked the other lanes. THE DEFECT. 26 tables on the shared project carry RLS - all of auth.*, cron.*, storage.*. A dump role that lacks BYPASSRLS reads those tables through the policy and gets zero rows. pg_dump DOES NOT FAIL. It writes a well-formed artifact of plausible size, and the loss only becomes visible at RESTORE time, when auth.users comes back EMPTY. So the failure is silent at the moment it happens, detectable only by an act nobody performs until they are already in an incident. WHY THIS IS p1 DESPITE NOTHING BEING BROKEN TODAY. This is the exact shape the fleet has hit four separate times this week: an operation that succeeds while doing less than the caller assumes. A backup is worse than the others, because its whole value is being correct on a day when nothing else is. An artifact that exists, has a recent mtime and a believable byte count is indistinguishable from a good one without a restore. WHAT TO DETERMINE, per backup lane, and report per lane rather than as one verdict: which ROLE each backup job authenticates as, whether that role has BYPASSRLS or is a superuser (a superuser bypasses RLS inherently and is fine on this axis), and whether the artifact it produces actually contains auth.users rows. The last one is the only real answer - rolbypassrls is the cheap proxy, a row count from the artifact is the measurement. LANES IN SCOPE, all of them, not only the one that raised it: /opt/evolutiva-backups on whey (the mars/pluto/venus case-arms, plus enamel if that arm exists - ENAMEL-12 is open on exactly that contradiction), and venus-db-backup.service on venus, which matters most because VENUS-350 established it is venus prod ONLY restore path - pitr_enabled=false, backups=[]. If that one dumps without RLS bypass, venus has no restore path at all rather than a degraded one, and nobody would know. ALSO CHECK THE CONNECTION PORT while you are in each job, same class of silent trap: pg_dump against the :6543 pooler is known-bad; session mode :5432 is required. db-enamel-cc flagged it independently. DO NOT ROTATE OR REPLACE ANY CREDENTIAL ON THE STRENGTH OF THIS ROW. Measure and report. If a lane needs a new role the shape is login, INHERIT, pg_read_all_data, BYPASSRLS - but minting it is a separate decision with an owner, and VENUS-350 carries a standing measure-and-report-only directive that this row does not override. Report the coverage limit explicitly: which lanes you could NOT reach from whey, and say so rather than omitting them.
  • wi-cli-venus completed · 5w ago
    CLOSED 2026-09-11. BOTH LEGS MEASURED, HAZARD NOT ARMED ANYWHERE. No credential touched, no remediation, VENUS-350 directive not overridden. WHEY LANES (bin-whey-cc): all four DSNs authenticate as role postgres, rolsuper=f but ROLBYPASSRLS=TRUE; all on aws-1-sa-east-1.pooler.supabase.com:5432, the session pooler, none on :6543. Artifact row counts against live auth.users: mars 514/514, pluto 400/400, venus 42/42, enamel 9/10 (timing - the dump ran 09:00; an RLS failure reads 0, never 9). VENUS LANE (bin-venus-cc): venus-db-backup.sh v1.6, port 5432, role postgres with rolbypassrls=t. Artifact venus-prod-20260911-1207.sql.gz.gpg, 2,936,021 bytes, decrypting to 20,511,970 bytes of SQL containing exactly one COPY auth.users block of 42 rows against a live count of 42. Decrypted copy SHREDDED, not trashed - trash would have left a full plaintext patient database recoverable in ~/.local/share/Trash. THE METHOD TRAP, in both shapes, and it must travel with any repeat of this check: the check fails in a way indistinguishable from the defect it hunts. pg_restore -a -n auth -t users WITHOUT -f - errors and prints nothing, reading as zero rows. A COPY-block count has the same property - a missing table yields no block AND no error, also reading as zero. ASSERT PRESENCE BEFORE MEASURING ABSENCE, in both shapes. bin-whey-cc hit the first and caught it; bin-venus-cc pre-empted the second. PROXY NARROWED TWICE, keep both: rolbypassrls is necessary but NOT sufficient. A role can be effectively unfiltered by OWNING the tables instead - auth.* here is owned by supabase_auth_admin, so a job authenticating as anything else needs the flag regardless. And FORCE ROW LEVEL SECURITY applies policies to the owner too, so a lane that PASSES a rolbypassrls check can still dump zero rows if it relied on ownership. The artifact row count is the only measurement that closes it. THE FINDING THAT OUTLASTS THIS ROW: venus 42/42 was read as two independent lanes corroborating each other. It is ONE DATABASE COUNTED TWICE - both lanes target ref fjnhfjwskixzsilxrkph. bin-venus-cc inferred difference from surface spelling (different env file, destination and cipher) without comparing the one field that identifies the target, then found and reversed it unprompted. Two independent-looking measurements agreeing EXACTLY is a reason to check whether they share a source, not a reason for extra confidence. Consequence is good news and is recorded on VENUS-350: venus prod has one database with TWO rails whose failure modes are disjoint. DO NOT CONSOLIDATE THEM. NOT COVERED, stated rather than omitted: restore-drill.sh also calls pg_dump and was not checked - the backup lane was, the drill lane was not. Non-postgres lanes carry no RLS surface (backup-gdrive-to-hdd, whey-state-backup, venus-backup-pull, api-images-backup are file copies; the mariadb rail has no RLS concept). Separately escalated to Elazar by bin-whey-cc and deliberately not folded in here: ~/.local/bin/backup-whey-to-ssd.sh v1.1.1 advertises a pg_dump that the unit actually running (v2.0 at /opt/maintenance/bin/) does not contain - nil consequence today since whey postgresql is masked, but a tracked file claims a capability that runs nowhere.
task
5w ago by wi-cli-venus
5w ago
2026-09-11 13:20