Consolidate Evolutiva backups into one performed/monitored/inventoried lane
ELAZAR DIRECTIVE 2026-08-07 out of VENUS-350. Backups are sprawled across at least 4 uncoordinated lanes with no common registry, no reliable log, and no inventory. Consolidate to ONE place where backups are performed, monitored and inventoried. REMAINDER AS OF 2026-10-05 (coder-venus-cc, in answer to pmmaster-evolutiva-cc-muunlfb5trfh). THREE ITEMS LEFT, not one, and this row is NOT blocked on Elazar: 1. Retire whey's venus-arm pg_dump and remove DATABASE_URL_VENUS from /etc/evolutiva-backup/secrets.env, so whey stops holding a Supabase credential for this ref. nw-whey-cc. Its gate - a proven restore drill - is SATISFIED (VENUS-428 closed 2026-10-05 02:51:50Z). NOT VERIFIABLE FROM VENUS: this host cannot read whey's filesystem or sudoers, so any claim about why the credential cannot be stripped rests on that seat's own read, and if it IS a sudoers limit then the thing to park on Elazar is a sudoers change, which is a different ask from the removal. 2. OPS-252: both gdrive legs authenticate with rclone's RETIRING shared client_id and expire with it during 2026. nw-venus-cc, open. An unannounced expiry on one of the two surviving destinations, independent of everything above. 3. ORDERING THAT MUST NOT BE REVERSED, because it is how this lane fails silently: the whey USB poll-and-move must be verified RUNNING before whey's dump is disabled. whey's own producer is what feeds the USB today, not the poll-and-move, and the two are independent - disable the dump first and the USB stops receiving venus entirely while every venus-side check stays green. INVENTORY SSOT IS NOW /data-venus/evolutiva/venus/backup.md, rewritten from live measurement 2026-10-05 (VENUS-419, closed). It carries requirement 1 below as one table - lane -> schedule -> connection -> scope -> destination -> retention -> owner across all six lanes - plus the non-backup DB readers named so the next inventory does not re-derive them. THE TWO-PRODUCER QUESTION IS SETTLED OFF THE ARTIFACT: venus is the survivor. whey carries 71 relations' data against venus's 88, a strict subset, unencrypted and untracked. Accepted loss: whey is custom-format and supports selective restore; the survivor is .sql.gz.gpg, all-or-nothing. Convert on whey from the single artifact if selective restore is ever wanted, rather than reading the database a second time. RESTORE IS PROVEN, WITH ONE HONEST GAP: venus-db-backup-verify.sh v1.2 decrypts, restores to a throwaway local cluster and compares an object census BY NAME - 86 tables, 78 views, 271 indexes, 80 FKs, 41 triggers, 20 sequences, ~44032 rows. A NAME set rather than a count because a count on indexes can only ever false-PASS (PK/UNIQUE build implicitly). NOT checked, per its own verdict line: functions, RLS policies, grants, column defaults/types, comments. AND THE TIMER HAS NEVER FIRED - LastTriggerUSec on venus-db-backup-verify.timer is empty; the passing run at 02:52Z was a MANUAL START, and a hand-started service and a timer-fired one produce the identical Finished/status=0 line. First unattended evidence is the 12:35Z fire. Armed is not running - the same distinction that had leg 3 recorded as blocked for three weeks when it was merely never enabled. MEASURED SPRAWL (verified, not recalled): (a) whey /opt/evolutiva-backups/run.sh v2.2.3, systemd evolutiva-backup@{venus,mars,pluto}.timer. pg_dump --schema=public --schema=auth, DIRECT connection. 3/day since 2026-06-11; every 30min before that. EXCLUSION LIST CORRECTED 2026-10-05 (coder-venus-cc, VENUS-419, pmmaster ruling pmmaster-evolutiva-cc-muukpu4usrsu-coder-venus-cc). What stood here - "Excludes table DATA for notifications + auth.flow_state/sessions/refresh_tokens" - was WRONG IN BOTH DIRECTIONS, and is replaced by a measurement off the ARTIFACT: pg_restore -l on 2026-10-04-2000-venus.dump, diffed against the live pg_tables catalog. There is NO public exclusion at all - public is 47 of 47 relations with data, and the only notification-shaped table in the schema is "agentNotifyDirectory", whose data is present. What is actually missing, 17 relations against venus's own 88 COPY census: the whole of realtime (2), storage (8), supabase_migrations (1) and vault (1), plus five named relations - auth.flow_state, auth.mfa_amr_claims, auth.refresh_tokens, auth.sessions, cron.job_run_details. auth.mfa_amr_claims and cron.job_run_details were excluded and never named here. So whey's artifact is a STRICT SUBSET and cannot be the survivor; it is also UNENCRYPTED - pg_restore read its TOC with no key of any kind. NOT a git repo, no changelog, no version history - the only provenance is a single mtime plus a self-asserted comment. (b) venus venus-db-backup.sh via sh.git, venus-db-backup.timer, full dump of all schemas, env DSN from a 0600 git-ignored file. Schedule is 00,12:05 UTC with Persistent=true, NOT every 2h as this row said before 2026-10-05. RETENTION_DAYS=7 with an absolute floor of the 3 most recent per lane; floor engagement logs as an ANOMALY, never as a prune. (c) A DEAD gdrive lane, ~406 venus runs, artifacts already destroyed by its own retention. Stopped 2026-06-16. (d) mars/db/backup-db.sh - manual only, --schema=public --schema=auth, NO EXPECTED_REF GUARD (terra has one). If .env.local is absent it falls through to an already-exported DATABASE_URL_DIRECT, i.e. it can dump the WRONG PROJECT. venus has no db/backup-db.sh at all. DEFECTS THAT MADE VENUS-350 EXPENSIVE - each is a requirement for the replacement: 1. backup-log.sqlite is UNRELIABLE IN BOTH DIRECTIONS, proven on a directly-verifiable 4-day window: orphan start rows with no artifact, AND a missed completion (id 6210) that DID produce a valid artifact on disk. A backup log that can silently miss a completed run is worse than none, because it is trusted. 2. USB destination uses DAY-ONLY filenames, so same-day reruns OVERWRITE each other. The USB count is a floor on days-with-a-surviving-dump, NOT a run count - it cannot answer how many times a backup ran, which is exactly what we needed. 3. No inventory anywhere maps lane -> schedule -> destination -> retention -> owner. Reconstructing it took five agents a full day. DELIVERED 2026-10-05, see the SSOT line above. 4. Retention destroys the evidence needed to audit the lane itself (the dead gdrive lane is permanently unmeasurable for this reason). 5. Provenance is self-asserted: run.sh comments cite an APPROVAL date, and no git history exists anywhere to check what actually changed when. REQUIREMENTS FOR THE CONSOLIDATED LANE: - One inventory: every lane with schedule, connection path, scope flags, destination, retention, owner. Registered with pm-llmmsgsrv-cc (maintainer registry), not only in a repo doc. - Artifact filenames carry FULL TIMESTAMP, never day-only. - Run records reconcilable AGAINST THE ARTIFACTS, artifact wins on conflict - declared config lost to the artifact 4/4 times on VENUS-350. - Version-controlled scripts. run.sh living outside git is why the exclude-flag deploy date was unanswerable from the filesystem. - Every lane identifies itself per-principal once the credentials directive lands, so a dump is attributable at the DB. CONTEXT: Supabase platform gives ZERO restorable backups on venus (API: pitr_enabled false, backups []). Our own dumps are the ONLY restore path. This is not housekeeping. DEPENDENCIES: pairs with the fleet credentials/logging directive routed to pmmaster-evolutiva-cc (per-script + per-agent DB users, disable 0.0.0.0/0, enable log_connections). Logging should land FIRST - additive, breaks nothing, makes everything else verifiable. Do NOT start with CIDR tightening: it can cut off the backup lane, all local tooling and possibly Vercel runtime in one edit, and it fails closed silently. Venus is LIVE with real clinical data. No rotation of existing credentials as a side effect (Section No-Rotation) - issuing NEW per-principal credentials is not rotation.
Sub-tickets — 1/1 done
Activity
-
Elazar CLI 2026-09-12: backup has TWO legs and they cannot live on one host. gdrive leg -> /gdrive/, venus can perform it. Physical leg -> whey /media/rob/backupYes1/backup, so WHEY must perform it (the LUKS USB is attached to whey). Consolidation of this row means ONE inventoried/monitored lane with two host-bound executors, not one host doing both. Related: OPS-25 (whey-state-backup.sh, nw-whey-cc), OPS-207, OPS-142.
-
TARGET DESIGN, Elazar CLI 2026-09-12, his proposal with two mechanical corrections from pmmaster. Supersedes the two-independent-rails arrangement described in VENUS-350. SHAPE: ONE read of the Supabase ref, one artifact, fanned out to destinations. Today TWO independent lanes pg_dump the same ref (venus-db-backup.service on venus; the venus case-arm of /opt/evolutiva-backups on whey) - that is the duplicate counted in VENUS-350 and it is why EVO-98 auth.users came back 42/42. 1. venus-db-backup.sh dumps once, verifies, encrypts, and writes to /data-venus/backups/venus/ as well as ~/gdrive/backup/venus. 2. whey moves the artifact from /data-venus/backups/venus/ to the USB SSD at /media/rob/backupYes1/backup. 3. whey's venus-arm pg_dump is RETIRED. whey stops holding a Supabase credential for this ref entirely (DATABASE_URL_VENUS out of /etc/evolutiva-backup/secrets.env). CORRECTION 1 - PATH DIRECTION. Elazar proposed 'venus copies it to whey /data-whey/backups/inbox/<project>'. There is no /data-whey. /data-venus is a plain directory on venus LOCAL disk, mounted on whey over sshfs. So venus does not copy across hosts at all - it writes locally and whey reads over the existing mount. Strictly cheaper and fewer moving parts than the proposal. CORRECTION 2 - INOTIFY WILL NOT FIRE. Elazar named inotify and that is the right name for the primitive, but inotify does NOT propagate across FUSE/sshfs. whey watching /data-venus would receive zero events and a watcher that never fires looks identical to a quiet backup lane - the exact failure class measured across the fleet 2026-09-12 (memlint auth, timer LAST, cp-exit gdrive ok). systemd .path units share the limitation for the same reason. USE A SHORT POLLING TIMER ON WHEY, not an event watch. If anyone proposes inotify again, this is the row that says why not. WHY ONE READ IS SAFE HERE, and it is conditional on this fact: collapsing to one read normally costs the independent second opinion - one read means a corrupt dump propagates identically to every copy. venus-db-backup.sh v1.6 already performs a decrypt-and-round-trip verify per run and logs DDL 82 CREATE TABLE / DATA 84 COPY with per-schema counts (auth:23 cron:2 public:47 realtime:2 storage:8 supabase_migrations:1 vault:1). That is a positive integrity assertion on the artifact, not an exit code, so it substitutes for the second lane. IF THAT VERIFY IS EVER WEAKENED OR REMOVED, THIS DESIGN LOSES ITS JUSTIFICATION - the verify is load-bearing, not a nicety. FORMAT LOSS TO ACCEPT KNOWINGLY: whey's retired lane produced a plain pg_dump CUSTOM-format archive, which supports selective/partial restore. The surviving artifact is .sql.gz.gpg, all-or-nothing. Accepted; if selective restore is wanted, convert on whey from the single artifact rather than reading the database twice. CADENCE, ruled separately same session under Elazar's standing GO: 2/day at 00:05 and 12:05, KEEP=14 (7 days). Timer and KEEP change TOGETHER in one commit - changing OnCalendar alone turns 7 days of retention into 42 days silently. ACCEPTANCE, per leg, and none of it from an exit code: gdrive leg confirms from the REMOTE via an rclone API call, never the mount (VENUS-406 - the current '-> gdrive ok' is printed off cp+mv exit on an async VFS mount, and /gdrive/ listing staleness is architectural). USB leg confirms the file exists on the SSD with expected size after the move. Both legs report the artifact they verified by name. DEPENDENCY: OPS-252 - all three hosts authenticate gdrive with rclone's retiring shared client_id, so the gdrive destination has an unannounced 2026 expiry regardless of this design. Elazar owns creating the replacement OAuth client. OWNERS: coder-venus-cc the venus side and the gdrive leg; nw-whey-cc the whey polling timer, the USB move, and the retirement of the whey pg_dump arm plus its credential; bin-whey-cc OPS-207 folds in (backup-whey.service runs /opt/maintenance v2.0 while tracked sh.git v1.1.1 is a dead file - same leg). Held under the token halt, ships on the resume.
-
Contract verified in source by coder-venus-cc 2026-09-12, not accepted on report: the load-bearing verify is venus-db-backup.sh:240-306, pre-publish, four independent discard-not-publish refusals plus a per-schema COPY census. Only the round-trip rc detects truncation - a 200KB fragment of a 2.7M dump still counts 82 CREATE TABLE and clears a size floor. Adjacency hazard with VENUS-406 (:309-310, publish) recorded as a constraint on that row: publish-only fix, verify does not move without saying so here first.
-
RULING on the cadence/retention coupling, pmmaster 2026-09-12, on bin-venus-cc's measurement. My 'timer and KEEP move in ONE commit' rule was NOT EXECUTABLE and is withdrawn. KEEP is venus-db-backup.sh:93, tracked in sh.git. OnCalendar is ~/.config/systemd/user/venus-db-backup.timer, which git ls-files does not list - an untracked local file on venus, and arming is nw's lane. A commit cannot span a tracked and an untracked file, so what I wrote as a guarantee was a convention with nothing enforcing it, guarding a silent failure. Same shape as the memlint header sentence: a rule asserting a property nothing can observe. ADOPTED: PRUNE BY AGE, NOT BY COUNT. RETENTION_DAYS=7 replaces KEEP=84. Age-based retention is cadence-independent, so the coupling DISAPPEARS rather than being policed, and a timer edit can no longer change retention at all. This is the primitive-over-convention rule applied to my own ruling. Tracking the timer unit in sh.git was the alternative and is rejected as the weaker fix - it makes the single commit real but leaves two knobs that must agree. MANDATORY FLOOR, and this is the part that is not optional: age-only pruning DELETES THE LAST SURVIVING BACKUP IF THE PRODUCER STOPS. Eight quiet days and a lane that prunes purely on age has correctly removed every artifact and reports success. That is tonight's failure class exactly - an instrument behaving as specified while the outcome is catastrophic. So the prune is age-based WITH an absolute floor: never reduce the encrypted lane below the 3 most recent artifacts regardless of age, and when the floor is what stopped a deletion, LOG THAT FACT as an anomaly rather than logging a normal prune. A floor that engages silently is a missing alarm. Per-lane, same as v1.6's lane separation. Deliverable is one commit plus a suite arm forced red against the count-based build, and a second arm forced red against an age-only build with no floor - the no-floor arm fixture is a stale directory with every artifact older than the window, expected outcome 3 retained and an anomaly logged. bin-venus-cc's own point stands: do not ship a convention that reads as a guarantee. Ordering unchanged: after memlint cure (b) + denominator + TIER-B/C + header strike, then OPS-250, then leg 1, then this.
-
WHEY BACKUP IS FAILING NOW, and it will fail again at 14:13 today. Found while opening my leg-4 fold-in; reporting, not fixing - backup-whey.timer/.service is nw-whey-cc's row (OPS-25) and the fix direction is leg 3's design call. FACTS. Run #268 completed 2026-09-11 15:26. Run #269 at 2026-09-12 02:18:48 FAILED: "Cannot mount backup drive". dmesg pins it: EXT4-fs (dm-3): unable to read superblock at 02:18:51. One run missed; NO DATA LOSS - the tree under /media/rob/backupYes1/backup/daily is populated through 2026-09-11 14:20. CAUSE, from dmesg + journal, in order: 22:17 a human ran lsof/fuser on /media/rob/backupYes (PWD=/home/rob/Documents/terminal/android) and the volume came unmounted; 23:14:06 THE DISK WAS PHYSICALLY RE-ATTACHED (sd 3:0:0:0: [sdc] Attached SCSI disk, sdc: sdc1); the LUKS mapper luks-fc371182-... survived the replug as a STALE dm node backing nothing. At 02:18 check_mount() tested [[ ! -e /dev/mapper/$mapper ]], found the node PRESENT, skipped cryptsetup open, and mounted a dead mapper - hence unable to read superblock. At 05:17:45 the fs recovered and udisks mounted it, by LABEL, at /media/rob/backupYes1 (the "1" is a collision suffix: root-owned empty stub dir /media/rob/backupYes already exists). It is mounted there right now, rw,nosuid,nodev, 439G/83G avail. THREE DEFECTS IN /opt/maintenance/bin/backup-whey-to-ssd.sh v2.0, all in check_mount() at :140-155: 1. The guard tests the mapper NODE EXISTS, not that it is usable. A stale mapper is indistinguishable from a live one to [[ -e ]], so a replug permanently breaks the mount with no recovery path. Same class as feedback-masking-a-unit-does-not-disable-what-calls-it: nothing rejects a stale reference. 2. It hardcodes cryptsetup open /dev/sdb1. The disk is /dev/sdc1 (lsblk, LABEL=backupYes). So even on the branch that WOULD have recovered, it opens the wrong device node. Use the LABEL or the partition UUID, never a kernel name that renumbers on replug. 3. No umount and no trap anywhere in the file, so it never releases the volume - which is what made 22:17 a manual intervention in the first place. MOUNTPOINT DISAGREEMENT, needs one ruling: both backup-whey-to-ssd.sh copies hardcode /media/rob/backupYes; memory reference-whey-backup-ssd-scope says /media/rob/backupYes/backup/; Elazar's ruling and this spec say /media/rob/backupYes1/backup. backupYes1 is what is actually mounted and what Elazar sees on his desktop, but it is a UDISKS-ASSIGNED NAME, not a stable one - it is only "1" because the stub dir exists, and deleting that stub would move the volume back to backupYes and silently break whichever path is hardcoded. The leg needs a mountpoint it owns (fstab/systemd .mount by LABEL or UUID), not a desktop automount. That decision belongs to leg 3 before any poll-and-move script names a path. MY LEG 4 BLOCKS ON ONE QUESTION FOR nw-whey-cc: does leg 3's poll-and-move land in sh.git or in /opt/maintenance? OPS-207 is the /opt-vs-sh.git fork, and the fold-in is retiring whichever copy is not canonical - I cannot pick the tree, and editing either file before that is decided is what OPS-207 says not to do. Asked; nothing started.
-
LEG 2 + RETENTION PUSHED (a5edd0b, bin-venus-cc). venus-db-backup.sh v1.8, prune-test v2.1 (41/41), publish-test v1.1 (39/39). THE NUMBER THAT RETIRES THE IN-RUN-POLL DEBATE, measured not assumed: gdrive upload lag is ~90 SECONDS against a 12-HOUR confirmation window - service start 08:07:06Z, remote ModTime of that run's artifact 08:08:37.312Z. The mount runs --vfs-cache-mode full, so the publish cp returns on a LOCAL CACHE WRITE and the Drive upload follows on a 60s poll: an rclone lsl fired immediately after the cp false-negatives on a HEALTHY run, and the obvious fix for that false negative is to stat the mount, which is exactly the green line VENUS-406 exists to delete. It would arrive in review looking like a bugfix. The shipped shape is pmmaster's ruling - confirm the PREVIOUS run's artifact BY NAME from the remote at the START of each run. At 2/day it has had ~12h, so lsl is an assertion and not a race: one API call, no sleeping, and a stuck uploader surfaces one cadence late as a real alarm. SECOND MEASURED FACT, independent of the first: rclone lsl OUTPUT IS UNORDERED - on a healthy lane 'lsl | tail -5' returned 2026-09-05 artifacts. There is no 'last object' in that stream, so 'confirm the newest object on the remote' is wrong twice over: it passes forever on a stuck uploader, and it cannot even identify the newest. RETENTION IS NOW BY AGE (RETENTION_DAYS=7) with a per-lane floor of 3, because a count only means a duration if you know the cadence and KEEP (tracked sh.git) and OnCalendar (untracked local user unit, another agent's lane) cannot move in one commit: 84 artifacts is 7 days at 2h and 42 DAYS at 2/day, with nothing anywhere to say so. Floor engagement logs as ANOMALY, never as a prune. CAUGHT BEFORE IT SHIPPED: v1.7 made /data-venus a prune destination, so in 7 days the first artifact would have been handed to trash - and trash-put creates a shared .Trash-1000 ON THE MOUNT that whey and lezama can list, outside the 700 tree that is this lane's only access control. Fires once, quietly, a week after deploy. v1.8 removes /data-venus artifacts with plain rm and logs it as the CORRECT removal, not a degraded fallback. NEXT: cadence to 2/day (00:05, 12:05), sequenced last so it cannot move retention.
-
was: ELAZAR DIRECTIVE 2026-08-07 out of VENUS-350. Backups are sprawled across at least 4 uncoordinated lanes with no common registry, no reliable log, and no inventory. Consolidate to ONE place where backups are performed, monitored and inventoried. MEASURED SPRAWL (verified today, not recalled): (a) whey /opt/evolutiva-backups/run.sh v2.2.3, systemd evolutiva-backup@{venus,mars,pluto}.timer. pg_dump --schema=public --schema=auth, DIRECT connection. 3/day since 2026-06-11; every 30min before that. Excludes table DATA for notifications + auth.flow_state/sessions/refresh_tokens. NOT a git repo, no changelog, no version history - the only provenance is a single mtime plus a self-asserted comment. (b) venus venus-db-backup.sh via sh.git, venus-db-backup.timer every 2h, full dump, env DSN. (c) A DEAD gdrive lane, ~406 venus runs, artifacts already destroyed by its own retention. Stopped 2026-06-16. (d) mars/db/backup-db.sh - manual only, --schema=public --schema=auth, NO EXPECTED_REF GUARD (terra has one). If .env.local is absent it falls through to an already-exported DATABASE_URL_DIRECT, i.e. it can dump the WRONG PROJECT. venus has no db/backup-db.sh at all. DEFECTS THAT MADE VENUS-350 EXPENSIVE - each is a requirement for the replacement: 1. backup-log.sqlite is UNRELIABLE IN BOTH DIRECTIONS, proven on a directly-verifiable 4-day window: orphan start rows with no artifact, AND a missed completion (id 6210) that DID produce a valid artifact on disk. A backup log that can silently miss a completed run is worse than none, because it is trusted. 2. USB destination uses DAY-ONLY filenames, so same-day reruns OVERWRITE each other. The USB count is a floor on days-with-a-surviving-dump, NOT a run count - it cannot answer how many times a backup ran, which is exactly what we needed. 3. No inventory anywhere maps lane -> schedule -> destination -> retention -> owner. Reconstructing it took five agents a full day. 4. Retention destroys the evidence needed to audit the lane itself (the dead gdrive lane is permanently unmeasurable for this reason). 5. Provenance is self-asserted: run.sh comments cite an APPROVAL date, and no git history exists anywhere to check what actually changed when. REQUIREMENTS FOR THE CONSOLIDATED LANE: - One inventory: every lane with schedule, connection path, scope flags, destination, retention, owner. Registered with pm-llmmsgsrv-cc (maintainer registry), not only in a repo doc. - Artifact filenames carry FULL TIMESTAMP, never day-only. - Run records reconcilable AGAINST THE ARTIFACTS, artifact wins on conflict - declared config lost to the artifact 4/4 times on VENUS-350. - Version-controlled scripts. run.sh living outside git is why the exclude-flag deploy date was unanswerable from the filesystem. - Every lane identifies itself per-principal once the credentials directive lands, so a dump is attributable at the DB. CONTEXT: Supabase platform gives ZERO restorable backups on venus (API: pitr_enabled false, backups []). Our own dumps are the ONLY restore path. This is not housekeeping. DEPENDENCIES: pairs with the fleet credentials/logging directive routed to pmmaster-evolutiva-cc (per-script + per-agent DB users, disable 0.0.0.0/0, enable log_connections). Logging should land FIRST - additive, breaks nothing, makes everything else verifiable. Do NOT start with CIDR tightening: it can cut off the backup lane, all local tooling and possibly Vercel runtime in one edit, and it fails closed silently. Venus is LIVE with real clinical data. No rotation of existing credentials as a side effect (Section No-Rotation) - issuing NEW per-principal credentials is not rotation.
-
pmmaster ROUTING + RULING RECORDED (pmmaster-evolutiva-cc-muukpu4usrsu-coder-venus-cc), on the VENUS-419 comparison. OPS-207 closed 09-12, /opt/maintenance v2.0 canonical, leg 3 lands there. Mountpoint: own it via a .mount unit or fstab keyed to the LUKS partition UUID, never the udisks name. Sequencing: leg 3 must be VERIFIED RUNNING, not code-complete, before the venus-arm dump or DATABASE_URL_VENUS is removed. Restore drill: do NOT hand whey the GPG passphrase. TWO CORRECTIONS FROM nw-whey-cc (nw-whey-cc-muukq1fo542l), both accepted - I had the whey side wrong in one place and over-tight in another. THE USB PATH DOES NOT EXIST AS WRITTEN ANYWHERE ON THIS ROW. Actual mountpoint is /media/rob/backupYes, no "1". Every artifact on this row, the spec event, Elazar's 09-12 ruling and my own DM say backupYes1, which is a udisks collision suffix. pmmaster's UUID ruling makes the name moot going forward, and that is the right fix - but note WHY the wrong name survived three weeks unchallenged: it was copied between documents, never resolved against a mount table. The same class as the exclusion list this row's description just lost. AND LEG 3 IS NOT BLOCKED, IT IS UNARMED. venus-evolutiva-backup-pull.timer/.service plus ~/.local/bin/venus-evolutiva-backup-pull.sh EXIST and are disabled, inactive, Trigger n/a, last touched 09-12, and have NEVER RUN ARMED. So the question was never the mountpoint or the /opt fork: nobody enabled it. A written-but-never-fired unit is the green line this row keeps rediscovering - it reads as a built leg in every inventory that lists files rather than runs. MY FINDING 1 STANDS AND IS NOW SHARPER, because nw-whey-cc also reports whey's OWN producer (evolutiva-backup@venus.timer, unencrypted .dump straight to USB) is live and current - last run 17:00:43 today. So the USB is fed by the unencrypted producer, NOT by the poll-and-move, and the two are independent. That makes the ordering constraint concrete rather than theoretical: disable the dump before the pull timer is armed and proven, and the USB stops receiving venus entirely while every venus-side check stays green. ON THE RESTORE DRILL, pmmaster asked bin-whey-cc/nw-whey-cc which host should own the repoint. One venus-side fact they need and nobody has put on a row: venus can host a real restore, with no new credential anywhere. scripts/pg-harness.sh (in-repo, Aug 16) does initdb + pg_ctl start + load against a throwaway cluster and prints a DSN, pg_restore 17.11 is on the host, and the GPG passphrase already lives here - so a venus-side drill needs nothing that is not already on this machine. I am NOT claiming the drill should land on venus; the harness is sized for schema work, not a 3MB clinical restore under retention, and that sizing is unmeasured. It does mean "repoint it to where the passphrase lives" is implementable rather than aspirational, and it should be weighed against moving the passphrase.
-
LEG 3 IS ARMED AND VERIFIED RUNNING, reported by nw-whey-cc from the USB itself, not from a unit-state read. 15 .gpg files present on /media/rob/backupYes/backup/evolutiva/venus/, 7 of them moved by the armed poller, and venus-prod-20261005-0008.sql.gz.gpg is 2973425 bytes on BOTH sides. The dev-lane pair differs by two bytes (65025 local vs 65027 on the USB) which is the live dev DB changing between the two dumps, not a transfer defect - the prod-lane byte-for-byte match is the control that distinguishes those. TWO CORRECTIONS THAT TOUCH EVERY PRIOR ARTIFACT OF THIS ROW. (1) The mountpoint is /media/rob/backupYes with NO trailing 1. backupYes1 is wrong everywhere it appears, including Elazar 2026-09-12 ruling and my own DM - udisks names the mount from the LUKS partition, so the number is an artifact of which attempt created the directory and must never be hardcoded; key the path to the partition UUID. (2) Leg 3 was never BLOCKED, it was UNARMED: the units existed, disabled, and had never run armed. Those are different failures and the row said the wrong one - a disabled unit present on disk is exactly the shape that reads as a control and enforces nothing. THE SEQUENCING GATE THEREFORE MOVES OFF LEG 3 AND ONTO VENUS-428 (bin-venus-cc, restore drill), per pmmaster ruling pmmaster-evolutiva-cc-muukse0e5rtt. Leg 3 running is what the venus-arm dump removal was waiting on, but retiring whey arm would delete the fleet ONLY venus restore exercise, so the drill has to exist before anything is removed. Nothing is removed yet: the venus-arm dump still runs and DATABASE_URL_VENUS is still in place.
-
REMAINDER, measured 2026-10-05 in answer to pmmaster (pmmaster-evolutiva-cc-muunlfb5trfh). THE PREMISE OF THE QUESTION IS WRONG IN TWO PLACES AND I AM NOT ACCEPTING IT AS ASKED. (1) VENUS-428 IS NOT A PENDING GATE - IT IS DONE, closed 02:51:50Z, and it is bin-venus-cc + nw-venus-cc, not nw-venus-cc alone (script leg bin-venus-cc, unit leg nw-venus-cc, per the 161/163/164 script-vs-service split). So the sequencing gate this row was waiting on is CLEARED. (2) BUT THE GATE IS SATISFIED BY A DRILL THAT HAS NEVER RUN UNATTENDED, and that is this row own recurring failure. systemctl show -p LastTriggerUSec on venus-db-backup-verify.timer is EMPTY and list-timers shows LAST as a dash: the timer has never fired. The passing run at 02:52Z was a MANUAL START - a hand-started service and a timer-fired service produce the identical Finished/status=0 journal line, so "last run exited 0" is evidence the script works and no evidence at all that the schedule does. First unattended evidence is the 12:35Z fire. The drill itself is real and good work (decrypt -> restore to a throwaway local cluster -> object census BY NAME: 86 tables, 78 views, 271 indexes, 80 FKs, 41 triggers, 20 sequences, ~44032 rows; a NAME set rather than a count because a count on indexes can only false-PASS, PK/UNIQUE build implicitly; NOT checked, per its own verdict line: functions, RLS policies, grants, column defaults/types, comments). I am recording the distinction, not disputing the close. WHAT I DID RATHER THAN HAND THE ROW BACK: VENUS-419 was still open and still mine, so it is now delivered and closed. /data-venus/evolutiva/venus/backup.md is rewritten from live measurement and carries what this row asked for in requirement 1 and never had - ONE table mapping lane -> schedule -> connection -> scope -> destination -> retention -> owner across all six lanes, plus the non-backup DB readers named so the next inventory does not re-derive them. The duplicate question is settled off the ARTIFACT: whey carries 71 relations of data against venus 88, a strict subset, unencrypted and untracked, so venus is the survivor; the format loss (whey custom-format supports selective restore, the survivor is all-or-nothing) is recorded as accepted rather than discovered later. SO THE ACTUAL REMAINDER IS THREE ITEMS, not one: 1. Retire whey venus-arm pg_dump and remove DATABASE_URL_VENUS from /etc/evolutiva-backup/secrets.env. nw-whey-cc. Gate now satisfied. I AM NOT CONFIRMING THE SUDOERS CLAIM: venus cannot read whey filesystem or sudoers, so "whey sudoers cannot strip it" rests entirely on that seat own read and must be attributed to them, not asserted by this row. If it IS a sudoers limit, the row to park on Elazar is a sudoers change, which is a different ask from the credential removal itself. 2. OPS-252 - both gdrive legs authenticate with rclone retiring shared client_id and expire with it during 2026. nw-venus-cc, still backlog. That is an unannounced termination date on one of the two surviving destinations and it is named in this row own DEPENDENCIES paragraph; it is not blocked on Elazar and it is not done. 3. The ordering constraint, unchanged and still the way this lane fails silently: the USB poll-and-move must be verified RUNNING before whey dump is disabled, because whey own producer is what feeds the USB today and the two are independent. NOT PARKING THIS ROW ON ELAZAR. Item 1 is awaiting another agent, not a human decision, until nw-whey-cc states the limit themselves; item 2 is an open agent-owned row. A blocked-on-Elazar label on either would record a decision nobody has been asked for.
-
was: ELAZAR DIRECTIVE 2026-08-07 out of VENUS-350. Backups are sprawled across at least 4 uncoordinated lanes with no common registry, no reliable log, and no inventory. Consolidate to ONE place where backups are performed, monitored and inventoried. MEASURED SPRAWL (verified today, not recalled): (a) whey /opt/evolutiva-backups/run.sh v2.2.3, systemd evolutiva-backup@{venus,mars,pluto}.timer. pg_dump --schema=public --schema=auth, DIRECT connection. 3/day since 2026-06-11; every 30min before that. EXCLUSION LIST CORRECTED 2026-10-05 (coder-venus-cc, VENUS-419, pmmaster ruling pmmaster-evolutiva-cc-muukpu4usrsu-coder-venus-cc). What stood here - "Excludes table DATA for notifications + auth.flow_state/sessions/refresh_tokens" - was WRONG IN BOTH DIRECTIONS, and is replaced by a measurement off the ARTIFACT: pg_restore -l on 2026-10-04-2000-venus.dump, diffed against the live pg_tables catalog. There is NO public exclusion at all - public is 47 of 47 relations with data, and the only notification-shaped table in the schema is "agentNotifyDirectory", whose data is present. What is actually missing, 17 relations against venus's own 88 COPY census: the whole of realtime (2), storage (8), supabase_migrations (1) and vault (1), plus five named relations - auth.flow_state, auth.mfa_amr_claims, auth.refresh_tokens, auth.sessions, cron.job_run_details. auth.mfa_amr_claims and cron.job_run_details were excluded and never named here. So whey's artifact is a STRICT SUBSET and cannot be the survivor; it is also UNENCRYPTED - pg_restore read its TOC with no key of any kind. NOT a git repo, no changelog, no version history - the only provenance is a single mtime plus a self-asserted comment. (b) venus venus-db-backup.sh via sh.git, venus-db-backup.timer every 2h, full dump, env DSN. (c) A DEAD gdrive lane, ~406 venus runs, artifacts already destroyed by its own retention. Stopped 2026-06-16. (d) mars/db/backup-db.sh - manual only, --schema=public --schema=auth, NO EXPECTED_REF GUARD (terra has one). If .env.local is absent it falls through to an already-exported DATABASE_URL_DIRECT, i.e. it can dump the WRONG PROJECT. venus has no db/backup-db.sh at all. DEFECTS THAT MADE VENUS-350 EXPENSIVE - each is a requirement for the replacement: 1. backup-log.sqlite is UNRELIABLE IN BOTH DIRECTIONS, proven on a directly-verifiable 4-day window: orphan start rows with no artifact, AND a missed completion (id 6210) that DID produce a valid artifact on disk. A backup log that can silently miss a completed run is worse than none, because it is trusted. 2. USB destination uses DAY-ONLY filenames, so same-day reruns OVERWRITE each other. The USB count is a floor on days-with-a-surviving-dump, NOT a run count - it cannot answer how many times a backup ran, which is exactly what we needed. 3. No inventory anywhere maps lane -> schedule -> destination -> retention -> owner. Reconstructing it took five agents a full day. 4. Retention destroys the evidence needed to audit the lane itself (the dead gdrive lane is permanently unmeasurable for this reason). 5. Provenance is self-asserted: run.sh comments cite an APPROVAL date, and no git history exists anywhere to check what actually changed when. REQUIREMENTS FOR THE CONSOLIDATED LANE: - One inventory: every lane with schedule, connection path, scope flags, destination, retention, owner. Registered with pm-llmmsgsrv-cc (maintainer registry), not only in a repo doc. - Artifact filenames carry FULL TIMESTAMP, never day-only. - Run records reconcilable AGAINST THE ARTIFACTS, artifact wins on conflict - declared config lost to the artifact 4/4 times on VENUS-350. - Version-controlled scripts. run.sh living outside git is why the exclude-flag deploy date was unanswerable from the filesystem. - Every lane identifies itself per-principal once the credentials directive lands, so a dump is attributable at the DB. CONTEXT: Supabase platform gives ZERO restorable backups on venus (API: pitr_enabled false, backups []). Our own dumps are the ONLY restore path. This is not housekeeping. DEPENDENCIES: pairs with the fleet credentials/logging directive routed to pmmaster-evolutiva-cc (per-script + per-agent DB users, disable 0.0.0.0/0, enable log_connections). Logging should land FIRST - additive, breaks nothing, makes everything else verifiable. Do NOT start with CIDR tightening: it can cut off the backup lane, all local tooling and possibly Vercel runtime in one edit, and it fails closed silently. Venus is LIVE with real clinical data. No rotation of existing credentials as a side effect (Section No-Rotation) - issuing NEW per-principal credentials is not rotation.
-
coder-venus-cc
-
Sole remainder: strip DATABASE_URL_VENUS from /etc/evolutiva-backup/secrets.env on whey. Genuinely Elazar-only - nw-whey-cc confirmed rob's sudo on whey needs an interactive password for arbitrary file edits (NOPASSWD allowlist is narrow: systemctl/apt/kill/nohup/vpn-specific + one single-file tee, no secrets.env entry). Needs either (a) a scoped NOPASSWD sudoers line for this one file, or (b) Elazar stripping the one line himself interactively. Everything else done+verified: leg2 (gdrive) done, leg3 (USB poll-and-move) verified running, leg4 timer disabled+confirmed inactive, VENUS-428 restore-drill gate closed+verified via forced-failure test.