backup-whey-to-ssd.sh: tracked sh.git v1.1.1 is a dead file; backup-whey.service runs /opt/maintenance v2.0 with no pg_dump

Sub-tickets

Activity

• wi-cli-venus created · 5w ago
• wi-cli-venus note · 4w ago
  Elazar CLI ruling relayed by pmmaster-evolutiva-cc 2026-09-12, verbatim: 'backup takes place in /gdrive/ (venus can do it) and in whey /media/rob/backupYes1/backup thus whey must do it.' Recorded on VENUS-357. That settles the canonical-copy question this item was escalated on and was waiting for: the whey physical leg is whey's to execute because the backupYes1 LUKS volume is attached here and no other host can reach it, so the executing copy is authoritative and the sh.git v1.1.1 file is the one that must stop asserting a capability. pmmaster directs that this be folded into the whey backup leg (nw-whey-cc owns that leg, OPS-25) rather than solved twice as a separate file fix. NOT STARTED: the fleet stop still holds, and Elazar's lift covers jobreg only - this ruling settles the design, it is not a start signal. One requirement carried forward and it is the part I would otherwise get wrong: the fix must assert the DESTINATION WAS WRITTEN THIS RUN, read back from the destination, not inferred from the script's exit code or from a summary line printed unconditionally over a loop that failed. A backup is the worst place to inherit that shape.
• wi-cli-venus note · 4w ago
  Read VENUS-357's spec event (2026-09-12) rather than the summary. OPS-207 is a sub-leg of it and does not stand alone any more. What binds my part: whey's venus-arm pg_dump is RETIRED under that design and whey stops holding a Supabase credential for the ref entirely, so the surviving whey code path is a POLL-AND-MOVE from /data-venus/backups/venus/ to /media/rob/backupYes1/backup - not a dumper. That is what makes the sh.git v1.1.1 dead file actively misleading rather than merely stale: it advertises a pg_dump of one-db telemetry plus a verify-restore, on a leg whose whole point is now that whey does not read a database at all. nw-whey-cc owns the timer, the move and the retirement; my fold-in is the tracked-file half. TWO CORRECTIONS FROM THAT ROW WORTH NOT RE-DISCOVERING: there is no /data-whey - /data-venus is local disk ON VENUS mounted to whey over sshfs, so nothing crosses hosts; and INOTIFY DOES NOT PROPAGATE ACROSS FUSE/sshfs, so a whey watcher or a systemd .path unit on /data-venus receives zero events and is indistinguishable from a quiet backup lane. Use a short polling timer. ACCEPTANCE, and it is the same requirement already on this item: USB leg confirms the file on the SSD by NAME AND SIZE after the move, never from an exit code. One contract to carry: the single-read design is justified ONLY by venus-db-backup.sh v1.6's decrypt-and-round-trip verify per run; if that verify is ever weakened the design loses its basis. Dependency OPS-252 (rclone shared client_id expiry, Elazar's to create). Still NOT STARTED - held under the halt, ships on the resume.
• wi-cli-venus note · 4w ago
  WHEY BACKUP IS FAILING NOW, and it will fail again at 14:13 today. Found while opening my leg-4 fold-in; reporting, not fixing - backup-whey.timer/.service is nw-whey-cc's row (OPS-25) and the fix direction is leg 3's design call. FACTS. Run #268 completed 2026-09-11 15:26. Run #269 at 2026-09-12 02:18:48 FAILED: "Cannot mount backup drive". dmesg pins it: EXT4-fs (dm-3): unable to read superblock at 02:18:51. One run missed; NO DATA LOSS - the tree under /media/rob/backupYes1/backup/daily is populated through 2026-09-11 14:20. CAUSE, from dmesg + journal, in order: 22:17 a human ran lsof/fuser on /media/rob/backupYes (PWD=/home/rob/Documents/terminal/android) and the volume came unmounted; 23:14:06 THE DISK WAS PHYSICALLY RE-ATTACHED (sd 3:0:0:0: [sdc] Attached SCSI disk, sdc: sdc1); the LUKS mapper luks-fc371182-... survived the replug as a STALE dm node backing nothing. At 02:18 check_mount() tested [[ ! -e /dev/mapper/$mapper ]], found the node PRESENT, skipped cryptsetup open, and mounted a dead mapper - hence unable to read superblock. At 05:17:45 the fs recovered and udisks mounted it, by LABEL, at /media/rob/backupYes1 (the "1" is a collision suffix: root-owned empty stub dir /media/rob/backupYes already exists). It is mounted there right now, rw,nosuid,nodev, 439G/83G avail. THREE DEFECTS IN /opt/maintenance/bin/backup-whey-to-ssd.sh v2.0, all in check_mount() at :140-155: 1. The guard tests the mapper NODE EXISTS, not that it is usable. A stale mapper is indistinguishable from a live one to [[ -e ]], so a replug permanently breaks the mount with no recovery path. Same class as feedback-masking-a-unit-does-not-disable-what-calls-it: nothing rejects a stale reference. 2. It hardcodes cryptsetup open /dev/sdb1. The disk is /dev/sdc1 (lsblk, LABEL=backupYes). So even on the branch that WOULD have recovered, it opens the wrong device node. Use the LABEL or the partition UUID, never a kernel name that renumbers on replug. 3. No umount and no trap anywhere in the file, so it never releases the volume - which is what made 22:17 a manual intervention in the first place. MOUNTPOINT DISAGREEMENT, needs one ruling: both backup-whey-to-ssd.sh copies hardcode /media/rob/backupYes; memory reference-whey-backup-ssd-scope says /media/rob/backupYes/backup/; Elazar's ruling and this spec say /media/rob/backupYes1/backup. backupYes1 is what is actually mounted and what Elazar sees on his desktop, but it is a UDISKS-ASSIGNED NAME, not a stable one - it is only "1" because the stub dir exists, and deleting that stub would move the volume back to backupYes and silently break whichever path is hardcoded. The leg needs a mountpoint it owns (fstab/systemd .mount by LABEL or UUID), not a desktop automount. That decision belongs to leg 3 before any poll-and-move script names a path. MY LEG 4 BLOCKS ON ONE QUESTION FOR nw-whey-cc: does leg 3's poll-and-move land in sh.git or in /opt/maintenance? OPS-207 is the /opt-vs-sh.git fork, and the fold-in is retiring whichever copy is not canonical - I cannot pick the tree, and editing either file before that is decided is what OPS-207 says not to do. Asked; nothing started.
• wi-cli-venus note · 4w ago
  RESOLVED BY RULING, 2026-09-12, pmmaster-evolutiva-cc - and the answer is the opposite of the one I was given an hour earlier, so record the REASON rather than the verdict. /opt/maintenance/bin/backup-whey-to-ssd.sh v2.0 IS CANONICAL. The sh.git copy ~/.local/bin/backup-whey-to-ssd.sh v1.1.1 is retired BY DELETION, not resurrected and not synced. THE GROUND IS A PRIVILEGE BOUNDARY, NOT TIDINESS, which is why it beats the "sh.git is where scripts live" default: this script runs as ROOT (cryptsetup open, mount). A root-executed script living in a user-writable ~/.local/bin checkout means anyone holding rob's account has a root execution path. That is what /opt/maintenance exists for. The rule generalises - SPLIT BY WHAT THE SCRIPT NEEDS, NOT BY WHICH TREE IS TIDIER: leg 3's poll-and-move needs no privilege, it moves files as rob, so that one belongs in sh.git. nw-whey-cc first answered "sh.git" for this file; pmmaster overruled it on the privilege argument and I agree - the earlier answer is superseded, not a second opinion. WHAT I WAS ABOUT TO GET WRONG, recorded because the deletion looks trivial and is not: I had scoped leg 4 as MIGRATING v2.0 into sh.git, on the reasoning that /opt v2.0 is strictly ahead of v1.1.1 - it carries the sqlite run ledger at /var/lib/backup-whey/backup.sqlite, log_event/fail_run, the cleanup+prune pass and the odaia_gdrive leg, none of which exist in v1.1.1 - so a plain delete of /opt would have silently downgraded the running backup to v1.1.1 and resurrected the dead pg_dump. That reasoning was right about the danger and wrong about the direction. Under the ruling the danger disappears: nothing moves, the running code stays where it runs, and only the dead copy goes. DELETION MECHANICS, and they are not optional here: a pushed deletion in sh.git is UNDONE by the next merge from a host that has not pulled, because the merge is newest-wins by mtime and a resurrected file does not error. So this needs ITS OWN COMMIT and a read-back on whey, venus AND lezama before it is called done. I am already holding one deletion of exactly this shape - jobreg.tsv, OPS-220, waiting on Elazar naming the file - and the two must NOT be bundled into one commit: bundling makes a partial resurrection unreadable afterwards. HELD, not done: pmmaster gated leg 4 behind ruling 2 (the LABEL/UUID systemd .mount, nw-whey-cc's, OPS-25) and directed me to hold. The deletion itself has no path dependency, so when it is released it is one commit plus three read-backs, with this reason in the message.
• wi-cli-venus completed · 4w ago
  CLOSED. Deletion verified on ALL THREE HOSTS, which is the gate this item actually needed - sh.git merges newest-wins by mtime, so a pushed deletion is undone by the next merge from a host that has not pulled, and a resurrected file does not error. READ-BACK: whey absent at 4449296 (the commit itself). lezama pulled 4449296, confirmed absent (bin-lezama-cc 06:01). venus at ed08834 which carries 4449296 - absent on disk, 0 in git ls-files, 0 refs in index.md (bin-venus-cc 06:05). Three independent confirmations, each naming a commit, not an exit code. RESOLUTION: /opt/maintenance/bin/backup-whey-to-ssd.sh v2.0 is canonical; sh.git's v1.1.1 is deleted. THE GROUND IS A PRIVILEGE BOUNDARY, NOT TIDINESS (pmmaster-evolutiva-cc, overriding an earlier 'sh.git canonical' answer): the script calls cryptsetup open and mount, so it runs as root, and ~/.local/bin is a user-writable checkout three hosts pull into - a root systemd unit pointed there means anyone holding rob's account has a root execution path, and a git pull changes what root executes. Generalisation: SPLIT BY WHAT THE SCRIPT NEEDS, NOT BY WHICH TREE IS TIDIER. VENUS-357 leg 3's poll-and-move needs no privilege and correctly lives in sh.git. I HAD THE DIRECTION BACKWARDS AND THE MEASUREMENT IS WHAT CORRECTED IT: I scoped this as MIGRATING v2.0 into sh.git, having measured that v2.0 is strictly ahead - sqlite run ledger, log_event/fail_run, cleanup+prune, the odaia_gdrive leg, none in v1.1.1 - so a naive delete of /opt would have silently downgraded the running backup and resurrected a pg_dump that runs nowhere. Right about the danger, wrong about which copy to keep. The same measurement settles it the other way: the living code is in /opt, the dead file was in sh.git. index.md's reference was removed in the SAME commit rather than left to dangle, and backup-whey.service ExecStart was confirmed already reading /opt before anything was deleted - an orphaned pointer outlives the file it names. SPUN OUT, not closed here: the whey backup outage found while opening this item (run #269 failed 02:18 on a stale LUKS mapper the [[ -e ]] guard could not distinguish from a live one) is OPS-25, nw-whey-cc's, holding action taken. The watchdog row that made it silent is armed and is now nw-whey-cc's. The 15 fleet-wide sense=review rows are OPS-255.
• wi-cli-venus completed · 4w ago
  CLOSED ON A THREE-HOST READ-BACK, NOT ON A COMMIT HASH. sh.git merges newest-wins BY MTIME, so a deletion is undone by the next merge from any host that still has the file on disk - which makes 'the commit is in history' the wrong evidence and 'the file is absent from every checkout' the right evidence. whey: HEAD 0c864c8, 4449296 in history, backup-whey-to-ssd.sh absent from disk. venus: absent at ed08834 (pmmaster). lezama: HEAD 68ed23d, merge-base --is-ancestor 4449296 HEAD rc=0, ls says No such file or directory - absent from DISK, not just from the tree at HEAD (bin-lezama-cc, 10:46). I could not verify lezama from whey and did not try to: 68ed23d does not resolve in my checkout (whey is behind) and there is no standing whey-lezama connectivity for agents. Asked for the two lines instead of inferring them, and asked specifically for the on-disk ls as well as the ancestor test, because a commit deletes from the repo and not from a checkout. THE SURVIVING COPY IS /opt/maintenance/bin/backup-whey-to-ssd.sh and the rule generalises: a script that runs as root does not live in a user-writable checkout that three hosts pull into, because a git pull then changes what root executes. Split by the privilege the script NEEDS, not by which tree is tidier - VENUS-357's poll-and-move runs as rob and correctly stays in sh.git, two files from one workstream in two trees. The deletion direction was the part I had backwards and it is worth keeping: /opt v2.0 is strictly AHEAD of the sh.git v1.1.1 (sqlite run ledger, log_event/fail_run, cleanup+prune, gdrive leg), so the fork closed by deleting the dead copy. Deleting the other side would have silently downgraded a running root backup.