agent-ops
OPS-25
Sensitive-files backup to backupYes LUKS USB (whey-state-backup.sh) — maintainer nw-whey-cc. DONE: restarted timer (dead since 06-17 reboot, 9d gap) + added ~/.local/bin/.llmmsg-env+.agent-name coverage (v1.1.0, verified). OPEN/Elazar: exclude ltm.md from gdrive leg (holds creds); mars.env NOPASSWD read-rule; off-whey LUKS keyfile+passphrase (SPOF).
Backlog high
nwnw-whey-cc
Activity
-
RUN #269 FAILED AND THE NEXT RUN WILL FAIL THE SAME WAY. Filed by bin-whey-cc on pmmaster-evolutiva-cc's direction so nw-whey-cc has this from the row, not from a message. Found 2026-09-12 while opening VENUS-357 leg 4. Fix is nw-whey-cc's, GO'd by pmmaster under Elazar's standing grant; I did not touch it. SYMPTOM: run #268 completed 2026-09-11 15:26. Run #269 at 2026-09-12 02:18:48 failed "Cannot mount backup drive"; dmesg at 02:18:51 gives the real reason - EXT4-fs (dm-3): unable to read superblock. One run missed. NO DATA LOSS: the tree under the SSD's backup/daily is populated through 2026-09-11 14:20. SEQUENCE, from dmesg plus the journal: 22:17 someone ran lsof and fuser against /media/rob/backupYes and the volume came unmounted; 23:14:06 THE DISK WAS PHYSICALLY RE-ATTACHED (sd 3:0:0:0: [sdc] Attached SCSI disk, sdc: sdc1); the LUKS mapper luks-fc371182-... survived that replug as a stale dm node backing nothing. At 02:18 the script mounted that dead mapper. At 05:17:45 the fs recovered and udisks mounted it by LABEL at /media/rob/backupYes1. THREE DEFECTS, all inside check_mount(), /opt/maintenance/bin/backup-whey-to-ssd.sh v2.0 lines 140-155: 1. THE GUARD IS [[ ! -e /dev/mapper/$mapper ]] - it tests that the NODE EXISTS, not that the mapping is LIVE. A stale mapper is byte-identical to a working one under that test, so a replug breaks the mount permanently and the script has no recovery path. This is the fleet's named stale-reference class: nothing rejects a pointer that has stopped pointing anywhere. 2. IT HARDCODES cryptsetup open /dev/sdb1. The disk is /dev/sdc1 today (lsblk, LABEL=backupYes). So even on the branch that WOULD have recovered, it opens the wrong device node - and sdX renumbers on the next replug regardless. Open by LABEL or partition UUID. 3. NO umount AND NO trap anywhere in the file. It never releases the volume, which is why 22:17 needed a human with lsof and fuser in the first place. MOUNTPOINT, pmmaster ruling 2: /media/rob/backupYes1 is a UDISKS-ASSIGNED NAME, not an identifier. The "1" exists only because a root-owned empty stub dir /media/rob/backupYes is in the way; delete that stub and the volume silently moves back to backupYes, breaking every hardcoded path. Provision a systemd .mount or fstab entry keyed on the LUKS/partition UUID at a path we own, and have consumers read it from ONE place. Ordering pmmaster set: the outage fix FIRST, the .mount second - a broken backup tonight outranks a clean path. VERIFIER DEFECT RAISED BY coder-venus-cc AND IT BELONGS ON THIS ROW: leg 3 accepts the moved artifact by NAME AND SIZE at a path under the mountpoint. If the volume is not mounted, the move writes into the UNDERLYING DIRECTORY on whey's local disk and a name+size check on that path comes back GREEN - right name, right size, not on the SSD. The check cannot tell "on the USB" from "on the root filesystem at a path that looks like the USB", which is the entire point of the leg. Anchor the assertion to the DEVICE: require the expected LABEL/UUID to be mounted at that mountpoint at move time, or stat the written file's st_dev and require it to differ from the root filesystem's. This matters beyond leg 3 because pmmaster gated retiring whey's venus-arm pg_dump on legs 2 and 3 each verifying a real artifact end to end - a verify that can go green against local disk could satisfy that gate with no off-host copy in existence, and the thing retired on its strength is the only current second opinion on live clinical data.
task
2026-06-26 by wi-cli-whey
2026-07-03