DB backup: no restore ever tested, no failure alerting
Shared /opt/evolutiva-backups infra (mars/pluto/venus), whey host - not pluto-owned to fix. Ownership + facts as of 2026-08-22: MAINTAINER: bin-whey-cc now owns this rail end-to-end (run.sh, restore-drill.sh, mark-failed.sh, backup-log.sqlite, the evolutiva-backup@{project}.timer/service units) per Elazar GO 2026-08-22. Route change requests to bin-whey-cc, do not edit units/scripts directly. CADENCE CHANGED: was 8h/3x-day per project (9/day fleet-wide, all three firing simultaneously). Now daily, staggered - pluto 04:00 ART. RPO is now 24h, accepted by Elazar pending revisit. FLAGGED to bin-whey-cc 2026-08-22: pluto carries live clinical práctica data since go-live 2026-06-09 (see pluto-real-data-cutover memory) - a 24h-loss window is a real clinical-record exposure, not hypothetical pre-launch risk. Awaiting his/Elazar's read on whether pluto needs a tighter cadence than mars/venus. RESTORE DRILL NOW LIVE (was the confirmed gap - nothing had ever been restored, only pg_dump exit-0 checked): weekly, pluto's slot Tue 07:30 ART, restores into a scratch DB and verifies CONTENT via per-table row counts (query_to_xml), not reltuples estimates. Mars/venus first runs already landed (66 tables/107907 rows, 70 tables/37391 rows respectively); pluto's first result due Tuesday - do not close this WI until that number is in. BACKUP-LOG INTEGRITY BUG found and fixed (run.sh 2.4.0): sqlite3's zero busy-timeout plus 3 simultaneous timers meant the log was silently losing rows since 2026-06-10 - fleet-wide 339/600 inserts lost. 8 pluto rows sit stuck at status='start' with no terminal row, spanning 2026-05-02 to 2026-08-05; left un-rewritten deliberately (Elazar's call, not backfilled to look clean). Any historical query counting status='fail' to conclude 'no failures' was reading a record with holes in it. ORIGINAL FINDING (still accurate): in-band tier failures DO alert via run.sh's notify_failure() (notify-send + email) - monitoring was never fully absent. Tier-1 (/gdrive/backup/evolutiva/pluto/) is genuine off-host storage (rclone Google Drive mount, 2-day retention, run.sh guards against writing if unmounted). Tier-2 (/media/rob/backupYes, USB LUKS, 90-day retention) is whey-local. The real pre-fix gap was narrower: a crash or timeout-kill that never reached notify_failure() was silent - now covered by the OnFailure= reconciliation bin-whey-cc added.
Activity
-
was: Verified via nw-whey-cc 2026-08-22 while answering coder-terra-cc/TERRA-7: evolutiva-backup@pluto timer runs reliably (zero fail rows in backup-log.sqlite since 2026-08-19), but (1) no restore has ever been executed from any dump - only pg_dump flag-semantics comments in docs, no restore script/log exists; (2) zero monitoring - no OnFailure=, stdout/stderr null, nothing polls backup-log.sqlite, a silent stop would only surface when a restore is needed and fails; (3) both backup tiers (gdrive rclone, USB LUKS) are on whey - if whey is lost both go with it, only gdrive's own Drive-side replication is truly off-host. Same failure shape as applog-listen@enamel (16.79-day silent stop, nothing reported it). This is shared evolutiva-backup@ infra (likely affects other projects too, not pluto-specific) - fix belongs to whoever owns /opt/evolutiva-backups, not pluto's own lane.
-
db-pluto-cc
-
was: Verified via nw-whey-cc, corrected by bin-whey-cc 2026-08-22 while answering coder-terra-cc/TERRA-7: evolutiva-backup@pluto timer runs reliably (217 ok rows/30d, zero fail). CONFIRMED gap: no restore ever drilled - this is todo #2 in the orchestrator's own todos table (filed 2026-05-01, done=0, never actioned; schema.sql:30 seeds 'Phase 2: monthly restore-drill into throwaway DB, assert row counts'). 653 ok dumps proves pg_dump exits 0, nothing more. CONFIRMED gap, narrower than first filed: unit-level has no OnFailure= and stdout/stderr null (a crash is silent), BUT run.sh:99 notify_failure() does alert (notify-send+email) on in-band tier failures (e.g. USB copy fail at run.sh:201) - the real unmonitored class is a crash or RuntimeMaxSec=1800 kill that never reaches the handler. Also unfiled: todo #3, stale-backup alarm (>3h since last ok/project) - this is what actually catches a stopped timer, since notify_failure can't fire if the run never starts. 2 venus 'start' rows with no terminal row in the 30d log are exactly this silent-death class. CORRECTION - destination tiers are NOT both same-host: Tier 1 (/gdrive/backup/evolutiva) is an rclone mount of Google Drive = genuine off-host storage; Tier 2 (/media/rob/backupYes) is whey-local. run.sh:129 already refuses to write to a bare mountpoint if /gdrive isn't mounted - the off-host-copy risk is already guarded. Do NOT scope work as 'no off-host copy exists' - that's already handled. /opt/evolutiva-backups is bin-whey-cc's lane; timers are Elazar's, no action without his GO. Proposed order if/when he rules: todo #3 (stale alarm) before todo #2 (restore drill) - a drill on a silently-stopped stream tests a stale dump.
-
was: --append
-
Resolved: evolutiva-restore-drill@pluto (bin-whey-cc, v1.4.1) runs weekly Tue, content-asserted, history since 2026-08-22, latest 2026-09-29 all ok. Both failure-alert paths wired (notify_failure() content-check, OnFailure= dead-drill). Original gap (no restore ever tested, no failure alerting) closed.