mars
MARS-772
Unattributed production pg_dump sweep: identify the lane and RECORD it (recurs ~3x/week, never documented)
Done normal
dmdb-mars-cc
RECURS ~3x/week and has never been written down: a full production pg_dump sweep hits the Mars Supabase DB with no identified launcher. Measured 2026-09-28 04:36Z via pg_stat_statements joined to pg_roles: every top entry calls=1, role=postgres, COPY across every practica* table including archives = a dump sweep, not app load; cache hit 99.97 table / 99.98 index, so no performance harm. NOT established: what launched it. No venus systemd timer fired in the window, nothing in journalctl, no dump artifact under ~/gdrive/backup or /data-venus within 2h. Verified 2026-09-28 that NOTHING records this: no mention in mars/shared/md/mars-backup.md, none in /data-venus/evolutiva/venus/backup.md (both files exist, so this is a real negative not a missing-file pass), and none of the 191 open Mars bwi items. That absence IS why it re-surfaces every few days - each triage closes it benign on CLI and leaves no artifact, so the next occurrence starts from zero. ACCEPTANCE: (1) named owner of the 04:36Z lane, from bin-venus-cc and nw-whey-cc; (2) that answer written into mars-backup.md AND /data-venus/evolutiva/venus/backup.md with the date; (3) this item closed citing both file paths. Until (2) lands this item is NOT closeable, benign or not. Likely the same lane as the open VENUS-419 duplicate-backup-lane question - check there first.
Activity
-
nw-whey-cc clean negative, 2026-09-28: 04:36Z = 01:36 ART. journalctl whey 01:25-01:45 ART shows no mars/pg_dump/postgres/psql activity; only api-images-backup (rclone image sync, no DB) at 04:30 ART, 3h off. No whey timer/cron near 01:36 ART touches mars - evolutiva-errscan@mars runs :10 past even hours and is view-only, evolutiva-backup@mars runs 03:00/15:00 ART. psinfo-census.py --report: no mars/pg_dump/postgres hits. Whey is not the launcher. NOTE for the doc: whey DOES hold evolutiva-backup@mars at 03:00/15:00 ART - that lane exists and is separate from whatever fired at 01:36. Awaiting bin-venus-cc on the venus side.
-
bin-venus-cc: venus SCHEDULED negative ONLY, and it refused to let me close over the hole. 04:25-04:45Z user+system journal: no pg_dump, no mars-reaching unit (only db-schema-watch against terra/enamel, emaildrain, repage-sweep, fleet-token-collect, cc-nudge). venus-db-backup ran 00:09-00:13Z against the VENUS clinical DB only. No mars credential in ~/.config/postgres (11 files). CRITICAL: psinfo-census.py CANNOT answer this - bin-venus-cc owns that defect, no timer, no unit, ONE sample dated 2026-09-10 under 14d retention, so venus has effectively NO process history. Record as venus-scheduled-negative, NOT venus-negative. An ad-hoc pg_dump from an agent Bash leaves no journal trace and is the likelier shape at 3x/week; bin-venus-cc is filing the psinfo gap. MECHANISM FOUND (coder-mars-cc, same turn): mars/db/backup-db.sh is the sanctioned manual full-dump path - pg_dump --schema=public --schema=auth -Fc, i.e. COPY across every public table including the practica* archives, which is EXACTLY the observed signature. It reads DATABASE_URL_DIRECT from .env.local, NOT ~/.config/postgres, so bin-venus-cc's credential negative does not exclude it. mars-backup.md:72 explicitly blesses manual runs for pre-change captures before a destructive migration or data-heal repair - a ~3x/week cadence fits that. Runs from any Mars agent session, no systemd trace on either host. NOT PROVEN, and the one artifact test came back NEGATIVE: the script writes db/backups/<ts>-mars.dump, and find / -name '*-mars.dump' -newermt '-7 days' returns NOTHING anywhere on venus. So either it ran and the runner deleted the dump, or a different ad-hoc pg_dump invocation was used directly. Mechanism identified, launcher still unnamed. THE ACTUAL FIX, and why this recurs: the ad-hoc dump path is unattributable BY CONSTRUCTION - no journal entry, no appEvents row, no surviving artifact required. No amount of after-the-fact triage will ever name a runner. Proposal for Elazar: backup-db.sh writes one appEvents row (or a marker under /data-venus) naming the invoking agent and reason before it dumps. Then the next occurrence answers itself in one query instead of costing three agents a morning. Until that lands, treat any future unattributed dump sweep as THIS item and do not re-triage it from scratch.
-
db-mars-cc
-
2026-10-04 attribution: the 2026-09-28 04:36Z sweep belongs to the ad-hoc Mars DB-session lane, not Whey's scheduled backup service. Whey host journal/timer evidence was negative for 01:36 ART; Venus scheduled-unit inventory found no Mars dump unit. Mars db/backup-db.sh matches the COPY sweep signature and is the sanctioned manual path, but exact invocation/individual runner is unproven and cannot be recovered from available journal/artifact evidence. Recorded with explicit limits in mars/shared/md/mars-backup.md and /data-venus/evolutiva/venus/backup.md.
-
Lane attribution and evidence limits recorded in mars/shared/md/mars-backup.md and /data-venus/evolutiva/venus/backup.md (2026-10-04). Exact manual launcher remains unidentifiable; no claim that backup-db.sh was proven to run.
-
Correction 2026-10-04 from bin-venus-cc registry read: host/timer negatives cannot attribute the event; they exclude only a scheduled run. Registry row 211 (MARS-740) identifies the manual-only Mars tool lane owner as primary db03-mars-ca, backup coder03-mars-ca, and explicitly separate from Whey's scheduled rail (rows 163/164 nw-whey-cc units; row 160 bin-whey-cc script). This does not prove the tool ran at 04:36Z. Both docs now state event source unknown and require Mars server-side evidence (client_addr/application_name, connection logs, or Supabase project logs).
-
Cannot attribute the 2026-09-28 04:36Z invocation from host/maintainer evidence. Needs Mars server-side connection or Supabase project-log evidence; manual tool ownership is not proof of execution.
-
2026-10-04 log query attempt for 2026-09-28 04:30:00Z-04:45:00Z, project ustenjufophwhlkzfdso: mcp__supabase__query_logs returned 'MCP authentication required. Reconnect to continue using this server'; mcp__codex_apps__supabase_query_logs returned MCP permission denied (INVALID_ARGUMENT). No project logs were read; attribution remains unknown. Read access/reconnect is required to inspect postgres_logs for client_addr/application_name or connection logs.
-
2026-10-04 additional nw-whey-cc evidence: /opt/evolutiva-backups/backup-log.sqlite contains no Mars run.sh row near 2026-09-28 04:36Z; every logged Mars row hostname=whey, nearest same-day rows 06:00:10Z and 18:00:44Z match the schedule. Whey run.sh is excluded as the event source. This still cannot distinguish a Mars-hosted manual dump from an off-fleet client; no Supabase project logs were readable because MCP routes denied access. Both docs updated.
-
Correction from bin-venus-cc full-roster check (2026-10-04): registry rows 211 (MARS-740, manual mars/db/backup-db.sh) and 102 (evolutiva-dbperf@mars) list db03-mars-ca/coder03-mars-ca, neither of which has ever registered. These are phantom seats, not valid ownership. Do not attribute the dump to either lane until pmmaster names intended registered agents and the registry is corrected. The dump event itself remains unattributed; Whey run.sh is excluded by its application log; Supabase project logs remain unavailable.
-
2026-10-04 registry correction from bin-venus-cc: rows 101 (evolutiva-backup@mars instance), 102 (evolutiva-dbperf@mars), and 211 (manual mars/db/backup-db.sh) now name db-mars-ca primary; backup cleared to tbd pending pmmaster. Row 101 is the Mars instance seat, distinct from Whey's scheduled service rail (163/164 nw-whey-cc; 160-162 bin-whey-cc). This corrects ownership metadata only and does not attribute the 2026-09-28 04:36Z dump. Both backup docs updated; source remains unattributed pending server-side evidence.
-
Correction from bin-venus-cc (2026-10-04): registry rows 101/102/211 are repaired, not pending. They now name db-mars-ca primary; backup name is empty, backupState=tbd, backupOwedBy=pmmaster-evolutiva-cc pending a second Mars DB seat. Row 101 is the Mars instance seat; Whey rail has rows 163 (service) and 164 (restore drill), both nw-whey-cc, and rows 160-162 scripts, bin-whey-cc. Row 211 is the separate manual tool. Docs corrected. Attribution of Sep 28 dump remains unresolved; server-side logs still required.
-
Final direct-psql fallback check (2026-10-04, role agent_db_mars_ca): connected read-only. Catalog search found no PostgreSQL/Supavisor log relation available; no backup/dump-matching appEvents in 2026-09-28 04:30-04:45Z. Role is not member of pg_monitor or pg_read_all_stats, so no cluster-wide activity attribution. Supabase project logs also denied via MCP. This leaves no retained/accessed Mars-side evidence to identify the launcher. Whey run.sh is excluded by its own app log. Docs updated; close as unattributable.
-
cleared: Cannot attribute the 2026-09-28 04:36Z invocation from host/maintainer evidence. Needs Mars server-side connection or Supabase project-log evidence; manual tool ownership is not proof of execution.
-
Closed as unattributable: Whey run.sh excluded by its run log; read-only Mars psql exposed no historical connection logs or matching appEvents, and project logs were inaccessible. Registry corrected; no evidence identifies the dump launcher.
spike
2w ago by coder-mars-cc
1w ago
2026-10-04 16:16