basquetWi + New ticket
agent-ops OPS-217

Venus Postgres records the failure side of a credential change and nothing of the change itself: log_statement=none, log_connections=off, track_commit_timestamp=off

Backlog normal nvnw-venus-cc

MEASURED 2026-09-12 by pmmaster-evolutiva-cc on venus, read-only, while trying to attribute the BIN_VENUS_CC_TELEMETRY_DSN rotation behind OPS-216. The answer is not recoverable, and the reason is a configuration gap worth fixing rather than an unlucky gap in the record. CORRECTION TO AN EARLIER REPORT FIRST. It was reported on the wire that venus has no postgresql log directory. It does: /var/log/postgresql/ holds postgresql-17-main.log plus ten weekly rotations back to 2026-07-05, it is actively written (last line 01:17 UTC), and it is group adm so it needs sudo to read, which is the likely source of the wrong conclusion. A permission denial is not an absence. The directory is not the problem. WHAT THE LOG DOES RECORD. Every authentication FAILURE, in full, with a role@database prefix and the matching pg_hba line: 26 FATAL password authentication failed for user bin_venus_cc, first ever at 2026-09-12 00:00:35.421 UTC, last at 01:16:44.127, and zero such lines for that role anywhere in the ten rotations before that. It also records ERROR-level statements, so a failed statement carries its own text. WHAT IT DOES NOT RECORD, AND THIS IS THE DEFECT. log_statement = none, so a SUCCESSFUL ALTER ROLE ... PASSWORD is never written. log_connections = off and log_disconnections = off, so a successful connection is never written either. track_commit_timestamp = off, so pg_xact_commit_timestamp() over the pg_authid row's xmin returns an error rather than a date. All three read live from pg_settings. The net property: the instrument faithfully records the alarm and is silent on the change that caused it. Every DDL that grants, revokes, renames, or re-passwords a role on this host is unattributable by construction, and nothing signals that. WHY xmin DOES NOT RESCUE IT, since it is the obvious next idea and it does not work. pg_authid.bin_venus_cc carries xmin=1419663, which sits inside the transaction range of other catalog writes on this host, so it LOOKS like it dates the change. It does not, for two independent reasons. First, xmin is cluster-global and every writing backend in every database draws from the same counter, so falling between two known writes is a temporal bound and never a causal one - the same shape as reading a usename out of pg_stat_activity without a correlated request. Second, xmin is a high-water mark and not a history: it records only the most recent write to that row, and a password change followed by any later unrelated role write leaves you looking at the second one. A pg_authid write is also not necessarily a password at all - VALID UNTIL, CONNECTION LIMIT, LOGIN, INHERIT and a rename all land in the same row. Do not attribute a rotation from xmin, and do not put an agent name on one. WHAT IS ACTUALLY TRUE ABOUT THE ROTATION. The on-disk DSN in ~/.local/bin/.llmmsg-env authenticates right now - verified by sourcing the overlay, running it through metrics-write.sh _metrics_load_env, and connecting: current_user=bin_venus_cc at 01:23:25 UTC. Successes and failures are INTERLEAVED within the same minutes on the same role (success 01:00:23 and 01:10:13, failures 01:08:46, 01:08:58, 01:16:00, 01:16:44), which is direct observation of two different credential values in flight at once rather than an inference, and it is exactly the picture OPS-216 describes: the file is right and long-lived sessions hold a dead copy. Who changed it, and at what moment, cannot be established from this host. A CONTROL WORTH RECORDING BECAUSE IT COST A WRONG READING. A first pass at that connection test returned psql exit 2 and looked like proof the on-disk credential was dead. The cause was my own harness passing PGSSLMODE as an EMPTY string. metrics-write.sh documents precisely this above _metrics_psql - libpq rejects an empty PGSSLMODE and an empty value is not the same as an absent one - so the script was already right and the test was wrong. Pass ssl variables only when set. PROPOSED FIX, NOT APPLIED - VENUS-350 is measure-and-report and this needs Elazar. log_statement = 'ddl' is the minimum that closes it and is close to free: it logs CREATE/ALTER/DROP only, so ordinary query traffic is unaffected and there is no password in the log text because Postgres redacts ALTER ROLE ... PASSWORD to ALTER ROLE ... PASSWORD <redacted> when log_statement is on. Adding log_connections = on gives attribution of WHO connected and costs one line per connection, which on this host is small. track_commit_timestamp = on is the only one requiring a restart and is the least valuable of the three, so it can be declined. First two are a reload, not a restart. ASIDE, ONE LINE, DO NOT CHASE IT: the logs carry 10 password authentication failures for a role telemetry_rw across 2026-07-03 and 2026-07-12, and telemetry_rw does not exist in pg_authid at all. Either it was dropped after those failures or it never existed. Noted so it is queryable; it is not part of this row's fix.

Sub-tickets

No sub-tickets.
+ Add sub-ticket

Activity

  • wi-cli-venus created · 4w ago
  • wi-cli-venus decision · 4w ago
    CORRECTION to this row's PROPOSED FIX - the redaction claim is FALSE and must not be acted on as written. I wrote that log_statement='ddl' costs no secret because Postgres redacts PASSWORD out of the log text. It does not. Redaction is a pgaudit/RDS behaviour, not a log_statement behaviour. PostgreSQL 17 CREATE ROLE carries the opposite in its own Caution: 'The password will be transmitted to the server in cleartext, and it might also be logged in the client's command history or the server log.' zgrep across all ten rotations on venus returns zero role statements, so there is no local sample that would have caught this either. Consequence if applied as originally written: the next ALTER ROLE ... PASSWORD '<literal>' lands in /var/log/postgresql/ and rides into rotations ten weeks deep - a hardening change that arms a cleartext-credential path is a regression. THE RECOMMENDATION SURVIVES, ONLY THE RATIONALE CHANGES. Still ask for log_statement='ddl' + log_connections=on. Reload not restart is now MEASURED rather than recalled: pg_settings.context is 'superuser' for log_statement and 'superuser-backend' for log_connections/log_disconnections; track_commit_timestamp is 'postmaster', the one needing a restart, still decline it. The true cost statement is two parts: (1) the log is mode 0640 postgres:adm on a single-operator box, so the exposure is bounded by host access, not by redaction; (2) role changes should pass a pre-computed SCRAM-SHA-256$... verifier instead of a literal, so what lands in the log is a verifier and not a secret. Part (2) is the durable half and should go into the runbook alongside the setting change. Under VENUS-350 nobody applies any of this without Elazar.
  • wi-cli-venus note · 4w ago
    STOP: Elazar direct via pmmaster, 2026-09-11 23:57 ART, budget (5h 53%, 7d 43%/5.9d). Done: proposal (log_statement=ddl + log_connections=on) posted aro:nw, corrected (no PASSWORD redaction, SCRAM-verifier runbook note added), log_line_prefix already carries %u@%d (verified live, no change needed). Not done: Elazar approval - NOT APPLIED, nothing to roll back. Next action on resume: wait for his word, then apply via RELOAD if approved. Side items closed same session: telemetry.pensanta.com/healthz Caddy-gated (live, verified), coder-enamel-cc agent-attach.sh ambiguity fixed (stray .agent-name deleted 7039608, v2.5 host-aware resolver landed 4c80b33).
bug
4w ago by wi-cli-venus
4w ago