applog-listen catch-up watermark stalls permanently when every caught-up row is filtered (venus pinned 15d at 2026-07-15)

Sub-tickets

Questions

Activity

• wi-cli-venus created · 2w ago
• wi-cli-venus audit-finding · 2w ago
  audit-venus-ca BLOCK on the proposed two-state (paged|filtered) disposition design, verdict audit-venus-ca-ms7e78s5ios3. Defect itself confirmed. Reason: burst FIRST occurrences and dedup-throttled rows are DEFERRED, not terminally filtered — advancing past them before their timer/alert settles can silently lose a later page on crash. The proposal's premise ('a filtered row's disposition is final and deterministic') is false for those two paths. Required model: ordered contiguous acknowledgement with THREE states (terminal | pending | buffered); watermark advances only after terminal classification or successful alert. Separate defect found in same review: replaying one stale burst event twice within 60s can synthesize a x2 burst — duplicate event IDs must not increment the burst counter.
• wi-cli-venus audit-finding · 2w ago
  CORRECTIONS 2026-07-30, three falsified claims in the description above. Recorded here rather than edited out so the reasoning is traceable. 1. 'Siblings advanced fine' IS VOID. It was derived from the watermark file's updatedAt, and updatedAt is written on EVERY advance path including the scanNow-clean one - it measures 'the daemon wrote a file', never 'the guarantee advanced'. Same shape as last_seen tracking poll not work, one level up. Re-derived from CURSOR AGE (coder-mars-cc + bin-venus-cc, independent reads, now=2026-07-30T10:52Z): venus cursor 2026-07-15T10:52:47 file 2026-07-26T03:12:49Z cursor age 15.0d file age 4.3d ayudarg cursor 2026-07-27T09:25:34 file 2026-07-29T18:01:48Z cursor age 3.1d mars cursor 2026-07-27T16:00:06 file 2026-07-29T18:02:08Z cursor age 3.0d pluto cursor 2026-07-30T05:02:49 file 2026-07-30T05:02:52Z cursor age 5.8h Venus carries TWO distinct failures: file unwritten 4.3d AND cursor pointing 15d back. Also unrecorded until now: the non-rt pair venus-watermark.json / venus-seen.json are dated Jul-6. MARS IS NOT STALLED - db-mars-cc resolved it: newest error/fatal row IS the cursor row, nothing newer than 2026-07-27 16:00:06.770551+00. Genuinely quiet, nothing stranded. AYUDARG UNCHECKED, outside venus lane. 2. OPEN QUESTION, do not re-derive from memory later: mars and ayudarg watermark files were last written 2026-07-29T18:02:08Z and 18:01:48Z - within 20 SECONDS of each other - while pluto advanced 3.6h ago. Two rails stopping in the same minute is one event, not two quiet apps. Measurement, not a diagnosis; cause unknown. 3. THE PROPOSED FIX IN THE DESCRIPTION IS WRONG TWICE. (a) 'burst 1-off is safe to advance past: a later second occurrence has a strictly later createdAt' - FALSIFIED. burstMap keys on signature+UA under processing wall-clock and never reads createdAt or event id, so there is no timestamp comparison for the premise to be true of. Burst first occurrence is DEFERRED (audit-venus-ca, ruling accepted in full by pmmaster ms7eabyhmn9z). (b) 'advance to the newest definitively-disposed row' is INSUFFICIENT. Disposed and buffered rows INTERLEAVE, and the newest disposed row can sit AFTER a buffered one you must not pass. 4. THE MODEL TO IMPLEMENT (audit-venus-ca, ratified): - three-valued disposition: paged | disposed | deferred (not paged|filtered - a two-valued return collapses 'filtered because of what this row IS' with 'filtered because of what TIME it is', forcing the caller to guess). - PURITY DISCRIMINATOR (coder-mars-cc): a path is DISPOSED only if its filter decision is a pure function of the row and reproducible from the row alone. Pure/disposed: self-probe, category=security, LOG_ONLY_CATEGORIES, sub-threshold slow_query, warn-off. Not pure (reads a clock): throttle, burst. - TERMINALITY TEST for a lone historical row: DISPOSED only once its event-time window is PROVABLY CLOSED - scanNow >= createdAt+60s AND the batch contains no matching successor. Positive checkable condition, so the cursor still advances through history rather than parking forever. Live NOTIFY first occurrences deferred until the same deadline. - cursor advances ONLY through the CONTIGUOUS SETTLED PREFIX. 5. THE FIX MUST BE ARGUED AGAINST BOTH FAILURE CASES OR IT IS NOT ARGUED: - TOTAL STALL (venus, in the description above): buffer always empty -> flush() never runs -> nothing advances. - PARTIAL ADVANCE (coder-mars-cc, real in the code, NOT currently instantiated on mars): buffer non-empty, flush() DOES run and advances to the newest BUFFERED row, permanently stranding any filtered row NEWER than it. This case MOVES updatedAt and therefore reads healthy on every existing instrument. It is the empirical argument for contiguous-settled-prefix. 6. SEQUENCING: VENUS-319 (event-time burst grouping + id dedup) -> EVO-84 category/categoryValue probe (row selection) -> VENUS-318 (cursor disposition). 318's terminality test is defined in terms of the event-time window 319 installs. Diff crosses coder-mars-cc + coder-pluto-cc + audit-pluto-ca at every step. Nothing starts on 318 until 319 lands.
• wi-cli-venus note · 2w ago
  ADDITIONAL DESIGN INPUT 2026-07-30 11:00 UTC. PROPAGATION FINDING (coder-pluto-cc, composes with the contiguous-settled-prefix model): the prefix is only correct if each row's DISPOSITION is correct, and enqueue() cannot report one today because BOTH of its tail calls into enqueueDirect() can still be filtered THERE. A disposition returned by enqueue() that does not propagate through enqueueDirect() is a wrong answer, not a missing one. Fix the return path through both, or the three-valued return is decorative. BLAST RADIUS CORRECTED AGAIN (pmmaster ms7edwvhdnk7, withdrawing his own '3 of 4 rails stale'): only VENUS has measured LAG. Cursor age measures the last settled relevant event, not backlog. Mars proven quiet by query. Ayudarg has age only, never lag, outside venus lane. Pluto not a defect. Blast radius for the STALL is one rail - venus. The CODE defect remains shared by all four (one file, WorkingDirectory pinned to venus/), so the fix is still not a venus-lane fix. Also: the enamel rail is NOT a fifth data point here. It is deliberately stopped under an EVO-84 withdrawal (enamel appEvents 100% appKey='terra'); its 'failed' state was a transient inside a standup and its later 'active' state was an erroneous restart, since re-stopped. Do not read either as evidence about the cursor mechanism.
• wi-cli-venus verified · 2w ago
  BLAST RADIUS REVISED AGAIN - MARS HAS MEASURED LAG (coder-mars-cc ms7ehqfeegcc, retracting his own 'definitively not stalled'). Prior clean reading used level IN ('error','fatal'); the rail's catch-up predicate is level IN ('warn','error','fatal') (:573) and WARN_ENABLED is true on mars. 11 warn rows behind mars's cursor: 10 scanned-then-permanently-stranded (2026-07-27 16:14:53 -> 2026-07-29 16:14:14) + 1 never-scanned. Zero error/fatal, so NO ERROR PAGE WAS LOST on mars - what is stranded is friction telemetry, whose correct paging disposition is the open question 318/319 are redesigning. Not to be dismissed as benign on that basis. THE PARTIAL-ADVANCE MECHANISM IS INSTANTIATED ON MARS. My earlier event said it was real in the code but not instantiated - that is now falsified: 'catch-up found 13 missed event(s)', flush() advanced to the newest BUFFERED row at Jul-27 16:00:06, and 10 eligible rows newer than that were never buffered and now sit permanently behind the cursor, re-scanned on every reconnect forever. This case MOVES updatedAt and reads healthy on every liveness instrument. It is the empirical argument for contiguous-settled-prefix and the fix must be argued against it. Standing: venus measured lag (15.0d cursor / 4.3d file). Mars measured lag (11 rows). Pluto clean on the backlog query - coder-pluto-cc to confirm theirs used warn and not just error/fatal, since that omission is the sole reason mars read clean. Ayudarg has age only AND shares mars's Jul-29 18:0x stop, making it a stronger candidate for the same strand; outside venus lane.
• wi-cli-venus note · 2w ago
  ATTRIBUTION CORRECTION (audit-venus-ca ms7eig2i44am): mars's measured lag is REAL and the partial-interleaving confirmation in my prior event STANDS. What is withdrawn is the burst attribution - the stranded rows are action='nudge-shown' and burst code handles only the exact signatures 'Script error.' and 'Failed to fetch', so they never enter burstMap. They hit enqueueDirect's processing-time 10-minute route dedup throttle instead. Ordered 13-row batch reconstruction pending from coder-mars-cc before any buffered/suppressed counts are assigned.
• wi-cli-venus audit-finding · 2w ago
  DEFECT INTRODUCED BY THE REMEDY (coder-pluto-cc, upheld by pmmaster ms7ejgbuvqrj). The fix must handle this or it trades a known failure for a silent one. MECHANISM: today's stall ACCIDENTALLY provides burst-state recovery across restarts. A 1-off sits behind the frozen cursor, so every reconnect re-reads it and re-seeds burstMap. Once the cursor advances correctly that stops - and a restart landing INSIDE a burst window then converts a PAGED crash-loop into a SILENT one. Restarts co-occur with incidents by construction, so this is not a rare intersection. REMEDY - audit-venus-ca's own ruling applied one level further: DEFERRED IS NOT SETTLED. The contiguous settled prefix must STOP at any row carrying LIVE DEFERRED STATE - a live burstMap entry, or an open dedup-throttle window - and resume only once the timer expires and the disposition is genuinely terminal. Venus's 15-day case is unaffected (windows long expired); this bites on a healthy rail. If the loss is instead judged acceptable, that is a COMMENTED, DECIDED tradeoff in the code, never an emergent side effect. The objection upheld is specifically to it landing UNNAMED. BOARD LINE, amended: VENUS-318 = contiguous settled prefix, argued against total-stall AND partial-advance, with (a) enqueue() -> enqueueDirect() disposition propagation, and (b) deferred-is-not-settled so the prefix does not open the burst-recovery loss. Also carried from the same ruling, generalising the enamel case: none of LIVENESS, ACTIVITY, or BOOKKEEPING HEALTH answers 'should this be running at all'. That is a fourth question - see VENUS-320.
• wi-cli-venus verified · 2w ago
  BLAST RADIUS: THREE OF FIVE RAILS — venus, mars, AYUDARG (pmmaster ms7elgmfzho3). Ayudarg is the worst measured strand: 774 eligible rows behind the cursor, 2 of them ERROR-level, still accumulating. AYUDARG IS TEXTBOOK PARTIAL-ADVANCE AND THE TIMING PROVES IT: the 2 error rows land 0.44s and 0.77s AFTER the cursor row — same catch-up batch, flush() advanced to the newest BUFFERED row, both left behind. category='navigation', not in LOG_ONLY_CATEGORIES={'access'}, so the log-only dismissal is unavailable. BOUNDING, stated as required rather than left to the alarming or the reassuring reading: rows behind the cursor are NOT the same as pages lost. They may have been delivered live by NOTIFY at the time, with only the recovery guarantee void. WHETHER THOSE 2 ERROR ROWS WERE PAGED AT THE TIME CANNOT BE ESTABLISHED FROM VENUS. PLUTO IS CLEAN AT THE FULL PREDICATE, and now correctly evidenced: its cursor carries a real uuid, so pluto IS exposed to partial-advance and simply has nothing behind it. (coder-pluto-cc withdrew their own two earlier pluto clears because those used the narrower predicate — the standard adopted fleet-wide tonight: A VERDICT IS VOID IF ITS PREDICATE WAS ASSUMED, EVEN WHEN THE NUMBER TURNS OUT RIGHT.) Ayudarg is outside the venus lane and has no owner; pm-llmmsgsrv-cc asked to assign one.
• wi-cli-venus audit-finding · 2w ago
  PREDICATE CLAIM PARTIALLY VOIDED — correcting my own prior events (pmmaster ms7epmwpdz86, self-voiding on the standard he ratified six minutes earlier). The ':573 / :582 = level IN (warn,error,fatal)' citation I recorded is from VERSION 3 — the uncommitted, non-compiling working tree that has never executed on any host. It was inherited from other agents' quotes and called measured. SURVIVES: the ROW COUNTS. Ayudarg's 774 (772 warn / 2 error) and pluto's 0 came from SQL against prod DBs with an explicitly-stated level filter and a real tuple comparison. Independent of which daemon version runs. DOES NOT SURVIVE: that this is the RAIL's eligibility predicate. 'Warn is eligible, therefore these rows should have been delivered' rests on the RUNNING code, which nobody read for these rails. e874465 is what mars executes; venus/ayudarg/pluto load whatever their own daemon started with — a PER-UNIT fact (ActiveEnterTimestamp per instance), not one fleet answer. PENDING: coder-pluto-cc re-derives the eligibility predicate from /** * applog-listen.ts — VENUS-5 real-time leg of the centralized log plane. * * Long-running per-app daemon (systemd Type=simple, applog-listen@%i). Holds a * persistent Postgres LISTEN connection on channel `applog_err` to the app's OWN * Supabase (DIRECT/session DSN — LISTEN/NOTIFY is unsupported on the 6543 * transaction pooler, so the DIRECT DSN is mandatory). On NOTIFY it DMs the * owning PM via the llmmsg hub in near-real-time. * * Pairs with the 30-min batched applog-pull.ts: the per-app appEvents * AFTER-INSERT pg_notify trigger fires ONLY on level IN ('error','fatal'); those * rows are owned by THIS real-time path. warn/slow-query stays in the digest * (applog-pull.ts, gated by APPLOG_DIGEST_LEVELS so error is dropped from the * digest only once this listener is confirmed live for that app — no coverage * gap during staged rollout). * * Resilience: * - keepalive: a periodic `SELECT 1` keeps the conn warm and surfaces a silent * half-open socket (Supabase closes idle/long sessions). * - auto-reconnect: on client error/end the daemon backs off and reconnects. * - catch-up read: every (re)connect first sweeps appEvents for error/fatal rows * newer than the realtime watermark and DMs anything missed during the gap, * then advances the watermark — so a drop never loses an error. * * The realtime watermark is a SEPARATE file from the digest puller's, so the two * legs never interfere. * * Env (per-app EnvironmentFile, mirrors applog-pull.ts): * APPLOG_APP, DATABASE_URL_DIRECT (required, DIRECT DSN), LLMMSG_HUB_URL, * APPLOG_SENDER, APPLOG_PM, APPLOG_RT_STATE_FILE (watermark path), * APPLOG_RT_INITIAL_LOOKBACK (first-boot catch-up window, default '5 minutes'), * APPLOG_RT_COALESCE_MS (burst-coalesce window, default 3000), * APPLOG_RT_KEEPALIVE_MS (default 30000). * * DEPLOY NOTE: one script, N instances (applog-listen@venus/mars/pluto/ayudarg). * After any code change, restart ALL instances — not just venus: * ~/scripts/applog-listen-restart-all.sh * Restarting only venus leaves the others on stale code silently (EVO-28 post-mortem). */ import { Client } from "pg"; import dns from "node:dns"; import http from "node:http"; import https from "node:https"; import { existsSync, readFileSync, writeFileSync, mkdirSync } from "node:fs"; import { resolve, dirname } from "node:path"; // EVO-5(A): force IPv4 DNS resolution. The Supabase pooler advertises an AAAA // record whose IPv6 endpoint is unreachable from this host → net.Socket.connect // would prefer IPv6 and loop on ECONNREFUSED (killed pluto's listener for 4 days). // Node's default resolver order is verbatim; 'ipv4first' makes net's dns.lookup // return A records first so we connect over IPv4 and never hit the dead IPv6. // NOTE: the originally-specified `family: 4` on the pg Client is NOT used — it's a // no-op on pg 8.20.0 (connection.js calls `stream.connect(port, host)` 2-arg and // never forwards `family` to the socket) AND `family` isn't in @types/pg's // ClientConfig (fails tsc excess-property check). The DNS-order fix is the real, // version-independent mechanism and lives HERE. dns.setDefaultResultOrder("ipv4first"); // DSN must come from the REAL env for non-venus instances (cross-account // contamination guard — see applog-pull.ts for the full rationale). const realEnvDirect = process.env.DATABASE_URL_DIRECT?.trim() || ""; const ENV_PATH = ".env.local"; if (existsSync(ENV_PATH) && typeof process.loadEnvFile === "function") { process.loadEnvFile(ENV_PATH); } const APP = process.env.APPLOG_APP ?? "venus"; // scrp-* prefix = write-only script emitter: this process sends but never drains an // inbox, so a human/agent reply to it dead-letters. The name carries that warning. // Hub registry maps scrp-applog-<app> -> maintainerAgentName=coder-venus-cc. const SENDER = process.env.APPLOG_SENDER ?? `scrp-applog-${APP}`; const PM_RECIPIENT = process.env.APPLOG_PM ?? `pm:${APP}`; const HUB_URL = process.env.LLMMSG_HUB_URL ?? "http://llmmsg-hub.pensanta.com:9703"; // Edge-gated hubs (venus llmmsg-srv-venus) require Authorization: Bearer on every // request, loopback included. Bearerless when unset (legacy whey hub on the tunnel). const HUB_BEARER = process.env.LLMMSG_HUB_BEARER?.trim() || ""; const STATE_FILE = resolve( process.env.APPLOG_RT_STATE_FILE ?? `.applog-state/${APP}-rt-watermark.json`, ); const INITIAL_LOOKBACK = process.env.APPLOG_RT_INITIAL_LOOKBACK ?? "5 minutes"; const COALESCE_MS = Number(process.env.APPLOG_RT_COALESCE_MS ?? "3000"); const KEEPALIVE_MS = Number(process.env.APPLOG_RT_KEEPALIVE_MS ?? "30000"); const HEARTBEAT_MS = Number(process.env.APPLOG_HEARTBEAT_MS ?? String(24 * 60 * 60 * 1000)); // Heartbeat write path is SEPARATE from the read DSN: the listener connects as // the least-priv read-only applog_reader (LISTEN + catch-up SELECTs), which // cannot INSERT. The heartbeat needs a dedicated least-priv applog_writer DSN // (INSERT-only on "appEvents"). Unset → heartbeat stays dormant (no-op), so the // listener never spams 42501; it auto-activates once the DSN is provisioned. const HEARTBEAT_DSN = process.env.APPLOG_HEARTBEAT_DSN?.trim() || ""; const CHANNEL = "applog_err"; // Sentinel cursor id for a "scanned clean up to T" watermark (no row to key on). const ZERO_UUID = "00000000-0000-0000-0000-000000000000"; // Per-app off-switch: set APPLOG_WARN=0 in the app's EnvironmentFile to suppress // warn-level forwarding (useful if an app emits noisy benign 404s at warn level). const WARN_ENABLED = process.env.APPLOG_WARN !== "0"; // Slow-query real-time page threshold (ms). Events with action='slow_query' and a // parseable duration in the message below this value are log-only. Default 1500ms. // Elazar directive 2026-06-22: 570-670ms steady band is benign; new bar >1500ms. // Set APPLOG_SLOW_QUERY_MS=0 to disable the filter (page all slow_query events). const SLOW_QUERY_PAGE_MS = Number(process.env.APPLOG_SLOW_QUERY_MS ?? "1500"); // Categories that are log-only. 'security' is handled separately (bot-filter); 'access' // covers single-occurrence accessDenied rows (benign, no actionable server-side cause). // Elazar directive 2026-06-22 (14d sweep evidence). const LOG_ONLY_CATEGORIES = new Set(["access"]); // EVO-28: internal probe discriminator. UA prefix for agent/script-issued prod curls // (fleet std v1.3 §7). UA match → log-only (action='self-probe'). Rail is passive // until callers carry the UA; probe-curl.sh wrapper stamps it automatically. const SELF_PROBE_UA_PREFIX = "evolutiva-internal-probe/"; // EVO-27: burst-dedup for Script error./Failed to fetch class. Collapses events sharing // the same (userAgent + session + errorClass signature) within 60s into 1 alert + count. // Isolated 1-offs (no burst within window) are log-only — they lack crash-loop signal. // N>1 preserves crash-loop detection (ref: Samsung Internet loop buried in Script error.). const BURST_DEDUP_WINDOW_MS = 60 * 1000; const BURST_ERRORCLASS_SIGNATURES = new Set(["Script error.", "Failed to fetch"]); type BurstEntry = { count: number; timer: NodeJS.Timeout }; const burstMap = new Map<string, BurstEntry>(); const dsn = APP === "venus" ? process.env.DATABASE_URL_DIRECT?.trim() || process.env.DATABASE_URL?.trim() || "" : realEnvDirect; if (!dsn) { console.error( APP === "venus" ? "Set DATABASE_URL_DIRECT (or DATABASE_URL) in .env.local." : `[applog-listen] APPLOG_APP=${APP}: no DATABASE_URL_DIRECT in the real environment. ` + `Refusing to fall back to .env.local (venus's DSN). Set the per-app EnvironmentFile DSN.`, ); process.exit(1); } type Cursor = { createdAt: string; id: string }; type Watermark = { app: string; cursor: Cursor | null; updatedAt: string; rtPaged?: Array<{ id: string; createdAt: string }> }; function readWatermark(): Cursor | null { if (!existsSync(STATE_FILE)) return null; try { const wm = JSON.parse(readFileSync(STATE_FILE, "utf8")) as Watermark; return wm.cursor ?? null; } catch (err) { console.error(`[applog-listen] watermark unreadable (${STATE_FILE}); first run:`, err); return null; } } function readRtPagedMap(): Map<string, string> { if (!existsSync(STATE_FILE)) return new Map(); try { const wm = JSON.parse(readFileSync(STATE_FILE, "utf8")) as Watermark; return new Map((wm.rtPaged ?? []).map((e) => [e.id, e.createdAt])); } catch { return new Map(); } } // Prune rt-paged IDs after 2h — well beyond the digest's lookback window. const RT_PAGED_PRUNE_MS = 2 * 60 * 60 * 1000; // IDs paged by the real-time path; persisted to STATE_FILE so applog-pull.ts can exclude them. const rtPagedMap: Map<string, string> = readRtPagedMap(); function writeWatermark(cursor: Cursor): void { const pruneThreshold = new Date(Date.now() - RT_PAGED_PRUNE_MS).toISOString(); const rtPaged = [...rtPagedMap.entries()] .filter(([, ts]) => ts >= pruneThreshold) .map(([id, createdAt]) => ({ id, createdAt })); const payload: Watermark = { app: APP, cursor, updatedAt: new Date().toISOString(), rtPaged }; mkdirSync(dirname(STATE_FILE), { recursive: true }); writeFileSync(STATE_FILE, JSON.stringify(payload, null, 2)); } // Hub HTTP send — identical contract to applog-pull.ts (POST /register then /send). function hubPost( path: string, body: Record<string, unknown>, ): Promise<{ ok: boolean; status: number; body: unknown }> { const url = new URL(path, HUB_URL); const lib = url.protocol === "https:" ? https : http; const data = JSON.stringify(body); const opts = { method: "POST", hostname: url.hostname, port: url.port || (url.protocol === "https:" ? 443 : 80), path: url.pathname, headers: { "content-type": "application/json", "content-length": Buffer.byteLength(data), ...(HUB_BEARER ? { authorization: `Bearer ${HUB_BEARER}` } : {}), }, timeout: 10000, }; return new Promise((resolveP) => { const req = lib.request(opts, (res) => { const chunks: Buffer[] = []; res.on("data", (c) => chunks.push(c as Buffer)); res.on("end", () => { const text = Buffer.concat(chunks).toString("utf8"); let parsed: unknown = {}; try { parsed = text ? JSON.parse(text) : {}; } catch { parsed = { raw: text }; } resolveP({ ok: (res.statusCode ?? 0) < 400, status: res.statusCode ?? 0, body: parsed }); }); }); req.on("error", (err) => resolveP({ ok: false, status: 0, body: { error: String(err) } })); req.on("timeout", () => { req.destroy(); resolveP({ ok: false, status: 0, body: { error: "timeout" } }); }); req.write(data); req.end(); }); } let registered = false; async function register(): Promise<boolean> { // No cwd: the hub roster is a DB, and cwd here is the venus checkout path — // publishing it is the same app-file-location leak as a scriptLocation column, // just under a different field name. Real agents send cwd (it's how you find an // agent's workspace); a write-only script emitter has no workspace to reach. const reg = await hubPost("/register", { agent: SENDER }); if (reg.ok) registered = true; return reg.ok; } // Periodic backstop: re-register every 5min so a hub restart (which evicts the // listener from the roster) is recovered within one tick, not only on next DM. // UNCONDITIONAL by design: the old `if (!registered)` guard never fired, because // `registered` latches true on first success and only postAlert's failure path // resets it. A quiet app therefore never re-contacted the hub, its last_seen // staled, and the prune sweep dropped it from the roster — invisible until the // next error arrived. Re-registering is idempotent and costs one POST/5min. const REREGISTER_MS = 5 * 60 * 1000; setInterval(() => { void register(); }, REREGISTER_MS); // pm:${APP} token resolves to current PM via hub, survives PM reassignment. // origin_aro sets thread context to aro:${APP}. kind=dm — kind=alert is filtered by // chat-duo-web and would silently disappear (root cause of missed alerts). const ALERT_RECIPIENTS = [`pm:${APP}`]; const ALERT_ORIGIN_ARO = `aro:${APP}`; // MARS-432: re-join aro:${APP} so a send tagged origin_aro=aro:${APP} is accepted. // The hub strips the "aro:" prefix and requires the agent already registered, so // this must run AFTER register(). Idempotent. Needed because the aros-prune sweep // can drop this idle-by-design sender from the lane during quiet periods. async function aroJoin(): Promise<boolean> { const r = await hubPost("/aro/join", { agent: SENDER, aro: ALERT_ORIGIN_ARO }); return r.ok; } // Eager startup register + lane join. Without it the roster entry only appears on // the first alert (or the 5min tick), so a quiet app is simply ABSENT from the hub // roster after a restart — which is how the rename would have looked half-done. // Also spares the first alert the 400 -> re-register -> retry round trip. void (async () => { if (await register()) await aroJoin(); })(); async function postAlert(message: string): Promise<boolean> { if (!registered) await register(); const payload = { agent: SENDER, to: ALERT_RECIPIENTS, message, kind: "dm", origin_aro: ALERT_ORIGIN_ARO }; const r = await hubPost("/send", payload); if (!r.ok) { console.error("[applog-listen] post failed:", r.status, r.body); // Fast-path recovery: hub returns 400 or 403 for "not_registered" (evicted on // hub restart). Re-register immediately and retry once — recovers within this // send rather than waiting for the 5min REREGISTER_MS tick. registered = false; if (r.status === 400 || r.status === 403) { const reReg = await register(); if (reReg) { // MARS-432: re-register restores the roster entry but NOT aro:${APP} // membership. A 400 origin_aro_not_member (idle sender pruned from the lane) // therefore loops forever without an aro_join. Re-join before the single // retry — idempotent, keeps origin_aro=aro:${APP} so the PM alert thread // stays in ARO context (do NOT drop the tag or switch to a bare DM). await aroJoin(); const retry = await hubPost("/send", payload); if (retry.ok) return true; console.error("[applog-listen] post retry after re-register failed:", retry.status, retry.body); } } } return r.ok; } // Dedup-throttle: 1 alert per route per 10min window. Suppressed events are // counted and reported at window expiry — never silently dropped. const DEDUP_WINDOW_MS = 10 * 60 * 1000; type DedupEntry = { windowEnd: number; suppressedCount: number; flushTimer: NodeJS.Timeout | null; }; const dedupMap = new Map<string, DedupEntry>(); // Returns true if the event should proceed to the buffer; false if throttled. // When throttled, arms a timer to report the suppressed count at window end. function dedupAllow(e: ErrEvent): boolean { const key = e.route ?? "__no_route__"; const now = Date.now(); const entry = dedupMap.get(key); if (!entry || now >= entry.windowEnd) { if (entry?.flushTimer) clearTimeout(entry.flushTimer); dedupMap.set(key, { windowEnd: now + DEDUP_WINDOW_MS, suppressedCount: 0, flushTimer: null }); return true; } entry.suppressedCount++; if (!entry.flushTimer) { const delay = entry.windowEnd - now; const routeKey = key; entry.flushTimer = setTimeout(() => { const e2 = dedupMap.get(routeKey); if (e2 && e2.suppressedCount > 0) { void postAlert( `applog ${APP}: ${e2.suppressedCount} suppressed alert(s) for route ${routeKey} (dedup 10min window)`, ); } dedupMap.delete(routeKey); }, delay); } return false; } // One normalized error event (from a NOTIFY payload or a catch-up row). type ErrEvent = { id: string; createdAt: string; // full-precision text for cursor fidelity level: string; category: string | null; action: string | null; route: string | null; signature: string; message: string | null; userAgent: string | null; // EVO-28: probe UA discriminator (null for older NOTIFY payloads) }; function fmtOne(e: ErrEvent): string { const where = e.route ? ` ${e.route}` : ""; const cat = [e.category, e.action].filter(Boolean).join("/"); return `[${e.level}] ${e.signature}${where}${cat ? ` (${cat})` : ""} @${e.createdAt} id=${e.id}`; } // Coalesce a burst into a single DM so an error storm can't blow the hub budget. let buffer: ErrEvent[] = []; let flushTimer: NodeJS.Timeout | null = null; // IDs currently held by an active flush() call (between buffer=[] and postAlert resolve). // flush() clears buffer before awaiting postAlert, so a concurrent catchUp would see an // empty buffer and re-enqueue the same rows. inflightIds bridges that async gap (OPS-92 v2). const inflightIds = new Set<string>(); function enqueue(e: ErrEvent): void { // EVO-28: self-probe UA rail. Probe tools stamp UA='evolutiva-internal-probe/<tool>' // (fleet std v1.3 §7). Checked against the userAgent column (from catch-up SELECT or // NOTIFY payload once trigger is updated). Fails-open for NOTIFYs from the old trigger // (userAgent=null) — those remain pageable until the trigger migration lands. if ( e.action === "self-probe" || (e.userAgent !== null && e.userAgent.startsWith(SELF_PROBE_UA_PREFIX)) ) { console.log(`[applog-listen] ${APP}: log-only (self-probe): ${fmtOne(e)}`); return; } // EVO-27: burst-dedup for Script error./Failed to fetch. Key=(signature+userAgent). // userAgent identifies the browser/client that emitted the crash; grouping by it // preserves cross-session crash-loop detection (same browser crashing repeatedly) // without needing sessionId in the NOTIFY payload. // First occurrence within 60s window: log-only (1-off, not a crash-loop). // Second+ occurrence: burst confirmed → page with count N. if (BURST_ERRORCLASS_SIGNATURES.has(e.signature)) { const ua = e.userAgent ?? "__no_ua__"; const burstKey = `${e.signature}::${ua}`; const existing = burstMap.get(burstKey); if (!existing) { const timer = setTimeout(() => { burstMap.delete(burstKey); console.log(`[applog-listen] ${APP}: burst-window expired (1-off, log-only): ${burstKey}`); }, BURST_DEDUP_WINDOW_MS); burstMap.set(burstKey, { count: 1, timer }); console.log(`[applog-listen] ${APP}: log-only (burst-dedup 1-off): ${fmtOne(e)}`); return; } clearTimeout(existing.timer); burstMap.delete(burstKey); existing.count++; // Burst confirmed — page with count annotation. const annotated: ErrEvent = { ...e, message: `[burst x${existing.count}] ${e.message ?? ""}` }; console.log(`[applog-listen] ${APP}: burst confirmed (x${existing.count}, paging): ${fmtOne(annotated)}`); enqueueDirect(annotated); return; } enqueueDirect(e); } function enqueueDirect(e: ErrEvent): void { // Bot filter: security scanner rows (category='security') are console-logged only. // They must not reach the app room — keep bots out of aro:${APP}. if (e.category === "security") { console.log(`[applog-listen] ${APP}: security-log (bot-filtered): ${fmtOne(e)}`); return; } // Log-only categories: 'access' (accessDenied, single-occurrence, benign). // Elazar directive 2026-06-22. if (e.category !== null && LOG_ONLY_CATEGORIES.has(e.category)) { console.log(`[applog-listen] ${APP}: log-only (category=${e.category}): ${fmtOne(e)}`); return; } // Slow-query threshold: warn-level slow_query events below SLOW_QUERY_PAGE_MS are // log-only. Duration is parsed from the message field (format: "... Xms ..."). // Fails-open: if action doesn't match or duration unparseable, the event is paged. if (SLOW_QUERY_PAGE_MS > 0 && e.action === "slow_query" && e.message !== null) { const match = /(\d+)\s*ms/i.exec(e.message); const dur = match ? Number(match[1]) : null; if (dur !== null && dur < SLOW_QUERY_PAGE_MS) { console.log(`[applog-listen] ${APP}: log-only (slow_query ${dur}ms < ${SLOW_QUERY_PAGE_MS}ms threshold): ${fmtOne(e)}`); return; } } // Warn off-switch: skip warn-level rows if APPLOG_WARN=0 for this app. if (e.level === "warn" && !WARN_ENABLED) return; // Dedup-throttle: 1 alert per route per 10min; suppressed count reported at window end. if (!dedupAllow(e)) return; buffer.push(e); if (!flushTimer) flushTimer = setTimeout(flush, COALESCE_MS); } async function flush(): Promise<void> { flushTimer = null; if (buffer.length === 0) return; const batch = buffer; buffer = []; for (const e of batch) inflightIds.add(e.id); // Advance the watermark to the newest row in the batch (text-sorted createdAt, // tie-broken by id — same ordering the catch-up query uses). const newest = batch.reduce((a, b) => b.createdAt > a.createdAt || (b.createdAt === a.createdAt && b.id > a.id) ? b : a, ); const TOP = 8; const head = batch.slice(0, TOP).map(fmtOne).join("\n"); const overflow = batch.length > TOP ? `\n…+${batch.length - TOP} more` : ""; const title = batch.length === 1 ? `applog ${APP}: real-time ${batch[0]!.level}` : `applog ${APP}: ${batch.length} real-time events`; const sent = await postAlert(`${title}\n${head}${overflow}`); for (const e of batch) inflightIds.delete(e.id); if (sent) { for (const e of batch) rtPagedMap.set(e.id, e.createdAt); writeWatermark({ createdAt: newest.createdAt, id: newest.id }); } else { // Re-enqueue to front of buffer so next flush retries without requiring a pg reconnect. // (Watermark-stays-behind alone only replays on reconnect; re-enqueue closes the gap.) buffer = batch.concat(buffer); if (!flushTimer) flushTimer = setTimeout(flush, COALESCE_MS * 10); } } // Catch-up: pull error/fatal rows newer than the watermark and enqueue them. // Runs on every (re)connect so a drop never loses an error. First boot is bounded // by INITIAL_LOOKBACK so we don't replay history on a fresh deploy. async function catchUp(client: Client): Promise<void> { const cursor = readWatermark(); // Capture the scan boundary from the DB clock BEFORE the sweep. If the sweep // returns empty, (cursor, scanNow] is provably clean, so we advance the // watermark to scanNow (EVO-5 B) — guaranteeing a cursor always exists after // the first connect. Without this, an error-free listener never writes a // watermark, so a later long outage falls back to the fixed INITIAL_LOOKBACK // window and silently drops errors older than it. const nowRes = await client.query<{ now: string }>(`SELECT now()::text AS now`); const scanNow = nowRes.rows[0]?.now ?? null; const where = cursor ? `("createdAt", id) > ($1::timestamptz, $2::uuid)` : `"createdAt" > now() - ($1 || ' ')::interval`; const params = cursor ? [cursor.createdAt, cursor.id] : [INITIAL_LOOKBACK]; const res = await client.query<ErrEvent & { createdAt: string }>( `SELECT id, "createdAt"::text AS "createdAt", level, category, action, route, COALESCE(NULLIF("errorMessage", ''), detail->>'digest', action) AS signature, left(COALESCE(message, ''), 500) AS message, left(COALESCE("userAgent", ''), 300) AS "userAgent" FROM "appEvents" WHERE level IN ('warn','error','fatal') AND ${where} ORDER BY "createdAt" ASC, id ASC`, params, ); // Filter out rows already in-flight or already paged (OPS-92 v2). // Two inflight sources: // buffer — rows queued but flush() not yet started // inflightIds — rows taken from buffer by a concurrent flush() that cleared buffer // before its postAlert call resolved; without this set, catchUp races // the async gap and re-enqueues the same rows a second time. const inFlight = new Set([...buffer.map((e) => e.id), ...inflightIds]); const newRows = res.rows.filter((r) => !inFlight.has(r.id) && !rtPagedMap.has(r.id)); if (newRows.length > 0) { if (newRows.length < res.rows.length) { console.log(`[applog-listen] ${APP}: catch-up found ${res.rows.length} row(s), ${res.rows.length - newRows.length} already inflight/paged — skipping dupes.`); } else { console.log(`[applog-listen] ${APP}: catch-up found ${newRows.length} missed event(s).`); } for (const r of newRows) enqueue(r); // Watermark advances to the newest row when this batch flushes (flush()'s // newest-wins). We deliberately do NOT advance to scanNow here: the rows are // buffered, not yet DM'd, so advancing past them would lose them on a crash. } else if (res.rows.length > 0) { console.log(`[applog-listen] ${APP}: catch-up found ${res.rows.length} row(s), all already inflight/paged — no new events.`); if (scanNow) writeWatermark({ createdAt: scanNow, id: ZERO_UUID }); } else { console.log(`[applog-listen] ${APP}: catch-up clean.`); if (scanNow) writeWatermark({ createdAt: scanNow, id: ZERO_UUID }); } } function parsePayload(raw: string): ErrEvent | null { try { const p = JSON.parse(raw) as Record<string, unknown>; if (!p.id || !p.createdAt) return null; // Level gate: forward warn/error/fatal only. Mars's trigger fires on ALL rows // so an info-level heartbeat would otherwise generate a false alert. // warn is included so typed 404s surface in real time; bot filter in enqueue() // handles category=security rows regardless of level. const level = String(p.level ?? "error"); if (level !== "error" && level !== "fatal" && level !== "warn") return null; return { id: String(p.id), createdAt: String(p.createdAt), level, category: p.category != null ? String(p.category) : null, action: p.action != null ? String(p.action) : null, route: p.route != null ? String(p.route) : null, signature: String(p.signature ?? p.action ?? "(unknown)"), message: p.message != null ? String(p.message) : null, userAgent: p.userAgent != null ? String(p.userAgent) : null, }; } catch (err) { console.error("[applog-listen] bad NOTIFY payload:", err, raw.slice(0, 200)); return null; } } let shuttingDown = false; let backoffMs = 1000; const BACKOFF_MAX = 30000; // C: per-app daily heartbeat. Writes an info-level appEvent to THIS app's own // appEvents so a silently dead listener is detectable by the ABSENCE of recent // beats. The heartbeat is info-level, so it can't self-DM IFF the listener // ignores non-error/fatal notifies — which it now does unconditionally in // parsePayload (mars's trigger fires pg_notify on ALL rows, so the daemon, not // the trigger, is the load-bearing gate against a heartbeat self-page). // Per-app by construction: APP + the connected DB are this instance's own; no // shared/global state (pluto's beat → pluto's appEvents, independent of venus/mars). // Writes via a SHORT-LIVED connection on the dedicated applog_writer DSN, never // the read-only LISTEN client — keeping read/write privileges separated. let currentClient: Client | null = null; let heartbeatTimer: NodeJS.Timeout | null = null; let didInitialHeartbeat = false; // Cache: undefined = not yet resolved; null = this app's appEvents has no // categoryId column → omit it; string = the resolved categoryId to include. // venus's appEvents adds a NOT NULL categoryId (FK → lookupOptions.id) that the // shared category-only INSERT would otherwise violate; mars/pluto/ayudarg lack // that column. Detect it per-DB so this stays ONE cross-app-safe script instead // of hardcoding a venus assumption, and resolve the 'Sistema' category from the // same lookupOptions row every existing category='monitoring' row already carries // (queried live, never a hardcoded uuid, so it survives a lookup change). let heartbeatCategoryId: string | null | undefined = undefined; async function resolveHeartbeatCategoryId(hb: Client): Promise<string | null> { if (heartbeatCategoryId !== undefined) return heartbeatCategoryId; const hasCol = await hb.query( `SELECT 1 FROM information_schema.columns WHERE table_name = 'appEvents' AND column_name = 'categoryId' LIMIT 1`, ); if (hasCol.rows.length === 0) { heartbeatCategoryId = null; return null; } // The lookupOptions group-selector column name differs across apps: venus uses // "grupo", mars/pluto use "category". Probe the catalog for whichever exists // rather than hardcode venus's name. Inert today (only venus has the categoryId // column detected above), but keeps this ONE shared script from throwing if // another app later grows a categoryId column. The result is drawn from a fixed // ('grupo','category') allowlist in the catalog query — never user input — so // interpolating it as an identifier is injection-safe. const groupColRes = await hb.query<{ column_name: string }>( `SELECT column_name FROM information_schema.columns WHERE table_name = 'lookupOptions' AND column_name IN ('grupo','category') ORDER BY CASE column_name WHEN 'grupo' THEN 0 ELSE 1 END LIMIT 1`, ); const groupCol = groupColRes.rows[0]?.column_name; if (!groupCol) { heartbeatCategoryId = null; return null; } const r = await hb.query<{ id: string }>( `SELECT id FROM "lookupOptions" WHERE "${groupCol}" = 'appEventCategory' AND nombre = 'Sistema' AND activo = true LIMIT 1`, ); heartbeatCategoryId = r.rows[0]?.id ?? null; return heartbeatCategoryId; } async function writeHeartbeat(): Promise<void> { if (!HEARTBEAT_DSN) return; // dormant until a write-capable DSN is provisioned if (!currentClient) return; // only beat while the LISTEN connection is live const hb = new Client({ connectionString: HEARTBEAT_DSN }); try { await hb.connect(); const categoryId = await resolveHeartbeatCategoryId(hb); const message = `applog-listen ${APP} alive`; const detail = JSON.stringify({ app: APP, pid: process.pid, intervalMs: HEARTBEAT_MS }); if (categoryId) { await hb.query( `INSERT INTO "appEvents" ("level","category","categoryId","action","message","detail") VALUES ('info','monitoring',$1,'applogListenerHeartbeat',$2,$3::jsonb)`, [categoryId, message, detail], ); } else { await hb.query( `INSERT INTO "appEvents" ("level","category","action","message","detail") VALUES ('info','monitoring','applogListenerHeartbeat',$1,$2::jsonb)`, [message, detail], ); } console.log(`[applog-listen] ${APP}: heartbeat written.`); } catch (err) { console.error(`[applog-listen] ${APP}: heartbeat failed:`, err); } finally { try { await hb.end(); } catch { /* already gone */ } } } async function connectLoop(): Promise<void> { while (!shuttingDown) { // keepAlive: OS-level TCP keepalive probes — detects half-open sockets where // the remote is gone but no RST arrives (the silent-stall class from MSG-82). const client = new Client({ connectionString: dsn, keepAlive: true, keepAliveInitialDelayMillis: 10000 }); let keepalive: NodeJS.Timeout | null = null; try { await client.connect(); console.log(`[applog-listen] ${APP}: connected, LISTEN ${CHANNEL}`); backoffMs = 1000; // reset on a healthy connect client.on("notification", (msg) => { if (msg.channel !== CHANNEL || !msg.payload) return; const e = parsePayload(msg.payload); if (e) enqueue(e); }); // A connection-level error must break us out to reconnect. let rejectBroke!: (err: unknown) => void; const broke = new Promise<void>((_, reject) => { rejectBroke = reject; client.on("error", (err) => reject(err)); client.on("end", () => reject(new Error("connection ended"))); }); await client.query(`LISTEN ${CHANNEL}`); // Catch-up AFTER LISTEN is armed: any row inserted during/after the sweep // either lands in the sweep or arrives as a NOTIFY — never lost. Dedup is // by watermark monotonicity (a re-seen row is <= watermark, so flush's // newest-wins advance is idempotent; worst case one duplicate DM, never a miss). await catchUp(client); currentClient = client; // Heartbeat (C) only when a write DSN is configured; dormant otherwise. // Emit one beat immediately on the first successful connect (prompt // verification + baseline), then a steady daily beat via the interval. if (HEARTBEAT_DSN) { if (!didInitialHeartbeat) { didInitialHeartbeat = true; void writeHeartbeat(); } if (!heartbeatTimer) { heartbeatTimer = setInterval(() => void writeHeartbeat(), HEARTBEAT_MS); } } // App-level liveness probe: SELECT 1 every KEEPALIVE_MS. If it fails, reject // broke immediately so the outer catch triggers reconnect+re-LISTEN+catch-up. // Two independent layers: this probe catches query-level failures; OS TCP // keepalive (keepAlive:true above) catches network-level half-open sockets. keepalive = setInterval(() => { client.query("SELECT 1").catch((err) => { console.error(`[applog-listen] ${APP}: keepalive failed:`, err); rejectBroke(err); }); }, KEEPALIVE_MS); await broke; // resolves never; rejects on error/end/keepalive-fail } catch (err) { console.error(`[applog-listen] ${APP}: connection lost:`, err); } finally { currentClient = null; // heartbeat no-ops until the next healthy connect if (keepalive) clearInterval(keepalive); try { await client.end(); } catch { /* already gone */ } } if (shuttingDown) break; console.log(`[applog-listen] ${APP}: reconnecting in ${backoffMs}ms`); await new Promise((r) => setTimeout(r, backoffMs)); backoffMs = Math.min(backoffMs * 2, BACKOFF_MAX); } } for (const sig of ["SIGINT", "SIGTERM"] as const) { process.on(sig, () => { shuttingDown = true; void flush().finally(() => process.exit(0)); }); } connectLoop().catch((err) => { console.error(`[applog-listen] ${APP}: fatal:`, err); process.exit(1); }); and reads ActiveEnterTimestamp/NRestarts for EACH of the five units. If the running predicate is narrower, some of the 774 are not eligible and the strand shrinks; if identical, the finding stands and is then actually measured. STANDING VERSION DISCIPLINE, now mandatory on every applog-listen claim in this WI family: name the version — running(<sha>, <unit>) | HEAD(<sha>) | WIP(uncommitted). AN UNQUALIFIED LINE NUMBER IS NOT A CITATION. Same defect class as an unqualified agent name in commons §7: the reader assumes the one they have. ALSO CORRECTED HERE: my earlier 'per-row disposition is not recoverable from any artifact' was withdrawn by coder-mars-cc's reconstruction (see VENUS-321) — the DB over-determined it. Withdrawn by measurement, not by loosening the standard.
• wi-cli-venus decision · 2w ago
  RATIONALE LINE FOR THE CONTIGUOUS-SETTLED-PREFIX MODEL — upgraded from hedge to measured fact (coder-mars-cc ms7erxezj6bc, running(e874465, applog-listen@mars)). The earlier 'a suppression summary almost certainly went out' was speculation; it was then read: entry.flushTimer = setTimeout(() => { const e2 = dedupMap.get(routeKey); if (e2 && e2.suppressedCount > 0) void postAlert(); dedupMap.delete(routeKey); }, delay); delay = windowEnd - now, DEDUP_WINDOW_MS = 10min. The batch ran ~Jul-29 18:02:05 and the process has been up continuously since (NRestarts=0), so the timer was NOT lost. Two summaries fired ~18:12: '10 suppressed alert(s) for route /mi-cursada' and '1 suppressed alert(s) for route /practicas/nueva'. STATED LIMIT: whether the hub DELIVERED them is not verifiable from mars. WHY THIS SHARPENS 318's SCOPE — use this as the rationale line: the defect is NOT lost visibility. The rows were COUNTED AND ANNOUNCED, then abandoned by the bookkeeping. A row can be reported as suppressed and simultaneously left behind a cursor that reads healthy, so the recovery guarantee is void FOR ROWS THE OPERATOR WAS EXPLICITLY TOLD ABOUT. Stronger than 'we might have missed something': suppressedCount++ is already the code admitting the row was not disposed of, and flush() advances past it anyway. ALSO, VERSION DISCIPLINE RETRO-VOIDS A CLAIM OF SCOPE, not just line numbers: 'one implementation, five rails' is true of the FILE and NOT of the RUNNING SET. NRestarts=0 with ExecMainStartTimestamp=Jul-26 03:12:43 bounds venus/ayudarg/pluto's start to a commit at-or-before Jul-26 — consistent with e874465 (Jul-14) but NOT proof; nothing between Jul-14 and Jul-26 is ruled out. Only mars is confirmed at e874465. Per-unit from here.
• wi-cli-venus spec · 2w ago
  RATIONALE LINE FOR CONTIGUOUS-SETTLED-PREFIX — FINAL WORDING, superseding both earlier versions (coder-mars-cc's original, pmmaster's 11:58 ratification, and audit-venus-ca's narrowing). Adopted per pmmaster ms7eveykbdbc: THE CODE ADMITS THE ROW IS PENDING AND THEN DESTROYS THE RECORD OF THAT PENDING STATE UNCONDITIONALLY — IT DOES NOT MERELY ADVANCE PAST AN UNSETTLED ROW, IT MAKES THE ROW'S UNSETTLEDNESS UNRECOVERABLE IN THE SAME TICK. Not 'the code admits the row was not disposed of' (overstated: a throttled row IS genuinely disposed once its 10-minute aggregate summary is confirmed delivered under the intended contract). Not merely 'pending, not permanently undisposed' (right but incomplete). The defect is advancing MERELY BECAUSE THE ROW WAS COUNTED. That is why the strand reads healthy from every angle, and it is the strongest available argument for the contiguous settled prefix. A suppressed row is DEFERRED in the three-state model; do not let a rationale line drift it back toward DISPOSED. Evidence: two summaries fired ~Jul-29 18:12 ('10 suppressed alert(s) for route /mi-cursada', '1 ... /practicas/nueva'), timer intact, NRestarts=0 — so the operator WAS EXPLICITLY TOLD about rows the cursor then declared settled. Delivery of those summaries is UNVERIFIABLE BY CONSTRUCTION (void at the call site + synchronous delete), not merely unverified from mars. Attach that limit every time the line is used. SCOPE STATEMENT — RESTART IS NOT A REMEDIATION, ON ANY RAIL. flush() is TOP=8 plus an overflow COUNT, so a backlog discharge is ONE DM, bounded in volume AND LOSSY IN CONTENT: ayudarg's would read 'applog ayudarg: 774 real-time events', name eight, and the other 766 would appear nowhere. Standing: 'the rows will be paged on the next restart' delivers a count, not the events. CORRECTED (pmmaster ms7etiao1tm1, withdrawing their own wider wording): this does NOT make the rows unretrievable. They remain queryable in the DB — every strand tonight was found by querying it. The correct claim is that THE CURSOR IS THE RAIL'S ONLY DURABLE AUTOMATIC RECOVERY POSITION, and 318 is the change that governs the cursor. CODE DEPENDENCY: 318 is now blocked on VENUS-321 by CODE, not review order. The producer of the paged|disposed|deferred signal is dedupAllow, which is 321's function; 318 consumes the signal and must NOT open dedupAllow. If 321 slips, 318 cannot proceed ahead of it. VERSION DISCIPLINE, corrected once more and this is the standing form (pmmaster ms7etw4v6dgc, from coder-pluto-cc): ExecMainStartTimestamp proves WHEN the process read the file, not WHAT was on disk at that instant. A tsx-from-checkout service loads the WORKING TREE, and a working tree's historical dirty state is unrecoverable. So running(e874465, <unit>) is NOT establishable for ANY rail, mars included. Say: running(<=Jul-26 boot, <unit>), consistent-with e874465, dirty-state-at-boot unrecoverable. Note the form is not a stylistic preference — a caveat placed AFTER a claim is optional to the reader, a caveat placed IN it is not; the second cannot be shortened without visibly deleting words. Everything that failed tonight failed at a summarisation boundary. What survives this: coder-mars-cc's 13-row reconstruction, because it is evidence FROM BEHAVIOUR (batch size + exact cursor, over-determined) rather than from a version claim.
• wi-cli-venus note · 2w ago
  CORRECTION TO A CORRECTION I PROPAGATED INTO THIS WI MINUTES AGO — 'UNVERIFIABLE BY CONSTRUCTION, ON EVERY HOST, PERMANENTLY' IS WITHDRAWN (pmmaster ms7ewdw66q6i, withdrawing their own ruling after audit-venus-ca ms7evqif56ds). THE LIMIT IS CALLER-SIDE ONLY. void + the synchronous dedupMap.delete destroy the CALLER's evidence. HUB-SIDE LOGS AND RECIPIENT-SIDE EVIDENCE ARE UNTOUCHED AND COULD INDEPENDENTLY PROVE RECEIPT OF THE TWO ~18:12 MARS SUMMARIES. NOBODY HAS LOOKED. ADOPTED WORDING, verbatim, replacing every instance in this WI: 'caller-side delivery outcome irrecoverable by construction; no such external evidence has been checked.' WHY IT IS WORTH A SEPARATE EVENT RATHER THAN A QUIET EDIT: the withdrawn version did not merely overstate a scope, IT ATTACHED AN INSTRUCTION TO STOP LOOKING — 'a limit you can imagine closing keeps someone looking; this one must not.' Self-sealing: it took a caller-side gap, declared it universal, and forbade the check that would have falsified it. Every other narrowing tonight left the evidence reachable; that one told the fleet the evidence did not exist. Same shape as invoking Unverifiable-State one layer in — the failure that rule exists to prevent. Hub-side logs remain an OPEN, UNEXAMINED path for anyone who wants those summaries resolved. ALSO: the void/delete finding itself is git show e874465, behaviourally consistent with the mars reconstruction — NOT an exact running-version proof. Correct form here as everywhere: running(<=Jul-26 boot, <unit>), consistent-with e874465, dirty-state-at-boot unrecoverable.
• wi-cli-venus note · 2w ago
  IMPACT HALF OF THE RATIONALE LINE IS WITHDRAWN — see VENUS-322. I recorded 'the operator WAS EXPLICITLY TOLD about rows the cursor then declared settled' and 'the defect is not lost visibility'. FALSE: the two ~18:12 summaries were addressed to scrp-applog-mars, the emitter itself. VISIBILITY WAS LOST. The rationale line stands on the CODE (the row is pending and the record of its pendency is destroyed in the same tick); the claim that the rows' existence was nonetheless reported to a human does not.
• wi-cli-venus decision · 2w ago
  PM RULING — 318'S TERMINALITY PREDICATE IS UNSOUND AS DESIGNED, AND 318 GAINS A SECOND DEPENDENCY. Raised by coder-venus-cc (ms7f2hvuqcle), who flagged it rather than proposing a scope change; the call is mine and this is it. THE DEFECT: flush() at :551 does const sent = await postAlert(...) and advances the persisted watermark on sent === true, where SENT IS A HUB OK. Per the EVO-86 recipient distribution the hub returns ok WHILE ROUTING THE ALERT TO THE SENDER. So venus's cursor-advance criterion has always been 'the hub accepted it', never 'a human can read it' — and 318 is being designed to treat exactly that boolean as TERMINAL DISPOSITION. WHY THIS IS WORSE UNDER 318 THAN IT IS TODAY, which is the part that makes it a blocker rather than a note: today a bad advance is repeated — the rail keeps re-erring and a later batch can still page. A contiguous settled prefix built on `sent` would DURABLY RECORD AS PAGED every row that was delivered to scrp-applog-venus. 318 IS THE CHANGE THAT CONVERTS A REPEATED LOSS INTO A PERMANENT ONE. Building the correct cursor on an unsound predicate is strictly worse than the incorrect cursor we have. RULING: 1. VENUS-318 MAY NOT SETTLE A ROW ON `sent` ALONE. `paged` requires a RESOLVED-RECIPIENT-VALIDATED delivery, not a hub ok. This is a hard acceptance criterion, not a caveat. 2. 318 IS NOW DEPENDENT ON EVO-86 AS WELL AS ON VENUS-321. Both dependencies are on the PRODUCER of its disposition signal — 321 supplies per-row identity and confirmed-delivery branching, EVO-86 supplies what 'confirmed' is allowed to mean. Neither is optional and 318 starts after both. 3. postAlert must assert the resolved recipient is a human seat. pmmaster ruled that into EVO-86 scope rather than 319's; venus does not duplicate it here. But 318 CONSUMES it, so 318's acceptance criteria cite it. NOTE ON THE SHAPE, because it is the night's shape one more time and this time it was caught before the change shipped rather than after: the hub returns a recipients array on EVERY send. postAlert returns r.ok and drops it. THE FAILURE WAS OBSERVABLE ALL ALONG — not unobservable, just unread. Every 'verify delivery' instrument anyone proposes tonight should be checked against that: the information may already be in a return value nobody looks at.
• wi-cli-venus verified · 2w ago
  THE UNSOUND-PREDICATE RULING IS NOT A FORWARD RISK — IT ALREADY HAPPENED, AND THE TWO EVENTS ARE THE SAME EVENT. coder-mars-cc ms7f3v1sjmrb, two artifacts agreeing to the second: hub messages 51881 2026-07-29 18:02:08 scrp-applog-mars -> scrp-applog-mars 'applog mars: 2 real-time events [error] fetch failed /practi...' mars watermark file 2026-07-29T18:02:08Z (mtime) flush() posted that batch, THE HUB RETURNED OK WHILE ADDRESSING IT TO THE EMITTER, sent===true, writeWatermark() advanced. So the cursor recorded as PAGED an error-level alert that reached no human, AND STRANDED THE TEN /mi-cursada ROWS BEHIND IT IN THE SAME CALL. This changes the relation between the two defects: mars's cursor advance is not ADJACENT to the delivery defect, IT WAS CAUSED BY IT. rtPagedMap and the persisted watermark are both keyed on the same poisoned boolean. CONSEQUENCE, sharpened (coder-mars-cc, and it is the reason this sits in 318's BLOCKING set and not its notes): today the rail re-strands rows on every restart from a cursor it wrote wrongly — a repeated loss. A contiguous settled prefix built on sent would write DEFINITIVELY DISPOSED for rows nobody read, AND THAT IS THE ONE STATE NO LATER FIX CAN DISTINGUISH FROM A REAL PAGE. The WI cannot define terminal disposition on postAlert's current return value regardless of how good the prefix logic is.
• wi-cli-venus audit-finding · 2w ago
  HARD DOWNSTREAM REQUIREMENTS from audit-venus-ca PASS:3a42160 (ms7h5u2v9lkc). Not blockers on 323; blockers on 318. (1) 318 needs its OWN durable disposition ledger. rtPagedMap is INTENTIONALLY pruned once the cursor passes a row, so ABSENCE MUST NEVER MEAN VERIFIED — it means the cursor moved past. Goes in 318's spec as a sentence, adjacent to the terminality sentence (pmmaster ms7h53o9s25p). The two are the same trap from opposite sides: one says a settled row may be weakly settled, the other says an unrecorded row is not settled at all. (2) SCAN-BOUNDARY RACE, must be resolved before any gap-free-recovery claim. appEvents.createdAt DEFAULT now() is TRANSACTION-START time. A transaction can start before scanNow, remain invisible to the SELECT snapshot, then COMMIT after decideSweepAdvance/write; its NOTIFY has not reached `pending` yet. A crash in that interval leaves a row with createdAt <= cursor UNSEEN. The new pending check closes already-buffered NOTIFYs, NOT future commits from transactions absent from both snapshot and memory. 318 must either PROVE this cannot occur or use a DB/overlap boundary that survives it.
• wi-cli-venus spec · 2w ago
  pmmaster ruling ms7h6ewr9wbl — REQUIREMENT (2) CHANGES WHAT 318 IS. THE WATERMARK'S MONOTONIC-CURSOR MODEL IS NOT MERELY INCOMPLETE, IT IS UNSOUND AGAINST MVCC. A value ordered by createdAt cannot be a safe high-water mark when createdAt is assigned at TRANSACTION START and visibility is decided at COMMIT. Those are two DIFFERENT ORDERINGS. Every "contiguous settled prefix" argument 318 rests on assumes they are the same one. Mechanism: a txn begins before scanNow, is invisible to the SELECT snapshot, commits after the sweep decides and writes, carrying a createdAt ALREADY BEHIND the cursor. A crash in that interval loses the row PERMANENTLY — no cursor-based recovery can find it, because the cursor's own contract says everything at or before it is done. The pending check added in 3a42160 closes already-buffered NOTIFYs; it cannot close a future commit from a txn absent from both snapshot and memory. 318 therefore does NOT get to prove gap-free recovery on this model. It must either (a) prove the interleaving cannot occur — requires an argument about COMMIT ORDERING, not about the rail; pmmaster doubts it — or (b) change the boundary to one that survives it. Candidates for (b) belong to the DB LANE: commit-time-ordered sequence, xmin/snapshot-based boundary, or an explicit overlap window that re-scans BEHIND the cursor and relies on the disposition ledger for idempotence. (b)'s overlap variant is safe ONLY BECAUSE requirement (1) exists — re-scanning behind the cursor is a duplicate storm without a durable disposition ledger to suppress against. THE TWO REQUIREMENTS ARE ONE DESIGN. ROUTE: DB-semantics question FIRST, rail question second. coder-venus-cc takes it to the venus db lane BEFORE writing any 318 code, and brings pmmaster THE BOUNDARY CHOICE WITH ITS ARGUMENT — not the implementation. Better re-specified now than shipped as a proof that quietly assumes MVCC away.
• wi-cli-venus blocked · 2w ago
  SPEC HOLD, formal (pmmaster ms7h6zpu2mo8): VENUS-318 DOES NOT OPEN until the boundary question has an answer WITH AN ARGUMENT ATTACHED. Do NOT scope, estimate, or dispatch 318 in the meantime — a spec that exists gets worked on, and filing a soundness problem as a follow-on is how it becomes a footnote in a shipped design. Routing already done by pmmaster direct to coder-venus-cc -> venus DB lane. Deliverable back to pmmaster is THE BOUNDARY CHOICE WITH ITS REASONING, not an implementation and not a plan. Treat requirements (1) and (2) as ONE design, not two tickets. BUDGET NOTE: the model may not survive the answer at all. If createdAt ordering and commit ordering cannot be reconciled, 318 IS NOT A BOUNDED FIX TO A CURSOR — IT IS A DIFFERENT CURSOR. Plan for that rather than being surprised by it. Restart plan is separate and must carry: thaw order across the four rails, per-rail backlog volume, page-storm signature, abort condition. Ayudarg's 774 stranded rows ride in it and still have no owner (with Elazar).
• wi-cli-venus decision · 2w ago
  GATE 1 CLOSED (pmmaster ms7hm1xqurts). Boundary = xid8: insertXid < pg_snapshot_xmin(pg_current_snapshot()). Not an INFERENCE about commit ordering — it IS Postgres's visibility primitive, so boundary and cursor share ONE order and there is no interleaving left to argue about. This is a DIFFERENT cursor, not a patched one. DISCARDED PROPOSAL IS THE MORE VALUABLE HALF -> COMMONS: A PRIVILEGE-MASKED xact_start AND A GENUINELY QUIET DATABASE ARE THE SAME BIT PATTERN. applog_reader holds neither pg_read_all_stats nor pg_monitor, so min(xact_start) FROM pg_stat_activity returns NULL across 17 invisible backends, and the guard's own rule reads NULL as SAFE. A FAIL-OPEN guard whose failure mode is indistinguishable from the safe state — 4th appearance of that shape in this WI and the first that would have been undetectable from inside the rail forever. Found ONLY by measuring AS THE RAIL'S ACTUAL ROLE rather than as the owner: a permission-dependent query must be measured under the EXACT role that will run it; the owner's result is NOT evidence about the consumer's. Keeper (db-venus-cc): "confident wrong beats honest unknown, in the wrong direction." Bigint-identity demoted: identity is assigned at INSERT, so identity order is NOT commit order — same hole. Tiebreak WITHIN a boundary only, never the cursor unit. NEW PRODUCTION RISK, raised by neither lane: ADD COLUMN ... DEFAULT pg_current_xact_id() is a VOLATILE default -> ACCESS EXCLUSIVE on appEvents + FULL TABLE REWRITE. appEvents is the hot write path for every app action; that lock blocks every INSERT for the duration, and a blocked appEvents write is a blocked USER-FACING MUTATION, not a background delay. DO NOT RUN AS WRITTEN. Cutover shape to pmmaster before any DDL: nullable column, NO default (metadata-only) -> batched backfill -> set default for new rows; or an explicit maintenance window with a MEASURED rewrite time on 7744 rows + lock_timeout. 7744 is small enough the rewrite may be sub-second, in which case the safe path costs nothing to prove and the risky one is unnecessary. MEASURE; do not assume either way. Class-A schema change, full migration discipline. BACKFILL RELOCATION ACCEPTED: ADD COLUMN gives every pre-existing row the ALTER's SINGLE xid, so those rows carry NO ORDER AMONG THEMSELVES — cursor at 0 re-pages all 7744 in one boundary; cursor after the ALTER silently DISCARDS the backlog. Both are exactly the two failure modes the freeze exists to prevent (a sign the analysis is complete). One-time createdAt-bounded catch-up with cursorXid initialised to snap ONLY on completion is correct, and belongs in GATE 3 (thaw sequencing), not buried in gate 1. STANDING: gate 1 closed; gate 2 (318 durable disposition ledger) OPEN; gate 3 (thaw plan) now carries the cutover pass, the DDL shape, and the measured volumes. No restart, no DDL, freeze holds. Gate 3 must LEAD with the volume figure that contradicts what has been repeated tonight — a correction to a repeated number outranks the plan around it.
• wi-cli-venus spec · 2w ago
  GATE 3 THAW PLAN DRAFT (coder-venus-cc ms7hmbf7bpbf). Not executed. All numbers measured today. TWO CORRECTIONS TO THE SHARED PICTURE: (1) THE RAILS ARE RUNNING, NOT STOPPED. All four units active running since 2026-07-26 03:12:43, live InvocationIDs. The freeze is a no-RESTART freeze — correct — but it means all four are EXECUTING PRE-VENUS-323 CODE and the fix lands ONLY AT RESTART. Reading "frozen" as "stopped" has the blast radius backwards: the risk is at restart, and restart is the only way to get the fix. (2) THERE ARE FIVE RAILS, NOT FOUR. applog-listen@enamel.service is inactive dead — no unit instance — yet ~/.local/state/applog/enamel-rt-watermark.json was WRITTEN TODAY 10:54:59. Something ran that rail OUTSIDE SUPERVISION. Invisible to a list-units sweep, no restart policy, no owner. Flagged, not adopted. MEASURED BACKLOG (cursor from each state file; err/warn/fatal is an UPPER BOUND on pages, not a page count — downstream filters not modelled): pluto cursor 2026-07-30 05:02:49 16 rows after 0 pageable venus cursor 2026-07-15 10:52:47 1672 rows after 3 pageable mars cursor 2026-07-27 16:00:06 516 rows after 11 pageable ayudarg cursor 2026-07-27 09:25:34 781 rows after 776 pageable enamel (unsupervised) cursor 2026-07-30 10:54:58 6 rows after 0 pageable THE BRACED-FOR FIGURE IS 776 AND IT IS ALL ONE RAIL. Ayudarg is 70x the other four combined. Number correction: 774 -> 776 live. And VENUS's cursor is 15 DAYS stale, not 4 — the frightening-looking one — but worth 3 PAGES. Volume and staleness are on DIFFERENT rails; a plan written from memory gets that backwards. THAW ORDER: pluto (0) -> venus (3) -> mars (11) -> AYUDARG LAST AND NOT AT ALL YET. Enamel excluded until it has a unit and an owner. Ascending blast radius; ayudarg is the only rail where gate 2's absence is LOAD-BEARING — 776 pages is where a missing durable disposition ledger becomes a storm rather than a nuisance. STORM SIGNATURE: (a) sustained >1 page/sec or >20 pages in 60s; (b) any page whose event createdAt precedes process start by more than the initial lookback; (c) the SAME EVENT ID PAGED TWICE ACROSS A RESTART — the re-page defect audit-pluto-ca blocked, arriving in production. Plus THE FAIL-CLOSED TWIN THAT WILL NOT LOOK LIKE ANYTHING: the cursor silently ceasing to advance. OPEN, UNDIAGNOSED, BLOCKS VENUS'S THAW: venus's watermark HAS NOT BEEN WRITTEN since six seconds after boot on 07-26, while pluto's was written today. Four days, three pageable events, ZERO state writes. Not called benign. Belongs IN FRONT OF venus's thaw, not after. ABORT: systemctl --user stop applog-listen@<app> on (a)/(b)/(c), or if the watermark has not advanced within two sweep intervals of a page. DO NOT DELETE THE STATE FILE TO RECOVER — a deleted state file is a FRESH RAIL and a fresh rail RE-PAGES. That is the entire content of the last BLOCK. BLAST RADIUS, one sentence: worst case is 776 DMs into ayudarg's PM seat — the same rail whose seat resolution was already measured wrong (delivering to db-ayudarg-cc) — so thawing it today points 776 pages at a seat nobody has claimed. AYUDARG STILL HAS NO OWNER. coder-venus-cc is not it and will not thaw it; requesting owner assignment as a PRECONDITION rather than carrying it silently. GATING: gate 1 RULED (xid8), gate 2 OPEN (318 ledger), plus the xid8 cutover's one-time backlog pass (coder's, unspecified). No rail restarts on this message.
• wi-cli-venus spec · 2w ago
  CUTOVER SHAPE ACCEPTED AS SPEC (pmmaster ms7ho5jai9z2). The correct move was to REMOVE THE REWRITE ENTIRELY, not measure its lock window: NOT NULL + volatile default is what forces the rewrite; nullable ADD COLUMN then SET DEFAULT is CATALOG-ONLY in both statements. Do not spend a rewrite proving the timing of a path we are not taking. Three ratified properties: 1. BOTH statements in ONE txn — separately, rows inserted between them take NULL and are misfiled as pre-cutover. 2. lock_timeout before it — the catalog update still takes ACCESS EXCLUSIVE, so queuing behind a long reader blocks every appEvents INSERT, i.e. blocks USER-FACING MUTATIONS. Fail-and-retry beats stall. 3. NO BACKFILL, BECAUSE NULL IS THE CORRECT VALUE AND NOT A GAP. NULL = pre-cutover; those rows belong to the one-time createdAt-bounded catch-up; insertXid >= cursorXid AND insertXid < snap excludes NULL automatically, so the two consumption paths PARTITION the table — no overlap, no seam to police. This also DELETES the gate-1 ALTER-xid problem: no ALTER-xid block exists, so neither re-page-7744 nor discard-the-backlog can arise. KEEP THE FRAMING: A DESIGN WHERE THE AWKWARD CASE STOPS EXISTING BEATS ONE WHERE IT IS HANDLED. Nit (preference, not correction): CREATE INDEX CONCURRENTLY ... WHERE "insertXid" IS NOT NULL — pre-cutover rows index as NULL under a plain btree and are never selected by the boundary predicate, so a partial index is smaller and strictly sufficient. Keep the full index instead if future queries on the column want it; state the reason. FIVE-DB SCOPE FINDING RESIZES GATE 1. The rail is ONE FILE; the column is FIVE SCHEMAS. venus/mars/pluto/ayudarg/enamel each hold their own appEvents, so GATE 1 IS CLOSED FOR VENUS AND OPEN FOR FOUR RAILS. pmmaster's "gate 1 ruled" was a claim about the DESIGN that read as a claim about the FLEET. Record corrected: the boundary is decided; its DEPLOYMENT is four DB lanes wide, none of them venus's. THAW ORDER FALSIFIED ON A SECOND AXIS: pluto-first is right on VOLUME and WRONG ON REACHABILITY — only venus has the boundary. Revised sequencing is VENUS-FIRST-OR-NOTHING; the other three lanes need the DDL specced by their OWN db agents before their rails can thaw at all. Bigger coordination surface than the thaw plan assumed — goes to pmmaster as a SCOPE, not as four dispatches; he routes. AYUDARG now blocked on THREE things: no owner, 776 pages at a seat measured wrong, no DB lane to run this DDL. Stays frozen; owner question escalated to Elazar in those terms. STANDING: no DDL, no restart. db-venus-cc spec is FOR REVIEW ONLY — executes once the sweep logic is final, which it is not. GATE 2 (318's durable disposition ledger) IS NOW THE LONG POLE. Venus's four-day-silent watermark remains a precondition ahead of venus's own thaw.
• wi-cli-venus decision · 2w ago
  CUTOVER RULING (pmmaster ms7hosp4kihz) — two proposals crossed in flight. coder-venus-cc's NULLABLE shape ships. db-venus-cc's measurement accepted and it KILLS pmmaster's own argument for the nullable shape — but that argument was never the strongest one. MEASUREMENT (db-venus-cc): 3.392ms on a real clone, 4936 kB total, 652 bytes/row. BYTES govern rewrite cost; "7746 rows" was the wrong unit and pmmaster was the one repeating the row count. Three orders of magnitude under sub-second => the LOCK-DURATION objection is DEAD. BUT THE NULLABLE SHAPE DOES NOT WIN ON LOCK DURATION. IT WINS BECAUSE NULL IS A MEANINGFUL VALUE IN IT. Under the rewriting form every pre-existing row gets the ALTER's ONE xid, so 7746 rows carry NO ORDER AMONG THEMSELVES and the cursor has exactly two starting positions: below that xid (re-page all 7746) or above it (silently DISCARD the backlog) — the two failures this freeze exists to prevent. Under the nullable form NULL MEANS pre-cutover, insertXid >= cursorXid AND insertXid < snap excludes NULL automatically, and the two consumption paths PARTITION the table with no seam. THE AWKWARD CASE STOPS EXISTING RATHER THAN BEING HANDLED. Correctness argument, not a performance one. Consequence: "the safe path costs real complexity — two-step migration, batch backfill, second DDL" is NOT the trade here. The nullable form has NO BACKFILL AND NO BATCH LOOP AT ALL: ADD COLUMN nullable + SET DEFAULT, both catalog-only, both in one txn, nothing ever backfilled. SIMPLER than the rewriting form, not more complex. That read came from assuming nullable implies a backfill to fill it in; HERE THE NULLS ARE THE ANSWER. SCOPE CORRECTION TO THE MEASUREMENT, matters if anyone later reasons from the number: "indexes aren't touched by ADD COLUMN regardless" is FALSE FOR A REWRITING ADD COLUMN — a table rewrite REBUILDS ALL INDEXES. The clone had NO indexes, so 3.392ms measures a HEAP-ONLY rewrite and omits ~1256 kB of index rebuild the live table would also do. At this size the conclusion holds, but THE NUMBER IS A FLOOR, NOT THE FIGURE, measured on a table shaped unlike the target in exactly the dimension excluded. Night's shape again: a sound measurement of a NARROWER OBJECT than the one asked about. ADOPTED REGARDLESS OF SHAPE: (a) lock_timeout defensively — better reasoning than pmmaster's: not because the lock is long, but because ACCESS EXCLUSIVE briefly queues writers and something unexpected holding a lock should FAIL LOUD rather than block user-facing mutations; (b) low-traffic timing as standard practice, for the QUEUE-EMPTIER reason not the lock-length reason; (c) RE-MEASURE SIZE IMMEDIATELY BEFORE RUNNING rather than trusting tonight's number if gate 3 lands later — correct instinct on a growing table, generalises past this DDL. STANDING: SPEC, NOT A GO. Nothing executes until the sweep logic is final and gate 2 lands. DDL is PER-DB ACROSS FIVE DATABASES; gate 1 closed for VENUS ONLY. mars/pluto/ayudarg/enamel each need their own db lane, routed by pmmaster as a SCOPE not four dispatches.
• wi-cli-venus note · 2w ago
  GATE 1 RESCOPED AS A FIVE-DB MIGRATION PROGRAMME, NOT A VENUS DECISION WITH FOUR FOLLOW-ONS (pmmaster ms7i6phn26w9). NOT STARTED. Explicitly NOT on VENUS-324's path. MEASURED (coder-mars-cc): NO xid, insertXid, or xmin-materialising column exists on mars appEvents — a %xid%/%xmin% filter returns nothing. So "complete the xid-aware contract on every DB first" is NEW DDL IN db-mars-cc's LANE, a scheduled migration against already-serialized numbering — NEVER a same-change fix. pm-mars-cc + db-mars-cc: this does not start now. STATE: venus is the ONLY DB where the boundary is decided; mars measured and BLOCKED; pluto and ayudarg UNREAD (coder-pluto-cc owns pluto's); enamel excluded. THE MVCC PREMISE IS NOT VENUS-LOCAL — CONFIRMED ON A SECOND DB BY A SECOND ROUTE. mars public."appEvents"."createdAt" is timestamptz DEFAULT now(), and now() IS transaction_timestamp() — transaction-START, stable for the whole transaction. audit-venus-ca's schedule is constructible on mars: a tx starting T+30s, uncommitted when a sweep takes scanNow > T+60s, committing after, lands createdAt = T+30s behind an already-settled boundary — INVISIBLE TO THE SWEEP THAT SETTLES IT, AND VISIBLE TO NOBODY AFTERWARD. scanNow settlement is unsound on mars too, and mars is one of the rails that would have executed it. THE CORROBORATION IS WORTH MORE THAN A RE-READ BECAUSE IT CAME FROM THE ORDERING SIDE, NOT THE VISIBILITY SIDE: mars's earlier independent finding that all rows written in one transaction share an identical createdAt is THE SAME FACT WEARING DIFFERENT CLOTHES. Two DBs, two routes, ONE property — corroboration in the sense the dilution rule demands, not a second instrument that could not fail. COVERAGE LIMIT CARRIED AND THE REFUSAL ENDORSED: the interleaving was NOT constructed live — that needs two concurrent sessions against prod appEvents, and declining is right. A property confirmed by SCHEMA plus an INDEPENDENT ORDERING OBSERVATION does not need a live race constructed against production to be actionable.