Venus-consolidation: unified service on host venus for Vercel/Supabase background work

Sub-tickets — 2/2 done

Activity

• wi-cli-venus created · 2026-08-16
• wi-cli-venus assigned · 2026-08-16
  coder-venus-cc
• wi-cli-venus note · 2026-08-16
  POC selection rationale (Elazar directive via db-pluto-cc): assigned coder-venus-cc, kept as-is not reassigned. Checked maintainer registry SSOT (v_maint_active, host=venus) for all candidates already running venus services. Strongest structural match: coder-venus-cc's applog-pull@{mars,pluto,venus} fleet — live systemd timer units already implementing exactly the target shape (venus-side scheduler hitting each project's authed endpoint), which is the identified migration path replacing the pg_cron */5 pollers. Considered and rejected: coder-enamel-cc (telemetry-ingest is inbound-only, not a scheduler analog), coder-pluto-cc (pluto-authconfig-drift.timer is single-project), pm-llmmsgsrv-cc (explicitly out of scope per Elazar), db-venus-cc/evolutiva-dbperf@venus (periodic collector, not a cross-project HTTP scheduler).
• wi-cli-venus descriptionChanged · 2026-08-18
  was: Elazar directive 2026-08-16: analyze and migrate cron-like/background work now running in Vercel and/or Supabase across all projects (mars, pluto, venus, terra, enamel, ...) to an existing or new unified app on host venus (flat-rate VPS, 24/7, caddy+postgres, runs nodejs/rust/go/python/etc). Trigger: pluto Supabase Disk IO Budget email 2026-08-15; measured evidence: the */5min email-outbox pollers are the dominant non-domain WAL/IO churn on BOTH pluto (job_run_details #2 WAL, 36MB) and venus (#1 WAL, 104MB, unpruned). Analysts: coder-venus-cc + db-venus-cc (pmmaster structure); other projects feed inventories, project coder/db agents verify. Strongest item: replace both */5 pg_cron outbox pollers with a venus-side scheduler hitting the existing authed drain endpoints (zero app change; auth shapes differ: venus POST+Vault bearer vs pluto GET+cron_auth_header). POC agent: TBD - being selected from maintainers of apps already running on host venus. Assignment to coder-venus-cc provisional until POC chosen.
• coder-venus-cc note · 1w ago
  STATUS CHECK for pmmaster before a GO (pmmaster-evolutiva-cc-muukh9szspwo). Measured on the live venus cluster tonight, not recalled. THE ROW'S STRONGEST ITEM IS STALE FOR VENUS. It names "replace the */5 pg_cron outbox pollers"; venus's no longer exists. VENUS-396 moved the email drain off pg_cron entirely (0a23afe, d0884ae). fn_drainEmailOutboxGuard still EXISTS as a function and still contains net.http, but NO cron job calls it any more. WHAT IS ACTUALLY LEFT ON VENUS, from cron.job: 8 jobs, and only TWO are DB-resident outbound HTTP - jobid 15 fn_applogHeartbeatWatchdogGuard at 15 * * * * and jobid 16 fn_drainWatchdogGuard at */15. Both test positive for net.http in pg_get_functiondef. The other six are DB-local SQL (appEvents/dbperf/job_run_details retention) and are explicitly OUT of this row's migration set. AND THE TICK COUNT IS NOT A CALL COUNT. 100 scheduled ticks/day between the two, all "succeeded" in cron.job_run_details, but a guard that returns early on an empty queue logs succeeded without issuing a request - so the outbound volume is UNMEASURED and is an upper bound of 100, probably far below it. Do not let a child item justify itself on the tick count; that is the same move the withdrawn IO sentence made. NEXT CONCRETE INCREMENT, and it is venus-only and unblocked: move jobs 15 and 16 to venus-side systemd timer instances hitting the same authed endpoints - the applog-pull@{mars,pluto,venus} shape I already own and the reason this row was assigned to me. Zero app change, no cross-project credential, and it produces the reference implementation every other leg copies. THEN delete the Vault bearer and stop net use for those two paths, because dropping the cron job alone leaves the database still holding the secret and still able to call out - the architecture justification is only half-delivered by an unscheduled job, and that half is the half this epic is sold on. WHAT IS BLOCKING IT THAT IS NOT VISIBLE ON THE ROW - the reason it has sat six weeks: 1. NO SEAT IS NAMED TO ISSUE THE CREDENTIALS, and I cannot issue them. Every non-venus leg means venus holds that app's drain bearer, and a remote app cluster's credentials are provisioned by that project's own db-*-cc - not by me, not by nw-venus-cc. The row names analysts and a POC but nobody who hands over a secret. Precedent exists and is worth citing: repage-sweep@ already holds mars and pluto DSNs on venus (terra/enamel DSNs in hand, not wired), so the pattern is proven for DSNs, not for bearers. 2. THE COORDINATING SEAT CANNOT VERIFY ITS OWN INPUTS. I am host-bound and DB-bound: I cannot query mars, pluto, terra or enamel to confirm any cron inventory. So every non-venus line in this epic is another seat's report, which is exactly the channel the withdrawn IO claim travelled on when it propagated into three lanes as established fact. A fleet sweep assigned to one seat reproduces that shape. 3. A TICK MOVED ONTO VENUS NEEDS ITS OWN HEARTBEAT, not just an alert. A dead venus timer and a healthy idle one are identical from the app's side, and the app loses the only evidence it had (a pg_cron row) the moment the job leaves the database. 4. MARS-378 is already named here and still blocks the mars leg: jobid 6 inlines CRON_SECRET as plaintext in cron.job.command, which is a live problem the moment another host holds that credential. RECOMMENDED GO SHAPE: GO the venus-only increment above as the POC and keep each other app's leg behind its own db seat issuing a credential to venus. Do not GO a fleet sweep to this seat - it cannot verify a single non-venus fact in it.
• coder-venus-cc note · 1w ago
  GO RECORDED from pmmaster-evolutiva-cc (pmmaster-evolutiva-cc-muukpzkdfwgt): the VENUS-ONLY increment is authorized as the POC - move pg_cron jobs 15 and 16 to venus-side systemd timers in the applog-pull@ shape, then delete the Vault bearer and stop net use for those paths. A fleet sweep from this seat is explicitly NOT authorized. THE TWO BLOCKERS, recorded on the row at pmmaster's instruction so the next agent does not re-walk them. They are the reason this epic sat six weeks and neither is visible from the description. 1. NO SEAT IS NAMED TO ISSUE THE PER-APP BEARERS, and the seat this row is assigned to cannot issue them. Every non-venus leg means venus holds that app's drain bearer, and a remote app cluster's credentials are provisioned by that project's own db-*-cc - not by coder-venus-cc and not by nw-venus-cc. The row names analysts and a POC and no credential issuer. Precedent that narrows it: repage-sweep@ already holds mars and pluto DSNs on venus (terra/enamel DSNs in hand, not wired), so cross-project credential placement is proven for DSNs - but a DSN is not a drain bearer and the Vault-held bearers have no established handover path. 2. THE COORDINATING SEAT CANNOT VERIFY A SINGLE NON-VENUS FACT IN THIS EPIC. coder-venus-cc is host-bound and DB-bound: mars, pluto, terra and enamel refs are not queryable from this seat, so every non-venus cron inventory here is another seat's report. That is exactly the channel the withdrawn IO claim travelled on when it propagated into three lanes as established fact - and it is why the fix is per-app legs owned by each app's own seats, not one sweep. 3. A SMALLER ONE WORTH HAVING ON THE ROW: a tick moved off pg_cron onto venus needs its own HEARTBEAT, not only a failure alert. A dead venus timer and a healthy idle one are identical from the app's side, and the app loses the evidence it did have - a cron.job_run_details row - the moment the job leaves the database.
• coder-venus-cc commit · 1w ago
  VENUS LEG BUILT AND PUSHED: 4c97173 on origin/main, 7 files, 232 insertions, scripts/systemd/{cron-tick.sh,crontick-failnotify.sh,venus-crontick@.service,venus-crontick-failnotify@.service,venus-crontick@.timer} plus a cadence drop-in per instance. Nothing is installed, armed, or unscheduled; no pg_cron job and no Vault secret was touched. I AM CORRECTING MY OWN MEASUREMENT ON THIS ROW, from the note at 01:30:48Z. I wrote that the pg_cron tick count was an UPPER BOUND because the guard returns early on an empty queue. That is true of fn_drainEmailOutboxGuard and FALSE of jobs 15 and 16. pg_get_functiondef on fn_drainWatchdogGuard and fn_applogHeartbeatWatchdogGuard shows the net.http_post is UNCONDITIONAL - the only early return is when the Vault secret venus_cron_drain_bearer is absent - and net._http_response holds 30 rows, all status_code 200. So this is ~100 real HTTP calls a day leaving the database, not a ceiling, and the migration removes real net use rather than hypothetical net use. The wrong reading made the payoff look smaller than it is. FORCED RED, not sampled green. Two prod success cases (drain-watchdog 200 coverage=fresh ageMs=103217; applog-watchdog 200 coverage=fresh ageMs=4864755) and five must-fail controls, each with its own distinct exit so the failures are distinguishable in the journal: bad bearer -> exit 22 plus the route own 401 body; CRONTICK_VENUS_BASE_URL unset -> 78; VENUS_CRON_SECRET unset -> 78 (it refuses rather than falling through to another app secret); malformed instance -> 64; nonexistent endpoint -> 22 on a 404. THE ALERT PATH WAS PROVEN BY SENDING. crontick-failnotify.sh hand-run with DELIBERATE_BREAK=1 delivered to coder-venus-cc for both instances and named venus-crontick@venus-drain-watchdog.service and venus-crontick@venus-applog-watchdog.service correctly; the immediate re-fire suppressed on the 900s cooldown with suppressed=1; the other instance was NOT suppressed by the first state file, which is the per-instance keying working; a missing CRONTICK_INSTANCE fails closed instead of paging about an unnamed unit. This is also why the file is not applog-failnotify.sh: that one hardcodes UNIT=applog-${SOURCE_LEG}@${APPLOG_APP}.service and would have paged a correct alert about a unit that does not exist. BLOCKER 3 FROM MY OWN LIST IS ONLY HALF ANSWERED AND I AM NOT CLAIMING OTHERWISE. OnFailure= covers a failing tick. ABSENCE of a tick still reports as nothing - a dead timer and a healthy idle one are identical - and the app loses the cron.job_run_details row it did have. So the handoff below asks for systemctl --user list-timers evidence, not an enable confirmation, and a heartbeat for the host leg stays open on this row. TWO HALVES ARE NOT MINE and are filed as children: install+arm (nw-venus-cc, ~/.config/systemd/user is their lane) and cron.unschedule + Vault secret deletion (db-venus-cc, I hold SELECT only). THE ORDER IS LOAD-BEARING: host leg armed and VERIFIED RUNNING first, DB leg unscheduled second. Both legs ticking in the overlap is correct and harmless because these are idempotent watchdog reads; the reverse order leaves the watchdogs silent, and silent reports exactly like healthy.
• coder-venus-cc note · 1w ago
  SCOPE CORRECTION ON THE "STOP NET USE" GOAL, from the EVO-109 work. Jobs 15 and 16 are unscheduled and the host leg is carrying both watchdog ticks, but venus has NOT stopped making HTTP calls out of the database. trg_emailOutboxWake on public."emailOutbox" (AFTER INSERT, FOR EACH STATEMENT, tgenabled=O) calls net.http_post on every outbox INSERT via fn_emailOutboxWakeTrigger, reading the same venus_cron_drain_bearer Vault secret. It is not a cron job, so migrating pg_cron jobs - which is how this epic is framed - cannot reach it, and that framing is why it was invisible here until the secret deletion tripped over it. IT IS A LATENCY ACCELERATOR, NOT A TRANSPORT: evolutiva-emaildrain@venus.timer already drains every 5 minutes (*:00/5, LAST 02:15:26 UTC). Removing the trigger costs up to 5 minutes, ~2.5 average, on every venus email including password resets. That is a user-visible trade and needs a GO; it is parked on EVO-109 with the measurement, not done. THE GENERAL LESSON FOR THE REMAINING PER-APP LEGS: an inventory that enumerates cron.job will MISS the net.http_post callers that are triggers, and each one of those is also a Vault-secret holder, so "delete the bearer" is the step that discovers them. Enumerate by scanning pg_proc.prosrc for net.http_post, which is how db-venus-cc found 4 readers where the row expected 3 - not by listing scheduled jobs.
• coder-venus-cc verified · 1w ago
  VENUS LEG RE-VERIFIED LIVE TONIGHT (2026-10-06 00:2x UTC), not recalled from the prior close. On the venus cluster: a pg_proc.prosrc scan for net.http_post or net.http_get returns exactly ONE object fleet-wide on this DB, extensions.grant_pg_net_access, which is INVOKER and posts nothing; pg_proc has 0 rows named fn_emailOutboxWakeTrigger; the only non-internal trigger left on public."emailOutbox" is trg_emailoutbox_updatedat; and a join of pg_trigger to pg_proc for any trigger function whose body mentions net. returns NONE. So the venus database makes no outbound HTTP call by any path - scheduled, triggered or orphaned-SECDEF. THE ITEM I PARKED ON 2026-10-05T02:19 IS CLOSED AND HAD ITS GO: that note said trg_emailOutboxWake was still live and that removing it was a user-visible latency trade (up to 5 min, ~2.5 avg, on every venus email including password resets) needing authorization. It was authorized - option A from pmmaster-evolutiva-cc, tag pmmaster-evolutiva-cc-muumg62ibmvs - and applied as venus86 (2fa4b4f), with the three orphaned SECDEF guards dropped afterwards as venus87/VENUS-430. I am recording this because the park note is still the most recent narrative event on this epic and reads as open work; it is not. THE STRONGEST REMAINING ITEM OF THIS ROW IS BEING EXECUTED ON ANOTHER EPIC, which is the disposition question I am raising rather than deciding. This description names "replace the */5 pg_cron outbox pollers with venus-side PER-APP systemd timer instances" as the strongest item; that is EVO-93 (EPIC: move app email drains off Supabase pg_cron to venus systemd timers), status inProgress, assignedAgent coder-mars-ca, with per-app children already filed and assigned - PLUTO-733 (blocked, coder-pluto-cc) is the pluto leg verbatim. Starting that work from EVO-86 would duplicate an in-progress epic held by another seat. WHAT IS STILL UNEXERCISED AND NOT A CODE GAP, carried from 2fa4b4f so it is not lost: the post-change venus email path has never been exercised. emailOutbox holds zero rows and the newest appEvents row with action outboundEmailSent is 2026-09-26. A drain over an empty queue and a working send are indistinguishable from the timer exit code, so "venus email still works after the accelerator was removed" is verified STRUCTURALLY ONLY. The first real venus email is the test. NOT VERIFIED BY ME, unchanged: the Vault deletion. My role has has_schema_privilege FALSE on schema vault, so the secret being gone rests on db-venus-cc own read; re-checking it needs a role with vault USAGE. Also blind on cron.* entirely from this DSN.
• pmmaster-evolutiva-cc completed · 1w ago
  Superseded by EVO-93 (per-app systemd timers replacing pg_cron outbox pollers, inProgress, coder-mars-ca) on the remaining migration scope. Venus leg done+verified: no DB outbound HTTP call (prosrc scan, extensions.grant_pg_net_access only, INVOKER, posts nothing), fn_emailOutboxWakeTrigger removed, applied venus86/2fa4b4f + venus87. PM decision 2026-10-05 per coder-venus-cc's report — avoids duplicating EVO-93's live epic.