basquetWi + New ticket
evolutiva EVO-97

Cada proyecto serializa sus propios pushes y ninguno serializa contra los otros: cuatro gates de pre-push en 4 vCPUs llevaron a venus a load 47 y ningun proyecto pudo publicar

Done urgent bvbin-venus-cc

GO given by pmmaster-evolutiva-cc on Elazar behalf 2026-09-11. p0: this blocked every project from reaching production. ASSIGNED bin-venus-cc. RULING ON SHAPE, asked for and answered: make it DURABLE IN THE PUSH TOOLING, not an operational hold. I am running the hold today; a convention decays the moment attention moves. The gate must not be able to start unless the lock is held, enforced by the push path itself. MEASURED, not reported. 2026-09-11 13:08 UTC: load average 44.60 on 4 vCPUs (20 -> 32 -> 47 across the morning), 12 of 15 GB used, THIRTEEN build processes concurrent across three projects - terra eslint + tsc + check-migration-citations, pluto eslint + next build, venus tsc - alongside 31 claude sessions. Fell to 35.59 within two minutes of ordering lanes to stop starting new gates, and to 18.90 within ten. MECHANISM AS FIRST DIAGNOSED: each project serializes its own pushes with its own flock (mars-push.lock, venus-push.lock, pluto-push.lock) and there is NO cross-project lock, so N projects run pre-push gates simultaneously on the same four cores. Every individual lock is correct; the COMPOSITION fails, and no project can see it from inside itself. SECOND MECHANISM, FOUND LATER THE SAME HOUR BY db-pluto-cc AND IT IS WORSE: the per-project lock is not enforced EITHER. Running gitpush.sh RAW does not take ~/.cache/evolutiva/pluto-push.lock - only the wrapper does. db-pluto-cc ran it raw while coder-pluto-cc held that lock for the eq#61 push, so TWO full pre-push suites fought over ONE WORKING TREE at load 39.8. Through flock it passed first try. So the design must not assume the existing per-project locks hold; an unwrapped invocation bypasses them silently. THE COST THIS ALREADY IMPOSED, both measured: four consecutive mars push attempts failed, three timing out inside the vitest leg after 2-10 minutes with NO OUTPUT, leaving mars 4 commits ahead of origin at v2.79.47 with three of those being version bumps that published nothing. And db-pluto-cc was about to escalate a "pre-existing test timeout" that was contention, not a defect - a false defect report is the second-order cost of a capacity problem nobody owns. TWO CONSTRAINTS ON THE FIX, both binding. (1) An aborted gate must NOT commit a version bump. A failed push leaves the tree as it found it; the three published-nothing mars commits came from exactly that. (2) Whatever you build needs the TTL property the hub advisory lock already has, so one wedged project cannot block the fleet indefinitely. Design is yours - host-wide flock, the hub lock called from gitpush.sh, or a concurrency ceiling on the gate workers - state which and why. INTERIM CONVENTION IN FORCE until this lands, announced to aro:evolutiva-management: take the hub advisory lock on resource venus-host.push-gate before any local gate, release after the push lands or aborts. It 409s with the holder, DMs the waiter on release so nobody polls, and expires a wedged holder. Extend only from real steps in the work, never on a timer - a timer renews while wedged. Silent holder: lock_steal, never a second gate. Observed working the same hour: coder-mars-cc held it, db-enamel-cc took a 409 and registered for the release DM rather than polling. DIAGNOSTIC RULE THAT CAME OUT OF THIS AND SHOULD SURVIVE THE FIX: NO OUTPUT IS NOT EVIDENCE OF A HANG. The silent vitest was misdiagnosed as a hang caused by a pg 8.23.0 bump and killed and restarted three times - on a starved box the worst available move, since each restart added load. The control test cleared pg (reverting to 8.20.0 hung identically); the variable was always the machine. uptime before kill is the cheap discriminator and was not run until the fourth attempt. THE REAL CEILING IS HOST CAPACITY, not scheduling: 31 claude sessions plus N build toolchains on 4 vCPUs. Serialization buys time, it does not add cores. Say so plainly in whatever you deliver so nobody reads a working lock as headroom. STRICT REPO: gitpush.sh --only, never add -A.

Sub-tickets

No sub-tickets.
+ Add sub-ticket

Activity

  • wi-cli-venus created · 5w ago
  • wi-cli-venus descriptionChanged · 5w ago
    was: GO given by pmmaster-evolutiva-cc on Elazar behalf 2026-09-11. p0 because it is blocking every project from reaching production right now, not because the code is hard. MEASURED, not reported. 13:08 UTC: load average 44.60 on 4 vCPUs, climbing all morning per coder-mars-cc at 20 then 32 then 47. Memory 12 of 15 GB used, 2 available. Thirteen build processes running CONCURRENTLY across three projects - terra eslint plus tsc plus check-migration-citations, pluto eslint plus next build, venus tsc - alongside 31 claude sessions. MECHANISM. Each project serializes its own pushes with its own flock - mars-push.lock, venus-push.lock and so on. There is NO cross-project lock. So N projects run their pre-push gates, each of which spawns vitest, tsc and lint workers, simultaneously on the same four cores. Every individual lock is working correctly; the composition is what fails, and no single project can see it from inside itself. MEASURED CONSEQUENCE. Four consecutive mars push attempts failed, three timing out inside the vitest leg after 2-10 minutes with NO OUTPUT. Each aborted attempt still bumped package.json and COMMITTED, so mars sits 4 commits ahead of origin at v2.79.47 with three of those being version bumps that published nothing. The work - pg unhold plus migrations 146, 147 and 148 - is done and reviewed and cannot reach production. SECOND-ORDER DAMAGE WORTH RECORDING, because it will recur: the silent vitest was misdiagnosed as a hang caused by a pg 8.23.0 bump, and killed and restarted three times. On a starved box that is the worst available move - each restart added load. The control test cleared pg, since reverting to 8.20.0 hung identically, so the variable was always the machine. NO OUTPUT IS NOT EVIDENCE OF A HANG. uptime before kill is the cheap discriminator and it was not run until the fourth attempt. INTERIM RULE ALREADY IN FORCE, announced to aro:evolutiva-management: take the hub advisory lock on resource "venus-host.push-gate" before running a gate, release after the push lands or aborts. It 409s with the current holder, it DMs the waiter when the resource frees so nobody polls, and its TTL expires a wedged holder. Extend only from real steps in the work, never on a timer, because a timer renews while wedged. That is a convention, not an enforcement - which is exactly why this row exists. WHAT THIS ROW WANTS: the serialization moved into the push path itself so it does not depend on every agent remembering. gitpush.sh is a sh.git PATH script and yours. Whether that is a host-wide flock, the hub lock called from the script, or a concurrency ceiling on the gate workers is your design call - say which and why. TWO THINGS THE FIX MUST NOT DO. It must not let an aborted gate commit a version bump; three published-nothing commits on mars came from exactly that, and a failed push should leave the tree as it found it. And it must not serialize so hard that one wedged project blocks the fleet indefinitely - whatever you build needs the TTL property the hub lock already has. STRICT REPO: use gitpush.sh --only, never add -A.
  • bin-venus-cc note · 1w ago
    SHIPPED cc3037c 2026-09-11, then SILENTLY REGRESSED: 7cf82ee (2026-09-20, 'gitpush.sh v5.0: trim 2810 -> 308 lines') deleted the gate's entire call site. gitpush.sh v5.2 had zero references; gpc_push_gate_acquire/release stayed intact in gitpush-core.sh with NO caller except the test suite, which calls the library functions directly and never drives the script - so 16 green checks survived the removal of a p0 control for two weeks. RESTORED eb125d9: gitpush.sh v5.3 (acquire + EXIT trap at cc3037c's placement - after the arg/strict guards, before the first index mutation, bump and commit), gitpush-core.sh v1.7 (gpc_hub_creds, identity from agent_trailer_resolve per MSG-320, overlay lookup deliberately two places so /tmp fixtures cannot reach the real bearer), gitpush-gate-test.sh v1.4 case [13] which drives the SCRIPT end to end and was forced red against v5.2 first, gitpush-onlystage.test.sh v1.5 (its hub-neutralisation note described a control that had not run since v5.0). Gate suite 20/20; onlystage, bumporder, scoped-anyrepo, gitpushrepo, gitcommit-gates all pass. The push that shipped it was refused twice by the restored gate.
  • bin-venus-cc completed · 1w ago
    completed
task
5w ago by wi-cli-venus
1w ago
2026-10-05 00:27