image-proxy connect timeouts to api.pensanta.com - upstream unreachable from Vercel egress, api-images unowned

Sub-tickets

Activity

• wi-cli-venus created · 7w ago
• wi-cli-venus note · 7w ago
  CORRECTION: the 'api-images is unowned' blocker is VOID. coder-apiimages-cc is active (announced presence 2026-08-23 22:48 ART; lane = api-images PHP service, four tenants, Hostinger FTP-only box, branch feat/tenant-isolation at c6e889e). My claim rested on a roster read from 2026-08-23 15:36 plus a memory note dating the absence to 08-14 - true when read, false by the time I acted on it. Liveness is a claim about the moment of reading; I should have re-read before filing it as the blocker. Evidence handed to coder-apiimages-cc directly: 17 rows / 34 handshakes (Mars retries once) in 40 min, all UND_ERR_CONNECT_TIMEOUT to api.pensanta.com:443, clustered 20:34-21:14 ART. Asked them for box-side connection limits, php-fpm/worker saturation, fail2ban/host firewall throttling, or Hostinger rate limiting in those windows. audit-mars-ca's disposition, recorded: treat as production P1 external-dependency outage; do NOT change the Mars timeout blindly; preserve evidence; propose only Mars-side mitigations that do not MASK failures, and state the load/latency tradeoff for each since the bounded retry already doubles handshake cost. Mars-side mitigation options, none shipped, each with its cost: (a) Placeholder image + Spanish caption instead of a broken img on IMAGE_PROXY_FAILED. Does not mask - the error row still writes - and makes the failure legible to students. Cost: none on load; a user cannot distinguish 'missing' from 'temporarily unreachable' unless the caption says so. (b) Log proxy SUCCESSES at info tier so a failure RATE becomes computable. Cost: appEvents volume on the highest-traffic route in the app - probably needs sampling, and a sampled denominator is a different instrument than a full one. (c) Custom undici dispatcher with a longer connectTimeout. UNVERIFIED that this is settable on Vercel, and it directly worsens the doubled-handshake cost audit flagged: a longer connect ceiling times two attempts is a longer user-visible hang. Do not pursue before (a).
• wi-cli-venus note · 7w ago
  Mitigation (a) SHIPPED at 8f5245b (v2.79.4, live confirm pending), per audit-mars-ca's approval of (a) as the only immediate Mars mitigation. WHAT: new client component src/components/proxied-image.tsx wraps every /api/image/[id] img. On the img error event it swaps to an inline placeholder with role=img, aria-label 'Imagen no disponible temporalmente', caption 'No se pudo cargar. Reintenta en unos minutos.' Call sites converted: image-viewer.tsx (thumbnail + full viewer), practica-rx-button.tsx, mi-cursada/[id]/editar/edit-form.tsx. user-avatar.tsx does NOT get the caption - it falls back to the existing initials, already legible - it only gained the onError state so no broken glyph can render there either. Styles .image-unavailable / .image-unavailable-text in globals.scss. DOES NOT MASK: nothing retries, suppresses or swallows. The server still writes the error-tier IMAGE_PROXY_FAILED row on every failure; the client change is render-only with no telemetry of its own. WHY THE WORDING IS THE POINT: the caption says TEMPORARY. A student who reads 'missing' re-uploads a radiografia that is fine - that is the harm the broken-image glyph was causing. TEST: src/components/proxied-image.test.tsx, 4 cases, passing. The load-bearing one asserts that after the error event container.querySelector('img') is NULL and the temporary Spanish caption is present. Two more pin that the copy never says missing/deleted, and that fallbackClassName/style are honoured. COVERAGE LIMIT: jsdom never loads images, so the failure is driven by firing the same error event a real 500 produces - this pins the RENDER BRANCH, not the network behaviour, and it cannot see whether /api/image actually 500s. STILL OPEN, unchanged by this push: the upstream connect timeouts (coder-apiimages-cc owns), option (b) success-rate instrumentation (audit deferred until recurrence; denominator/sampling to be defined at emission AND reporting), option (c) connect timeout (audit: not before owner evidence). Depth artifact: DEPTH=deep CHECK=2c REASON='not provably inert: edit-form.tsx' SAMPLED=no.
• wi-cli-venus note · 7w ago
  coder-apiimages-cc got SSH onto the box (Elazar supplied creds; the project doc's 'FTP-only, no SSH, no PHP CLI' is FALSE - that was a property of the jailed FTP credential, not the host). Hostinger BR-812 (br-asc-web812.main-hosting.eu) ran SCHEDULED MAINTENANCE 2026-08-24 03:00-06:00 UTC; from inside the box curl to 127.0.0.1:80 and :443 were both connection-refused, which rules out routing, per-source filtering and our config. Scope was all 13 domains on the account, not just api.pensanta.com. THIS DOES NOT EXPLAIN MARS-677 AND MUST NOT ABSORB IT (coder-apiimages-cc's own correction, recorded verbatim in substance): MARS-677 is ConnectTimeoutError singletons 08-11 through 08-23 plus evening bursts, all BEFORE the window opened. Last good request on the box was 02:25 UTC, 35 min before the window. A maintenance window that explains a 3-hour hard-down explains nothing about intermittent connect timeouts over two weeks. STILL OPEN, unchanged: upstream connect-timeout cause (coder-apiimages-cc owns, awaiting box-side connection limits / php-fpm saturation / fail2ban / Hostinger rate-limiting evidence for the 20:34-21:14 ART clusters); option (b) success-rate instrumentation; option (c) undici connectTimeout. No Mars-side action available until that evidence lands.
• wi-cli-venus note · 7w ago
  RECURRENCE 2026-08-24T03:53 UTC, reported by audit-mars-ca. Two correlated rows retained: 077a0427-3a2a-4ea3-97cf-63fb4d9e9971 image-proxy/fetch-failed 03:53:10.710108+00 8362ae4b-1e99-4126-80fe-84777428725e suppressed 'fetch failed', route=__no_route__ 03:53:11.163796+00 (+0.45s) The 0.45s gap is consistent with MARS-669's bounded retry (one retry per proxy call), so these are ONE user-visible failure logged twice, not two independent failures - the same 2x understatement already recorded on this WI. Not suppressing either ID; the second row's __no_route__ is the expected shape for a throw outside a request-route scope. TIMING MATTERS AND CUTS AGAINST THE MAINTENANCE EXPLANATION: 03:53 UTC falls INSIDE the BR-812 window (03:00-06:00 UTC), so this particular recurrence is most cheaply explained by the window and should NOT be counted as fresh evidence of the MARS-677 intermittent-timeout pattern. Recorded here for completeness rather than as a new data point. The discriminator when the window closes at 06:00 UTC: MARS-677's signature is ConnectTimeoutError/UND_ERR_CONNECT_TIMEOUT (dropped packets, handshake never completes); a maintenance hard-down gives connection-refused/RST. The error CLASS on post-06:00 rows is what separates them - do not merge on route or timestamp alone. OPEN: whether these two rows carry ConnectTimeoutError or a refused/RST class. Not yet read - that read is what decides which bucket they belong in.
• wi-cli-venus note · 7w ago
  CLASSIFIED, per audit-mars-ca's ask. Both recurrence rows are ECONNREFUSED, NOT ConnectTimeoutError: 077a0427 / 8362ae4b 03:53:10.710 and 03:53:11.163 UTC causeCode=ECONNREFUSED, causeSyscall=connect, causeName=Error causeMessage='connect ECONNREFUSED 45.132.157.187:443', targetHost=api.pensanta.com, isImageApiError=false Both carry category=image-proxy action=fetch-failed level=error route=null. NOTE: the second row is NOT a distinct 'suppressed fetch failed / __no_route__' event as reported - it is the same category/action/detail as the first. The __no_route__ label came from the alert-rail rendering of route=null, not from a different event shape. THE SPLIT IS CLEAN ACROSS THE WHOLE TABLE. Grouping every image-proxy/fetch-failed row ever written by causeCode: ECONNREFUSED 4 rows, ALL on 2026-08-24 03:53:10 -> 04:07:53 UTC - entirely inside the BR-812 window (03:00-06:00 UTC) UND_ERR_CONNECT_TIMEOUT 32 rows across 08-17, 08-19, 08-20, 08-22, 08-23, 08-24 00:00-00:14 - ALL outside it null 25 rows on 08-15 and earlier (back to 05-22) DISPOSITION: the 4 ECONNREFUSED rows go in the MAINTENANCE bucket and are NOT corroboration of MARS-677. The refused/RST vs dropped-packet discriminator held exactly as predicted, and it is now measured rather than argued. MARS-677's own evidence is unchanged at 32 timeout rows. COVERAGE LIMIT, and it bounds any 'MARS-677 started on 08-17' claim: causeCode is NULL on every row from 08-15 back to 05-22, so those 25 rows CANNOT be classified either way - the field was not being captured yet. The apparent 08-17 start date is the instrumentation's start date, not necessarily the defect's. Do not date the onset from this table.
• wi-cli-venus note · 7w ago
  PENDING ACTION, agreed with audit-mars-ca (approved tag mt6pv06yp6qh): ONE sweep after 06:00 UTC 2026-08-24 (03:00 ART) recording the complete maintenance-window set - all image-proxy/fetch-failed rows 03:00-06:00 UTC with causeCode, retry pairs identified, and the user-visible failure count (rows/2). Per-row round trips are STOPPED by agreement; ECONNREFUSED-to-api.pensanta.com inside the window is pre-classified as maintenance and needs no ping. TWO ESCALATION DISCRIMINATORS, either one is significant and neither is predicted by the current framing: (1) any UND_ERR_CONNECT_TIMEOUT row INSIDE the window = MARS-677 continuing through a hard-down (2) any ECONNREFUSED row AFTER 06:00 UTC = the window did not close on schedule; stops being someone else's maintenance and becomes an outage Sweep query: SELECT id, "createdAt", detail->>'causeCode' FROM "appEvents" WHERE category='image-proxy' AND action='fetch-failed' AND "createdAt" >= '2026-08-24 03:00:00+00' ORDER BY "createdAt"; NOT ARMED AS A TIMER - self-arming a poll is banned; this is recorded so the next active session picks it up rather than it living only in a chat thread.
• wi-cli-venus note · 7w ago
  NEW FAILURE CLASS IN THE WINDOW - WRITE PATH, NOT JUST READ. audit-mars-ca surfaced d2d34178-9cab-4f71-ace5-ec0ad8fc6490, 2026-08-24 04:46:23.439 UTC, category=practica action=create route=/practicas/nueva level=error. SAME ROOT CAUSE, DIFFERENT AND WORSE CONSEQUENCE: causeCode=ECONNREFUSED, causeSyscall=connect, 'connect ECONNREFUSED 45.132.157.187:443', targetHost=api.pensanta.com. Maintenance bucket, inside the 03:00-06:00 UTC window. But every other row on this WI is the image-proxy READ path (a broken img on an existing practica). This one is the WRITE path: a student tried to CREATE a practica with a photo and could not. VERIFIED NO ORPHAN ROW: queried practicas for actorUserId 8d36f109 after 04:00 UTC - two rows exist, 04:22:44 and 04:32:46, both BEFORE the failure; nothing at 04:46. So the create aborted whole and left no partial practica. That is the upload-before-validate ordering behaving correctly (MARS-69) - the student lost a form submission, not data integrity. MESSAGING WAS CORRECT AND IS WORTH RECORDING AS A PASS: detail.userMessage = 'No pudimos guardar las imagenes por un problema temporal del servidor. Proba de nuevo en un momento.' It says TEMPORARY and says RETRY - the same property MARS-677's mitigation (a) was shipped to get on the read path. The write path already had it. Image context preserved in detail.context (pre ope.jpg, 1168x1562, 91888 -> 89591 bytes sanitized). BLAST RADIUS INSIDE THE WINDOW, all api.pensanta.com/ECONNREFUSED errors since 03:00 UTC: image-proxy/fetch-failed 4 rows, 1 distinct user, 03:53:10 -> 04:07:53 practica/create 1 row, 1 distinct user, 04:46:23 Two users affected, 3 user-visible failures. Small because it is the middle of the night in ART. CONSEQUENCE FOR THE POST-06:00 SWEEP: it must NOT be scoped to category='image-proxy'. That scoping would have missed this row entirely. Corrected sweep predicate = level='error' AND (detail->>'targetHost'='api.pensanta.com' OR detail->>'causeCode'='ECONNREFUSED'), which catches every route rather than the one we happened to be watching.
• wi-cli-venus note · 7w ago
  POST-WINDOW SWEEP COMPLETE (agreed with audit-mars-ca, tag mt6pv06yp6qh). Route-agnostic predicate as corrected: targetHost='api.pensanta.com' OR causeCode IN (ECONNREFUSED, UND_ERR_CONNECT_TIMEOUT, ETIMEDOUT), from 03:00 UTC to now (09:40 UTC). BOTH ESCALATION DISCRIMINATORS ARE CLEAN - this is the headline: (1) ZERO UND_ERR_CONNECT_TIMEOUT rows inside the window. MARS-677's intermittent pattern did NOT continue through the hard-down. (2) ZERO ECONNREFUSED rows after 06:00 UTC. The BR-812 window closed on schedule; last refusal 05:12:12.496. So the maintenance explanation held for its whole span and stopped exactly when it should have. Neither escalation fires. COMPLETE MAINTENANCE SET, 03:53:10.710 -> 05:12:12.496 UTC, all ECONNREFUSED to 45.132.157.187:443: image-proxy/fetch-failed 33 rows, 2 users practica/create 4 rows, 1 user, route /practicas/nueva 37 rows total, 3 distinct users affected RETRY DEDUP, and the instrument's limit is stated because the number depends on it: grouping rows <=3s apart per (category, action, actorUserId) gives image-proxy 11 distinct failures / 22 retry rows, and practica/create 4 distinct failures / 0 retry rows. THE 3:1 RATIO ON image-proxy DOES NOT MATCH MARS-669's SINGLE BOUNDED RETRY, which predicts 2:1. Two readings I cannot separate from this table: either the retry fires twice, or a 3s window is collapsing genuinely-separate failures from a page loading several images at once. DO NOT quote 11 as measured - it is 33 rows under one dedup assumption. Resolving it needs a per-image-id read, not a time-gap heuristic. THE 4 practica/create ROWS ARE ONE STUDENT RETRYING BY HAND, NOT OUR RETRY: 04:46:23, 04:49:37, 04:57:58, 05:01:43 - minutes apart, zero automatic retry rows. Corrects my earlier note on this WI, which saw only the 04:46 row because I queried before the rest existed and reported '1 user-visible failure'. It was four failed attempts by the same student over 15 minutes. No practica row was ever created for them; every attempt aborted whole. ALSO CORRECTED: audit reported '28 suppressed fetch-failed alerts 05:10:00.650 -> 05:12:12.496'. Those are inside the 33, not additional.
• wi-cli-venus note · 7w ago
  PER-IMAGE-ID READ (Elazar nqla 7a). The 3:1 dedup question is SETTLED, and my own 11-distinct-failures figure was WRONG. Retract it. THE LOG CALL CARRIES resourceId. src/app/api/image/[id]/route.ts:157 logs resourceId: id inside the single catch, so every row names its image and one REQUEST produces exactly ONE row regardless of how many handshakes it made. MEASURED, 03:00-06:00 UTC: 33 rows across 15 DISTINCT resourceIds, 2 users. 4 images x 4 rows, 3 images x 3 rows, 3 images x 2 rows, 5 images x 1 row THE GAPS KILL THE RETRY EXPLANATION OUTRIGHT. Within a single resourceId the intervals are 14s, 18s, 26s, 43s, 57s, up to 1m42s - e.g. 2026_21675523 at 05:10:00.276, 05:10:18.709, 05:10:26.406, 05:11:43.725. MARS-669's bounded retry fires in ~10s inside one request. Nothing at these spacings is an in-request retry: each row is a SEPARATE REQUEST, i.e. a user or a browser re-requesting the same image. SO THE CORRECT STATEMENT IS: 33 failed REQUESTS over 15 images, not 11 failures and not 16. One user (bd491417) accounts for 29 of the 33, repeatedly reloading a gallery of ~13 images across two minutes; the other user (8d36f109) has 4 rows over 2 images, 14 minutes apart. WHY MY EARLIER NUMBER WAS WRONG, worth keeping because the error is reusable: a <=3s time-gap heuristic keyed on (category, action, actorUserId) MERGED genuinely distinct images that failed within the same page load, and SPLIT the same image re-requested minutes later. It got the answer wrong in both directions at once. The identity was in the table the whole time - resourceId - and I reached for a time heuristic instead of the key. Do not dedup by time when an identity column exists. WHAT SURVIVES UNCHANGED: this WI's standing claim that each row represents TWO failed handshakes (~20s of hang) is about handshakes PER ROW and is still correct - the retry lives inside the try, so one catch = one row = up to two handshakes. 33 rows therefore = up to 66 handshakes. Alert counts understate HANDSHAKES 2x; they do NOT overstate failures 3x.
• wi-cli-venus assigned · 7w ago
  coder-mars-ca / coder
• wi-cli-venus note · 7w ago
  REASSIGNED to coder-mars-ca on coder-mars-cc's retirement 2026-08-24. Root cause is NOT the Mars lane's to chase - coder-apiimages-cc owns it and owes box-side evidence (connection limits, php-fpm saturation, fail2ban, Hostinger rate limiting) for the 20:34-21:14 ART timeout clusters. They now have SSH on the box, so saturation is directly inspectable rather than inferred client-side. No Mars-side action is available until that lands; do NOT pursue option (c) (undici connectTimeout) before it.
• wi-cli-venus assigned · 6w ago
  coder-mars-cc / coder
• wi-cli-venus verified · 5w ago
  New instance, and it is a DIFFERENT failure mode from the connect TIMEOUTS this item tracks - recording it here because the host is the same. appEvents 6f2e38ea-2bcb-4816-9e8a-192d41342826, 2026-09-10T13:38:48.959925Z, level=error category=practica action=create route=/practicas/nueva. Cause is ECONNREFUSED, not a timeout: 'connect ECONNREFUSED 45.132.157.187:443', targetHost api.pensanta.com, on a 241799-byte JPEG that had already sanitized to 168959 bytes. A refusal means nothing was listening, which is a service-down window rather than the slow-connect class. The student saw the Spanish temporary-server-problem copy and the practica create failed - the upload happens before validation, so a retry is the only recovery. RECOVERED: I probed api.pensanta.com four times ~1h later over both IPv6 and IPv4, all 200. So the outage window was transient and is closed; nothing to fix in Mars code. Worth noting the host is NOT venus - api.pensanta.com resolves to 2a02:4780:13:912::/Hostinger, while venus is 2a01:4f8:..., so a venus-side Caddy check says nothing about it and the maintainer api-images-evolutiva-cc is not on the roster. If this recurs, the discriminator to record is causeCode: ECONNREFUSED = the Hostinger service was down, ETIMEDOUT/connect timeout = the MARS-677 class proper.
• wi-cli-venus note · 5w ago
  mars-side discriminator for coder-apiimages-cc's IPv6/AAAA hypothesis, measured from appEvents over 14 days (20 failure rows, all targetHost=api.pensanta.com). VERDICT: nothing mars logged supports address-family selection as the cause, and one row points the other way. (1) No failing attempt names the v6 literal. Zero occurrences of '2a02:' in the corpus. Only two rows carry any address at all: 2026-09-10 13:38Z 'connect ECONNREFUSED 45.132.157.187:443' - the A record, actively REFUSED rather than hung - and an 11-row cluster on 2026-08-30 16:52-18:09Z all reading 'Connect Timeout Error (attempted address: api.pensanta.com:443, timeout: 10000ms)'. undici's ConnectTimeoutError prints the HOSTNAME, not a resolved literal, so those 11 do not discriminate family in either direction. (2) autoSelectFamily is not set anywhere in the tree. Whole-tree grep of src/ for autoSelectFamily, dispatcher and undici Agent returns no configuration, so mars calls bare global fetch and inherits Node's default (true on Node 20+). There is no Happy-Eyeballs disable to blame. Temporal shape, which fits an upstream cycling better than an unreachable family: the connect timeouts are ONE 75-minute burst, not a standing condition. Outside that window the failures are bare TimeoutError 'The operation was aborted due to timeout' at 1-2/day - that is mars's own AbortSignal firing, it carries no cause and therefore no diagnosis - plus the single ECONNREFUSED. A refusal and a hang on the same IPv4 address inside one corpus is the signature of a service going up and down, not of a family that cannot be reached. COVERAGE LIMIT, stated so it is not read as a refutation: networkFailureDetail (src/lib/image-api.ts) walks the cause graph breadth-first and RETURNS THE FIRST NODE CARRYING A CODE, discarding its siblings. If undici raised an AggregateError from happy-eyeballs with both a v6 and a v4 member, we persist whichever has a code and lose the other. So 'no v6 literal logged' does NOT establish 'no v6 attempt was made'. Closing that gap means changing the helper to keep every coded member rather than the first - a mars-side change, offered to coder-apiimages-cc, not yet made. Also worth recording separately: the bare-TimeoutError rows are the majority of this row's evidence and they are diagnostically empty by construction. mars aborts on its own signal before undici produces a cause, so the dominant failure class in MARS-677 has never been attributable. That is the mars-side half pmmaster assigned (degraded render instead of an unhandled timeout) and it is also a telemetry defect: the abort should record what it was waiting on.
• wi-cli-venus verified · 5w ago
  DISCRIMINATOR RUN, coder-apiimages-cc's first-traffic-watermark test against mars's own failure timestamps. RESULT: SILENCE in both windows. Verdict written to name what the instrument read, not what it implies (pmmaster correction, 11:15 ART). QUERY: appEvents level='error' over 2026-09-09 00:00:00Z to 21:52:22Z and 2026-09-10 00:00:00Z to 10:46:26Z - the two intervals ending at the two LATE purge watermarks on the api-images box. Window 2 (2026-09-10 00:00-10:46Z): ZERO error rows. Zero rows of ANY level until 10:00:54Z. Window 1 (2026-09-09 00:00-21:52Z): 6 error rows, none image-related - 1 client unhandled-error and 5 server unhandled-server-error, every one with detail->>'targetHost' NULL and detail->>'causeCode' NULL. No api.pensanta.com failure of any kind. Plus 1 warn row. SO THE ANSWER IS SILENCE, AND THE TWO SILENCES ARE NOT WORTH THE SAME. This is the part that must not be collapsed into 'no outage': Window 2's silence is WORTHLESS as evidence. mars logged nothing at all - no error, no warn, no info - until 10:00:54Z, i.e. for the first ten hours of the window. An app that made no calls cannot have observed an upstream failure, so the absence of failure rows there is the absence of TRAFFIC, not the presence of reachability. The late watermark and mars's silence have the same trivial explanation on that day: it was the middle of the night in ART and nobody was using either system. Reporting window 2 as a clean read would be counting a broken component as a control. Window 1's silence carries some weight - mars was active across the day (7 rows spread 01:36 to 19:28Z) - but its denominator is still unmeasurable, because mars logs image fetches only when they FAIL. There is no success row to count against, so 'no failures' cannot be turned into 'N successful fetches'. Stated as a coverage limit rather than a clear. INSTRUMENT REACH, carried in from pmmaster's correction so the row does not overclaim: the purge watermark samples ONCE PER DAY and can therefore only detect an outage that spans midnight UTC. The 2026-08-30 16:52-18:09Z connect-timeout burst is invisible to it BY CONSTRUCTION. So even a positive match on 09-09/09-10 would not have made ECONNREFUSED and the 08-30 burst one family - it would have made them consistent with one family over the only two days the instrument can see. The 16 clean days are not 16 days of proven reachability. NET: no evidence here of an upstream outage spanning either midnight. The AAAA/IPv6 hypothesis is dead by agreement (coder-apiimages-cc withdrew it on mars's corpus: zero v6 literals in 14d, the single address we hold is an IPv4 ECONNREFUSED, autoSelectFamily unset so Happy Eyeballs is on). Root cause remains unattributed, and the honest reason is recorded above: the dominant failure class on this row is bare TimeoutError from mars's own AbortSignal, which carries no cause and never has. That is the mars-side half to fix - degrade the render and record what the abort was waiting on - and it is mine. NOT CLEARED, coverage carried from coder-apiimages-cc: LVE entry-process faults unmeasurable from that shared account (/proc/lve/list EPERM, no lveinfo); csf/imunify360 unreadable, so an egress-range block is not excluded from either side.
• wi-cli-venus note · 5w ago
  Mars-side half WRITTEN (tsc clean, vitest 789/789), pending push. Upstream half was closed by coder-apiimages-cc this morning: api.pensanta.com IS reachable from Vercel egress - the 2026-09-10 ECONNREFUSED 45.132.157.187:443 is a TCP RST, so packets left egress and were answered; a silent egress block produces a HANG, not a refusal. They also retired their watermark signal as resolving to nothing, and their residual (host connection-accept behaviour under peak, /proc/lve EPERM, hPanel Elazar-only) is a Hostinger support item, not code. WHAT I FIXED, src/lib/image-api.ts fetchImage(): AbortSignal.timeout() rejects with a BARE TimeoutError DOMException - no cause, no code, no errno. networkFailureDetail() walks the cause graph and returns the first node carrying code ?? errno, so on OUR OWN timeout it has nothing to walk and the appEvents row records a name and nothing else. That is the DOMINANT failure class on this path (11 timeout rows vs the one coded refusal), and it is undiagnosable BY CONSTRUCTION rather than by bad luck: the row cannot say whether the 20s were spent never completing the connection or waiting on a peer that accepted and then went quiet - and those two have different owners (network/egress vs upstream capacity, i.e. exactly the Hostinger ceiling coder-apiimages-cc cannot measure). So the abort now records what it was waiting on. A bare TimeoutError is rethrown as ImageFetchTimeoutError with code=MARS_IMAGE_FETCH_TIMEOUT, phase, attempt and elapsedMs, original kept as cause. networkFailureDetail therefore returns a coded node instead of nothing, and an upstream-coded reset still classifies exactly as before - this only fills the case where there was nothing to read. phase is bounded honestly at 'connect-or-ttfb' and NOT split further: reaching that catch means no response headers arrived, so the wait ended in connect or TTFB. It can never be mid-body, because the signal covers the body read too and that abort surfaces later at the CALLER's arrayBuffer(), in the caller's catch. elapsedMs separates a timeout that burned the full 20s budget (a real stall) from one that fired early because the signal was already spent (which would be a bug in our own retry path). Also closed a smaller hole in the same function: the RETRY's failure propagated raw - only the first attempt's error was ever inspected. A retried connect timeout that then times out on TTFB is a different story from one that never connected twice, and the row could not tell them apart. Both attempts are now annotated with their own attempt number and their own elapsed clock. STILL OPEN, stated rather than implied - two items, both mine: (1) The CALLER-side body-phase abort. The 20s signal covers the body read, that abort lands at the consumer's arrayBuffer(), and it is still a bare TimeoutError there. Annotating it needs the same treatment at the call site, where phase='body' is knowable and is the discriminator that separates an upstream that never accepted from one that accepted and stalled mid-stream. Not done here; this change does not reach that catch. (2) The degraded render. This makes the failure DIAGNOSABLE, not invisible - a timeout is still an unhandled throw to the caller. The user-facing half (render a degraded image slot instead of surfacing a timeout) is untouched. Coverage limit: nothing here reduces the timeout RATE, and it cannot. It converts the rows from name-only to coded-with-phase-and-duration, so the NEXT occurrence is attributable to a side. Until traffic produces a post-deploy row the claim is about the code path, not a measurement.
• wi-cli-venus note · 5w ago
  CORRECTION to my previous note on this row, and it matters because the note claimed a result the code did not produce. The first version of that annotation put code=MARS_IMAGE_FETCH_TIMEOUT on the THROWN error. networkFailureDetail seeds its breadth-first walk at error.cause, NOT at the error itself - so the wrapper carrying the code was skipped entirely, the walk descended into the bare TimeoutError which has no code, and the row came out exactly as blank as before. tsc passed and all 789 tests passed. Neither instrument could see it, because nothing in the suite asserted the END of the pipeline; they asserted the throw site, where the annotation was plainly present. FIXED, three changes in src/lib/image-api.ts: 1. The walk now seeds at the thrown error rather than its cause. Behaviour is unchanged for every upstream failure - undici carries its code on a cause node, and a root with no code falls through to the same queue as before - and an error WE construct is now diagnosable, which is the only case that was broken. 2. NetworkFailureDetail gains causePhase and causeElapsedMs, additive and null on anything that is not one of our own aborts. The five existing keys keep their shape, so detail->>'causeCode' queries are unaffected. These are STRUCTURED rather than left in causeMessage on purpose: a discriminator that exists only in message text cannot be grouped on, which is the same undiagnosable-by-construction property this helper exists to remove. 3. Two tests added to image-api-fetch-retry.test.ts that assert the END of the pipeline - networkFailureDetail(await fetchImage(...).catch(e=>e)) yields causeCode MARS_IMAGE_FETCH_TIMEOUT, causeName ImageFetchTimeoutError, causePhase connect-or-ttfb and a numeric causeElapsedMs - plus one that pins an upstream-coded failure to its unchanged classification (UND_ERR_CONNECT_TIMEOUT / ConnectTimeoutError, both new fields null), so the seed change cannot silently repoint an existing diagnosis. 21/21 pass in those two files. VOCABULARY CHANGE, recorded because a query spanning today will split: causeName for our own timeout was the bare TimeoutError and is now ImageFetchTimeoutError. causeCode is the discriminator from here on. The 11 existing rows carry the old name; a query covering both sides of 2026-09-11 must accept both names, or better, key on the code. Comment at image-api.ts:805 updated - it still told the reader causeName was the attribution. NOT PUSHED YET: venus-host.push-gate is held by coder-pluto-cc (PLUTO-688, acquired 14:38Z, 2400s TTL). Registered for the release DM, not polling. MARS-206 and MARS-106 went out under my earlier hold as 8321731 and 7e839a6, v2.79.52.
• wi-cli-venus note · 5w ago
  Mars-side half LIVE: 50a58e6, v2.79.53, live-version confirmed against the REMOTE package.json on main (both 2.79.53) at 15:03:48Z, deploymentId dpl_7h848yraytP2gjiGUkzjN5h8wXC7. review-depth.sh 50a58e6 = DEEP (CHECK=2c, not provably inert: src/lib/image-api.ts, SAMPLED=no). Cold brief owed to coder02-mars-cc; that lane is frozen until 17:00 UTC and pmmaster's projection has the cap exhausting near or before then, so this SHA may ship without cross-model review. Recorded as such rather than reclassified shallow - the classifier says deep and a missing reviewer does not change the diff. ROW STAYS OPEN. Two named items remain, both mine, neither touched by this commit: (1) The CALLER-SIDE body-phase abort. The 20s signal covers the body read, that abort surfaces at the consumer's arrayBuffer(), and it is still a bare TimeoutError there. phase='body' is knowable at that call site and is the discriminator that separates an upstream which never accepted from one that accepted and stalled mid-stream. (2) The DEGRADED RENDER. This commit makes the failure diagnosable, not invisible: a timeout is still an unhandled throw to the caller, and the user-facing half is untouched. COVERAGE LIMIT, so the row does not read as stronger than it is: nothing here reduces the timeout RATE and nothing could. It converts the rows from name-only to coded-with-phase-and-duration so the NEXT occurrence is attributable to a side. Until prod traffic produces a post-deploy row, the claim is about the code path, not a measurement - the two tests assert the pipeline end, which is evidence the plumbing works, not evidence about production.
• wi-cli-venus note · 5w ago
  ITEM (2), THE DEGRADED RENDER, WAS ALREADY DONE AND I WROTE TWICE TODAY THAT IT WAS NOT. It shipped at 8f5245b on 2026-08-24 - src/components/proxied-image.tsx, adopted at all four consumer sites (image-viewer.tsx twice, mi-cursada/[id]/editar/edit-form.tsx, user-avatar.tsx's enlarged view). The single remaining raw img, user-avatar.tsx:52, is a documented and audited EXCEPTION: a 2rem thumbnail has no room for a sentence, so it falls through to the initials block, which is legible and is not a broken-image glyph. My 14:28Z and 15:04Z notes on this row both asserted 'the user-facing half is untouched'. That was false, and I never grepped before writing it - the same unmeasured-claim class this row has already caught twice, made in the direction of understating completed work rather than overstating it, which is why nothing flagged it. ITEM (1), THE BODY-PHASE ABORT, IS NOW WRITTEN. src/app/api/image/[id]/route.ts:137 is the ONLY consumer of fetchImage. The 20s AbortSignal covers the body read, so an upstream that accepts, sends headers and then stalls mid-stream aborts at arrayBuffer() rather than inside fetchImage, and arrived there as a bare TimeoutError - the identical undiagnosable-by-construction row, one call site later. The body read now has its own try and its own clock and rethrows through annotateAbortPhase with phase='body'. The clock starts at the body read rather than the request because the quantity that matters is how long the peer stalled AFTER answering. attempt is null here by construction and is omitted from the message entirely rather than rendered as 'attempt null', which would read as a real attempt number in a log search. FIRST POST-DEPLOY PRODUCTION MEASUREMENT, which this row has been waiting for and which arrived unprompted on the applog rail at 16:35:50Z: 'Image fetch aborted after 20003ms waiting on connect-or-ttfb (attempt 1)', appEvents b2b36a87-b3a8-4438-ab79-56c28f4d7365. So 50a58e6 is working in prod, not merely in tests. Two things that row now says which no prior row could: the full 20s budget burned (20003ms), so it is a genuine stall rather than a spent-signal bug in our own retry path; and connect-or-ttfb on attempt 1, meaning nothing was ever accepted. This is still ONE occurrence and says nothing about the rate. coder02-mars-cc RETURNED PASS ON ALL FIVE HELD SHAs - PASS:1cbc9cf shallow, PASS:8321731 deep, PASS:7e839a6 deep, PASS:50a58e6 deep, PASS:68221e6 deep - with two findings against 50a58e6 that I accepted and fixed rather than dismissed. M1 IS THE SAME DEFECT ONE LAYER OUT AND IT IS THE MOST IMPORTANT THING ON THIS ROW. annotateAbortPhase attaches four fields; NetworkFailureDetail declared only causePhase and causeElapsedMs, and the walk read only candidate.phase and candidate.elapsedMs. Nothing anywhere read candidate.attempt. So THE ATTEMPT NUMBER NEVER REACHED THE ROW while 50a58e6's message says both attempts carry their own number and their own clock. Attached, then dropped at the boundary - exactly the shape I had just finished correcting, and MY OWN TWO NEW END-OF-PIPELINE TESTS DID NOT CATCH IT because they asserted the two fields that DID get plumbed. Coverage of the fields you remembered is not coverage of the one you forgot. FIXED: causeAttempt added to the type, plumbed through all three return paths, attempt added to the candidate shape, and asserted in three places - 1 on connect-or-ttfb, null on body, null on the upstream-coded path. M2 accepted as a BOUND rather than a change, because the current behaviour is right: annotateAbortPhase returns the error untouched unless name==='TimeoutError', and the retry's own connect timeout arrives as undici TypeError wrapping ConnectTimeoutError. Relabelling it would destroy a real upstream code to substitute one of ours. So attempt=2 is recorded ONLY when the retry blows our own AbortSignal, which is narrower than 'on the retry'. Documented in the header, together with coder02's reading that the upstream-coded test PERMITS this rather than merely forbidding a regression. Also fixed on coder02's LOW against 68221e6, and it was the commons rule I was citing while breaking it - A RETRACTION REACHES THE SENTENCE, NOT ITS COPIES. pm/06-passwords-and-settings.md line 26 'Plan: Free' was established FALSE by this work and left standing unedited, in the file the commons tells people to read before touching env or DB config. Now annotated in place: falsified, the measurement that falsifies it, that the replacement is NOT known and must not be guessed, and a pointer to the unverifiable-state entry. Annotated rather than struck, because a blank would read as unknown-by-nobody rather than unknown-by-measurement. 799/799 tests pass, tsc clean. The new route test was verified to FAIL against the prior behaviour before being trusted, and only that test failed - a control that has not been seen to fail is a green line, not a control. NOT PUSHED YET: venus-host.push-gate is held by coder02-mars-cc; registered for the release, not polling. The push will also publish db-mars-cc's handoff e529df3 (MARS-711 migration 150), which a branch push carries regardless of my --only scope.
• wi-cli-venus note · 5w ago
  PTD PASSED, and this one is a REAL version match rather than an inert one. Live /api/health = 2.79.56, remote package.json on main = 2.79.56, read at check time via gh api and not from the local checkout. deploymentId dpl_Bcz76GwYBNbrZ6x1Tjj3Dm5CQHK9, ts 17:10:09Z. git ls-remote origin main = 0e8fef0. Published range 68221e6..0e8fef0 carried five commits: my 9cc8157 (this work) and 0e8fef0 (the range bump), coder02-mars-cc's 2b707a7 (MARS-716), and db-mars-cc's e529df3 + 2d29d82 (MARS-711 migration 150). coder02-mars-cc returned PASS:9cc8157 deep (classifier on the real SHA: DEPTH=deep CHECK=2c REASON=not provably inert src/app/api/image/[id]/route.ts SAMPLED=no) and PASS:0e8fef0 shallow. Two non-blocking findings, BOTH FIXED in the working tree, comment-only, waiting on the gate: P2 IS THE GOOD ONE AND IT IS A LIVE MISDIAGNOSIS RISK, not a style point. causeElapsedMs MEANS TWO DIFFERENT THINGS BY PHASE and the surviving comment described only the first. On connect-or-ttfb the clock and the 20s signal start together, so a SHORT value means the signal was already spent - a bug in our own retry. On BODY the signal was created at the fetch of the winning attempt while the clock starts at the body read, so elapsedMs is (budget minus time-to-headers) and CAN NEVER APPROACH 20000. A body row reading 3000 does NOT mean an early-firing signal; that is structurally impossible there. It means headers took about 17s - a slow-header upstream. So the old comment pointed a triager at the one cause that cannot occur in that phase. THE INVERSE IS THE WIN AND IT IS NOW DOCUMENTED: TIME-TO-HEADERS = 20000 - causeElapsedMs is derivable from any body row, and nobody would ever have computed it while the comment said otherwise. This is the first thing on this row that yields a number we could not previously obtain at all. P3 accepted as a WORDING fix. I wrote that attempt is null on the body phase 'by construction'. It is not - fetchImage knows whether it went through the retry and discards that at 'return response', so it is NOT-PLUMBED, a choice, not a structural bound. Corrected in both the helper header and the causeAttempt type comment. The argument against INVENTING a number stands and is kept; what was wrong was elevating a decision to an impossibility. Diagnostic value is low anyway since the body read follows whichever attempt succeeded, so the remedy is the wording, not the plumbing. Also recorded from coder02's M1 re-check, because it is load-bearing and was implicit: causeAttempt arrives ONLY because the coded branch is the single one of networkFailureDetail's three return paths that reads candidate.attempt - the other two hardcode null - which holds only because annotateAbortPhase stamps a code on the annotated ROOT and the walk is seeded at the thrown error rather than at error.cause. Move that seed back and causePhase, causeElapsedMs and causeAttempt all silently go null together. Now stated in the type comment so the next person to touch the walk sees what depends on it. ROW NOT CLOSED YET. Both original items are done - (1) body-phase annotation shipped in 9cc8157, (2) the degraded render was already live since 8f5245b - but the P2/P3 comment fix is written and UNPUSHED, and definition of done is pushed and working. Gate is held by coder-venus-cc; registered for the release. Closing after that lands.
• wi-cli-venus note · 5w ago
  SHIPPED AND VERIFIED. b1d7995, v2.79.57. Live /api/health 2.79.57 == remote package.json on main 2.79.57 read at check time, deploymentId dpl_8F28nTTZnyfr1dP4dvVusfKWuozS, git ls-remote origin main = b1d7995. Classifier on the real SHA: DEPTH=deep CHECK=2c REASON=not provably inert src/lib/image-api.ts SAMPLED=no - deep despite being comment-only, because the classifier cannot prove a .ts edit inert, and I did not reclassify it by eye. Cold brief sent to coder02-mars-cc; its PASS is the only thing outstanding and it is not a gate on the work being live. CLOSING CONDITION MET. Both items this row has carried since 2026-08-23 are done and pushed: (1) Body-phase abort annotated - 9cc8157, v2.79.56. The one call site the earlier fix structurally could not reach. (2) Degraded render - ALREADY LIVE since 8f5245b, 2026-08-24, adopted at all four consumer sites, with user-avatar's 2rem thumbnail a documented exception that falls through to initials rather than a broken-image glyph. THE FINDING WORTH CARRYING OFF THIS ROW, and it came from coder02-mars-cc's P2 rather than from me: causeElapsedMs MEANS A DIFFERENT THING IN EACH PHASE, and the comment I shipped described only one of them. On connect-or-ttfb the clock and the 20s signal start together, so a SHORT value means the signal was already spent - a bug in our own retry. On body the signal was created at the fetch of the winning attempt while the clock starts at the body read, so the value is (budget minus time-to-headers) and CAN NEVER approach 20000. A short body value is structurally incapable of meaning an early-firing signal - which is exactly where my comment pointed a triager, so it would have burned a real investigation on a cause that cannot occur there. THE INVERSE IS THE ACTUAL WIN: TIME-TO-HEADERS = 20000 - causeElapsedMs, derivable from any body row. Everything else on this WI made an EXISTING number attributable; that one CREATES A MEASUREMENT we could not previously take at all. WHAT THIS ROW DID NOT DO, restated at close so nobody reads the closure as more than it is: NOTHING HERE REDUCES THE TIMEOUT RATE AND NOTHING COULD. api-images capacity is not Mars's to fix. What changed is that the next occurrence is attributable to a SIDE - network/egress versus upstream capacity - where every row before 50a58e6 carried a name and nothing else. One post-deploy production row has been observed (16:35:50Z, 20003ms, connect-or-ttfb, attempt 1, appEvents b2b36a87-b3a8-4438-ab79-56c28f4d7365), which confirms the pipeline works in prod and says nothing about the rate. No body-phase row has been observed yet, so that half is verified by test and by construction, not by traffic. MY OWN ERRORS ON THIS ROW, recorded because they are the reusable part: (a) I shipped an annotation that produced NO code at all, with tsc clean and 789 tests green, because the walk was seeded past the wrapper - fixed at 50a58e6; (b) after fixing that I left attempt attached-but-unread at the type boundary, claiming in the commit message that it reached the row - caught by coder02, not by my own new end-of-pipeline tests, which asserted the two fields that DID get plumbed; (c) I twice wrote that the degraded render was untouched without grepping, when it had been live for eighteen days. All three are the same shape: a claim about a field or a file that nobody measured at the point it mattered.
• wi-cli-venus completed · 5w ago
  Both items shipped and live-verified. Body-phase annotation b1d7995/9cc8157 (v2.79.56 then v2.79.57, live-version confirmed against the REMOTE package.json at check time); degraded render was already live at 8f5245b since 2026-08-24. An image-fetch abort now reaches appEvents with causeCode MARS_IMAGE_FETCH_TIMEOUT plus causePhase, causeElapsedMs and causeAttempt at BOTH call sites, where before it carried a name and nothing else. Coverage limit, stated rather than implied: this does not reduce the timeout RATE and could not - api-images capacity is not Mars's to fix. It makes the next occurrence attributable to a side (network/egress vs upstream capacity). One post-deploy prod row observed (20003ms, connect-or-ttfb, attempt 1); no body-phase row observed yet, so that half is verified by test and construction, not by traffic. coder02-mars-cc PASS:9cc8157 deep and PASS:0e8fef0 shallow; PASS:b1d7995 outstanding and not a gate on the work being live.
• wi-cli-venus audit-finding · 5w ago
  PASS:b1d7995 deep (reviewer coder02-mars-cc, cold brief from coder-mars-cc, depth per scripts/review-depth.sh: DEPTH=deep CHECK=2c REASON='not provably inert: src/lib/image-api.ts' SAMPLED=no). Comment-only verified: every changed line in b1d7995 sits inside a // block; the only other change is the package.json bump. TWO NON-BLOCKING FINDINGS, reported on-channel. P2, same class as the P2 this commit was fixing: the new body-phase comment says elapsedMs 'CAN NEVER approach 20000' and that a near-budget value is 'structurally impossible' there. That is false, and the formula in the same paragraph implies it - elapsedMs = 20000 - time-to-headers, so as time-to-headers goes to 0, elapsedMs goes to 20000. Measured out of band (local headers-then-hang server, 400ms budget): headers at 18ms, body abort at 385ms = 96% of the budget; scaled to the real 20s budget a body row reads about 19980. That is the fast-header plus fully-stalled-body upstream - a healthy connection to a peer that accepted and then went quiet - which is exactly the failure the phase discriminator was added to name, and the comment now tells a triager it cannot occur. Remedy: keep the inverse formula, drop the impossibility claim. P3: the inverse yields the WINNING attempt's time-to-headers only, because the AbortSignal is fresh per attempt (image-api.ts:966 and :978). On a retry row it omits attempt 1's roughly 20s and so is not user-visible latency; and since attempt is null on the body phase, the row cannot say whether a retry happened at all. POSITIVE COVERAGE FINDING: verified empirically that a real mid-body abort rejects with root name === 'TimeoutError' - no undici 'TypeError: terminated' wrapper - so annotateAbortPhase's gate genuinely fires in the body phase. Previously unproven: the suite hands the helper a hand-built TimeoutError, so it could not have caught a wrapper. Bound stated honestly: repro ran on local Node v22.23.2 while prod reports v24.19.0 (/api/health) - same undici family, different major, so indicative rather than conclusive. Verdict recorded here because the ledger is the verdict SSOT (evolutiva-coder-commons Push-to-Done item 8); b1d7995 asserts no verdict of its own.
• wi-cli-venus note · 5w ago
  POST-CLOSE CORRECTION, comment-only, written and pending push. Not reopening the row - the shipped BEHAVIOUR is correct and live at v2.79.57; what was wrong is a sentence in a comment on that commit. coder02-mars-cc returned PASS:b1d7995 deep and then caught it. I ASSERTED AN IMPOSSIBILITY THAT MY OWN FORMULA, ON THE SAME LINE, DISPROVES. I wrote 'elapsedMs is (budget minus time-to-headers)' and then 'CAN NEVER approach 20000'. As time-to-headers goes to zero, elapsedMs goes to the budget. No external evidence was required to catch this - just reading my own sentence against my own equation. coder02 also measured the empirical form against a headers-then-hang server: headers at 18ms, body abort at 385ms of a 400ms budget = 96%, which scales to a body row reading about 19980 out of 20000. AND IT IS THE SAME CLASS AS THE DEFECT IT WAS FIXING, which is the part worth keeping. The original P2 was a comment pointing a triager at a cause structurally impossible in that phase. My fix pointed a triager AWAY from the one shape the phase discriminator exists to name: FAST HEADERS PLUS A FULLY STALLED BODY - a healthy connection to an upstream that accepted and then went quiet. I inverted the error while believing I was correcting it, and shipped it under a note calling the finding I was fixing 'the best finding either of us produced today'. CORRECTED SHAPE, per coder02's suggestion - keep the inverse, drop the impossibility: HIGH (near 20000) means headers came back fast and the body stalled essentially the whole budget; LOW means slow headers, with time-to-headers = 20000 minus elapsedMs; and a short value NEVER implies an early-firing signal, because that diagnosis belongs only to the connect-or-ttfb phase. The retracted claim is named in the comment together with coder02's measurement, so the next reader cannot re-derive it. P3 APPLIED IN THE SAME HUNK, and it bounds the inverse I had oversold: each attempt gets a FRESH signal, so time-to-headers is the WINNING attempt's only. On a retried request it omits the first attempt's roughly 20s, so it is NOT user-visible latency - and because causeAttempt is null on the body phase, the row cannot even say whether a retry happened. COVERAGE GAIN FROM coder02 THAT THE SUITE STRUCTURALLY CANNOT PROVIDE, recorded as a named residual rather than a clear: it verified out-of-band that a REAL mid-body abort rejects with root name === 'TimeoutError' and no undici 'TypeError: terminated' wrapper, so annotateAbortPhase's gate genuinely fires on the real thing. Our tests hand that helper a HAND-BUILT error, so they prove the annotation GIVEN a TimeoutError and say nothing about whether a real body abort IS one. Bound on that check, stated by coder02: Node v22.23.2 locally versus prod v24.19.0 per /api/health - indicative, not conclusive. THREE OVERSTATEMENTS FROM ME ON THIS ROW IN ONE DAY: an annotation that produced no code at all; attempt attached-but-unread while the commit claimed it reached the row; and now an impossibility contradicted by the adjacent line. Every one was caught downstream rather than by me, and this last one needed no instrument at all.
• wi-cli-venus note · 5w ago
  POST-CLOSE CORRECTION SHIPPED. 614934f, v2.79.58. Live /api/health 2.79.58 == remote package.json on main 2.79.58 read at check time, deploymentId dpl_Eb1amfZLavh2GrR6tN7NtkxWjRph, git ls-remote origin main = 614934f. Classifier: DEPTH=deep CHECK=2c REASON=not provably inert src/lib/image-api.ts SAMPLED=no. Cold brief sent to coder02-mars-cc. Row stays DONE - the shipped behaviour was correct throughout; what was wrong was a sentence in a comment, and it is now retracted in place rather than silently rewritten. The retracted claim is named in the file together with coder02's measurement, deliberately, so the next reader cannot re-derive it: 'a body-phase causeElapsedMs can never approach 20000' is FALSE, and the formula on the same line disproves it. coder02 measured the shape against a headers-then-hang server - headers 18ms, abort at 385ms of a 400ms budget, 96 percent - which scales to a body row reading about 19980 of 20000, i.e. fast headers plus a fully stalled body. That is a healthy connection to an upstream that accepted and went quiet, which is exactly the failure the phase discriminator was built to name. P3 in the same hunk, bounding an inverse I had oversold: each attempt gets a FRESH signal, so time-to-headers = 20000 minus elapsedMs yields the WINNING attempt's figure only. On a retried request it omits the first attempt's roughly 20s, so it is NOT user-visible latency - and causeAttempt is null on this phase, so the row cannot say whether a retry happened at all. NAMED RESIDUAL, kept as a residual rather than folded in as a clear: coder02 verified out-of-band that a REAL mid-body abort rejects with root name 'TimeoutError' and no undici 'TypeError: terminated' wrapper, so annotateAbortPhase's gate genuinely fires on the real thing. OUR SUITE STRUCTURALLY CANNOT SHOW THIS - it hands the helper a hand-built error, proving the annotation GIVEN a TimeoutError and saying nothing about whether a real body abort IS one. Bound stated by coder02: Node v22.23.2 locally against prod v24.19.0, indicative not conclusive. Still no body-phase row observed in production, so that half remains verified by test, by construction and now by one out-of-band local reproduction - never by traffic. GATE NOTE for anyone reading this row for push mechanics: venus-host.push-gate was held by coder-pluto-cc for 4.7 hours and EXPIRED rather than being released; I waited 16813s. The hub DOES fire a release DM on expiry, not only on an explicit unlock - I had the opposite recorded and have corrected it. The notification is explicitly NOT a reservation, so call lock immediately on receipt.
• wi-cli-venus audit-finding · 5w ago
  PASS:614934f deep (reviewer coder02-mars-cc; depth artifact DEPTH=deep CHECK=2c REASON='not provably inert: src/lib/image-api.ts' SAMPLED=no). Comment-only confirmed: all changed lines inside // blocks plus the package.json bump. Independent PTD reads (own, not relayed): git ls-remote origin main = 614934f, live /api/health version 2.79.58, node v24.19.0. 614934f retracts the false impossibility I flagged as P2 on b1d7995 and applies the P3 bound. All four corrected claims verified against the code: elapsedMs = budget minus time-to-headers; HIGH near 20000 = fast headers plus fully stalled body; LOW = slow headers via time-to-headers = 20000 minus elapsedMs; short value never implies an early-firing signal (that belongs to connect-or-ttfb only). The fresh-signal-per-attempt bound is correct - AbortSignal.timeout is constructed per attempt at image-api.ts:978 and :990, so the inverse yields the winning attempt's time-to-headers only. TWO NEW P3 FINDINGS, both non-blocking, reported on-channel. P3 (the substantive one, NEW rather than leftover): the corrected comment hardcodes the literal 20000 three times (:914, :918, :922) while the budget lives at IMAGE_FETCH_TIMEOUT_MS = 20_000 (:858). That coupling was harmless while the number was prose and is not harmless now that the comment is a triage instrument a human computes with: raise the constant to 30s and the comment still says a body row of 3000 means about 17s of headers when it would mean 27s - wrong by the whole delta, silently, and in the direction of UNDER-stating an upstream problem. Remedy: name the constant in the formula, keep the single literal at :858. Separately, :695 carries an unrelated bare 20000 as a minimum byte size - same digits, different quantity, a grep hazard rather than a defect. P3 (marginal, record-only): the inverse is stated unconditionally but holds only because the abort came from OUR signal; annotateAbortPhase gates on name === 'TimeoutError' and cannot tell which signal fired. True today because this path creates no other signal, false the moment a second or caller-supplied signal appears, and the row would then read as a header latency it is not. Verdict recorded here because the ledger is the verdict SSOT (evolutiva-coder-commons Push-to-Done item 8); 614934f asserts no verdict of its own. Prior verdict on this WI: PASS:b1d7995 deep, PASS:9cc8157 deep.
• wi-cli-venus note · 5w ago
  coder02-mars-cc returned PASS:614934f deep with two P3s, both accepted and shipped at b955437 (v2.79.59). P3-1, real and new rather than a leftover: the corrected comment hardcoded the 20000 budget three times while the constant is IMAGE_FETCH_TIMEOUT_MS = 20_000 at :858. That coupling was harmless while the reading was prose and is not harmless now that the comment is a triage instrument someone computes with - change the budget to 30s and the comment still says a body row of 3000 means 17s headers when it would mean 27s, wrong by the whole delta, silently, and in the direction of UNDER-stating an upstream stall. Every formula in the block now reads in units of IMAGE_FETCH_TIMEOUT_MS and the one literal lives at the const. P3-2, record-only: the HIGH/LOW inverse is asserted unconditionally but holds only because the abort came from OUR signal - annotateAbortPhase gates on name === TimeoutError and cannot tell whose signal fired. True today (the route creates no signal of its own and fetchImage accepts no caller-supplied one); a second or caller-supplied signal would make a body row read as a header latency it is not. Bounded in place with the condition named, so the next author sees the arithmetic that has to be re-derived. Left alone deliberately: the bare 20000 at :695 is a min byte size, a different quantity that happens to share the digits - coder02 flagged it only as a grep hazard. PTD b955437: live /api/health 2.79.59 == remote package.json 2.79.59 read at check time, ls-remote main = b955437, deploymentId dpl_BDud9yaJtyxRo1eCStzXEnNutVaM. review-depth.sh b955437 = DEPTH=deep CHECK=2c SAMPLED=no.
• wi-cli-venus audit-finding · 5w ago
  PASS:b955437 deep (coder02-mars-cc). Comment-only; both P3s from 614934f addressed. Two new P3s, no blocker. (1) The new rule is falsifiable by grep in its own file: :909 declares EVERY FORMULA BELOW is in units of IMAGE_FETCH_TIMEOUT_MS and NEVER a literal, and :947 sits below it saying 'The 20s signal covers the body too'; :855 and route.ts:139 carry the same literal outside the declared scope. No formula computes wrong - the cost is that the next author greps 20s, finds hits under the rule, and reads the rule as non-literal. (2) The second bound names one hazard and describes the other's consequence. annotateAbortPhase gates on error.name !== 'TimeoutError' against the ROOT error with no cause walk, so a caller-supplied AbortController.abort() (AbortError) fails the gate and the body error returns UNANNOTATED - no body row at all, which is the undiagnosable-by-construction row the annotation exists to remove, not a misread inverse. The inverse is corrupted only by another TimeoutError-producing signal: a second AbortSignal.timeout or AbortSignal.any([...]) containing one. Two opposite failures, both real, one named. Worth keeping: the 'at today's 20s budget, 3000 means ~17s headers' worked example is dated to the budget rather than stated as a formula, so it goes stale loudly. Coverage: diff, the const at :858, the gate at :975, both fetch call sites, the route's body catch and its absence of any signal. Not exercised: no runtime run on this SHA - the arithmetic was measured empirically on 614934f.
• wi-cli-venus note · 5w ago
  coder02-mars-cc returned PASS:b955437 deep with two more P3s. Both accepted; fix written, push REFUSED at the gate (held by bin-venus-cc, median hold 79s) and registered for the release DM. P3-1 IS THE RULE FALSIFYING ITSELF BY GREP. I wrote 'EVERY FORMULA BELOW IS IN UNITS OF IMAGE_FETCH_TIMEOUT_MS, NEVER A LITERAL' as an unscoped absolute while '20s' survives four times in this file and once in the route's body catch. None of them is a formula, so nothing computes wrong - the damage is to the next author, who greps 20s, finds hits sitting under an absolute rule, and concludes the rule was never meant literally. Fixed by SCOPING the rule (this comment block plus the route's body catch) and naming the survivors as what they are: arguments ABOUT choosing 20s, which would read as circular in units of themselves. The route's body catch now names no number at all. P3-2 WAS A REAL ERROR, NOT A WORDING NIT, AND IT IS MY FOURTH ON THIS ROW. I named a caller-supplied AbortController as the hazard and then described a DIFFERENT hazard's consequence. The gate reads error.name on the ROOT error with no cause walk, so an AbortController.abort() yields AbortError, FAILS the gate, and the body error returns UNANNOTATED - no body row at all. That is not a misread inverse; it is the undiagnosable-by-construction row this whole annotation exists to remove, silently restored. The inverse is corrupted only by a second TIMEOUT-producing signal - another AbortSignal.timeout, or AbortSignal.any([...]) containing one - which PASSES the gate while the arithmetic no longer holds. Two opposite failures, both real, now stated as opposites. THE PATTERN ACROSS ALL FOUR IS ONE THING, AND IT IS NOT CARELESSNESS ABOUT FACTS: every one was an OVERREACH IN A SENTENCE I WROTE TO CORRECT AN OVERREACH. 'attempt reaches the row' (it did not), 'can never approach 20000' (the adjacent formula disproved it), 'NEVER A LITERAL' (falsified by grepping the same file), and now a bound that named one mechanism and costed another. Each was written in the confident half-turn right after diagnosing the previous one. The cheap guard that would have caught three of the four: after writing a universal - never, every, cannot, impossible - grep or evaluate it against the same file or the same paragraph BEFORE shipping, not after. coder02's own framing worth keeping, because it is the positive half: the dated worked example ('at today's 20s budget a body row reading 3000 says headers took roughly 17s') is the RIGHT treatment of a literal - pinned to the budget so it goes stale LOUDLY instead of silently. Named in the comment as the pattern to copy if another example is ever added. Stated coverage bound from coder02 on this review: read the diff, the const, the gate, both fetch call sites and the route's body catch; NO runtime run on this SHA.
• wi-cli-venus note · 5w ago
  497b1d8 LIVE, v2.79.60. Both b955437 P3s fixed and verified: live /api/health 2.79.60 == remote package.json on main 2.79.60 read at check time, git ls-remote main = 497b1d8, deploymentId dpl_CXPeZtGfPK4ump8QAk46FgJVgmVm. review-depth.sh 497b1d8 = DEPTH=deep CHECK=2c SAMPLED=no. Gate was refused on the first attempt (bin-venus-cc held it), released after a 132s wait, taken and used on the retry - no version burned, the refusal happens before the bump. Cold brief owed to coder02-mars-cc.
• wi-cli-venus audit-finding · 5w ago
  PASS:497b1d8 deep, NO FINDINGS (reviewer coder02-mars-cc; DEPTH=deep CHECK=2c SAMPLED=no). Both P3s confirmed fixed at the mechanism rather than the wording. It verified rather than relayed: the route's body catch names no number and points at the const with 'not exported - read the budget there', which is the only correct treatment since IMAGE_FETCH_TIMEOUT_MS is not exported and the route cannot reference it; and it counted the surviving 20s occurrences itself, arriving at the same four I claimed, with :695's 20000 correctly set aside as the min-byte-size. ONE RESIDUAL, NAMED AND DELIBERATELY NOT FIXED. My enumeration lists three survivor categories; :118 is a fourth occurrence sharing :895's category from 780 lines away, inside the detail-keys block rather than the triage block. A grepper landing there first is outside both the rule's stated scope and the enumeration's line of sight. Nothing computes from it. coder02 raised it explicitly so the count is complete rather than to request a fifth revision, and recommended stopping; I agree and am stopping. A fifth comment-only push would cost a version, a deploy and a review cycle to close a gap that misleads nobody - and the rule this whole sequence produced is that an absolute claim must be checked against its own file, which the enumeration now satisfies for every occurrence inside its scope. MARS-677 IS DONE. Five commits carried the annotation work and its corrections: 9cc8157 (body-phase annotation), 0e8fef0, b1d7995, 614934f, b955437, 497b1d8. Verdicts on the row: PASS:9cc8157 deep, PASS:0e8fef0 shallow, PASS:b1d7995 deep, PASS:614934f deep, PASS:b955437 deep, PASS:497b1d8 deep. No BLOCK was ever issued; every finding was a P2 or P3 accepted and fixed, none silently dismissed.
• wi-cli-venus audit-finding · 5w ago
  PASS:497b1d8 deep (coder02-mars-cc). No findings. Both P3s on b955437 fixed at the mechanism rather than reworded. Verified independently: the route's body-phase catch now names no number and points at IMAGE_FETCH_TIMEOUT_MS with 'not exported - read the budget there', which is the only correct treatment since the const is not exported and the route cannot reference it; the whose-signal bound now states the two hazards as the opposites they are, with the right consequence on each - AbortController.abort() yields AbortError, fails the name === TimeoutError gate on the ROOT error with no cause walk, and the body error returns UNANNOTATED so there is no row at all, while a second AbortSignal.timeout (or AbortSignal.any containing one) passes the gate with the arithmetic broken. Counted the survivors myself: 20s appears at :118, :847, :875, :895 outside the rule block's self-references and the dated worked example - exactly the four claimed - and :695's bare 20000 is the minimum byte size, a different quantity. One residual named for completeness, not a defect: the in-file enumeration lists three survivor categories while :118 is a fourth occurrence sharing :895's 'what the row can say' category from 780 lines away, inside the detail-keys block rather than the triage block, so a grepper landing there first is outside both the rule's stated scope and the enumeration's line of sight. Nothing computes from it and I recommended leaving it rather than a fifth revision. Coverage: the full diff on both files, the surviving literal census, the gate at :975, the const at :858. Not exercised: no runtime run on this SHA.
• wi-cli-venus note · 3w ago
  RECURRING AFTER CLOSE, AND THE FAILURE MODE HAS CHANGED. Raised by support-evolutiva-cc from the applog rail 2026-09-20 (scrp-applog-mars, id b904c40f-1a1a-418e-b596-27bdbfad90fe). Not reopening - the BLOCKER on this row is still the operative fact and is Elazar's, not a coder's. Putting it to him on CLI today. EVIDENCE, 72h of appEvents category=image-proxy action=fetch-failed, 7 rows in 3 bursts: 2026-09-19 00:01:26 x2 (v2.79.79), 2026-09-19 03:41:47 x2 (v2.79.79), 2026-09-20 11:33:40-41 x3 (v2.79.81). Every row: errorName ImageFetchTimeoutError, causeCode MARS_IMAGE_FETCH_TIMEOUT, causePhase connect-or-ttfb, causeAttempt 1, targetHost api.pensanta.com, causeElapsedMs 20002-20004. route and requestId both NULL, consistent with this row's 'logged outside a request-route scope'. THE MECHANISM IS NOT THE ONE THIS ROW DIAGNOSED. MARS-677's evidence was ConnectTimeoutError / UND_ERR_CONNECT_TIMEOUT at ~10000ms - undici's DEFAULT connect timeout, per non-fix (1) on this row - meaning the handshake never completed. Today's rows abort at ~20003ms, which is OUR AbortSignal.timeout budget, and carry no undici connect error at all. If connect were failing, undici's 10s connect timeout would have fired first and produced the old error shape. It did not. So connect COMPLETED and the stall is in TTFB - the upstream accepted the connection and went quiet. The 'connect-or-ttfb' label cannot discriminate these, but the two timeout ceilings can, and they point at TTFB. Anyone triaging this class should read the elapsed value first: ~10000 with UND_ERR_CONNECT_TIMEOUT is the old connect failure, ~20000 with ImageFetchTimeoutError is this one. THE PEAK-CAPACITY READING IS WEAKENED. This row recorded bursts clustering at evening peak (20:34-21:14 ART) as 'consistent with upstream capacity rather than a code regression'. Converting the three new bursts to ART: 21:01, 00:41 and 08:33. Only the first is evening peak; 08:33 ART is a weekday morning. Three bursts is far too few to assert a distribution, so treat this as the peak hypothesis no longer being supported rather than as a new pattern - but it should not be carried forward unexamined. BURST SHAPE: each burst is 2-3 DISTINCT resourceIds landing within ~100ms of each other, all at the same ~20s ceiling. That is one stall window catching every concurrent image fetch on a page render, not N independent failures. So row count tracks images-per-page, and 7 rows is 3 user-visible incidents, not 7. WHAT I DID NOT ESTABLISH. No attempt-2 row exists on any of the 7. With MARS-669's retry live that is either a retry that succeeded, or a retry that did not run on this path - the proxy logs no SUCCESSES (this row's own COVERAGE LIMIT), so appEvents cannot tell the two apart and neither can I. This is the second time that missing success tier has blocked a conclusion, which strengthens MARS-SIDE OPTION (b) from this row. MY PROBE PROVES NOTHING, stated so it is not mistaken for evidence: api.pensanta.com from venus just now returned HTTP 200 three times, connect 0.22-0.33s, ttfb 0.70-0.80s. Per non-fix (2) on this row, venus->Hostinger is a different network from Vercel->Hostinger. It excludes 'origin is hard down right now' and nothing else.
• wi-cli-venus note · 3w ago
  RATE IS RISING, and today's second burst shows a user retrying into the same failure. Appending to my 11:35Z note rather than restating it. Second burst today: 2026-09-20 13:00:38.643 / 13:00:39.654 / 13:00:39.756, then a SECOND sub-burst 13s later at 13:00:52.281 / 13:00:52.307. Five rows, five distinct resourceIds, all ~20002-20003ms, all v2.79.81. That makes 8 rows today in two bursts 1h27m apart (11:33 and 13:00), against 7 rows in three bursts over the preceding 60 hours. 12 rows / 5 bursts in 72h now. THE 13-SECOND GAP IS THE NEW INFORMATION. Every prior burst was a single cluster inside ~100ms - one page render's concurrent fetches all hitting the ceiling together. This one is two clusters 13s apart with no overlap in resourceId. A page render does not fetch the same set twice 13 seconds later on its own, so the plausible readings are a user reloading after the images broke, or a second user landing on adjacent content in the same stall window. Either way the 20s ceiling fired again on the retry, so the stall outlasted it. I cannot distinguish the two readings from appEvents - actorUserId is not populated on these rows and there is no success tier to correlate against. RATE CAUTION, stated because I got the peak-clustering reading wrong on this row earlier today: five bursts is still too few to assert a trend, and I am reporting the count change, not calling an escalation. What can be said without a distribution: the gap between bursts fell from ~28h and ~8h to 1h27m, and both of today's are outside the evening-peak window this row originally recorded. NOTHING HAS CHANGED ABOUT THE BLOCKER, which is the point. api-images / api.pensanta.com is still unowned, coder-apiimages-cc still absent from the roster, and no agent can specify a fix. This is now the second consecutive day of user-visible broken images on a service nobody holds. Raised to Elazar on CLI as an owner-naming decision; re-raising with today's rate.
• wi-cli-venus note · 3w ago
  Recurrence 2026-09-20, MECHANISM CHANGED - support-evolutiva-cc, from mars appEvents category='image-proxy', full 72h population (12 rows, not a sample). Bursts (group by date_trunc('minute'), since route is NULL on every image-proxy row and the whole class shares one __no_route__ applog dedup bucket): 09-19 00:01 x2, 09-19 03:41 x2, 09-20 11:33 x3, 09-20 13:00 x5. Rising; the two 09-20 bursts are 1h27m apart. Every one of the 12: causeCode=MARS_IMAGE_FETCH_TIMEOUT, causeName=ImageFetchTimeoutError, causePhase=connect-or-ttfb, causeSyscall=null, causeElapsedMs 20002-20004, targetHost=api.pensanta.com, causeAttempt=1. This is NOT the UND_ERR_CONNECT_TIMEOUT this WI was filed on. fetchImage (src/lib/image-api.ts:1003) retries ONLY when isConnectTimeout(error) is true and annotates the retry as attempt 2; there are ZERO attempt-2 rows in 72h. No custom undici dispatcher exists on the image path, so undici's connect timer (shorter than our 20s - demonstrated on this very path by this WI's original rows) would have fired first if connect were hanging. It did not; our own AbortSignal.timeout(IMAGE_FETCH_TIMEOUT_MS=20_000) did. Connect succeeds and the upstream sends no headers for the full budget: a TTFB stall. Follows: this WI's note that 'each logged row is TWO failed handshakes ~= 20s of hang, alert counts understate attempts 2x' no longer holds. The retry is unreachable on the current mode - one row is one 20s hang, counts are 1:1. Not reopening: the blocker is unchanged and is not technical - api-images has no owner (coder-apiimages-cc off the roster since at least 2026-08-14), so no fix can be specified. Naming an owner is with Elazar on CLI.
• wi-cli-venus note · 3w ago
  probe
• wi-cli-venus note · 3w ago
  CORRECTION to my note minutes ago, and it is the sentence anyone would act on. I wrote 'coder-apiimages-cc off the roster since at least 2026-08-14'. Wrong, and this WI's own event log disproves it: on 2026-09-10 that agent closed the upstream half of this very row (reading the ECONNREFUSED 45.132.157.187:443 as a TCP RST, hence egress-reachable). It is absent from the hub roster as of 2026-09-20T13:20Z, so the lane IS vacant now - but it went quiet within the last ten days rather than in August. A recently-vacated lane and a month-dead one are different asks when someone is deciding who picks it up. Also ignore the one-word 'probe' event above it - mine, checking whether wi note accepts a body on a done row. It does.
• support-evolutiva-cc note · 2w ago
  support-evolutiva-cc 2026-09-28: cross-reference, no reopening. This row's ECONNREFUSED 45.132.157.187:443 of 2026-09-10 13:38Z recurred on 2026-09-28 23:03:50.796Z, same mode, ~105s of refusal then self-recovery. coder-apiimages-cc suggested reopening here; I am not, because this row closed on 'the box is reachable', which was TRUE and simply does not speak to intermittent listener loss - reopening would give one failure mode two rows. The recurring mode is tracked as EVO-103, which carries the Hostinger shared-hosting ownership (br-asc-web812.main-hosting.eu, account u105991616), the 35-day uptime that rules out a reboot, the load average near 30 that was not our footprint, and the fact that proving it needs a Hostinger ticket only Elazar can open.