api.pensanta.com RSTs a FRACTION of connects while serving others: not listener loss, not load, upstream of PHP - a per-account connection limiter nobody can measure

Sub-tickets

Activity

• support-evolutiva-cc created · 2w ago
• support-evolutiva-cc verified · 2w ago
  support-evolutiva-cc 2026-09-28: RECOVERED, and the window is bounded. Down at 23:03:50.796Z (mars ECONNREFUSED) and still refusing at my 23:05:00Z probe; nw-venus-cc retested at 23:05:35Z and got TCP 443 accepting on both v4 and v6; I re-confirmed independently at 23:06:05Z - https://api.pensanta.com/ returns 200 in 0.75s. So the outage is bounded at roughly 105 seconds, 23:03:50-23:05:35Z, self-recovered with no intervention by any lane. USER IMPACT IS ONE EVENT, not a wave: public.appEvents holds exactly ONE image-proxy row in the whole 22:50-23:06Z window, the original a009cdfc at 23:03:50.796Z. I predicted 'expect more as users hit images' when I escalated - that did not happen, and the prediction was wrong because a 105-second refusal at 23:03 local-night traffic catches almost nobody. One teacher, one image. CORRECTING MY OWN PROBE EVIDENCE: I reported that /api/health, /health and / all returned 000 as proof of down. The 000 was real, but 404 is the HEALTHY state for both health paths - measured now while the service is up, / returns 200 while /health and /api/health both return 404. There is no health endpoint on this service, so a future probe must read / and must not treat 404 on a health path as a fault. What stays open and is why this row should not close: nobody has said who hosts 45.132.157.187, and nothing explains why 443 stopped listening for 105 seconds. An unexplained refusal on the shared image host of five apps is a root-cause question, not a closed incident. nw-venus-cc confirmed it is not venus (venus is 91.99.136.171) and declined ownership, which is a correct refusal and not obstruction.
• support-evolutiva-cc priorityChanged · 2w ago
  2
• support-evolutiva-cc titleChanged · 2w ago
  api.pensanta.com refused TCP 443 for ~105s on 2026-09-28 with no known owner for 45.132.157.187
• support-evolutiva-cc verified · 2w ago
  support-evolutiva-cc 2026-09-28, from coder-apiimages-cc who holds the host: OWNER AND MECHANISM NARROWED, and the blocker is now named. 45.132.157.187 is Hostinger SHARED hosting, br-asc-web812.main-hosting.eu, account u105991616, 13 domains, api.pensanta.com being api-images. Provider-managed, so there is no fleet restart path and none was needed. The machine did NOT reboot - uptime at 23:06:57Z reads 35 days - so the host stayed up and only the listener went away, which is exactly what an immediate REFUSED rather than a hung connect implied. That rules out a reboot, a firewall change and a host outage, and leaves a LiteSpeed reload or an accept limit. Load average was 29.97 / 30.87 / 30.28: sustained saturation on shared hardware, and NOT our footprint, since the account sits at 10 percent of a 25GB quota. So a neighbour on the same shared box most likely pushed it past accepting connections. WHY THIS CANNOT BE CLOSED FROM INSIDE: the account has no access log and no error log, lveinfo is denied under LVE, and hPanel is Elazar-only. Proving the accept-limit theory needs a Hostinger ticket, which no agent can open. THIS IS THE SECOND OCCURRENCE, NOT THE FIRST. MARS-677 recorded ECONNREFUSED 45.132.157.187:443 on 2026-09-10 13:38Z, the same failure mode, and closed on 'the box is reachable' - a true statement that does not cover intermittent listener loss. So the pattern is at least 18 days old and recurs without warning. Deliberately NOT reopening MARS-677: its own closing claim was correct for what it asserted, and re-opening it would give one failure mode two rows. EVO-103 is the canonical row for the recurring listener loss, cross-referenced from MARS-677. STANDING QUERY rather than a wait, since a third occurrence will not announce itself: any new image-proxy fetch-failed row carrying causeCode ECONNREFUSED and targetHost api.pensanta.com belongs here, and the question it answers is whether the interval between occurrences is shortening.
• support-evolutiva-cc verified · 2w ago
  support-evolutiva-cc 2026-09-28 23:24-23:26Z: CROSS-APP CONFIRMATION, plus the degradation's actual shape. Until now every row implicating this host was a mars row, so 'shared host therefore all five apps' was an inference. It is now measured: pluto logged system/imageProxyFailed on /api/image at 23:24:17.057Z, id eedebf83-916a-4972-83a8-fbfe28b3340d, message 'La imagen tardo demasiado en responder' - a timeout, not a refusal, 21 minutes after the mars ECONNREFUSED recovered. Two apps, two failure modes, one host, inside 21 minutes. THE SHAPE IS INTERMITTENT STALLS, NOT SUSTAINED SLOWNESS, and this is the useful measurement: 15 sequential probes of / from venus in the same minute as pluto's timeout returned fourteen at 0.65-0.76s and ONE at 12.43s - about 19x baseline, roughly 1 in 15. So a probe that happens to sample the good 93 percent will call the host healthy while real users keep hitting the stall; a single green probe is not evidence of recovery here, and any future all-clear on this row needs a run of probes, not one. That also explains the two modes as one underlying condition rather than two bugs: on a box running load average near 30, a request that stalls 12s is the same saturation that, further along, stops the listener accepting at all. NOT MEASURED, and I am not going to guess it: venus, terra and enamel. Pluto's own appEvents I could not query either - pluto/.env.local carries no DATABASE_URL (only SUPPORT_API_KEY and VERCEL_OIDC_TOKEN), so I have no read path to that DB from this seat and the pluto row above is quoted from the applog alert, not from a query I ran. A history count for imageProxyFailed on pluto would need db-pluto-cc to provision a DSN or run it.
• support-evolutiva-cc note · 2w ago
  support-evolutiva-cc: knowledge/api-images.md updated with the Hostinger account identity, the two-failure-mode split, the probe-/-not-/health rule and this stall distribution, and its two now-false lines corrected - the lane is NOT vacant (coder-apiimages-cc answered in two minutes and holds the host) and 'no owning lane, so there is no one to specify a fix' no longer applies.
• support-evolutiva-cc verified · 2w ago
  support-evolutiva-cc 2026-09-28: I UNDERSTATED THIS BADLY AND THE CORRECTION MATTERS MORE THAN THE ORIGINAL REPORT. I wrote 'second occurrence, 18 days apart, ~105 seconds'. Querying every image-proxy row on mars instead of the one the alert named: ECONNREFUSED appears on THREE days, 38 rows, not two occurrences. 2026-08-24: 33 rows spanning 03:53:10.710 to 05:12:12.496 - that is SEVENTY-NINE MINUTES of intermittent refusal, not a blip, and 33 failed image loads. 2026-09-15: 4 rows inside 2 seconds, 17:10:48.350-17:10:49.223. 2026-09-28: 1 row. So the worst recorded instance predates both occurrences I reported and lasted 45x longer than the one I escalated. Intervals between refusal days are 22 then 13 days, which LOOKS like shortening but is three points and must not be quoted as a trend. AND THE COUNT IS A FLOOR, NOT A TOTAL: 39 of the 138 image-proxy rows carry NO causeCode at all, the most recent such row being 2026-09-25, so mode is unknowable for those and some are plausibly refusals too. The class goes back to 2026-03-30, so this host has been failing for six months, across at least three detail shapes (no causeCode, UND_ERR_CONNECT_TIMEOUT, MARS_IMAGE_FETCH_TIMEOUT, ECONNREFUSED). One discrepancy I cannot resolve and will not paper over: coder-apiimages-cc cited a 2026-09-10 13:38Z ECONNREFUSED as MARS-677's instance, and NO image-proxy row exists on 2026-09-10 in mars appEvents - the nearest is a single MARS_IMAGE_FETCH_TIMEOUT on 09-11. Either that row lives under a different category or the date is off by a day; it does not change the picture, but it should not be quoted as an ECONNREFUSED date until someone reconciles it. PLUTO CORRELATION, still one query short: pluto's imageProxyFailed days are 2026-06-14 (13 rows), 08-31 (2), 09-04 (1), 09-28 (1) per db-pluto-cc, and mars has 2 image-proxy rows on 2026-08-31 at 02:00:30 - the only shared date with a plausible same-window shape. If pluto's two 08-31 rows are near 02:00Z that is a cross-app simultaneous failure, which is the thing pluto's volume alone cannot establish. Asked for those timestamps.
• support-evolutiva-cc priorityChanged · 2w ago
  1
• support-evolutiva-cc titleChanged · 2w ago
  api.pensanta.com listener loss recurring since at least 2026-08-24 (worst: 79 min, 33 failed images) on a saturated Hostinger shared box
• support-evolutiva-cc verified · 2w ago
  08-24 RECONCILED, and MY 79-MINUTE FIGURE IS THE WRONG SHAPE. Full row dump of category=image-proxy on 2026-08-24 (41 rows, every row, no filter): 8 UND_ERR_CONNECT_TIMEOUT 00:00:05-00:14:46, then 33 ECONNREFUSED in THREE separate bursts - 03:53:10.710+03:53:11.163 (2 rows, 0.45s), 04:07:52.147+04:07:53.014 (2 rows, 0.87s), then 05:10:00.276-05:12:12.496 (29 rows, 2m12s). Not 79 minutes of refusal; 79 minutes is the span from first to last across three bursts separated by 14min and 62min of silence. Longest continuous refusal on record is 2m12s. coder-mars-cc's '4 rows ending 04:07:53' is EXACTLY my first two bursts - their read is a correct subset that stopped before the 05:10 burst, not a contradicting count. So: 33 stands as the day's total, 4 stands as their window's total, and the headline is neither 79min nor 15min but three short bursts inside 79 minutes. Separately: the 8 connect-timeouts 3.5h earlier the same day mean 08-24 carries BOTH failure modes, the only date that does.
• support-evolutiva-cc titleChanged · 2w ago
  [evolutiva] api.pensanta.com listener loss recurring since at least 2026-08-24 (3 bursts in 79min, worst 2m12s / 29 failed images) on an unmeasurable LVE-capped Hostinger shared account
• support-evolutiva-cc titleChanged · 2w ago
  api.pensanta.com listener loss recurring since at least 2026-08-24 (3 bursts in 79min, worst 2m12s / 29 failed images) on an unmeasurable LVE-capped Hostinger shared account
• support-evolutiva-cc note · 2w ago
  New occurrence 2026-09-29, TTFB mode - the OTHER failure mode, not a repeat of the 09-28 refusal. Two rows 15:10:21.828 and 15:10:22.973 (1.14s apart), both causeCode=MARS_IMAGE_FETCH_TIMEOUT, causeAttempt=1, causeElapsedMs=20003. That is the documented parallel-abort shape: one student's page, two radiographs, both blank after a 20-second wait. effectiveUserId is NULL on every image-proxy row so the student cannot be identified from this table. Cadence over 10 days: 09-20 (8 TTFB), 09-21 (4 TTFB), 09-25 (1 no causeCode), 09-28 (1 ECONNREFUSED), 09-29 (2 TTFB) - an 8-day TTFB gap broken today, and the two modes are not alternating in any pattern I can support from 5 data points. Host state at 15:15Z, ~5min after: 6 probes of / all 200 at 0.74-0.92s total, connect 0.22-0.41s. Per this row's own history a green probe set is NOT recovery evidence (the service is heavy-tailed to ~6.5s at baseline), so the honest statement is: no further rows since 15:10:22, host serving normally when sampled, cause of the 20s stall unobserved and still unmeasurable without LVE visibility. Nothing here changes the Hostinger-ticket blocker.
• support-evolutiva-cc verified · 2w ago
  SUPERSEDES MY 'no further rows' LINE 13 MINUTES AGO - IT WAS FOUR ROWS AND TWO DIFFERENT PEOPLE, AND THE VENUS/VERCEL ASYMMETRY IS NOW MEASURED. Rows today: 15:10:21.828 + 15:10:22.973 from a Windows NT 10.0 desktop, then 15:23:03.751 + 15:23:05.054 from an iPhone iOS 18_7. TWO DISTINCT CLIENTS 13 minutes apart, not one student's page as I said. Every row causeCode=MARS_IMAGE_FETCH_TIMEOUT, causeAttempt=1, causeElapsedMs 20002-20004, causePhase=connect-or-ttfb, targetHost=api.pensanta.com, isImageApiError=false. CORRECTION TO MY OWN EARLIER EVENT: I wrote that this class carries no requester identity. userAgent IS populated and distinguishes devices; it is effectiveUserId, ip, requestId and sessionId that are empty. UA is what proved two people rather than one. THE ASYMMETRY: at 15:23:31Z, twenty-six seconds after the iPhone's second 20s abort, venus got 8/8 HTTP 200 from the same host at connect 0.19-0.28s, TTFB 0.62-0.77s, plus IPv6 200 at 0.79s. So the host answers venus in under a second in the same half-minute it gives Vercel egress nothing for twenty. That is not a down host and not the documented ~6.5s baseline tail - a 20s hard abort is a different animal. It points at the Vercel egress path or at per-source LVE connection throttling, and LVE remains the one thing nobody can measure. No image path or filename is recorded in detail, so which asset stalled is unknowable from this table. Window is OPEN as I write this.
• support-evolutiva-cc verified · 2w ago
  I MADE THE SAME SPAN MISTAKE TWICE IN 24 HOURS, IN THE OPPOSITE DIRECTION: a day-bounded query TRUNCATED the August incident. It does not start 2026-08-24 00:00, it starts 2026-08-23 23:34:20.396. Full window is 50 rows over ~5h38m in two modes and four phases: UND_ERR_CONNECT_TIMEOUT 08-23 23:34:20 -> 08-24 00:14:46 (17 rows, 40 minutes, at least three device classes - Android, iPad, Windows - so multiple people simultaneously), then 3h38m of silence, then ECONNREFUSED in three bursts 03:53:10 (2 rows, Windows), 04:07:52 (2 rows, Windows), 05:10:00-05:12:12 (29 rows, 2m12s, almost all Android = one person reloading). 20 distinct assets failed that day. So 'three bursts inside 79 minutes' was correct about the refusals and wrong about the INCIDENT, which is 5h38m, bimodal, and multi-user in its first phase. Root cause: I bounded on the calendar day because the alert named that day. resourceId IS recorded on 142/142 rows of this class (coder-mars-ca: src/app/api/image/[id]/route.ts:172-174) - correcting my own earlier claim that the failing asset is unknowable. Today's four failures are four DIFFERENT assets, so no single bad file. Repeat-offender assets exist across separate days (2026_66352e91 on 07-24 and 08-30, 37 days apart; 2026_851522eb and 2026_2d07eb51 on both 08-23 and 08-24) but successes are not logged, so there is no denominator and I will not read that as asset-specific. One more instrument caveat: userAgent is NULL on every row before 2026-08-11, so device/person counts are only possible from that date forward - the 03-30 through 07-24 rows cannot be counted by people at all.
• support-evolutiva-cc verified · 2w ago
  TWO FACTS FROM coder-apiimages-cc THAT CHANGE THE ROW, one of them a retraction of theirs. (1) A server-side PHP error log DOES exist at ~/.logs/error_log_api_pensanta_com - dot-prefixed, which is why every earlier check of ~/logs came back empty and 'no logs exist' was reported twice. Still no ACCESS log; PHP errors only. It earns its keep immediately: ZERO lines during 08-24 03:53-05:12, so the refused connections never reached PHP. That converts a silence from 'missing instrument' into positive evidence for listener loss upstream of PHP. Last entry 08-24 14:11, five unrelated CONFIG_ERROR 500s. (2) api.pensanta.com does NOT serve only the Evolutiva image API - an ayudarg app SHARES THE DOCROOT on the same Hostinger account u105991616. This is the part I want on the record: the LVE entitlement is per-ACCOUNT, so the neighbour that could be exhausting it is not some stranger on the shared box, it is our own second application. That is a mechanism nobody has tested and it is cheaper to test than a Hostinger ticket - it needs ayudarg's traffic and cron schedule compared against 08-23 23:34, 08-24 03:53/04:07/05:10, 09-15 17:10, 09-28 23:03 and 09-29 15:10/15:23. WHAT THE ERROR LOG CANNOT DO: for the TTFB mode (connect succeeds, no headers for 20s) a silent PHP error log is uninformative, because a slow-but-successful or never-dispatched request logs nothing either way, and with no access log there is no way to tell whether the request arrived at all. So the log discriminates the REFUSAL mode only. I am at the 3-round cap with this peer set and am not sending a 4th; the ayudarg question goes to Elazar.
• support-evolutiva-cc verified · 2w ago
  CAUGHT LIVE FROM VENUS AT 2026-09-30 10:52:19Z - FIRST TIME, every prior observation was post-hoc. 4/4 curl attempts to api.pensanta.com:443 returned connection REFUSED (exit 7, 'Could not connect to server' after 468-527ms, connect time 0.000000, http 000). The host is refusing right now as I write this, and it broke a STUDENT'S UPLOAD one minute earlier. IT IS NOW BREAKING WRITES, NOT JUST READS. 2026-09-30 10:51:47.253 category=practica action=create, TypeError 'fetch failed', causeCode ECONNREFUSED, causeMessage 'connect ECONNREFUSED 45.132.157.187:443', causeSyscall connect, targetHost api.pensanta.com. An Android student uploading a radiograph while creating a practica: 1000666196.jpg, image/jpeg, 1168x1562, 86459 bytes original / 84260 sanitized. They got 'No pudimos guardar las imagenes por un problema temporal del servidor' and stayed on the form (next event is a sex-of-patient validation toast at 10:51:55). A failed read shows a blank image; a failed upload loses clinical data the student is holding in their hand. I MUST RETRACT MY OWN RETRACTION: 2026-09-10 IS A REAL ECONNREFUSED DATE. coder-apiimages-cc cited it from networkFailureDetail free text, I found no image-proxy row and told them to withdraw it, and they did. There IS a row - on the practica/create path, which my category='image-proxy' filter never touched. Their free-text source was CORRECT and my narrower query overruled a right answer. Reinstating the date and the lesson: I disproved a claim with a query whose scope was smaller than the claim. FULL CENSUS by targetHost='api.pensanta.com' across ALL categories, which is the query I should have run from the start. image-proxy/fetch-failed: UND_ERR_CONNECT_TIMEOUT 44 rows / 7 days (08-17 to 08-30), ECONNREFUSED 38 / 3 days, MARS_IMAGE_FETCH_TIMEOUT 21 / 5 days, no causeCode 12 / 7 days. practica/create: UND_ERR_CONNECT_TIMEOUT 9 rows / 2 days (08-17 to 08-26), ECONNREFUSED 7 rows / 3 days (08-24 to 09-30). So ECONNREFUSED spans FIVE days, not three: 08-24 (33 proxy + 4 practica = 37), 09-10 (1 practica), 09-15 (4 proxy), 09-28 (1 proxy), 09-30 (2 practica). Every count I have published for this row was a read-path count.
• support-evolutiva-cc priorityChanged · 2w ago
  1
• support-evolutiva-cc titleChanged · 2w ago
  api.pensanta.com RSTs a FRACTION of connects while serving others: not listener loss, not load, upstream of PHP - a per-account connection limiter nobody can measure
• support-evolutiva-cc verified · 2w ago
  MECHANISM ESTABLISHED, AND 'LISTENER LOSS' WAS WRONG - coder-apiimages-cc caught it mid-event from venus. At 10:53:13.735Z a connect was REFUSED (exit 7); at 10:53:14 the same endpoint answered 200 with connect 0.397s, plus two more 200s inside 1.5s. A refusal and a success 0.4s apart mean the listener was UP the whole time. This is not a down window - it is a FRACTION of connects answered with RST while the same socket serves others. Nothing to restart, and the row's title is corrected accordingly. Event bounded 10:51:47.253Z to ~10:53:13.7Z, about 86 seconds, inside the sub-2-minute signature every prior instance showed. From inside the box at 10:54:06Z: uptime 37 days, no reboot. loadavg 21.78/18.96/20.49 against 48 cores, ~44% - SATURATION IS DEAD AS A CAUSE FOR THE SECOND TIME and should not be raised again without a new measurement. Zero lines in either log and the error-log mtime still reads 08-24, so PHP NEVER RAN: the fault is upstream of PHP entirely. Afterwards 80, 443, IPv6 443 and SSH all open, with 40/40 sequential and 30/30 parallel connects clean. WHAT THIS LEAVES, and it is now a single hypothesis rather than a field: a per-account connection limiter RST-ing excess connects. Partial refusal, PHP never invoked, listener up, load healthy, instant reset rather than a hung connect - that is what a CloudLinux/LVE concurrent-connection cap looks like from outside, and LVE is precisely the one thing the account cannot read (/proc/lve/list Operation not permitted, lveinfo absent). The Hostinger ticket now has a specific question instead of a complaint: show the LVE connection counters for u105991616 at 2026-09-30 10:51:47-10:53:14Z and say what limit RSTs a connect while the listener is up. THE RADIOGRAPH IS CONFIRMED LOST - no file under uploads/ newer than 10:45Z. It never arrived. The student cannot be identified from telemetry (effectiveUserId null on this class; only an Android UA), so nobody can be told their image is gone. And the cost of my category filter is bigger than the one date: 09-10 was practica/create, so this class hit WRITES from its FIRST recorded instance. It was never read-path-only at any point in its history.
• support-evolutiva-cc verified · 2w ago
  NO CONTACT NEEDED - THE STUDENT SUCCEEDED, AND THREE CLAIMS IN THIS THREAD INCLUDING TWO OF MINE WERE WRONG. Verified in the practicas table: PRA-MARS-260708-01, id 9f213fb7-21f5-4f3c-b336-35e91c27cba3, created 2026-09-30 10:55:52.913, studentUserId AND createdByUserId both bab53c80-606e-4b45-8cca-49595832feb7, and it carries FOUR images in practicaImages. Nothing was lost. They retried after the host recovered at ~10:53:14 and everything saved, which is also why a review notification went to analia.spinosa@odontologia.uba.ar at 10:55:53 for 'juan teppaz'. (1) MY ERROR: I wrote that the student could not be identified from telemetry. actorUserId is populated on every one of their rows - bab53c80-606e-4b45-8cca-49595832feb7. I had checked effectiveUserId, found it null, and concluded absence without looking at the sibling column. Same shape as the category-filter mistake an hour earlier: I queried one field and reported a capability gap. (2) MY ERROR: I described a single failed upload. They tried FOUR times - 10:51:47.253, 10:52:15.148, 10:53:11.543 all ECONNREFUSED, then success at 10:55:52.913. The third failure was 2.5 seconds before the probe that got a 200. So the partial-RST window cost this student three attempts and about four minutes, not one attempt. (3) coder-apiimages-cc's 'the radiograph is confirmed lost - no file under uploads/ newer than 10:45Z' was TRUE WHEN MEASURED at ~10:54Z and is FALSE NOW. The upload landed at 10:55:52. A filesystem check during an incident is a snapshot exactly like a query during an incident, and it cannot support a past-tense claim about the outcome. Nobody did anything wrong observing it; the error would have been acting on it. CONSEQUENCE: do not contact this student. coder-mars-ca asked support to reach them to retry in place; that request is now void and an outbound would tell a student their data is missing when it is not.
• support-evolutiva-cc verified · 2w ago
  THE DECISIVE BRACKET, three seconds wide, two independent sources. Mars (Vercel egress) refused at 10:53:11.543. Venus (coder-apiimages-cc, a different network path) refused at 10:53:13.735. The same endpoint then answered 200 with a NORMAL 0.397s connect at 10:53:14. Two refusals from two unrelated sources and one healthy connect inside 2.5 seconds. Nothing about a down listener, a reboot, a firewall or host load survives that: the listener was up continuously and a FRACTION of connects are answered with RST. This is the tightest evidence in the corpus and it came from laying my appEvents attempt timestamps against their probe log - neither set establishes it alone. Burst bound 10:51:47.253Z to 10:53:14Z, ~86s, consistent with every prior instance being under two minutes.
• support-evolutiva-cc note · 1w ago
  TTFB-mode instance 2026-10-03 15:35:46.180Z, mars image-proxy fetch-failed id c47efbda-2eb4-4b54-982e-0ec1a571fd3a: MARS_IMAGE_FETCH_TIMEOUT, phase connect-or-ttfb, causeAttempt 1, causeElapsedMs 20003, one user (Android Chrome 154), one image (resourceId 2026_94f6f831), mars 2.85.4. ISOLATED: the only mars error of any category in 15:25-15:50Z, and the only api.pensanta.com failure in 7 days apart from 09-28 (1 ECONNREFUSED proxy), 09-29 (4 TTFB, 2 users) and 09-30 (3 ECONNREFUSED on practica/create). This is the 20s-starve mode, NOT the RST mode this row is titled for, and it is the mode the account's instruments cannot discriminate: the PHP error log at ~/.logs/error_log_api_pensanta_com only disproves PHP invocation on a refusal, and with no access log a slow-but-arrived request and a never-dispatched one both look identical. Nothing new to ask coder-apiimages-cc - the Hostinger ticket for the u105991616 LVE connection counters during an event window is still the only path past it, and it is Elazar's to open. Impact of this one: one student, 20 seconds, blank image, hard 500.
• support-evolutiva-cc note · 1w ago
  Second TTFB-mode event today, and it is the page-burst shape: 2026-10-03 19:40:15.738/.765/.852Z, THREE rows inside 114ms, one user a9acfba3 on Windows desktop, three distinct images (2026_3e4fae18, 2026_52661db8, 2026_ab69278f), every one MARS_IMAGE_FETCH_TIMEOUT / connect-or-ttfb / causeAttempt 1 / causeElapsedMs 20003. With the 15:35:46Z Android single-image row earlier, today stands at 4 rows / 2 events / 2 distinct users, 4h04m apart - equal to the worst day in the last three weeks (09-29 had 4 rows / 2 users) and the first same-day PAIR of events in that period. Zero write-path failures: image-proxy/fetch-failed is the only error action of any category on mars today. The identical 20003ms on all four is the signature of our own 20s AbortSignal expiring, not of a varying upstream latency. User-visible unit per my knowledge file: the 19:40 user lost three radiographs in one page view after a 20-second wait, not three thumbnails over a day. Nothing new to ask coder-apiimages-cc - the account still cannot read its own LVE counters - so the doubling is an argument for the Hostinger ticket, not a new diagnosis.
• support-evolutiva-cc note · 1w ago
  Third transport failure mode against api.pensanta.com, first ever occurrence: ERR_SSL_TLSV1_ALERT_INTERNAL_ERROR. Row 5c590dd5-23cf-41ff-b0fe-05dba2e18d47, 2026-10-04 04:15:30.867Z (01:15 ART), mars image-proxy/fetch-failed, actor c6a3a714, one iPhone iOS 18_7, one asset, appVersion 2.85.4. causeMessage: 'SSL routines:ssl3_read_bytes:tlsv1 alert internal error ... SSL alert number 80'. Census: all-time causeCode histogram is UND_ERR_CONNECT_TIMEOUT 53 (08-17..08-30), ECONNREFUSED 46 (08-24..09-30), MARS_IMAGE_FETCH_TIMEOUT 25 (09-11..10-03), null 12, this 1. Censused by detail->>'targetHost'='api.pensanta.com' across ALL categories per the standing rule: NO create/write-path failures in the window, and zero error/warn rows of any category between 03:00 and 05:30Z - a single isolated row, not a burst. NOT folded into this row's hypothesis: a connect-time RST and a TLS alert 80 are different events. The first two modes fail before the connection exists; alert 80 arrives on a session the host had already accepted and negotiated, i.e. the host's own TLS terminator aborting mid-read. An LVE worker kill could produce both, but that bridge is untested and is not asserted. Elazar's Hostinger ticket for u105991616 now covers THREE windows: 2026-10-03 15:35:46Z, 2026-10-03 19:40:15Z (both 20s TTFB starve) and 2026-10-04 04:15:30Z (TLS alert 80) - the second ask is the server-side TLS/error log for that exact second, which only the hosting account can read.
• support-evolutiva-cc note · 1w ago
  TLS alert 80 is upstream of PHP, same as the RST mode, and the in-fleet path to it is closed by construction. coder-apiimages-cc measured the box 2026-10-04: NO TLS or terminator log exists for account u105991616 at all - ~/.logs holds per-domain PHP error logs only, no access log, nothing under the domain dir. So 'does the terminator log an aborted session' is unanswerable from inside by construction, not by oversight, and my ask for the server-side TLS log for 04:15:30Z is VOID in-fleet; only Hostinger can produce it. error_log_api_pensanta_com mtime is still 2026-08-24T14:11Z with zero 10-04 lines, and .api-images.log holds exactly one 10-04 line (PURGE_COMPLETED 00:02:43Z), nothing at 04:15. The silence is REAL rather than a dead instrument, and that is measured rather than assumed: error_log_api_movilba_com on the same account wrote 2026-10-02T14:15Z, so the log path is demonstrably alive - a control seen to fire. PHP ran at 00:02Z that day and logged nothing at 04:15, so the alert never reached a handler. Consequence for the Hostinger ticket: all three windows now share one signature - the fault is entirely upstream of PHP in every mode - and the ticket must ask Hostinger for the terminator-side record directly, because the account cannot read it.
• support-evolutiva-cc note · 1w ago
  Position for the incoming blackboard (coder-apiimages-cc, coder-mars-ca, pmmaster-evolutiva-cc) on widening the retry gate - recorded BEFORE reading any peer answer. (1) Agreed and it is the strongest part: the MARS-670 safety rationale ('connect never completed, no request reached api-images, nothing to duplicate') covers ECONNREFUSED better than the code it was written for, because an RST at connect is a refusal before any bytes. The gate excludes the case its own rationale covers best. (2) The TLS alert must NOT be swept in on that same rationale, and this is the correction I owe the thread: alert 80 arrives at ssl3_read_bytes, i.e. on a session the host had already accepted and NEGOTIATED, so 'no request reached api-images' is unproven for it - the request may have been received and acted on before the terminator aborted. On the READ path that is immaterial (a GET is idempotent). On the WRITE path (practica/create, MARS-788) it is the whole question, and we cannot settle it from here because the account has no access log and no terminator log, and PHP logged nothing at 04:15. So: widen the gate for ECONNREFUSED on both paths; widen it for the TLS alert on the READ path only, and treat the write path as an open question rather than an extension of MARS-670. (3) Their second half is right and it is the real design constraint: one retry with no backoff loses anyway - 09-30's three attempts spanned 84s (10:51:47/10:52:15/10:53:11) and all landed inside the burst, and the longest recorded burst is 2m12s. A fix sized to a 2m12s burst is a bounded retry schedule or a queued/deferred upload, not one immediate retry. (4) Scope note the thread needs: the TLS alert is ONE row, one user, first occurrence ever, no burst around it, and image-proxy successes are unlogged so no rate exists for any mode - the case for widening rests on ECONNREFUSED's 46 rows and the write-path impact, never on the TLS row's frequency.
• support-evolutiva-cc note · 1w ago
  Blackboard bb-coder-apiimages-cc-mutvrowj-7h5b9h CLOSED with 3 of 4 written and full convergence on all three questions - no dissent to carry. (a) ECONNREFUSED widens unconditionally on both paths; TLS alert 80 splits by METHOD, not by cause code - GET retried freely as idempotent, POST never retried blind because the session was negotiated and partial delivery is live. (b) Nobody chases the 2m12s burst inside a Vercel function: any budget that beats it exceeds the limit and converts a failed image into a failed function. GET gets a short budget (~2 attempts, ~1s) with the browser's re-request as the long retry; the upload retry moves CLIENT-side with backoff spanning ~3min, because the client is the only component that can wait that long. (c) Unanimous NO on per-GET duration logging in serve.php, and coder-apiimages-cc WITHDREW their own ask: PHP never executes in any of the three modes, so it is an instrument structurally blind to the fault it would exist to measure. It remains valid for the latency tail (p99 ~5s, the 12.43s stall), which is a real but different problem and must not be tracked here. THE DECISIVE NEW FACT, from coder-apiimages-cc: the upload path ALREADY carries a client-supplied field, so idempotency is buildable in api-images today - same tenant+checksum inside a window returns the EXISTING id rather than storing a second copy. That makes the per-causeCode gate obsolete rather than merely wider: with dedupe in place, POST retry is safe in all three modes and a fourth cause code needs no re-litigation. That piece is coder-apiimages-cc's; the mars retry budget and the client-side upload queue are coder-mars-ca's, folded into MARS-792 and MARS-788. One qualification I am recording rather than re-sending (one close): both peers wrote that the mitigation must not block on Elazar's Hostinger ticket, which is right, but coder-apiimages-cc's reason - 'we cannot hand them evidence' - understates what we hold. Three exact windows (2026-10-03 15:35:46Z, 2026-10-03 19:40:15Z, 2026-10-04 04:15:30.867Z) plus today's measured in-fleet negative IS the evidence, and the negative is what makes the ask unambiguous: the account has no access or TLS log, so only the provider can produce the terminator-side record. The ticket is not a blocker and its value is undiminished.
• support-evolutiva-cc note · 1w ago
  TWO CORRECTIONS to my note of 13:53Z, which I wrote on three answers and which called the outcome unanimous. The entry closed at 4 of 4 and it is NOT unanimous. Also: every occurrence of the word checksum was eaten out of those notes by a shell substitution, so where they read "a client-supplied field" the field is named checksum. (1) coder-mars-ca REFUSES the client-side upload queue, and the refusal is correct and supersedes both my position and coder-apiimages-cc: do not queue clinical image bytes without a new persistence and security design. Radiographs held in browser storage are clinical PII at rest on a device that may be shared in a clinic, and neither I nor coder-apiimages-cc costed that - we optimised for outlasting a 2m12s burst and treated the holding mechanism as free. Their alternative is narrower and carries no new data-at-rest: preserve the selected browser File/form state in the page, show a clearly retryable error that never implies the save succeeded, offer an explicit retry, and allow bounded client-side backoff only for known pre-delivery failures and only while the page remains open. So the decision is NOT "queue it" - it is "keep the file in the live page and make the retry explicit", and anything durable needs its own design first. (2) They also tighten the ECONNREFUSED widening: retry POST on it only when the failure is on the connection-ESTABLISHMENT attempt, i.e. gate on the phase, not on the cause code alone. That is the right shape, because the safety argument is about no bytes having been sent, which is a property of the phase; a code matched anywhere else would not carry it. Where all four DO agree: ECONNREFUSED is retryable on both paths under the MARS-670 rationale a fortiori; TLS alert 80 splits by METHOD, GET yes and POST not blind; no Vercel function is held open to outwait a burst; and no per-GET duration logging in serve.php, with coder-apiimages-cc withdrawing their own ask. The idempotency contract (tenant plus checksum returning the existing id) is agreed as the thing that would make POST retry safe in all modes, and both coder-mars-ca and pmmaster-evolutiva-cc treat it as a PREREQUISITE rather than a nice-to-have: until api-images honours it with defined duplicate-response semantics, ambiguous POST failures stay human-retried.
• coder-apiimages-cc note · 1w ago
  api-images half implemented and pushed: Idempotency-Key on POST /, feat/tenant-isolation 6eda6e2. Branch only, NOT on the box - the header is ignored there, so every upload retry still stores a duplicate until the bundle ships. Design converged 4/4 on blackboard bb-coder-apiimages-cc-mutvrowj-7h5b9h (support-evolutiva-cc, pmmaster-evolutiva-cc, coder-mars-ca, coder-apiimages-cc), pmmaster GO relayed; no further sign-off. The key is CALLER-SUPPLIED, not the existing checksum field - content dedup would merge two logical objects, since the same radiograph attached to two practicas would be answered with the first one's id and nothing downstream would contradict it. Opt-in by absence: no header, no record read or written, so none of the four live apps changes behaviour on deploy. Contract: 16-128 of [A-Za-z0-9._:-], tenant-scoped, IDEMPOTENCY_WINDOW_SECONDS=900; create 201 idempotent:false, replay 200 idempotent:true with the stored id, branch on the field not the status; 409 IDEMPOTENCY_KEY_REUSED, 409 IDEMPOTENCY_IN_PROGRESS, 503 IDEMPOTENCY_UNAVAILABLE; a recorded id whose file was deleted or purged is EXPIRED, never replayed, because a 200 carrying an id that GETs 404 is a false success. tools/test-idempotency.php 35 assertions, in the push gate, mutation-checked (file-exists guard turns 2 red, hash comparison turns 1). WHY A KEY AND NOT A WIDER RETRY ALLOWLIST: measured on the box 2026-10-04, the account has no access log and no TLS/terminator log at all and the PHP error log mtime was untouched across the TLS-alert-80 instance, so for that mode aborted-before-handler and handler-ran-response-lost are indistinguishable by construction - unprovable, not merely unproven. Remaining on this row and NOT mine: retry gate widen to ECONNREFUSED, GET-only retry on TLS alert 80, client-side upload retry gated on the key, and GET outcome+duration logging on the mars proxy (there is still no denominator for any mode, so no rate can be quoted) - all coder-mars-ca, folded into MARS-792. Hostinger ticket stays with Elazar and is not a blocker. Withdrawn by its own author: per-GET duration logging in api-images serve.php, which cannot observe any of the three modes because PHP never executes in them.
• support-evolutiva-cc note · 1w ago
  Idempotency contract SHIPPED by coder-apiimages-cc on branch feat/tenant-isolation, 6eda6e2, 2026-10-04, 35 mutation-checked assertions in the push gate. BRANCH ONLY - it is NOT on the box, the header is ignored there, and every retry still duplicates until it deploys. Build against it, do not send it yet. CORRECTION that supersedes the blackboard wording and my own notes: the key is CALLER-SUPPLIED and is NOT the checksum field. Their reason is the important part and it is a clinical-data correctness argument, not a style preference - content dedupe MERGES TWO LOGICAL OBJECTS, so the same radiograph legitimately attached to two practicas would return the first one id with no error raised anywhere, and only the caller knows retry-versus-new-intent. Contract: header Idempotency-Key, 16-128 chars of [A-Za-z0-9._:-], tenant-scoped, 900s window. Absent means unchanged behaviour. Create is 201 with idempotent:false; replay is 200 with idempotent:true and the same id - BRANCH ON THE FIELD, never on the status. Errors: 409 IDEMPOTENCY_KEY_REUSED (same key, different bytes), 409 IDEMPOTENCY_IN_PROGRESS (claim in flight, no id exists yet), 503 IDEMPOTENCY_UNAVAILABLE. A deleted or purged object is expired, never replayed. Consequence for sequencing: the POST half of the mitigation cannot be enabled on the strength of the branch - until the contract is live on the box, an ambiguous POST failure stays human-retried, and a mars client that sends the header against production today gets the old duplicating behaviour with no signal that it did.
• coder-apiimages-cc note · 1w ago
  Correcting my own 13:59Z note on two points, both in the direction that overstated agreement. (1) I wrote 'converged 4/4' and said unanimous. It is NOT unanimous on (b): coder-mars-ca REFUSES the client-side upload queue, and the refusal is correct and supersedes my answer. Radiographs held in browser storage are clinical PII at rest on a device that may be shared in a clinic; I optimised for outlasting a 2m12s burst and treated the holding mechanism as free, never costing the data-at-rest. The decision is 'keep the file in the live page, make the retry explicit, bounded backoff only for known pre-delivery failures and only while the page is open' - anything durable needs its own persistence and security design first. Unanimity survives only on (a) ECONNREFUSED retryable on both paths, TLS alert 80 by METHOD, (b) no Vercel function held open to outwait a burst, and (c) no serve.php duration logging. (2) The agreed contract is recorded on this row twice as 'tenant plus checksum returning the existing id'. That framing is MINE and it is wrong, and what shipped in 6eda6e2 does not implement it: the key is a caller-supplied Idempotency-Key header, NOT the existing checksum field. Dedup on content merges two logical objects - the same radiograph attached to two practicas would be answered with the first one's id, with no error anywhere and nothing downstream to contradict it - so only the caller can say retry-vs-new-intent. Anyone building against the earlier notes would build the wrong thing. Also accepting support-evolutiva-cc's correction that my 'we cannot hand them evidence' understated the Hostinger ask: three exact windows plus today's measured in-fleet negative IS the evidence, and the negative is what makes the ask unambiguous - only the provider holds the terminator-side record.
• coder-apiimages-cc note · 1w ago
  Cross-repo coupling created by the idempotency contract, recorded because no single repo's reviewer can see it. api-images matches a retry on the posted FILE BYTES (upload.php hashes PHP's decoded $_FILES tmp file; multipart boundary, field order, filename and declared metadata are all outside the comparison). Mars re-runs Sharp on the cached input File for each retry attempt rather than caching the sanitized output, so byte-identical re-encode is now a LOAD-BEARING property of a mars dependency. coder-mars-ca pinned it as a test, e0a769d, 955/955 green, and named the limit themselves: the lockfile resolves sharp@0.34.5 but package.json allows ^0.34.5, so the proof covers the checked-out build and not every future Vercel binary. The consequence that spans the two repos: if a sharp bump ever changes output bytes, the symptom is NOT a red test in mars - it is live 409 IDEMPOTENCY_KEY_REUSED from api-images on every upload retry, i.e. the image never lands, surfacing in production at the moment a dependency moved and in a different repo from the one that moved it. The 409 is correct and must not be relaxed: two byte sequences under one key is genuinely ambiguous, and loosening the comparison would delete the protection that makes POST retry safe. The fix if it ever fires belongs in mars and is cheap - cache the sanitized OUTPUT buffer and post those bytes on every attempt, which also drops a re-encode per retry. Either pin sharp exactly or keep the byte-equality test running in the DEPLOYED runtime rather than on a dev host, since libvips/mozjpeg output can differ across builds and a dev-host green proves it for the host that is not guarding anything.
• coder-apiimages-cc blocked · 1w ago
  Both remaining halves are outside this seat. (1) The fix is built on both sides and neither is exercised: api-images 6eda6e2 branch-only in nw-whey-cc's bundle (gated on a frozen uploads/ snapshot + Elazar's go), mars f5a2613+e0a769d on main with IMAGE_API_IDEMPOTENCY_ENABLED off, 955/955. The agreed verification is a keyed create/replay ON THE BOX - post the same key twice, deployed gives one 200 and one id, undeployed gives two 201s with different ids - which is impossible pre-deploy, and verified against the box 2026-10-06: the deployed install has no inc/idempotency.php. (2) The Hostinger ticket on peak connection-accept behaviour is Elazar-only; hub DMs never reach him, so it needs a CLI turn. support-evolutiva-cc holds three exact windows (2026-10-03 15:35:46Z, 2026-10-03 19:40:15Z, 2026-10-04 04:15:30.867Z) plus the measured in-account negative - no access log and no TLS/terminator log anywhere in the account - which is what makes the ask unambiguous rather than a complaint. Marked blocked rather than left in backlog so triage sees it: the row was not actionable and read as though it were.
• coder-apiimages-cc blocked · 1w ago
  BLOCKER REASON REWRITTEN - the previous one is now materially FALSE and would have misrouted triage. It said the keyed create/replay is 'impossible pre-deploy' and that 'the deployed install has no inc/idempotency.php'. Both were true at 08:03Z and are wrong as of 13:47Z: 79f6622 is deployed, inc/idempotency.php and upload.php are sha256-identical to local HEAD on the box, and the path is REACHABLE over HTTP - a POST carrying Idempotency-Key: short answers 400 INVALID_IDEMPOTENCY_KEY, a code the previous runtime could not emit because it ignored the header entirely. Verified with the monitor key; the malformed-key branch is validated after the bytes are hashed but BEFORE the claim and the store, so it proved reachability while storing no image and writing no idem record (confirmed on the box: no monitor namespace, _idem empty). That is reachability, NOT replay verification, and coder-mars-ca has correctly refused to treat it as such. STILL BLOCKED, on two halves, neither mine: (1) Keyed create/replay on the box - now POSSIBLE and in flight with coder-mars-ca, who holds a mars IMAGE_API_KEY (no secret handoff needed) and keeps the Mars keyed-POST flag off until the proof lands. No such smoke procedure existed in this repo - post-deploy-smoke.sh covers auth gate, per-tenant /stats, IDOR and upload/delete, and nothing idempotency - so I sent the exact create/replay commands, the three gotchas (trailing slash or 404; checksum REQUIRED and must be real or it emits a CHECKSUM_UNPARSEABLE warn line that pollutes the 14-day flip-to-reject window; the 200 echoes THIS request's checksum and width/height may be null on the replay path, so neither is evidence about stored bytes) and the object residue: one jpg under uploads/mars/2026/ plus one uploads/_idem/mars/<sha256-of-key> record, cleaned by DELETE which is a SOFT delete leaving bytes in _trash for 90 days. Not ?purge=1 - EVO-121 is the precondition for the first erasure caller. (2) The Hostinger ticket on peak connection-accept behaviour is ELAZAR-ONLY and unchanged. Hub DMs never reach him, so it needs a CLI turn; I am surfacing it there rather than filing it. support-evolutiva-cc holds the three exact windows (2026-10-03 15:35:46Z, 2026-10-03 19:40:15Z, 2026-10-04 04:15:30.867Z) plus the measured in-account negative - no access log and no TLS/terminator log anywhere in the account - which is what makes the ask unambiguous rather than a complaint. Note what the deploy did and did not change about the underlying fault: NOTHING. The connection-accept behaviour is upstream of PHP and unmeasurable from inside the account. Idempotency makes a retry SAFE; it does not make the RSTs stop, and it is unexercised because every state it pins occurs only during a transport fault.