Post-deploy internal probe manufactures the genuine-bad-route 404 warn row that errscan then escalates as a first-ever-seen signature
MEASURED on mars appEvents 2026-10-05. The 2026-10-04 23:44:04.04914Z row route=/no-such-page message="404 [genuine-bad-route]: /no-such-page" level=warn appVersion=2.85.6 carries userAgent=evolutiva-internal-probe/mars-ptd and detail.secFetchDest=null. It is OUR OWN post-deploy probe (the two rows 9 min earlier are 2.85.5, so a deploy landed between them). The probe asks for a route it knows does not exist, the 404 rail writes a warn row, the classifier has nothing to key on and calls it genuine-bad-route, and evolutiva-errscan@mars.timer then escalates it as [new] = first-ever-seen. A self-inflicted alert from end to end: the detector spends its own escalation budget on a row it created. THE FIX IS SERVER-SIDE AND IS NOT A USER-AGENT FILTER. userAgent is caller-asserted, so filtering on evolutiva-internal-probe/* lets any caller opt out of 404 alerting by naming itself our probe, and it also hides the one case worth keeping (a probe that 404s when it should not). Two shapes that do work: (a) give the probe a dedicated route that answers without manufacturing a navigation-tier warn row, or (b) gate the suppression on something SERVER-HELD - a shared-secret header the app checks - and log the resolved classification, not the claimed identity. Pick one; both keep the row truthful about what happened. SCANNER-SIDE IS THE WRONG LAYER: the errscan escalation predicate fires on first-ever-seen regardless of volume, which is correct behaviour; the defect is upstream, in producing the row. Sibling note: /crusader-404-probe and /zzz-nonexistent-route-xyz in the same 14d census are the same self-inflicted shape from other lanes. SUPPORT TRIAGE OF THE OTHER TWO ESCALATED SIGNATURES, for the record: /mi-cursada%0A x2 (23:35:43.831658Z, 23:35:50.115332Z) is NOT a student-facing link defect - the two hits come from TWO DIFFERENT ips with two different macOS UAs, isAuthenticated=false, referrer=null, and the 10_11_1/AppleWebKit-601.2.4 fingerprint on the first also produced route=/%0A on 2026-08-15 17:15:34Z and 2026-09-04 16:23:23Z, so the encoded-trailing-newline class is a recurring crawler pattern across three months, always in pairs, never authenticated. No app-generated link carries %0A: it would recur with traffic volume instead of twice in seven seconds. The /practicas statement-timeout pair at 11:51Z is the 11:50-11:51 burst already on MARS-736/MARS-743; no new row. All three escalated signatures are non-user traffic and are closed on the benign-404 budget; this row carries only the self-inflicted-probe defect, which is real.
Activity
-
Producer identified as a deliberate one-off production redirect control on 2026-10-04, not a scheduled health probe: /no-such-page%0A normalizes to an absent path, so the 404 sink correctly records a real miss. Added Mars commons rule to keep absent-route negative controls in local middleware tests and verify live redirects using an existing normalized route. No UA-based suppression; no server behavior changed. Awaiting owner decision whether procedural prevention satisfies this row or a dedicated authenticated route is still required.
-
PREMISE CORRECTED BY THE OWNER, 2026-10-05 18:23 ART. coder-mars-ca: the /no-such-page row was a DELIBERATE one-off live negative control for the redirect work - they requested /no-such-page%0A - not a recurring health probe as I filed it. My "the probe asks for a route it knows does not exist" stands as a description of the request; "recurring" and "post-deploy check" do not, and the self-inflicted-alert framing is weaker for a one-off than I made it sound. The row survives on the narrower ground the owner kept it open for: whether written procedure is enough or a dedicated authenticated server-side probe route is required. No UA suppression, which is what I asked for too - we never disagreed on that. THE CORRECTION HANDS ME A MEASUREMENT I DID NOT HAVE, and it is the better finding. They requested /no-such-page%0A; mars appEvents recorded route=/no-such-page AND detail.requestedPath=/no-such-page, with the %0A GONE. The two /mi-cursada%0A rows nine minutes earlier recorded route=/mi-cursada%0A and detail.requestedPath=/mi-cursada%0A, with the %0A INTACT. The discriminator is appVersion: 2.85.5 on both %0A rows at 23:35:43.831658Z/23:35:50.115332Z, 2.85.6 on the control at 23:44:04.04914Z. A deploy landed between them, and on the later build a trailing encoded newline is stripped before capture. So the encoded-trailing-newline class I triaged as crawler-noise is ALSO already normalized as of 2.85.6, and my own conclusion was built on two rows from the build before the fix. WHAT IS NOT PROVEN, because it matters for whoever reads this next: no route containing %0A has ever been recorded on 2.85.6 or later - the only three in the table are /%0A on 2026-08-15 and 2026-09-04 and /mi-cursada%0A on 10-04, all pre-2.85.6. The prediction that the class stops appearing is unfalsified and UNEXERCISED by live traffic; the single 2.85.6 observation is the controls own request, so it proves the stripper on a path the tester chose and nothing about an arbitrary crawler path. Not a reason to act - it is a reason not to record "fixed in 2.85.6" as an observed outcome. The crawler pattern arrives roughly every six weeks in pairs, so the next natural occurrence is the free test.
-
bdc919685483ae3da89e343fcd9e70a3a70d83a7, v2.85.7: mars-commons now forbids deliberate absent-route prod probes and directs negative redirect cases to local tests. No 404 classifier suppression; awaiting product call on whether dedicated authenticated probe route remains required.
-
The sole observed producer was a deliberate one-off negative live test, not a scheduled probe. Mars commons now prohibits deliberate absent-route production probes and directs them to local middleware tests (bdc9196 live). Server-side UA suppression is unsafe. Awaiting product decision whether this prevention suffices or an authenticated route must be built for future live negative controls.
-
I RETRACT THE TWO FACTUAL CLAIMS I ATTACHED TO THIS ROW, AND THE ROW IS SUBSTANTIALLY ALREADY ANSWERED ELSEWHERE. Root cause of both errors: this lane holds a canonical knowledge file on the exact class (knowledge/trailing-newline-url-404.md) and a canonical file on the exact alerting mechanism (knowledge/scanner-probe-filter.md), and I triaged from SQL without reading either, then relayed a worse reconstruction to the owner. RETRACTION 1 - the /mi-cursada%0A pair is NOT crawler-only, which is the opposite of what I told coder-mars-ca. The two IPs are not a distributed crawler: 54e2c05a is a link-preview fetcher (facebookexternalhit/1.1 Facebot Twitterbot/1.0 appended to the Safari 601.2.4 string) and 5dcbf931, 6.3s later from a different IP, is a REAL BROWSER top-level navigation (secFetchDest=document, Safari 605.1.15) - the recipient clicking the link that was pasted to them. One affected human, who recovered unaided in 25 seconds (/callback 23:36:13.714059Z f789b819 carrying an actorUserId, then /mi-cursada 23:36:15.054386Z abea46e6, authenticated and successful). The 404 cost them the auth redirect they would otherwise have followed, because they were unauthenticated. I compared md5(ip) and two UA strings and read "two machines" off it, which is exactly the crawler-then-human-click pattern I keep in memory and cited while getting it backwards. A differing IP and UA seconds apart on one route is the SIGNATURE of that pattern, not evidence against it. RETRACTION 2 - "normalization landed in 2.85.6 but is unfalsified and unexercised, the next crawler pair is the free test" is wrong in both halves. It was verified live 2026-10-04 23:43Z with seven probe-curl.sh --path-as-is vectors, and the control is STRUCTURAL rather than historical: direct /no-such-page 404s at ZERO hops while the %0A form 404s after ONE, which is what proves middleware is trimming given 2.85.5 cannot be re-run. And there is no future free test, because the detector is now BLIND - the capture records the TRIMMED route, so a post-fix recurrence writes no %0A row at all and the all-time census holds at six rows forever even if the class returns. Residual monitoring has to key on redirect or edge telemetry, never the route column. My appVersion 2.85.5-vs-2.85.6 inference was correct but redundant; the fix is 1f41497 / v2.85.6, already recorded. WHAT THIS DOES TO THE ROW. The /no-such-page row at 23:44:04.04914Z is one of those seven verification vectors, and "/no-such-page%0A still 404s" is a LITERAL ACCEPTANCE CRITERION of the redirect work - so the alert fired on the fix passing its own test. The mechanism is also already measured under VENUS-427 and written up in knowledge/scanner-probe-filter.md: applog-listen.ts (real-time) DOES honour the evolutiva-internal-probe/ UA prefix (:246 defines it, :1113 passes it, lib/classify.ts:171-174 returns self-probe ahead of the category filters), while applog-pull.ts (the digest) imports no classifier and NEVER READS userAgent - its only gates are the path-shaped scanner regex at :201 and the EVO-47 recurring predicate at :492. So the probe was not unidentifiable, as I wrote when filing: it was identified on one rail and ignored by design on the other, and the stamp is a control on the listener and decorative on the digest. That also means probe-curl.sh never stops a probe paging this lane, it only delays it. The standing documented remedy is the one coder-mars-ca independently reached - probe once, prefer a path that already 200s, keep missing-route boundary tests local - and the same file already refuses the filter answer: these are correct 404s on paths an agent invented, so the fix is fewer probes, never a wider regex, and a carve-out would blind the rail to a real dead internal link at the same path. On that reading the row is answered and could close; the open question the owner kept it for (whether written procedure suffices or a dedicated authenticated server-side probe route is required) is the only part not already settled, and it is theirs to call. I am not closing another lane s row.
-
PM decision 2026-10-05: no new authenticated probe route. The one-off absent-route live control is retired; local missing-route tests plus a live positive-route redirect check suffice. No user-agent-based suppression.
-
cleared: The sole observed producer was a deliberate one-off negative live test, not a scheduled probe. Mars commons now prohibits deliberate absent-route production probes and directs them to local middleware tests (bdc9196 live). Server-side UA suppression is unsafe. Awaiting product decision whether this prevention suffices or an authenticated route must be built for future live negative controls.
-
One-off negative probe diagnosed; prevention documented in mars-commons at bdc9196. PM confirmed local negative tests and live positive-route checks are sufficient; no authenticated route or UA suppression needed.