Quiet well-known-path 404 noise (robots.txt, apple-touch-icon[-precomposed].png) in page_not_found telemetry

Sub-tickets

Activity

• wi-cli-venus created · 2026-06-21
• wi-cli-venus commented · 2026-06-21
  Post-PLUTO-180 organic traffic surfaces steady well-known-path 404 noise: /apple-touch-icon.png, /apple-touch-icon-precomposed.png (iOS Safari auto-probe), /robots.txt (crawlers). All genuine 404s, correctly logged by the new catch-all sink, warn/alert=false, 0 email - NOT a regression, the sink working as designed. But they pollute page_not_found telemetry with low-value organic noise every iOS visitor + every crawler generates. Options for coder (pick per intent, NOT yet prioritized): (a) add the real assets - a robots.txt (sensible for the public domain; disallow-crawl for an authed app) + apple-touch-icon; (b) matcher-exclude these well-known paths in src/proxy.ts (favicon.ico already excluded) so they never reach the capture sink; (c) classifier downgrade well-known-bot-paths to a quieter tier. Low priority, 0 user/email impact. Hold for Elazar/PM prioritization before implementing.
• wi-cli-venus commented · 2026-06-21
  Operational note: the well-known-path 404 alerts (apple-touch-icon, robots.txt) recur every few minutes from organic iOS/crawler traffic. A narrow real-time-suppression request DM'd to applog-pluto-cc went unactioned over 30+ min while alert emits continued - suggests applog-pluto-cc is a non-interactive emitter daemon (alive, forwarding) rather than an agent that reads/acts on DMs. So near-term PM-ping suppression likely needs maintainer-side daemon config, not a DM. This reinforces fixing at the SOURCE (option a: add robots.txt + apple-touch-icon assets, or option b: matcher-exclude in proxy.ts) over rail-side suppression. Still low-pri, 0 email/user impact, hold for Elazar prioritization.
• wi-cli-venus commented · 2026-06-21
  Audit security-triage (2026-06-21 17:12Z) surfaced an SEO angle within the well-known-path 404s: /robots.txt AND /sitemap.xml both 404 - these are legit SEO files that are simply not configured, so part of this WI is a real (minor) SEO gap, not pure noise. Recommend splitting the fix intent: (SEO) add a proper robots.txt (disallow-crawl appropriate for an authed app) + sitemap.xml if wanted; (noise) matcher-exclude apple-touch-icon* or add the icon asset. Separately, a credential-recon scanner hit prod same window (25 secret/config paths: .git/*, .env*, *credentials*.json, /api/config) - audit confirmed ALL 404 with zero exposure, NO incident; noted here only as the traffic context that surfaced the SEO gap. Security side needs no action; the scan rows log correctly as security/warn/alert=false/0-email.
• wi-cli-venus assigned · 7w ago
  coder-pluto-cc
• wi-cli-venus completed · 7w ago
  SUPERSEDED - all three named paths were already handled, by two later WIs, and none of them is 404ing today. Measured against the production origin 2026-08-22: /robots.txt 200 (PLUTO-195 added src/app/robots.ts, disallow-all, which is the correct posture for an auth-gated internal app); /apple-touch-icon.png 200 and /apple-touch-icon-precomposed.png 200 (public/ ships bare + 120x120/152x152/167x167/180x180, each in plain and -precomposed form). Sizes NOT shipped still 404 (-192x192, -76x76 confirmed) but were already down-tiered to info by ICON_PROBE_RE in not-found-classify.ts (PLUTO-184 era, admitted 2026-06-25 on measured recurring noise), so they never reach the operator either. So there is no telemetry noise left to quiet and nothing was added to a suppression list - which is the right outcome: the standing doctrine in NO_ROUTE_BY_DESIGN_RE is that a silenced 404 for something Pluto SHOULD serve is invisible forever, and the fix for robots.txt was to SERVE it, not to mute it. One real defect found while verifying, fixed in v2.23.59: the ICON_PROBE_RE comment claimed Pluto serves 'the bare 192x192 + 120x120 variants'. 192x192 404s. Wrong in the one direction that misleads, since the declared-and-missing vs never-advertised distinction immediately below is applied BY HAND against exactly that kind of list. Corrected with the measured set and a note that only 180x180 is actually advertised (layout.tsx icons.apple). Comment-only, no regex or behaviour change; 760/55 green. LIMIT: this is a live-origin measurement of what the paths return TODAY, not a read of the page_not_found corpus - I have no prod DB access, so I have not confirmed the historical rows the WI was filed on have stopped arriving, only that the routes no longer produce them.