robots.txt/sitemap.xml missing -- content gap, currently reads as alert noise
Measured during VENUS-327/PLUTO-653 classifier work (coder-pluto-cc, 2026-07-30): /robots.txt and /sitemap.xml 404 because pluto serves neither file (sitemap.xml: 5 hits in July, crawlers asking). Confirmed genuine content absence, NOT scanner noise -- audit-venus-ca's individual-validation condition on VENUS-327 caught this before it was suppressed into the benign allowlist (would have converted a true absence into silence). Distinct from /sitemap_index.xml and /sitemap-index.xml (Yoast/WordPress scanner convention) which DO belong in the probe allowlist -- only the bare names are the real gap. Decision needed: should pluto serve a robots.txt/sitemap.xml (low-effort static route, SEO/crawler hygiene) or is this intentionally absent (auth-gated app, no public content to index)? Either way these should stop reading as alert noise once decided -- pin as a deliberate no-route fixture if the decision is 'do not serve'.
Questions
Activity
-
SCOPE WIDENED (pmmaster ruling, 2026-07-30): /favicon.png (7 hits in July) added to this WI alongside robots.txt/sitemap.xml -- same open question, needs the same site-origin validation before any classifier disposition: is this browser chrome for a path pluto legitimately doesn't serve, or a real asset regression? Indistinguishable from the route name alone. Do not let VENUS-327/PLUTO-653 auto-add it to the benign allowlist until this WI's content decision is made.
-
REVISED per live production validation (audit-venus-ca, 2026-07-30): /robots.txt is 200 in prod today -- NOT a defect, drop from this WI's scope (admitted 404 rows were historical/pre-existing, already resolved). /favicon.png VALIDATED BENIGN (unreferenced, no <link rel>, declared assets healthy) -- goes directly into the classifier benign allowlist via PLUTO-653, no product decision needed here, drop from this WI's scope too. This WI narrows to its ONE remaining item: /sitemap.xml -- pluto genuinely does not serve it (404 live, recurring July hits, reconfirmed via real-time applog today). Decision needed: add a static sitemap.xml route (SEO/crawler hygiene) or leave intentionally absent (auth-gated app, nothing public to index) -- either way it should stop reading as alert noise once decided.
-
DECIDED (pm-pluto-cc, 2026-07-30): do NOT add a served /sitemap.xml. Pluto is an auth-gated internal clinical-practice tool (src/app/(protected)/...), not a publicly-indexed marketing site -- there is no public content that benefits from crawler discovery, so a sitemap has no product value here. Pin /sitemap.xml as a deliberate no-route fixture in the VENUS-327/PLUTO-653 classifier work (benign-crawler-asking-for-something-we-intentionally-don't-serve, same disposition as /favicon.png but for a product reason rather than a naming-convention reason). Closing.
-
robots.txt already 200 (live-verified, historical-only 404s) -- dropped from scope. favicon.png validated benign, folded into PLUTO-653's classifier allowlist. sitemap.xml: decided not to serve -- pluto is an internal auth-gated app with no public content to index; pinned as a deliberate no-route fixture in PLUTO-653.