Fetch + normalize service (html/txt/md/yaml/zip)
Build the fetch/normalize service itself. Input: URL + requested format(s). Output: html and/or txt and/or md and/or yaml and/or zip returned to the calling agent. Normalization is the value-add, not just proxying bytes: html->txt, html->md, and a yaml wrapper carrying metadata (source url, fetch timestamp, http status, final url after redirects, content-type, byte size, sha256) alongside the body - this is what makes the output llm-friendly and citable. zip when several formats or assets are requested at once. Must send a browser-realistic UA and Referer by default (that is the whole point: plain curl gets 403 from InfoLeg, browser UA gets 200) with per-request override. Decode correctly: detect charset from headers and meta, do not assume utf-8 - InfoLeg served cp1252 and a latin-1 guess silently ate the em dashes. Language/runtime is bin-venus-cc's call; state it in the blackboard answer.
Activity
-
parent=#3338
-
Q8 live instance, found by bin-venus-cc on the first URL tried. An InfoLeg fetch 302s to /infolegInternet/mostrarArchivoInexistente.do (literally: show nonexistent file) which returns HTTP 200 with 2389 bytes of HTML saying the document does not exist. A fetcher reporting {status:200, format:md} would hand the caller clean markdown of a not-found page, indistinguishable from a real document. bin-whey-cc named this defect class independently in the same round. Consequence for the design: final_url must ALWAYS be surfaced, and a redirect to a known interstitial must be a distinct outcome, not a success. This is the second and stronger reason the yaml metadata envelope ships in v1.
-
bin-venus-cc WITHDRAWS its recommendation to cut yaml from v1, on seeing the full brief. Reason: the specified yaml is a metadata envelope (source url, timestamp, status, final url, content-type, size, sha256 alongside the body), not a format conversion, and it is what carries the final-url and status that make the other formats safe to act on. Ship yaml in v1. The cut of zip stands - zip is a container, deferred until a named consumer states a shape.
-
Charset rule, established by bin-venus-cc against a live InfoLeg response, with standards basis. The response declares Content-Type: text/html;charset=iso-8859-1 but the bytes are cp1252. Bytes 0x80-0x9F are UNDEFINED in ISO-8859-1 and are exactly where cp1252 places the em dash (0x97), curly quotes and the ellipsis. So a latin-1 decode of a declared-latin-1 page succeeds silently and eats precisely those characters - this is the observed em-dash loss in the Ley 26.657 extraction on whey. RULE: when a page declares iso-8859-1, decode as cp1252. That is what the WHATWG encoding spec mandates and what every browser does, which is why the page renders correctly in a browser and wrongly through curl. Detection order: BOM, then meta charset within the first 1024 bytes, then the HTTP header with the latin-1 to cp1252 substitution applied, then a detector, then utf-8. Never decode with errors=replace silently. Record the chosen encoding AND its source in the metadata envelope; declare low-confidence detection rather than picking silently (bin-whey-cc).
-
API dropped by Elazar 2026-09-04 in favour of a local script. A network service was the wrong shape: the failure this was built for is a WAF User-Agent check, which a documented curl/python recipe fixes on every host at zero infra cost. The API's only real adds over that were a centralised egress IP, format normalisation and a shared cache - the first is a NEGATIVE on venus (datacenter ASN draws harder Cloudflare challenges than residential whey), the second is a script not a network service, the third was speculative. Replaced by getpage.py in ~/.local/bin (sh.git, already on PATH on whey/venus/lezama), tracked in project agent-ops. Net effect: no SSRF primitive on the fleet hub host, no DNS record, no TLS, no API key to issue or store, no systemd unit to maintain.