getpage.py: fetch a URL past UA-checking WAFs and emit html/txt/md/yaml (replaces cancelled getmethaturl API)

Sub-tickets — 0/1 done

Activity

• wi-cli-venus created · 6w ago
• wi-cli-venus verified · 6w ago
  Independent verification on whey by pm-medida-odaia-cc, getpage.py v1.0. GOLDEN PASSES: --format txt byte-identical to leyes/Ley_26657.txt minus that file's line-1 B.O. date, 29197 bytes, diff empty, 46 ARTICULO, 48 em dashes. NEGATIVES all pass: anexo 999999 rc=6 with zero bytes on stdout and the redirect named on stderr; es.wikipedia utf-8 19 n-tilde 0 mojibake, cp1252 rule correctly does not fire; g2.com rc=5 challenge, no fake success. FORMATS: html 40080 b verbatim, md 29197 b, yaml 29798 b parsing clean with the generic envelope (url/final_url/status/charset_declared/charset_used/charset_source/fetched_at/title/text) and its text block byte-identical to the golden txt. --min-text 0 correctly lets the interstitial through. Charset rule unit-tested on four fixtures against decode_body directly rather than via live pages: header-declared iso-8859-1 and meta-declared iso-8859-1 both yield used=cp1252 with the em dash surviving, declares-nothing yields used=ascii and is not relabelled, declared utf-8 stays utf-8. The bench change bin-whey-cc made (substitute on declared labels only, never detected) is correct and should stay. NOT EXERCISED: only whey. venus and lezama untested, and no dep beyond the whey-verified set was added so they are expected to work, but that is inference not measurement.
• wi-cli-venus audit-finding · 6w ago
  Comment defect, behaviour is correct, no functional impact - but it inverts the stated reason the golden test passes. MEASURED on the golden URL: HTTP header is a bare text/html with NO charset, and the document carries exactly one charset= occurrence, windows-1252, in a meta. getpage.py itself reports charset_declared=windows-1252 charset_source=meta charset_used=cp1252, so the substitution fires on the DECLARED path for InfoLeg, not via the detector. Three comments contradict that: L57 says InfoLeg declares iso-8859-1; the decode_body docstring at L102-106 still describes the REPLACED design (substitution applied to the resolved label whatever produced it) and asserts InfoLeg declares nothing at all on this URL with no meta, contradicting the code twenty lines below it; L130-131 repeats that InfoLeg does not need the rule to fire because the detector answers cp1252 on its own. Risk: the comments tell the next reader the declared-label branch is unexercised by the golden case and detection alone would carry InfoLeg, when the declared branch is the only thing making the golden test pass. Anyone trimming CP1252_LABELS or narrowing that branch on the strength of the comment breaks the corpus. Reported to bin-whey-cc with a fix suggestion (keep the rfc-editor evidence at L127-129, it is real and is the reason for the design; replace only the InfoLeg-declares-nothing sentences). Awaiting a --patch bump; will re-run golden plus the four fixtures on the drop.
• wi-cli-venus audit-finding · 6w ago
  BLOCKER for the odaia project use case, found by pm-medida-odaia-cc on v1.1 while looking for a second golden. getpage.py reports a present, complete 1.27 MB document as missing. MEASURED: anexo 235975 (https://servicios.infoleg.gob.ar/infolegInternet/anexos/235000-239999/235975/norma.htm) returns rc=6 with 'thin content - 36 characters of text from 1271460 bytes ... treat it as a failure, not as the document'. That page is the CODIGO CIVIL Y COMERCIAL DE LA NACION, Ley 26.994. bs4 get_text on the identical bytes returns 1158197 characters; getpage --format txt returns 37. CAUSE is extract_paragraph_lines, not the fetch and not the charset - the document is <br>-delimited. Tag census on the raw source: 13871 <br>, 451 <div>, 151 <td>, 51 <tr>, 8 <table>, exactly ONE <p>. The extractor is <p>-only so it sees one paragraph and declares the document absent. IMPACT here specifically: that document is the most-cited source in the custody case this project exists for. Confirmed present in the fetched bytes and unreachable through the tool: ARTICULO 653 (four-factor test for cuidado personal unilateral, the basis of the filing), 651, 607, 639, 700. TWO FIXES, do not merge them: (1) extraction must handle <br>-delimited and div/table-structured documents, with the hard constraint that the golden stays byte-identical - the empty-<p>-as-spacer rule for InfoLeg <p> pages must survive the generalisation, so golden re-runs every iteration not just at the end; (2) the thin-content heuristic misdiagnoses - 1.27 MB of markup yielding 36 chars is an EXTRACTOR failure, 2581 bytes yielding 0 is a genuine interstitial, same exit code today for opposite causes and the message actively misled. Branch on markup-to-text ratio, distinct rc, and stop saying not-found for a document that is present. Reported to bin-whey-cc. Related and accepted from bin-whey-cc: the InfoLeg corpus is MIXED - 175977 and 60036 declare windows-1252 (substitution is a no-op, python decodes it natively), 235975 declares ISO-8859-1 (substitution actually fires). Independently reproduced. So trimming iso-8859-1 from CP1252_LABELS is the dangerous edit, not windows-1252, correcting my earlier framing on the comment finding.
• wi-cli-venus verified · 6w ago
  v1.2 (334fca0) verified on whey by pm-medida-odaia-cc. BOTH FIXES WORK. Golden byte-identical, 29197 bytes, empty diff, re-checked after the extractor rewrite. CCyCN anexo 235975 now rc=0, 1165140 chars, 2687 ARTICULO hits, arts 653/651/607/639/700 all present, and 653 extracts with its four ponderación factors intact as a/b/c/d - the blocker is cleared. Three negatives still pass: interstitial rc=6, Cloudflare rc=5, es.wikipedia utf-8 rc=0 with 25 n-tilde and 0 mojibake. Four charset fixtures unchanged against decode_body directly: header-declared and meta-declared iso-8859-1 both to cp1252 with the em dash surviving, declares-nothing stays ascii and is not relabelled, declared utf-8 stays utf-8. ADOPTED anexo 235975 as a second golden - it declares ISO-8859-1 via meta, making it the first case that actually exercises CP1252_LABELS, since 175977 declares windows-1252 which python decodes natively. NOT EXERCISED: whey only, venus and lezama still untested.
• wi-cli-venus audit-finding · 6w ago
  Two defects found on v1.2, reported to bin-whey-cc. (1) SILENT SUCCESS, worst class: --format md on the CCyCN returns rc=0 with 109 characters against txt's 1165140, i.e. 0.009 percent of the document, and the whole output is a single nav link to verNorma.do. Verified WITHOUT --min-text 0 - the thin-content guard does not cover md, so it exits 0. Same failure class as the silent latin-1 decode this tool exists to prevent: the caller gets a plausible small file and no error, and nothing reports the document was dropped. Strictly worse than the txt extractor bug, which at least exited 6. Requested that Fix 2's ratio check cover EVERY format independently of when md gets the tree walk. bin-whey-cc had disclosed md as a known gap but characterised it as an under-read; measurement shows silent near-total data loss. (2) UNHANDLED RecursionError, rc=1, outside the documented 0/2/3/4/5/6/7 set: getpage.py on raw.githubusercontent.com/torvalds/linux/master/MAINTAINERS --format txt dies with a bs4 traceback from walk() at L246, 'Previous line repeated 991 more times'. Characterised with synthetic fixtures against extract_paragraph_lines: nesting depth 900 OK, 1000 RecursionError, sys.getrecursionlimit()=1000. The new tree walk recurses once per DOM level, so any ~1000-deep document crashes with a traceback instead of a diagnosis. Fix is an explicit stack, iterative - NOT a raised recursion limit, which moves the cliff and risks a segfault. The walk design is right, only its implementation is recursive. (3) RETRACTED, not a bug: w3.org .xsd returning rc=5 looked like a challenge false positive; checked headers directly and it is 403 with cf-mitigated: challenge, server: cloudflare, cf-ray present. Detection is correct. Priority given: crash first (no workaround), md silent-success second (workaround is txt or yaml), md walk third.
• wi-cli-venus verified · 6w ago
  v1.3 (94dc190 crash), v1.4 (98245f8 yield guard), v1.5 (6d84fd1 md walk) all verified on whey by pm-medida-odaia-cc. Golden byte-identical throughout, 29197 bytes, empty diff. CRASH FIXED: raw.githubusercontent MAINTAINERS rc=0 at 800739 chars; synthetic nesting depths 900/1000/5000/20000/50000 all pass against extract_paragraph_lines, where v1.2 died at 1000. The iterative stack was the right fix, not setrecursionlimit. md WALK: CCyCN md rc=0 at 1165371 chars with 2687 ARTICULO hits and 653 present, against 109 chars in v1.2. GUARD confirmed live in main() L522, FORMAT_YIELD_FLOOR=0.25, html exempt as raw source. Negatives rc=6 interstitial and rc=5 Cloudflare unchanged. NOT EXERCISED: whey only; venus and lezama still untested.
• wi-cli-venus note · 6w ago
  SELF-CORRECTION plus two findings, all on the same root cause. I challenged bin-whey-cc's claim that the 59 <b> tags on the golden page are empty after noise-stripping and that this is why golden md equals golden txt. I measured 59 NON-EMPTY and was about to report a silently disabled inline renderer. bin-whey-cc was right; I was wrong. I parsed with html.parser, main() parses with lxml at L462, and the parser is the entire difference. MECHANISM: the source markup is <B><P ALIGN=JUSTIFY>, inline opening before block, which is invalid nesting. lxml resolves it by closing <B> immediately, leaving 59 EMPTY <b> elements with the text as a sibling; html.parser keeps the text inside. Measured on the golden page with identical strip_noise, only the parser varying - lxml: <b>=59 non-empty=0, md '**'=0, md==txt True. html.parser: <b>=59 non-empty=59, md '**'=118, md==txt False. html5lib: <b>=105 non-empty=59, md '**'=118, md==txt False. CONSEQUENCE 1, md is silently lossy on this publisher and degrades exactly to txt: the bold lxml discards is not decoration, it is the semantic structure of every InfoLeg law - the title block, 'Capítulo I', and every 'ARTICULO N° —' header. The v1.4 yield guard cannot catch it because the two outputs are identical in LENGTH; the loss is structural, not volumetric - the same blind-spot class v1.4 fixed, one level up. Requested a documented known-limitation so nobody picks md over txt for InfoLeg expecting headings. CONSEQUENCE 2, lxml is load-bearing for the golden and nothing says so: same measurement on txt output gives lxml 28583 chars, html5lib 28583, html.parser 28629, so swapping to html.parser breaks the golden by 46 characters. The parser string at L462 is an unremarked implementation detail, lxml is not in the dep note, and html.parser is the stdlib default a future reader would reach for. Requested a comment at L462 carrying the 28583-vs-28629 number. Neither blocks use; txt and yaml are correct and are what this project uses.
• wi-cli-venus decision · 6w ago
  html5lib NOT adopted for now, recorded with evidence so it stays revisitable rather than becoming a dropped thread. bin-whey-cc measured that html5lib recovers the bold lxml discards AND still matches the golden (txt 28583, same as lxml), so it is strictly better on InfoLeg output quality, and it correctly declined to adopt it on a whey-only confirmation. DECLINED BY pm-medida-odaia-cc on grounds of use, not risk: this project standardised on txt + yaml for the /leyes/ corpus and does not consume md, so the recovered bold buys nothing today while the cost is a new dependency needing confirmation on three hosts. REVISIT TRIGGER: if md ever becomes a format anyone actually reads for InfoLeg, adopt html5lib, and confirm it on venus and lezama BEFORE shipping - not after.
• wi-cli-venus audit-finding · 6w ago
  Minor, last one, reported to bin-whey-cc. Now that lxml is load-bearing for the golden, I checked whether a host missing it would silently diverge into html.parser output - which would be the silent-divergence class this script exists to prevent, and would break the golden by 46 characters invisibly. It does NOT: bs4 raises FeatureNotFound, no silent fallback. Verified with a find_spec import blocker rather than asserted; my first attempt used the legacy find_module API that Python 3.13 ignores and returned a false 'silent fallback' reading, so the second measurement is the correct one and the first should be disregarded. HOWEVER L489 is unguarded, so on such a host this surfaces as an unhandled traceback with rc=1 - outside the documented 0/2/3/4/5/6/7 set and the same 'a traceback is not a diagnosis' case as the v1.2 RecursionError. Requested a wrap on the BeautifulSoup call catching FeatureNotFound, stating that lxml is required and load-bearing for output fidelity, naming the host and pointing at the install. Failing loudly is already the correct behaviour; it just should not present as a crash.
• wi-cli-venus note · 6w ago
  COVERAGE STATEMENT for everything pm-medida-odaia-cc verified across v1.0 through v1.6. EXERCISED, all on whey only: golden (InfoLeg 175977, byte-identical to leyes/Ley_26657.txt minus its line-1 date, re-run on every bump); second golden (InfoLeg 235975 CCyCN, the only case that actually exercises CP1252_LABELS since it declares ISO-8859-1 while 175977 declares windows-1252 which python decodes natively); the three negatives (interstitial rc=6, Cloudflare challenge rc=5 confirmed real via cf-mitigated header, es.wikipedia utf-8 with 0 mojibake); four charset fixtures against decode_body directly; recursion depths 900 through 50000 against extract_paragraph_lines; a large plain-text document (kernel MAINTAINERS, 800739 chars); all four formats; the FORMAT_YIELD_FLOOR guard read at its call site; three-parser comparison (lxml/html.parser/html5lib) on the golden; and bs4's behaviour when lxml is absent. NOT EXERCISED, and this is the remaining gap, owned by the bin lane not by me: venus and lezama. The script ships to both through sh.git and pm-medida-odaia-cc is host-bound so cannot test there. Dependency set needing confirmation on both hosts is yaml, bs4, lxml, requests, charset_normalizer - lxml especially, being the parser the golden now depends on. Also never exercised anywhere: a non-InfoLeg site with valid <b> nesting under lxml, so md's inline rendering is verified only on es.wikipedia (17 headings, 220 list items, 369 links per bin-whey-cc) and by minimal fixtures, not on a second real publisher. OPS-193 is done from the odaia side once the cross-host dependency confirmation lands.
• wi-cli-venus audit-finding · 6w ago
  TWO HOLES in the v1.7 EXIT_DEPENDENCY=8 guard, found by pm-medida-odaia-cc testing each dependency arm separately through the real CLI with a PYTHONPATH shim rather than only the two arms bin-whey-cc verified. lxml and requests both pass, rc=8 with the host named correctly. The other two do not. (1) yaml missing with --format yaml gives rc=1 AND A TRACEBACK, not rc=8: the guard covers module-level imports but 'import yaml' is local to to_yaml, so it escapes the guard and dies the exact way rc=8 was added to prevent. Invisible on --format txt, which is what makes it easy to miss - the failing combination is the missing library AND the single format that needs it. Repro: PYTHONPATH=<dir containing a raising yaml.py> getpage.py <golden url> --format yaml. (2) charset_normalizer missing gives rc=0 AND SILENT MOJIBAKE, which is worse: it is imported inside a try/except Exception in decode_body, so its absence degrades to label,used_source='utf-8','fallback' with no warning on any channel. Invisible on the golden because InfoLeg declares windows-1252 via meta so detection never runs and output stays byte-identical - which is exactly why a dependency inventory would never surface it. On a page declaring NOTHING, detection is the only thing between cp1252 bytes and garbage. MEASURED with charset_normalizer blocked, cp1252 bytes for 'niño — dash': used=utf-8, src=fallback, text='ni?o ? dash' with replacement characters, rc=0, no stderr. This is the tool's own founding failure mode - a decode that succeeds silently and eats the punctuation - reintroduced through the dependency path instead of the label path. The dependency message already lists charset-normalizer as required, so code and message disagree about whether it is optional. SUGGESTED SPLIT, not one fix: yaml joins the guarded set (module-level import, or guard the local one) so no format can outrun the check; charset_normalizer is bin-whey-cc's judgement call - either promote it to a hard rc=8 matching what the message already claims, or keep the graceful fallback but make it LOUD on stderr, saying detection is unavailable and an undeclared charset is being decoded as utf-8 unverified. It must not stay silent, because the caller cannot distinguish that run from a good one. Neither affects the golden, the corpus, or this project's use (txt and yaml on a declaring publisher). whey-measured only.
• wi-cli-venus note · 6w ago
  CROSS-HOST STATUS from bin-whey-cc, recorded. LEZAMA confirmed by run and it is the strongest evidence in the exercise: first run there returned rc=8 with 'missing dependency on lezama: No module named bs4' and empty stdout on a host that genuinely lacked bs4 and lxml - the v1.7 guard proven end to end in the wild, not induced with an import blocker. After install, golden 29197 bytes, sha256 b6e7f8bc84dfa4b3863b512dc7faa4b01687fb1315748f6b94b4d48fe6b27761, identical to whey's. Byte-identity across two hosts, not merely equal length. Version spread the golden survives: whey lxml 6.0.2 / bs4 4.13.4 / py 3.13.5; lezama lxml 6.1.3 / bs4 4.15.0 / py 3.11.2. bin-whey-cc corrected an offered 'cross-major' claim to minor - 6.0 vs 6.1 - and explicitly does not claim the lxml-major axis. VENUS confirmed by inventory only, not by run: bs4 not installed (requests 2.32.3, lxml 5.4.0, pyyaml 6.0.2, charset_normalizer 3.4.2 all present) and the file is not in that checkout yet. bin-venus-cc will not pull on a peer's say-so and bin-whey-cc will not ask - correct under the standing rule, and pm-medida-odaia-cc agrees no pull should be pushed for this; that half stays open until the venus checkout moves for its own reasons. INSTALL NOTE worth carrying fleet-wide: on lezama 'apt-get install python3-lxml' is impossible rather than merely absent - jammy candidate 4.8.0 Depends python3 (< 3.11) against a 3.11.2 interpreter, and PEP 668 blocks plain pip. Landed via pip --user --break-system-packages, nothing system-wide touched. 'Just apt it' is the wrong instruction for that host.
• wi-cli-venus verified · 6w ago
  v1.8 (9e8b41c yaml) and v1.9 (9d7e66c charset-normalizer) verified on whey by pm-medida-odaia-cc - both guard holes closed. Golden 29197 byte-identical on both. yaml blocked + --format yaml -> rc=8 naming host, package, and that other formats still work; yaml blocked + --format txt -> 29197, golden intact, so a missing optional dep no longer takes down formats that never needed it. charset_normalizer blocked -> loud stderr warning stating detection is unavailable, the page declares no charset, bytes are decoded utf-8 UNVERIFIED and output is NOT trustworthy. Catching Exception rather than ImportError is correct (a library that raises on import is unusable the same way as one that is absent) and keeping pyyaml a local import is right - hoisting it would fail html/txt/md on a host that never needed it. bin-whey-cc's own note that its v1.7 pass could not have caught the yaml hole, having only tested the two module-level imports, is the accurate account.
• wi-cli-venus audit-finding · 6w ago
  CONFIDENTLY WRONG CHARSET DETECTION - escalation of an observation bin-whey-cc reported unprompted and set aside as small-sample. It is not small-sample. bin-whey-cc saw charset_normalizer answer cp1250 instead of cp1252 on an 11-character fixture and framed it as detection quality on a tiny input. pm-medida-odaia-cc scaled it; it does not improve. Undeclared cp1252 bytes of realistic Spanish prose, detector PRESENT and working: 97 b -> cp1250, 485 b -> cp1250, 4850 b -> cp1250, 48500 b -> cp1250. MISDETECTED at every size, every n-tilde wrong. 49 KB is a real document, not a fixture. The em dash SURVIVES because cp1250 shares 0x97, which is exactly what makes it look benign - it is the letters that die: 'El niño español comió — año 2026' decodes as 'El nińo espańol comió — ańo 2026'. Correct punctuation, corrupted letters, rc=0, and NO warning, because the v1.9 warning fires when the detector is ABSENT, not when it is WRONG. That is worse than the hole just fixed: an absent detector announces itself, a confidently wrong one does not. NOT AFFECTING THIS CORPUS - InfoLeg declares windows-1252 via meta so detection never runs, golden re-verified byte-identical - but this project reads Argentine legal sources generally and Spanish-language sites that declare nothing are not exotic; each one silently loses its n-tilde. GAP IDENTIFIED, not a bug: there is no --charset override. Full flag set is --format --out --ua --referer --timeout --max-redirects --min-text. A caller who can SEE the detector is wrong has no recourse short of --format html and decoding by hand, which is the wheel this script exists to stop people reinventing. Requested a --charset flag that skips resolution entirely - cheap, and the honest escape hatch for a step the tool now documents as a guess. SECONDARY, offered as weaker and explicitly bin-whey-cc's call: for an undeclared page whose detected label is a Latin-family single-byte codepage, cp1252 is a better default than a detector's cp1250 - it is what browsers assume and what the WHATWG rule already encodes for declared labels. Not pushed, since it rests on one Spanish fixture and a --charset flag makes it survivable either way. whey-measured only.