scrp-sysstat-alert-venus: SWAP_THRESH measures occupancy, not pressure - and the alerter has no repeat suppression
REPORTED by nw-venus-cc 2026-09-12 20:00 ART: the 50% swap threshold has fired hourly for 14h on a stable 57-59% plateau. FILED-NOT-STARTED under the fleet-wide stop; no code change made. TWO INDEPENDENT DEFECTS, and raising the number fixes neither. (a) WRONG METRIC FOR THE CLAIM. Line 17 takes field 4 of 'sar -S 1 1' = %swpused, which is swap OCCUPANCY - how full the swap device is - and the alert reads as memory PRESSURE. Measured on venus 2026-09-12 ~23:00 ART: swap 4.7/8.0Gi = 59%, but vmstat swpd FLAT at 4934708 across 5 consecutive samples, si/so 0-10 and 0-21 kB/s, and sar -W for the day averages 5.67 pswpin/s and 12.61 pswpout/s. Pages resident in swap and never touched are not pressure; the paging RATE is, and it is ~0. Coverage: ~20h of sar -S occupancy, one day of sar -W, 5s of vmstat. Per-process swap attribution NOT measured - see reference-tmpfs-swap-does-not-attribute-to-a-process. (b) NO REPEAT SUPPRESSION. Neither scrp-sysstat-alert-venus.sh (28 lines) nor notify-nw.sh (223 lines) holds any state file, last-sent key or cooldown - grepped for state/last/dedup/seen/hash/cooldown/since and the only hits are changelog prose. So the flood is structural: ANY sustained breach re-sends every cycle at ANY threshold. This is the half that actually produced nw's complaint, and it reproduces at 75% the moment swap climbs there. Same family as project-sys-check-alert-leg-repair: severity IS the dedupe. DO NOT simply raise SWAP_THRESH. Occupancy is still worth keeping, re-scoped to CAPACITY/headroom rather than pressure - venus OOM-cascaded at 28 sessions and is now running 36. But a STATIC line cannot distinguish 'plateaued at 59%' from 'climbing through 59%' and those are opposite verdicts: today's sar -S went 37% at 06:30 to 59% by 13:30, then held 58-59% for nine hours. Direction wanted: alert on the paging RATE for pressure, and on headroom-plus-SLOPE for capacity, with per-condition suppression so an unchanging condition is stated once. Derive the lines from observed baseline, not from a copied number - reference-a-threshold-copied-from-a-cadence-goes-stale, reference-a-guc-is-not-a-baseline. BEFORE PATCHING: confirm the maintainer-agents registry row for scrp-sysstat-alert-venus.sh rather than inferring the seat from the bin lane.
Activity
-
Parked under the fleet-wide stop (2026-09-12 02:56Z). Triaged and measured same-turn; no code change. Resume needs the stop lifted.
-
bin-venus-cc
-
RELATED FIX SHIPPED 2026-09-18 in the same script, NOT this row's defect - OPS-264 (SWAP_THRESH measures occupancy, not pressure, and the alerter has no repeat suppression) is UNCHANGED and still deferred. scrp-sysstat-alert-venus.sh v1.3 fixes two things found while checking the venus twin after nw-lezama-cc reported a wrong-column bug in the lezama twin: 1. await was read from sar -d $8 = aqu-sz (QUEUE DEPTH), not $9 = await (LATENCY). The alert compared a queue length against a 20ms threshold, so it could neither fire on real latency nor be trusted if it did. THE TRAP: the header line carries a TWO-token timestamp ('07:56:19 AM') where the data line carries the single token 'Average:', so header $k lines up with data $(k-1). Counting names in the header and indexing the data is off by one every time - and the file's own comment listed the columns CORRECTLY while the code disagreed with it, so reading the comment would not have caught it. Mapped the header row onto the data row on this host instead. 2. An empty util/await now raises 'disk UNKNOWN' instead of passing silently. awk_gt on an empty string compares 0 against the threshold and returns false, so a sar that fails to run reads exactly like a healthy disk. The swap branch already treated 'no samples' as a breach; these two did not. BOTH FORCED, NOT OBSERVED - and the first attempt at (1) DID NOT DISCRIMINATE: on an idle disk $8 and $(NF-1) are both 0.00, so the control passed while proving nothing. Re-run under four parallel dd oflag=direct writers: aqu-sz 0.02 vs await 0.37, ~18x apart. (2) forced by shadowing sar with an exit-127 stub: v1.2 printed nothing and exited 0, v1.3 raises the breach. NOT PROPAGATED: the three sysstat scripts are INDEPENDENT implementations, not copies - which is why the same class of mistake had to be made three times instead of fixed once. whey and lezama both read await from $(NF-2), which is aqu-sz under venus's column layout, but sysstat versions differ and I cannot verify theirs from here. Routed to their own hosts' agents. COVERAGE: venus only. The swap-io branch and the normal notify path were not exercised beyond the forced UNKNOWN case.