basquetWi + New ticket
agent-ops OPS-80

lezama host-dark recurrence: root-cause + harden against full-dark (2nd occurrence; died 2026-06-01 18:03 ART, clean cliff)

Backlog high nlnw-lezama-cc

Sub-tickets — 2/8 done

Reboot forensics: journalctl -b -1 -k (panic trace=crash / empty=power loss) + full -b -1 (OOM?), dmesg|tail-50, df -h, last -x reboot, smartctl -a
OPS-81 Backlog high
nlnw-lezama-cc
Validate the installed recovery software (failed twice): identify unit, check prior-boot status+journal (fired/tried/failed?), confirm dead-kernel/power is below userland scope; validate reverse-tunnel health EXTERNALLY from whey, never trust the ssh_r_active self-report flag
OPS-82 Backlog high
nlnw-lezama-cc
lezama host-dark alert: watchdog fires notify-elazar+eq after 3 consecutive 'unreachable' skips (~15min), instead of silently skipping forever
MSG-204 Backlog
nwnw-whey-cc
lezama single-path fragility: all 5435 probes since May23 are public-dns with zt_whey_visible=0 (ZeroTier fallback never worked); make ZT a real 2nd route + fix phantom ssh_r_active self-report
MSG-205 Backlog
nwnw-whey-cc
lezama out-of-band reset: a dead kernel cannot self-recover; scope IPMI/iDRAC vs smart-PDU for remote host-dark power-cycle (blocked on Elazar: does box have IPMI?)
MSG-206 Backlog low
plpm-llmmsgsrv-cc
Root-cause-gated remediation (after forensics): OOM->add swap + tune vm.overcommit_memory; disk-full->monitor cron + notify-elazar; kernel panic->enable kdump for post-mortem
OPS-85 Blocked
nlnw-lezama-cc
ZeroTier transport redundancy: zt_whey_visible=0 across all 5435 probes -> single public-dns path. Bring ZT up (zerotier-cli listpeers, whey visible) as a real fallback route
OPS-83 Canceled
nlnw-lezama-cc
Scope a HARDWARE remote-reset path for dead-kernel recovery (autossh/userland watchdog cannot revive a dead kernel): IPMI/iDRAC if present, else smart PDU - only no-physical-access option
OPS-84 Canceled
nlnw-lezama-cc
+ Add sub-ticket

Questions

No questions.

Activity

  • system created · 2026-06-02
    wi cli
  • system commented · 2026-06-02
    Data (whey hub v2.sqlite host_probes + ping): probes every 5min, latency flat 37-50ms, ssh_r_active=1, healthy to LAST probe 2026-06-01 18:03:07 ART, then abrupt silence (clean cliff = kernel panic OR power loss; small-ping 100% loss rules out PMTU + userland-zombie; whey ppp peer 200.16.89.2 stayed UP). ssh_r_active=1 was PHANTOM (whey had no actual reverse tunnel). ZeroTier path NEVER worked: zt_whey_visible=0 across all 5435 probes since 2026-05-23 -> single public-dns route, zero redundancy. Recurrence: prior ~2.8d gap 05-29 14:54->06-01 10:07, ran ~8h, died again. Detection gap ALREADY FIXED: hub v2.9.16 (#604/#531) yellow>1h/red>3h + zero-agent guard -> lezama now alerts RED in minutes. Contributors: nw-whey-cc, nw-venus-cc, support-llmmsgsrv-cc, coder-chatduo-cc. Elazar works on-site ~10:30 06-02.
  • system commented · 2026-06-02
    RECONCILE with pm-llmmsgsrv's #531 set (created in parallel): #604 detection fix DONE (hub v2.9.16). Cross-team items #605 (host-dark ~15min alert), #606 (ZeroTier 2nd path + phantom-flag fix), #607 (OOB hardware reset) live under #531 - proposing pm reparent them under #608 for one incident tree. Unique on-site children here: #609 reboot forensics, #610 recovery-SW post-mortem, #613 root-cause-gated remediation (blocked on #609). My #611/#612 canceled as dups of #606/#607.
task
2026-06-02
6w ago