lezama-down email alert via Hostinger php-api heartbeat
GCABA VPN confirmed up (pod health endpoint reachable via ppp0) while lezama (10.78.42.168) unreachable, 2026-10-05 ~23:xx ART - whey-side check (proxy-mba-w-cc). Need a way to alert a few people by email that host lezama is down, without using whey as the monitoring host (whey itself can go down/lose VPN and would then falsely report lezama down, or simply be an unreliable single point of detection). Proposed design per Elazar: a systemd timer on lezama itself that curls the php-api on Hostinger (api.movilba.com, api-external/php-api) every ~20min as a heartbeat; the Hostinger side tracks last-seen and after missing 3 consecutive intervals (~60min) sends alert emails to a short list of people, then sends exactly one recovery email when lezama's heartbeat resumes (no repeat alerts while down, no repeat recovery emails). Needs: (1) lezama systemd service+timer unit for the heartbeat curl, (2) a Hostinger-side endpoint/cron in php-api that tracks last-heartbeat timestamp and alert/recovered state (likely a small table or flat state file), (3) recipient list and email-sending mechanism on the Hostinger side (send_email.php already exists per php-api agent's memory), (4) decide 3x20min=60min threshold is final or tunable. Out-of-scope note: api-external/php-api is out-of-repo per movilba CLAUDE.md - implementation belongs to coder-mbaexternalapi-cc-w (php-api maintainer), with lezama-side unit owned by whoever maintains lezama systemd units (nw-lezama-ca per agent-conventions).
Activity
-
researcher-mba-l-cc / solo
-
Researcher started at 2026-10-08T12:15:08-03:00.
-
researcher-mba-l-cc php-api is out-of-repo per CLAUDE.md (call/depend on, never edit) - this spec treats it as a black box with a contract, implementation belongs to coder-mbaexternalapi-cc-w. Lezama-side unit belongs to nw-lezama-ca. (1) LEZAMA-SIDE: systemd service+timer, naming convention matching the existing backup-movilba-db.timer/backup-movilba-dev-db.timer pattern: - `movilba-heartbeat.service` (Type=oneshot): `curl -fsS -m 10 --retry 2 -X POST https://api.movilba.com/heartbeat -d "host=lezama&token=<shared-secret>"` - use `-fsS` per the standing convention (no unbounded body capture on a non-2xx), short connect/total timeout since this must never itself hang the box it's monitoring, and 2 retries inside the same interval window rather than relying on the next tick alone (one dropped packet shouldn't look like a real outage). - `movilba-heartbeat.timer` (OnUnitActiveSec=20min, Persistent=false - a missed catch-up on boot would send a stale timestamp). - Secret: a shared token in the heartbeat payload, read from an env file, never a literal in the unit (matches "No inlined secrets in tracked files" - this unit isn't tracked in movilba-web at all since it's a lezama host unit, but the convention still applies). (2) HOSTINGER-SIDE (php-api, spec only, not implemented here): - One endpoint, e.g. `POST /heartbeat`, validates the shared token, upserts a single row/file: `last_seen_at`, `alert_sent` (bool), `recovered_pending` (bool, unused - see below). - One cron (existing php-api cron infra, if any - otherwise a new one) running every ~5min, checking `now() - last_seen_at > 60min`: - If over threshold AND `alert_sent` is false: send alert emails to the recipient list, set `alert_sent = true`. Never resend while still down (ticket's own "no repeat alerts" requirement) - the boolean gate is the whole mechanism, no need for a separate dedup table. - If under threshold (fresh heartbeat arrived) AND `alert_sent` is true: send exactly one recovery email, set `alert_sent = false`. This is naturally exactly-once because the flag flips on the same check that fires the email - no race if the cron is not run concurrently with itself (php-api's own cron runner needs to guarantee that, not this spec). - State storage: a one-row table is simpler and more robust than a flat file for concurrent cron+heartbeat-endpoint writes (file-based read-modify-write under cron has a classic lost-update risk if the heartbeat POST and the alert cron ever overlap); defer to php-api's own existing storage convention if it already has one for similar state. (3) RECIPIENTS + SEND MECHANISM: reuse the existing `send_email.php` (per php-api maintainer's own memory, already present) rather than building a second mail path. Recipient list: short, named by Elazar - not inferred here; store it as a config value on the Hostinger side, not hardcoded in the endpoint logic, so changing the list doesn't need a code deploy. (4) THRESHOLD: 3 missed 20-min intervals (~60min) per Elazar's own proposed design - make it a single named constant (`$HEARTBEAT_INTERVAL_MIN = 20`, `$MISSED_THRESHOLD = 3`) on the Hostinger side so it's tunable without touching the detection logic, but ship with these exact values - no new decision needed unless Elazar asks to retune. NOT MINE TO BUILD: steps (2) and (3)'s actual code land in php-api (coder-mbaexternalapi-cc-w), step (1)'s unit lands on lezama's systemd (nw-lezama-ca). This bwi note is the full contract both sides implement against.
-
Spec complete in bwi. Implementation split: lezama unit to nw-lezama-ca, php-api endpoint/cron/email to coder-mbaexternalapi-cc-w. Not mine to build.