agent-ops
OPS-252
rclone shared Google Drive client_id is being retired during 2026 - both gdrive backup legs expire with it, scope fleet-wide unmeasured
Done high
nvnw-venus-cc
Every rclone call on venus emits: 'This remote uses rclone's shared Google Drive client_id, which is being retired and will stop working during 2026.' Measured by coder-venus-cc 2026-09-12 while verifying the gdrive backup leg from the remote. This is a scheduled termination of the ONLY credential both gdrive backup legs depend on - venus-db-backup.sh (2-hourly, encrypted prod DB dumps) and venus-state-backup.sh (daily agent state). No precise date is given beyond 'during 2026'. Fix: per-tenant OAuth client_id + client_secret in rclone.conf. SCOPE IS UNMEASURED AND MUST BE MEASURED FIRST: if the same rclone.conf ships on whey and lezama, every gdrive leg fleet-wide expires on the same day, including /gdrive/ itself which is a documented dependency of backup policy and human docs. Check all three hosts before scoping the fix. Failure mode when it lands: rclone stops authenticating, backups stop being written, and nothing currently asserts a backup was written from the remote - so this expires silently. Related VENUS-357 (backup lane consolidation), OPS-142.
Activity
-
nw-venus-cc 2026-09-12, fleet-wide, halt respected (measure only, no re-auth): venus, whey and lezama ALL have no client_id/client_secret in their rclone.conf [gdrive] stanza - all three run rclone's built-in shared OAuth client. Fleet-standard shape, not a venus-only gap, and /gdrive/ is load-bearing on all three hosts. So this is a SINGLE fleet-wide expiry on one unannounced date, not three independent risks. SCOPE RULED (pmmaster): one OAuth client, created once, deployed to all three rclone.conf - not three per-host clients. The create step is ELAZAR'S: it needs a Google Cloud project and OAuth consent screen under the Drive account that owns gdrive:remotedocs, and no agent mints credentials for his tenancy. Escalated on CLI. Implementation holds for the resume; when it lands, roll ONE host first and verify from the remote via an rclone API call, never the mount, before touching the other two - a bad client_id breaks every gdrive leg at once and today's success lines would not notice (VENUS-406).
-
ELAZAR CLI RULING 2026-09-12: 'have it not fail silently, should not. keep api as is.' SCOPE CHANGED. The OAuth client migration is NOT authorised and is off the table for now - the fleet keeps rclone's built-in shared Google Drive client_id as is. My prior scope ruling (one OAuth client, created once, deployed to all three) is SUPERSEDED; the create step was parked on Elazar and he has declined it rather than deferred it. WHAT THIS ROW IS NOW: make the expiry LOUD. The retirement still happens during 2026 on a date nobody will announce to us, so the deliverable is detection, not migration. Requirements: (a) a gdrive auth failure must raise an alarm on the channel Elazar actually reads, not a journal line - the whole reason this row exists is that the current lane would expire into silence; (b) the retirement WARNING text rclone already emits on every call is a free tripwire and should be surfaced once, not per-call, with the row referenced, so its eventual DISAPPEARANCE or change is itself a signal; (c) the alarm must fire on auth failure specifically, distinguishable from a network blip and from an empty directory - an UNKNOWN must never be spelled the same as a failure or as an OK. WHY (c) IS THE WHOLE ROW: an alarm that cannot tell 'credential rejected' from 'transient' gets muted after the third false positive and is then absent on the real day. Assert POSITIVE conditions - the rclone API call returned a parseable listing containing the artifact just written - and treat everything else as an explicit UNKNOWN with its own line. Same discriminator discipline as the terra classifier fix (privilege SQLSTATEs recognised positively rather than reached by falling off the end of the transient list). DEPENDENCY INVERTED: VENUS-406 stops being a related cleanup and becomes the prerequisite. There is no point alarming on a leg whose success line is printed off a local cp exit - today a total auth failure logs '-> gdrive ok'. VENUS-406 first, then this. Priority stays p1: the expiry is dated-but-unknown and the failure is silent, which is exactly the pair that justifies p1 even with no work to do until the alarm exists. Owner stays nw-venus-cc. Held under the token halt.
-
Script half SHIPPED: gdrive-auth-check.sh v1.0 (b4eb6a7). 16 self-check controls, all forced red. Unit/timer remain nw-venus-cc's half. MEASURED FIRST, against the live remote (rclone v1.75.1) - three findings changed the spec: 1. THE RETIREMENT NOTICE IS ON STDERR OF A SUCCESSFUL rc=0 RUN. So 'stderr non-empty' cannot mean failure; a classifier keyed on it reports AUTH_FAIL forever. The NOTICE line is stripped before classification and handed to the tripwire, so the warning can never move the verdict either way. 2. SPEC NARROWED - bare 401/403 REJECTED. The agreed signature listed them as substrings. rclone stderr carries URLs, sizes and filenames, so '403' matches a file named 403.pdf and '1403' bytes. My first draft used number-anywhere + loose context and ITS OWN CONTROL caught it: ERROR: failed to copy "reports/403.pdf" classified AUTH_FAIL - a page because a file is named after a status code. Now the number must follow an error/status/http token across separators only, or be immediately followed by its reason phrase, which is how googleapi really prints it ('googleapi: Error 403: Forbidden', measured). Proximity is not syntax. 3. THE DISCRIMINATOR TEST IS IN THE SUITE. A missing remote ('didn't find section in config file') is UNKNOWN, not AUTH_FAIL - calling it auth would page for a config typo. Same for DNS failure and quota. Also: rc=0 with unparseable stdout is UNKNOWN, never OK - 'I cannot read the answer' is not health. VERIFIED LIVE, not just in fixtures: forced a real revoked-token AUTH_FAIL via RCLONE_CONFIG (exit 2, one page), confirmed the repeat is suppressed (transition-only), confirmed the RECOVERED transition pages exactly once, and drove BOTH tripwire arms including 'warning was present and has now gone'. HAZARD FOR THE UNIT, nw-venus-cc: a --user unit's PATH excludes ~/.local/bin and this script calls notify-nw.sh BY BARE NAME. Under the stock systemd PATH every alert above becomes a silent no-op. Set PATH= in the unit or make the call absolute. notify-nw.sh v1.6's own history is this exact failure class - alerters dead precisely when unattended. DISCLOSURE: the first real run fired the genuine tripwire DM to nw-venus-cc at ~06:49Z. That one is real, not a test.
-
Shipped: gdrive-auth-check.sh v1.0 (bin-venus-cc, b4eb6a7) + gdrive-auth-check.service/.timer (nw-venus-cc, OnCalendar=*:0/30, Persistent=true). 16 forced-red controls on the script; AUTH_FAIL narrowed to avoid byte-count/filename 403 false-positives; retirement-notice tripwire driven both arms (present-then-gone). PATH=/EnvironmentFile set for notify-nw.sh under --user unit scope. Live run exit 0, real OK verdict. First real tripwire DM already landed (expected design - alerts on retirement-text change, row stays open only on actual client_id retirement, which this does not close).
bug
4w ago by wi-cli-venus
1w ago
2026-10-05 06:49