evolutiva
EVO-119
Daily trash purge runs INLINE under LiteSpeed: one real customer request per day pays the full trash-walk
In progress low
cacoder-apiimages-cc
MEASURED on the box 2026-10-06 via authenticated /stats: runtime{sapi: litespeed, canFlushEarly: false}. fastcgi_finish_request() does not exist under LiteSpeed and the call site in index.php is guarded by function_exists(), so the flushed-response branch has NEVER been taken on this host - the daily trash sweep registered by register_shutdown_function runs INLINE, before the response reaches the client. Consequence: exactly one real customer request per calendar day pays the full trash-walk latency, and which tenant pays is whoever happens to be first that day. Cost scales with trash size, so it is worst exactly when the purge is behind. The purge is per ACCOUNT, not per tenant: one watermark at TRASH_DIR root (inc/purge.php:71, claimed BEFORE the sweep so a concurrent burst cannot each fire one) and one RecursiveIteratorIterator from $trashDir down (:29-32), so one request absorbs every tenant's trash in a single pass. DO NOT BUILD A REMEDY AGAINST AN ASSUMED NUMBER (pmmaster, explicit): measure what that request actually pays first. Trash today is 179 jpgs total (mars 150, terra 19, venus 10, pluto 0), so today's number is probably small and that is exactly why it must be measured rather than reasoned about - the figure that matters is the one at a 90-day backlog, not at 179 files. Measurement problem to solve first, and it is the real work here: there is no access log and no per-request duration anywhere in the account, so the latency of that one request is not observable from any existing instrument. Candidate remedy once a number exists: bin-venus-cc's weekly EVO-116 call is scheduled early and ahead of traffic, so it can BE the request that pays the walk by design - but weekly cadence cannot absorb a DAILY sweep, so that covers one day in seven unless the cadence changes, which is Elazar's call. Found while reading /stats runtime{} for something else; it had no row and would not have been found again by anyone looking for it.
Activity
-
Instrument BUILT, 9faa047, branch-only - deliverable changed from 'measure then fix' to 'add the instrument, remedy after' per pmmaster 2026-10-06, because the measurement is not available from any existing observer: no access log and no per-request duration exists anywhere in this account, so the latency cannot be recovered after the fact and only a self-instrumenting path can see it. PURGE_COMPLETED now carries scanned, trashMs, idemMs, inline and sapi beside files/bytes/retentionDays. scanned is counted BEFORE the retention test, which is the contract rather than an implementation detail: it is the denominator trashMs is attributed to, so it must include the files the sweep KEPT - beside the unlink it would report 0 on exactly the run worth explaining, a long walk over a backlog where nothing is old enough to delete yet, which is the shape trash takes at 90-day retention. inline says who paid: with an early flush trashMs is maintenance nobody waited on, without one it is latency charged to a person. Fields appended, never reordered - the log reader matches this event by prefix. tools/purge-trash.php also prints scanned + elapsedMs in dry-run, which is the only way to measure the walk without charging a customer, and it is runnable since SSH and PHP 8.4 CLI exist on the box. tools/test-purge-instrument.php: 21 assertions, in the push gate, mutation-checked - moving the counter turns exactly 4 red. NO NUMBER EXISTS YET and none can be quoted: the live box emits three fields, so forward coverage only, and nothing can be said about any day before the bundle ships. Remedy stays unproposed until a real figure lands. Keep on the row: bin-venus-cc's weekly call absorbing the sweep covers ONE DAY IN SEVEN, so it is the right side effect and never a mitigation - the other six days still land on a customer, and this row must not read as handled. A daily cadence is Elazar's call, not mine to invent.
-
nw-whey-cc's deploytohostinger.sh run of the waiting bundle (now 14 commits through 9faa047), gated on a frozen uploads/ snapshot + Elazar's go. The instrument is built and contract-pinned but emits nothing until it is on the box; I hold no FTP deploy creds. Same seat and same gate as EVO-103 and EVO-118.
-
Attribution, per pmmaster 2026-10-06: the inline-purge finding and the one-in-seven arithmetic are coder-apiimages-cc's, raised in DM muwec9dhyv8u. bin-venus-cc asked only whether the purge was keyed per tenant or per account - a good question, and the answer (per ACCOUNT: one watermark at TRASH_DIR root, one recursive walk) is what makes the weekly-vs-daily arithmetic obvious, but the flag was not theirs and they disclaimed it rather than accept the credit. Row created with --for pmmaster-evolutiva-cc because he ordered it filed; the finding is mine. Cause named by pmmaster and worth keeping, since it will recur: he batched one DM to two seats without naming which paragraph addressed which, so credit landed on whoever read it - and misassigned credit misroutes the follow-up. Also mine, same cause: 'registered for the gate DM instead of polling' was my .env.example push behind the venus push gate, not anything bin-venus-cc did.
-
Blocker VOID: the instrument is deployed. 79f6622 shipped inc/purge.php and sha256 matches local HEAD on the box. The snapshot gate that blocked it was removed (blackboard bb-coder-apiimages-cc-muwp3m2j-1ue5yk, 4/4) as a control that cannot fail against an operation with no --delete and uploads/ hard-excluded. NOT done, and the gap is exactly one day, which is the point worth stating rather than rounding away: the deliverable is a MEASUREMENT and no measurement exists yet. Today's sweep already fired at 2026-10-06T00:06:06Z - HOURS BEFORE the deploy - and the .last_purge_date watermark permits at most one per calendar day, so today's line carries the old three fields (files, bytes, retentionDays) and none of scanned/trashMs/idemMs/inline/sapi. The first instrumented line lands on the first request after midnight UTC, i.e. 2026-10-07, and it will be read from the box log, not inferred from the code being present. Deployed and measuring are a day apart here. Read trashMs against scanned when it arrives: a long walk over a zero-deletion day is a consistent pair, not a puzzle, which is why scanned is counted BEFORE the retention test. Read inline as who paid - true means a real customer request waited, and that is the only reading that makes the duration user-facing rather than housekeeping. Expect inline=true: sapi is litespeed and canFlushEarly false, so the fastcgi_finish_request branch has never been taken on this box.
-
DELIVERY PATH FIXED, a626318 - found one day before the first reading, by checking who reads the line rather than that it is written. agents/api-images-log-reader.sh (the only daily reader this service has, EVO-92 venus timer 12:00 UTC) reduced PURGE_COMPLETED to a BOOLEAN. Correct while the line carried files/bytes/retentionDays, since presence was the whole signal; a defect the instant 9faa047 appended scanned/trashMs/idemMs/inline/sapi. From 2026-10-07 the sweep would have written the only per-request duration that exists anywhere in this account, daily, and the reader would have scored it 1 and discarded the body. Nothing would have gone red. This is the LOG_FILE lesson one layer up, and it is already in CLAUDE.md: an observable can be correct at the call site and still never reach the reader. There the line never reached the file; here it reaches the file and dies at the reader. I built the emitter yesterday without checking who was listening - a self-reporting instrument still needs someone to read the report. Cost now goes to stdout unconditionally (journald retains it, so every day's figure is recoverable without spending a DM). trashMs=- means NO cost fields on the line, which is an uninstrumented sweep, not a zero-cost one. NO MS THRESHOLD, as a refusal rather than an omission: no baseline exists until tomorrow, and a threshold invented before the first reading is indefensible - this repo has twice shipped a control it never executed. The alert fires ONCE on the first inline sweep observed, to establish the baseline, then never again on its own; afterwards the cost rides along on reports triggered by something else. One-shot, not a cadence, and the script stays exception-only - a daily numbers message is how a channel gets muted before the line that matters. One-shot guarded on trashMs NON-EMPTY, not on the line existing, so a three-field line cannot burn it and leave the real first reading arriving with the marker already set. Armed against four real shapes, not reasoned about: the box's actual pre-instrument line (empty, safe), instrumented inline (fires), instrumented inline=false (does not fire - that duration is housekeeping nobody waited on), and trashMs=0 (present, not absent). Green arm end-to-end against the live log: purge_for_2026-10-05=1 trashMs=- inline=- new_warn_lines=0 days_observed=26, no DM sent. Neither the one-shot nor the ride-along is exercised yet - no instrumented line exists on the box - so both are armed on synthetic input only. Still inProgress. The first real reading is 2026-10-07 and arrives via the existing timer; nothing new was armed for it.
-
MEASURED, and charged to nobody - advanced today rather than waiting for tomorrow's inline line. Ran tools/purge-trash.php on the box over SSH with PHP 8.4 CLI, no --apply: would purge 0 file(s), 0 bytes (trash older than 90 days) [dry-run] walk: scanned 180 trash jpg(s) in 40.7 ms (real 0m0.145s wall, incl. PHP startup) Both files sha256-verified identical to local HEAD first (tools/purge-trash.php f6c62f5e..., inc/purge.php a07b474...), so this measured the DEPLOYED code and not a local copy. Pre-checked two things rather than assuming them, because either would have been destructive to the measurement I am waiting for: (1) unlink is reachable ONLY under $apply (inc/purge.php:57); the dry branch increments counters and touches nothing. (2) .last_purge_date is written only by maybe_run_daily_purge() (inc/purge.php:93), which this tool never calls - it calls purge_trash_older_than() directly. Confirmed by readback: watermark read 2026-10-06 BEFORE and 2026-10-06 AFTER. So this run did NOT claim the day and tomorrow's first instrumented inline line is NOT suppressed. Had it claimed the watermark, this measurement would have cost the one it was taken to anticipate. THE 0-AND-180 PAIR IS THE CONTRACT WORKING, not a puzzle: scanned counts every trash jpg stat()ed, before the retention test, so a long walk with files=0 is consistent. Retention is 90 days and nothing in trash is 90 days old yet, so NOTHING has ever expired under observation - trash grows monotonically until the first expiry and this 180 is the current floor, not a steady state. WHAT THIS DOES AND DOES NOT SETTLE. It is a defensible ORDER OF MAGNITUDE for the walk: tens of milliseconds at current trash size, ~0.23 ms/file, so ~4400 files would be needed to reach 1s. It is NOT the inline figure. Three reasons it cannot be substituted for one: PHP CLI is a different SAPI from litespeed, filesystem cache state differs, and the inline sweep also walks _idem (idemMs) which this tool does not report. The inline number arrives 2026-10-07 and is the authoritative one. CONSEQUENCE FOR THE REMEDY QUESTION, which was the whole reason pmmaster barred proposing one against an assumed number: the assumed number was wrong in the harmless direction. ~40ms paid by one request per calendar day does not justify a cron, a lock, or any added machinery, and the instrument plus this dry-run are the right deliverable. Revisit only if trashMs grows an order of magnitude, which the first-expiry transition is the moment to watch. SIDE EFFECT WORTH RECORDING: tools/purge-trash.php has now RUN AGAINST LIVE DATA for the first time in this service's life. CLAUDE.md filed it built-and-untested (and before 2026-08-24 'cannot run, BY CONSTRUCTION' on a false no-SSH premise). The DRY path is now exercised on the box; --apply is still never-run.
-
DATE CORRECTION, measured not predicted: the first instrumented PURGE_COMPLETED is READ on 2026-10-08, not 2026-10-07. Two separate dates and only one of them is the line's. Verified on the box just now: the 2026-10-06 line is 2026-10-06T00:06:06Z ... files=0 bytes=0 retentionDays=90 - the OLD 3-field shape, because the sweep fired hours before the 13:19:20Z deploy and the .last_purge_date watermark (reads 2026-10-06) allows one per calendar day. So the first line carrying scanned/trashMs/idemMs/inline/sapi will be DATED 2026-10-07, on the first request after midnight UTC. The reader asserts YESTERDAY, never today (deliberately - the sweep fires on the day's first request and today's line may legitimately not exist yet). Delivery path verified rather than assumed: the venus user timer api-images-log-reader.timer next fires 2026-10-07 12:02:05 UTC and its ExecStart is the repo checkout path itself, so it runs the fixed reader (a626318) and not a stale copy. But that run reads yesterday = 2026-10-06, i.e. the uninstrumented line. The 2026-10-07 line is read on the 2026-10-08 12:02 UTC run. That is also the one-shot baseline alert working as built: its guard is trashMs NON-EMPTY, not line-present, so the 10-06 line cannot burn the marker and leave the first real reading arriving with it already set. Confirmed against the real line rather than against the intent. Nothing to do until then and nothing to arm - no cron, no poll, no self-armed wakeup; the existing EVO-92 timer is the reader.
perf
1w ago by coder-apiimages-cc
1w ago