carlok — zsh — 88×30

cat _posts/2026-11-02-alien-hunt-a-way-to-test-whether-the-unavailability-term-chooses-anything.md

alien-hunt: a way to test whether the unavailability term chooses anything

In the five runs of alien-hunt whose records show what was actually hunted — the fourth on PyJWT and four on joserfc — the hunted set is exactly the candidates at or above the coherence floor, and the cap of 20 never cut one. So the floor chose and the unavailability term only set the order, which is the question the paper’s author put on the table: compare the combined score against coherence alone and against a random draw, under the same investigation budget (a802c44).

--arms N is that test. The floor goes off and each rule picks N candidates from the same pool — combined by the score at β = 0.7, coherence by the rating with ties broken at random, so it cannot quietly become the combined rule, and random a plain draw. The reviews and the pool are paid once, the rules share a number of hunts rather than a number of dollars, a pick two rules share is hunted once and counts for both, and the rules take turns so a run stopped by the budget cuts them about evenly. Since “confirmed” is a weak outcome, every hunt’s claim lands in local/arms-to-label.jsonl, shuffled and with no rule on it, to be labelled real or not before anyone opens the report or the rest of the run folder; alien-hunt arms then adds the runs up, counting hunts and cost, claims, confirmed issues and those whose functions no plain review named. On the last joserfc run the reviews and the pool cost 1.50 USD and a hunt about 0.05 USD, so --arms 10 should run to roughly 3 USD. It has not been run yet.

The same morning a hardening pass came across by hand — only what master lacked, since it had been built on the first commit and never saw the earlier pull requests (8c7e478). Confirmation got stricter: a reproducer must print DEFECT-OBSERVED from an if or except branch (or a try’s else), enter a function the claim names and exit 0, because “it called something and exited 0” had confirmed 7 claims on one run that were 3 trivial behaviours. The search now skips symlinks and anything resolving outside the checkout, refuses absolute and .. globs, bounds lines at 500 characters and resolves the paths a model names in a rating inside the checkout; the loop retries an unreadable submit_claims call once rather than reading it as an empty answer, skips and names a failed pass, ends after three in a row, and whatever way it ends — budget, model errors, Ctrl-C — confirms and reports the claims collected, with each claim written to local/claims.jsonl as it arrives. Containers gained a CPU limit and a kill on timeout, wheels are checked by sha256, and a failed install no longer voids every claim.

A second pass took a fresh review’s findings (204e60a): claims now merge within a rule rather than across rules, so one rule’s claim can no longer join two of another’s issues, and among confirmed issues only; a hunt the budget cut in the middle no longer counts as a hunt, though what it spent stays; reach counts hunted picks rather than picks; the rule that always went first now rotates and a run cut short is recorded; and “beyond the reviews” no longer compares function names by exact string, nor holds for a claim with no function. alien-hunt arms stops crashing on a file of the wrong shape and stops counting a run listed twice. That day’s joserfc run, at 1.98 USD and 9 candidates hunted against 1 in the run before, found the validate_exp boundary bug again — the first version of the stricter check refused it because its marker sat in a try body, which is why that form now counts (5f35be3) — and replaying the run in Podman with no model calls confirmed all six of its claims, though the two that came from hunts turned out to be markers in the right place rather than defects.