carlok — zsh — 88×30

cat _posts/2026-10-30-alien-hunt-is-public-sampling-the-defects-a-plain-review-of-a-repository-does-not-report.md

alien-hunt is public: sampling the defects a plain review of a repository does not report

alien-hunt went public on 30 October: a Python research prototype, MIT, that looks for defects a plain review of a repository does not already produce and then checks whether a reproducer can actually observe them. It is the cheap version of the sampler in The Alien Space of Science (code, MIT), with a model’s 1–5 coherence rating standing in for the paper’s trained models: score = (1 − β)·z(coherence) + β·z(unavailability), where unavailability is one minus the maximum cosine similarity to the claims three plain reviews and any local scanners had already made.

Candidates are built from atoms of the library itself rather than its tests, docs or examples — a source and a sink or invariant in the same file, else the same top-level folder — joined by up to three of 62 mechanisms, plus one drawn at random so the pool keeps unlikely pairs; only candidates rated at least min_coherence (3 by default) are hunted, the top 20 are kept at β = 0.7, and the ranking at β = 0.4 is stored beside it. A claim counts only when its reproducer imports the repository, calls the named function and exits 0 — a script that reads source text does not — and confirmation runs in Podman with --network none, the checkout mounted read-only, capabilities dropped and a pids and a memory limit, installing only wheels uploaded on or before the pinned commit and building nothing from source.

Nine runs on 30 October over three libraries logged about 10.7 USD at list prices (9e693c6, #8). On PyJWT 2.3.0 the plain reviews never reported GHSA-ffqj-6fqr-9h24 — an ssh-ed25519 public key accepted as an HMAC secret — and a directed hunt confirmed it in run 4, with the honest caveat in the README that the CVE is public and the model may have recognised it; asciitree drew no candidate above the coherence floor, which is what added the fallback to the best candidates rated 2; joserfc yielded a boundary bug in validate_exp that went upstream. Two of the day’s faults became features rather than results: sinks where a function’s own input sets the work or the memory arrived as two denial-of-service mechanisms (06cfa2b), and the container’s own CPython now decides which wheels to fetch (63634fd).