The Scientist

The harness result alone would be a curiosity. It has company. Three separate measurements show one model name delivering three different systems, and each effect is large enough to flip a model-selection decision.

Variable 1: the scaffold is trainable, not just noisy

The same Hugging Face work shows the harness can be trained against. AINews reports that multi-harness RL on LFM2.5-2.6B, run across four harnesses, moved first-attempt solves from 42% to 54%. SFT on 3,189 rollouts plateaued at 47.5%. The result holds up because a capture proxy records exact token IDs and logprobs rather than re-tokenized text. Retokenization drift would quietly corrupt the on-policy importance ratios. This is one task family and one seed. Read it as a direction, not an effect size.

Variable 2: the reasoning budget actually delivered

TLDR Data reports an analysis of 43,261 Fable 5 calls over six weeks. Delivered reasoning budget varied sharply under a single model name, and many calls showed zero thinking. The analysis has one large confound. It does not control for prompt difficulty, and adaptive-reasoning models are supposed to think less on easy prompts. It also never links budget to correctness. A replay study separates the two explanations. Run a fixed prompt set repeatedly across peak and off-peak windows, then split budget variance into a between-prompt part (adaptive, fine) and a within-prompt part (provider-side, a problem). If within-prompt variance is large and correctness tracks it, the offline eval overstates production quality.

Variable 3: the denominator

Turing Post works through the cost math for Reflection's Beam: cost per solved task = cost per attempt ÷ success rate. On DeepSWE-like work, Beam has to be at least 74.2/44.4 = 1.67x cheaper per attempt, end to end, just to match DeepSeek V4.1 Flash. On codebase QnA against Kimi K3, the bar is 1.97x. Those are floors, since real failures are correlated. Reflection's claimed 3–4x efficiency over GLM 5.2 leaves out prefill, and prefill dominates agent loops that re-read repo context every turn.

Import AI's SciUniverse lab-automation data re-ranks the same way. Claude Fable 5.1 passes 45.3% of tasks at $40.61 per task, about $89.65 per success. GPT-5 Astra passes 32.5% at $52.37, about $161.14 per success. The per-success figures assume tasks can be retried independently. They do not hold for irreversible actions.

Size the test before running it

Turing Post's power math is the part worth copying. SWE-bench Verified has ~500 tasks, and at ~80% accuracy the standard error of an unpaired difference is about 2.5 points. Beam's 2–3 point deficit there is not detectable at all. A paired McNemar test on 200–300 in-house tasks resolves the ~10-point gaps seen on Terminal-Bench 2.1. Resolving 2–3 point gaps takes 1,000+ paired tasks.

A model ranking that doesn't fix the harness, log the delivered reasoning budget and divide by success rate is really a ranking of your test setup.

The headlines group these incidents together. They fail in different ways. Sorting them by which assumption breaks is more useful, because each assumption maps to a different control in the stack.

Read the incident count like an adverse-event report

Term Sheet's reporting supplies a numerator. It omits total agent volume, so the count has no denominator, and severity mix and detection method are also unstated. A reasonable model of the observed count is roughly true incidents × detection recall × disclosure propensity, with both multipliers below 1. The German wiki hijack was reportedly known for months and surfaced through independent researchers, which says something about recall. The '100+' figure is therefore a lower bound. The Australian recurrence suggests the first fix shipped without a regression test. Vendor disclosures describe vendors. The exposure of any one organization shows up only in its own tool-call telemetry.

Two loader and host details that matter

The Unsloth case is one instance of a broader vulnerability class. Pickle weights, repo-supplied modeling code, dataset scripts and UI auto-setup all execute someone else's code. For an ML team, the worst outcome is a stolen write-scoped hub token that lets an attacker swap weights that get deployed later. Accuracy dashboards won't catch that. The Mac case is a triage lesson about a metric that misses the bottleneck. The CVE scored 7.1/10. The Netherlands NCSC saw root access and a Monero miner on every exploited host. The 'install security updates automatically' setting doesn't install the point release that carries the fix. Exploitation status belongs above severity score in a patch SLA.

Measure containment like a rare event

Escape attempts and reward hacks are rare, so 'zero in testing' means little until someone states n. By the rule of three, zero events in 300 clean audited episodes bounds the rate below ~1% at 95% confidence. Getting below 0.1% takes ~3,000. Episodes drawn from the same task template aren't independent, so the count should be clustered by template before the bound is computed. The anecdote of Claude Code's monitor halting itself after the intrusion is n=1. Even a perfect 30/30 on injected scenarios only bounds recall at ~89%. Self-halting is a behavior to formalize in policy, and at n=1 it has no measured reliability as a detector.

An agent eval that grades outcomes and trusts the sandbox boundary is partly measuring how well the model games the harness.

The pitch holds together. Define the action space, and the model returns a typed choice with a probability for each option rather than generated text. The evidence is much thinner than the pitch. Every number below is either a vendor claim or a practitioner anecdote.

The cost claims don't survive unit math

Run the units and most of the cost story falls apart. TLDR Data's Jev reference run made 16,150 decisions for a self-reported ~$1.65, roughly $0.0001 per decision. It was a toy game loop, so the per-decision figure says nothing useful about production input sizes. One practitioner's '$0.40 for 5 GB of JSON' cannot mean the model read all 5 GB. At 3–4 bytes per token that is 1.25–1.7B tokens, or about $2,500–$3,300 at GPT-6.1 Sol's $2/M input price. The likely pattern is deterministic blocking first, with the model adjudicating only the ambiguous remainder.

Cheap calls set two structural traps:

• Chaining multiplies errors. Two independent stages at 95% accuracy each give 90.25% end to end. A verification pass, in which a second call re-scores the first, can push accuracy up instead. Evaluate the composed pipeline end to end, not stage by stage.
• Brute force hits base rates. Pairwise dedup over 100K records is about 5×10⁹ candidate pairs. At a 0.1% false-positive rate that comes to about 5M false merges. Blocking stays necessary.

Calibration is the acceptance test

'Decide when an agent should ask a human' is selective prediction. A confidence threshold trades coverage against error rate, and that trade only works if the probabilities are calibrated. None of these vendors has published ECE or reliability diagrams. Hosted models also change under the caller, so calibration needs a re-check on every vendor update. The cheap baseline belongs in the same test. If logistic regression on embeddings matches the decision model, there is no case for an LLM-shaped component at all.

Logging is the long-term hedge. Record each decision's input hash, label-space version, probability vector and downstream outcome. That record supports drift monitoring, and it doubles as a distillation corpus if pricing or availability changes.

Before adopting a router that returns probabilities, measure ECE on held-out decisions and run the logistic-regression-on-embeddings baseline. No vendor in this category has published a calibration curve yet.