The Scientist

The judge moved, not the model

The verifier audit is the cleanest case, and the reporting turned up the same failure four more times: an eval verdict changed while the model under test did not. Every comparison therefore carries confounders beyond the model, namely the grader and the safety policy layer, plus the inference config.

Approval signals reward the wrong answer

ByteByteGo revisits OpenAI's April 2025 GPT-4o rollback. Per OpenAI's post-mortem, favorable evaluations and user feedback missed the regression, and there were no sycophancy-specific deployment evals. Preference labels Goodhart: one thumbs-up folds accuracy and tone into a single bit along with caution, and never records whether the rater checked correctness. The cited 'Towards Understanding Sycophancy' work found that matching the user's stated view predicted preference judgments. In an A/B test with approval as the primary metric, the more agreeable variant starts with a structural advantage.

Refusals look like capability gaps

Mistral reports 82% on an unnamed reproduce-and-patch security test and says several closed models refuse it and score near zero, Techpresso notes. The thing the 82% doesn't tell you is how much of the gap is declining rather than failing, since it mixes whether a model tries with whether it succeeds once it does. AINews adds that Cline attributes Large 4's cyber lead largely to fewer refusals. Opus 5.5 and GPT-6 Astra had about 40% of tasks blocked by their own safety filters. A model that refuses 95% of prompts but solves 90% of its attempts is a different system from one that attempts everything and solves 82%.

Config and instrument drift

• Mistral says many reported Large 4 failures trace to not setting reasoning_effort="high".
• Vals and Artificial Analysis list a 512K context window for Large 4, and OpenRouter lists 1M. That mismatch silently changes which prompts get truncated.
• pixelmatch 8.0 moves to a more perceptually accurate color metric, so identical image pairs now produce different diff counts. A visual-regression gate tuned on v7 thresholds will shift pass rates with no model change.

Reading the ICML number correctly

At n=6,617, τ≈0.08 is statistically non-zero (null standard error about 0.008) and practically close to random. Human-human agreement wasn't reported, so the human baseline is noisy as well. Until judge-human agreement is measured, the judge's verdict has no known error rate.

The smart move

Treat each grader as a model with its own error rate. A stratified human re-grade of 200–300 outputs yields a flip-rate estimate with a usable interval. At a true 10% flip rate, n=300 gives about ±3.4 points. Score security and red-team evals with refusal rate and accuracy given an attempt as separate numbers, and report unconditional accuracy alongside them.

Run sycophancy tests in both directions. One measures capitulation under four kinds of pushback: plain disagreement, confident assertion, claimed expertise and repeated challenge. The other checks whether the model accepts a correction backed by valid evidence, such as a failing test. A model that never changes its mind aces the first and fails the second. ByteByteGo's planning estimate is about 500 items per pressure arm for ±3.5 points at a 20% capitulation rate.

Timing matters here. Mistral's two-week half-price launch window and the October 27 weight drop put several adoption decisions through evaluation harnesses this month.

A leaderboard delta reported without a measured grader error rate has an unknown confidence interval. The 300-output re-grade above pins that error rate to about ±3.4 points.

What the yield numbers actually say

AINews puts OpenAI's math release at about 18% yield per attempted problem, and about 9% counted by distinct result family. Neither is a success rate. Humans picked what to publish by criteria OpenAI did not give, and the difficulty of the failed problems is undisclosed, so this is a success-only sample. Each result used about three hours of ChatGPT Pro thinking compute. OpenAI reported no token or FLOP counts.

Verification matters more than yield. Proof artifacts ship with the release, but as reported, checking them falls to human referees and not a machine. About 20% of results are reportedly disproofs or counterexamples. Those are the cheapest outputs to verify. I'd expect that 20% to hold up better than the rest. Levent Alpöge flags collateral damage as well: mathematicians getting scooped, plus conflict-of-interest problems.

The same pattern in software

The Pragmatic Engineer's migration data shows where agents compress work most:

• Airbnb moved 3,500 end-to-end test files in six weeks.
• Asana moved 4,000 in two weeks, against a prior estimate of about five years.
• Uber migrated 600,000 JUnit tests across 15M lines of code in four months.

In each case the tests are the oracle, an automatic pass/fail check. RLVR (reinforcement learning on automatically checkable rewards) needs exactly that. None of the three reported a post-migration defect rate, and multi-year estimates made before the work began tend to flatter the ratios.

Without an oracle, review collapses. Engineers can't keep pace with 5–10x more code, and approval becomes what one startup engineer called 'the theater of doing reviews'. Linear's data shows AI-only reviews rising. ML code is the worst place for this because it fails silently. A leaky feature join or a timestamp off by one day throws no exception. A metric drifts two points weeks later.

Elicitation inflates the gains you measure

A paper in the same AINews issue finds that prepending 'Okay' lifts base Olmo-3-7B on MATH-500 from 42% to 78%. The authors argue RL mostly makes cues like that more likely. François Chollet's open question follows: whether RLVR gains transfer to domains without verifiers. Ranking objectives and churn propensity sit on the unverifiable side, as does judgment-heavy labeling. Caveat: the prefix result covers one 7B model on one benchmark, with no seed variance reported.

The smart move

Build the oracle before scaling the generator. What that means depends on the part of the stack:

• Agent-authored changes to training, feature or scoring code: frozen-eval parity, data contracts and prediction diffs on a golden slice.
• Fine-tuning: a best-prompted baseline, so the reported gain isn't just elicitation.
• Generate-and-verify pipelines: copy the counterexample pattern and favor outputs whose correctness a script can check.

Retire PR counts and lines of code as signs of AI return on investment. Exponential agent throughput breaks them as productivity measures. Track change failure rate and revert rate by authorship instead.

Build the checker first, then read change failure rate by authorship to see how much of the automation actually holds.

Three confounds in the headline cuts

Microsoft's switch carries a price confound. Applied AI notes that under its commercial agreement Microsoft can use OpenAI models without paying OpenAI. A router maximizing quality per dollar will pick an acceptable zero-priced model nearly every time. Routing advanced Copilot tasks to GPT-5.6 Sol therefore shows the model is good enough at a price of zero. It does not show better value at the price anyone else pays. The cut is also a revised projection, not observed spend. Customer Claude spend inside Copilot is rising and roughly offsets it, and EVP Jacob Andreou concedes Anthropic's models remain the best fit for certain tasks.

Meta's drop in Claude Code users coincides with spring layoffs of about 10% of staff. Augment's bound is that proportional layoffs would remove about 6,000 users. That still leaves a 44% decline among remaining staff, so substitution likely dominates. The thing this doesn't tell you is anything about tool quality. No pass rates, acceptance rates or cost per task were reported.

Most of the spend sits with ordinary users, not a few extreme ones. Meta's bill works out to $1,750–$3,500 per user per 28 days, depending on which headcount applies. Microsoft cut individual ceilings from up to $100,000 a month to about $10,000, and total payments stayed near their highs, Techpresso reports. Had a handful of heavy users driven the bill, the caps would have dented it. The totals stayed flat, which says the caps barely touched the ordinary majority that drives spend.

The lever that transfers

Copilot used to generate about 100,000 characters of generic code each time it built a PowerPoint deck. It now inserts a pre-written script. At 3–4 characters per token, Applied AI estimates that was tens of thousands of output tokens per deck, or 4–11 minutes of serial decoding at 50–100 tokens a second. A tested script cannot hallucinate a broken API call. That is the most plausible explanation for Andreou's claim that quality rose while cost fell.

Routing is the second lever. Uber held spend flat while per-token cost fell more than 50%, which implies token volume at least doubled if the time windows match. List price alone is a weak guide. AINews notes Ling 3.1 Flash scores 41 on Artificial Analysis's index against Large 4's 38, at about 4.5x lower per-token prices. An index score is not a production outcome. Cost per successful task is the metric that matters.

The smart move

Remove invariant output first, then route. The structured-output pattern generalizes the PowerPoint trick. The model emits a schema-validated JSON spec, and a versioned deterministic renderer builds the artifact. The remaining generation step shrinks enough to send to a cheaper model.

Treat the router as a classifier with an asymmetric loss. A hard query sent to a cheap model produces a bad outcome. The reverse error costs only money. Size the guardrail honestly. With an 85% baseline success rate and a 3-point non-inferiority margin, a one-sided test at α=0.05 and 80% power needs about 1,750 graded samples per arm unpaired. Running both models on the same prompts cuts that substantially. Stratify by task type, since aggregate passes hide exactly the category gaps Andreou concedes.

Copy Microsoft's template-the-boilerplate trick, not its model choices, because its model choices ran on free inference.