The Scientist

Each component maps to a familiar regularizer

RRSI leaves the edit space fully open. The optimizer may change any prompt, tool or control-flow step. The constraint is on how the search moves, and Alejandro Saucedo maps its three core components onto classic regularizers:

• Annealed edit budget behaves like learning-rate decay. Search starts with bundled edits and ends with single edits, so late changes stay attributable.
• Diff critic is a leakage audit. It reads each diff before scoring and rejects edits that encode task names or benchmark-specific answers.
• Pruner works like an L1 penalty. It flags components whose removal does not lower the score, which keeps token cost flat.

TheSequence's summary lists three more pieces: evidence-aware proposals, a cost rule, and a noise-adjusted acceptance floor. The floor rejects any edit whose gain is smaller than the eval's own run-to-run variance. It is the easiest piece to copy. Rerun the unchanged configuration about five times, estimate the standard deviation, and accept only edits that beat roughly twice that.

The small gap is the finding

Beyond the headline score, the two accounts differ on these details:

On TheSequence's figures, held-out gain trails evolve-set gain by only 0.6 points. Saucedo adds that RRSI had the smallest evolve-set gain of the runs compared and the best held-out average. That pattern is what regularization is supposed to produce. Seeds, variance and the unregularized held-out score are not in either writeup, so pull the paper's tables before any number reaches a design review.

The production version of the same loop

Anthropic ran the same structure at product scale in an agent-assisted sprint on claude.ai, per Architecture Notes. Over two weeks, time until a user can type on a fresh page fell from 3.1s to 0.55s at p75. Agents iterated against a benchmark, each change was checked against real user latency, and kept wins became CI checks. Humans rejected changes whose complexity outweighed the milliseconds saved.

The mapping onto RRSI is direct. The benchmark is the evolve set and real-user latency is the held-out set. The CI check locks in each gain, and the human veto plays the cost rule. Only p75 is reported, with no sample size or tail percentiles. And 3.1s to 0.55s is about a 5.6x improvement, not the ‘3x’ in the headline, so the two figures probably measure different things.

Will Larson's Software Factory experiment points at the next step: an agent that reads live metrics to choose its work. No results are reported. Once an agent picks work by a metric, that metric becomes its evolve set, and Goodhart's law (a measure that becomes a target stops being a good measure) applies to the dashboards.

The smart move

For every optimizer in use, report evolve gain minus held-out gain beside the score. If a tuned harness's lead shrinks on unseen data, its reported gains were in-sample noise. The check costs about a day and protects every decision built on that harness.

Where the 18 points come from

NVIDIA and KAIST left both the TMAX-9B generator and its harness untouched on TerminalBench-Lite. They changed only what happens between proposing a terminal action and running it. TheSequence's breakdown shows how much of the gain depends on who does the checking:

Two findings matter. First, verification does not distill cheaply. Training the small model to verify like the frontier model recovered only about 40% of the frontier verifier's gain. Second, the economics rest on a cheap generator and an expensive judge. Every action costs N generator samples plus verifier calls. That is affordable with a 9B generator, and the bill looks very different with a frontier one.

The small gains sit inside the noise

The reported percentages are suspiciously clean fractions: 57.14% is 4/7 and 54.76% is 23/42. That pattern fits a benchmark of roughly 40–50 tasks, possibly averaged over a few runs. At about 42 tasks, one task moves the score by about 2.4 points. The binomial standard error near 50% is roughly 7.7 points. That makes the +4.76 self-verification gain about two tasks, indistinguishable from noise. The +18-point frontier-verifier gain clears that bar, but a proper paired test needs per-task results the summary does not give. This sizing is inferred from the reported fractions; the authors do not state the task count.

CorpusMap pays the join once, offline

Microsoft and KAIST's CorpusMap attacks a different cost. Without it, an agent works out on every query which documents mention the same entity. CorpusMap precomputes Entity Pages that link those documents once, which is the RAG equivalent of a materialized view. On EnterpriseRAG-Bench, WixQA and HERB, it beat raw-corpus search and four alternative navigation layers. Its gain was up to +11.74 Overall Quality with the Luna model, against +6.45 with GPT-5.5.

The result has gaps. The summary omits the offline build cost, the refresh strategy for a changing corpus, and sensitivity to entity-resolution errors. It also does not say how Overall Quality was scored, which is likely an LLM judge. Because the lift nearly doubles from one model to the other, the layer's value depends on the model that reads it.

The checkpoint is not the unit of evaluation

Birgitta Böckeler's four-week field test of local coding models, as Architecture Notes summarizes it, is the cautionary case. Speed was often acceptable, but correctness and tool calling stayed inconsistent. RAM, context limits and harness overhead mattered as much as the model itself. She reports no pass rates, so treat this as a field report. Its lesson matches both papers: you evaluate model × harness × configuration, not a leaderboard checkpoint.

The smart move

Run the verifier experiment on your own agent with enough tasks to resolve a 10-point difference. Price it per successful task rather than per token. Queue entity-page indexing for this quarter, and log build and refresh cost next to quality so you can count net token savings, not gross.

The valuation gap is a disagreement about one parameter

Divide each Nscale valuation anchor by the backlog and the argument reduces to a single number. The reported $35B IPO target values the company at about 0.34x its backlog. The Information's view, under roughly $17.5B, implies under 0.17x. The two anchors separate on one estimate: the probability that 12 facilities get financed and energized on schedule. The $35B target is reported and the lower anchor is editorial judgment, and neither side has published the model behind its estimate. In ML terms, the backlog is a leaderboard score with no held-out set. The score is large. The quantity it is supposed to predict is physical delivery, and that is still unmeasured.

Who holds the downside in each structure

The third row is the most informative. Lenders seek insurance on collateral when they doubt what it would fetch in a forced sale. A build-vs-rent model that assigns a residual value to owned accelerators is relying on a number that the people lending against those same chips doubt. The talks may not close; treat this as an early signal on collateral values.

The strain shows up across the stack

• Credit. The Information reports that credit for data center debt is tightening and that the debt is spreading through the financial system. Oracle borrowed heavily for AI data centers and is down 27% this year. Larry Ellison has pledged more than a third of his 38% stake as loan collateral.
• Colocation. Jim Chanos estimates that Digital Realty's $18.6B of debt, repriced at 5.3%, would cost almost $1B a year, enough to erase its trailing operating income. The multiplication checks out. The result depends on the whole debt stack repricing at once, which is the stress scenario, and the realistic path to a tenant is renewal pricing.
• CPUs. Gergely Orosz flags a new CPU shortage. He did not say how large or where. The mechanism is worth tracking anyway: when tokenization and data loaders cannot keep up, scarce CPUs turn into idle GPUs.

The reporting is narrower than the bubble talk around it. A valuation discount, preliminary insurance talks and tighter credit widen the error bars on future supply. They stop well short of a predicted default.

Where it enters a production pipeline

Anthropic's expansion now runs partly on its chip vendor's balance sheet, and its Nscale capacity ramps in 2027–2028. That puts a dependency chain into production stacks that few teams model: your features, then an LLM API, then the provider's capacity, then a construction and financing schedule. The hedge works whichever link breaks. It consists of a fallback model with a measured quality gap, checkpoints that resume elsewhere, and procurement terms that survive a slipped build.