ClarityBeta

Science & Analytics

The Scientist

7 min read · 1,477 words · 3 moves to watch
TUESDAY, SEPTEMBER 8, 2026
The Signal

Agents scoring 87.9% on rubric partial credit finish only 20.6% of full workflows.

The tasks were multi-stage scientific workflows that had to end in reproducible code, tables and figures. Partial credit measures how far a model got. The thing it doesn't tell you is whether anything at the end actually ran. Rubric-weighted scoring is the default in most internal agent harnesses, so the completion rate you're reporting upward is probably a progress rate wearing a better name.

In Play

  1. Partial Credit Overstates Agent Deployability

    FrontierChallenge scored frontier models at 87.9% on partial progress and 20.6% on full completion of multi-stage scientific workflows that must ship reproducible code, tables and figures. Rubric-weighted partial credit is the default in most internal agent harnesses, so your scorecard overstates deployability about fourfold on compound tasks. RealSWE adds that agents degrade further on short, informal user requests, and the cheapest fix is a required desired-behavior field in task intake.

  2. Coding-Agent Spend Is an I/O Bill

    Spotify's Portal splits coding-agent work into bulk-reader and code-writer modes on Gemini 2.5 Flash, and Dimitri Mazmanov measured roughly a 90% cut in Claude Code context tokens on Java monorepo tests. That lands on a real budget line: 25% of engineering leaders report $200-$500 per developer per month, with outliers past $2,000. The evidence is one engineer's token delta with no task-success rate and no ablation, so treat 90% as a hypothesis with a plausible mechanism behind it.

  3. Tool Selection Is the Injection Surface

    Johann Rehberger's writeup on breaking Claude Code Opus 5 auto mode describes a website that asks the agent to summarize it, nudges it off the constrained WebFetch primitive onto curl, then reuses the model's own generated Python decoder as the exploit primitive. The compromise happens in tool choice, so a guardrail that inspects tool arguments passes it cleanly. Frontier capability is not a robustness proxy, which means a model upgrade should not ship with wider tool scope until your adversarial suite passes at the prior model's rate.

  4. Session Cookies Are the Data-Platform Credential

    JSCeal, documented malware shipped as compiled V8 bytecode, bypasses Google authentication by replaying stolen session cookies rather than cracking credentials. That session lives in the same browser profile as Colab, the Vertex AI console, the BigQuery UI and the GCS browser, so password rotation and MFA are no-ops and only session invalidation plus token revocation work. The compiled-bytecode packaging also strips the lexical surface that source-token, AST and entropy features depend on, so any static JS detector needs per-slice recall reported by packing method.

  5. Serving Frontier MoE Without NVLink

    A community wiki now publishes reproducible Docker builds, runbooks and benchmarks for GLM-5, DeepSeek-V4 and Kimi on Nvidia RTX PRO 6000 Blackwell cards over PCIe alone, with no NVLink in the topology. Sparse mixture-of-experts layouts are why that can work: low active-parameter counts shrink the per-token collective payload that saturates the bus on dense tensor-parallel serving. Bloomberg separately reports DeepSeek plans at least 160,000 Huawei top-end accelerators at one Inner Mongolia site, so accelerator, driver and quantization recipe belong in your eval metadata now.

Deep Dives

The Denominator Every Cost Number Is Missing

Three unrelated disclosures land on the same replacement metric, and the arithmetic behind it says two models with identical accuracy can differ 40% in serving cost.

The bias runs one direction and has no bound. That asymmetry is the whole problem: every failure mode inflates the same metric, and nothing in the measurement pushes back the other way. An LLM call can return HTTP 200 and still be wrong in logical terms. A success rate keyed to transport status is measuring the layer where nothing interesting fails. The thing a 200 doesn't tell you is whether the response satisfied the contract. The source counts seven modes that qualify. The ones it names: instructions ignored, prose where JSON was expected, hallucinated products or policies, truncation at the token limit, spurious refusals, and answers that are simply wrong. There is no sourced figure for the size of the resulting overstatement, which is itself informative about how often anyone checks. Note the sampling problem underneath. Modes that get caught are the ones someone built a validator for. Schema violations get caught because JSON parsing fails loudly. Ignored instructions and quietly hallucinated policy text pass every automated gate that exists in most pipelines, so their observed rate is a function of instrumentation coverage rather than model behaviour. Correlating a clean dashboard with a healthy system is a mistake that this measurement design invites. The practical read: treat the unchecked modes as an open interval and report the range rather than the point estimate. A stated success rate is an upper bound on correctness, and the gap between the bound and the truth is exactly as wide as the validators are thin. Per-mode breakdowns are what make the number usable, and the enumeration above is where a breakdown would start.

What to do

  1. Add a binary end-to-end completion field to every agent scorecard this sprint and backfill it across your last three agent evals to quantify your own partial-credit-to-completion gap.

  2. Stand up a sampled semantic error budget within two weeks: draw ~750 responses daily, label against a four-part rubric, and publish the defect rate with a Wilson interval next to your 5xx and 429 rates.

  3. Make model_id, route_reason, degradation_tier, retry_count, validator_verdict and timeout_flag required fields on every inference event now, and gate new experiment readouts on their presence.

Measure the I/O Share Before You Touch the Router

Routing built before measurement is guesswork, and the same lesson arrives independently from a Postgres load, a Meta proxy and a starving training dataloader.

A credible mechanism, measured at n=1

The observation underneath Portal is concrete. A coding agent reads five files to answer a question about one method, so most of what it does is I/O rather than reasoning. Two declarative modes, bulk-reader and code-writer, run on a fast cheap model on an ephemeral runtime, and the frontier model is held back for work that genuinely needs reasoning. The evidence is one engineer's token delta on Java monorepo tests, with no task-success or latency numbers, and no ablation between bulk-reader and code-writer. The cost survey behind the spend figures states no sampling frame and no org-size stratification.

So instrument before routing. Log tokens by tool-call class, meaning file read, schema read, log read, plan and write, for two weeks, then compute the I/O share of spend. At 70-80%, the routing pattern has a real ceiling. At 30%, the binding constraint is context packing and retrieval precision, and routing will disappoint. After that, run a two-arm eval on a frozen task suite scoring tokens, first-pass task success and human-edit rate together, and adopt only if success holds within about two points. A 90% token cut that costs 15 points of first-pass correctness is negative ROI the moment engineer time is priced in.

Data movement, not compute

The finding travels a long way outside agents. A 100,000-row Postgres load took 41 minutes because every row became its own statement and its own transaction; COPY did it in 11 seconds, roughly 224x. Meta put a proxy in front of ZippyDB and cut connection fan-in 97-98% while gaining cross-client batching and coalescing. Vortex streams training data from S3 to GPU with cascading lightweight encodings, layout-aware pruning and zero-copy paths. Nobody involved was comparing notes. All three measure data movement rather than compute.

Two of the largest cost lines in an ML org sit exactly there. Agent token spend is inflated by bulk reads with near-zero reasoning value. GPU spend leaks whenever CPU or NVMe starves the accelerator. Both arrive on the invoice looking like capacity problems.

Where to look in your own stack

  • Write paths. Driver-level "batch" APIs frequently degrade to per-row statements and transactions, and the degradation never appears on a dashboard. Check feature-store materialization, the prediction logging sink, and offline scoring writes. Real batching removes network round trips, parse overhead and WAL flush barriers; driver flags and transaction chunking are what stop a bulk API from silently going row-by-row.
  • Read paths. Measure dataloader stall as a share of step time first. Below roughly 15%, a zero-copy S3-to-GPU format is not the problem, and it adds a new format dependency for nothing.
  • Interchange. Arrow is quietly consolidating as the only format that matters. Sail 0.7 persists shuffle output as compressed Arrow streams to object storage, and DuckFlight exposes a live DuckDB to Arrow Flight SQL clients with no copies. A feature pipeline still round-tripping through pandas-in-the-middle serialization is paying a tax the ecosystem stopped paying.

The cross-source tension is worth naming. One account presents the routing pattern as a reproducible playbook. Another grades it explicitly as a single practitioner test with no quality metric. Both readings are defensible. Expect the I/O split to reproduce and the 90% figure to stay workload-specific.

The agent bill and the GPU bill are both premium prices paid to move bytes, and both come down by moving fewer bytes rather than by buying differently.

What to do

  1. Instrument agent and analytics LLM calls by tool-call class for two weeks starting this sprint, then compute the I/O share of total token spend before changing any routing.

  2. Profile your highest-volume warehouse write path this sprint for per-row statement degradation and convert it to a true bulk path with explicit driver flags and transaction chunking.

  3. Run a paired routing A/B on a frozen task suite this quarter scoring tokens, first-pass success and human-edit rate together, and adopt only if success holds within two points.

Your Engagement Labels Are Mostly Machines

This is not a traffic-quality complaint; it is a label-noise, covariate-shift and A/B-validity problem that your drift dashboard is structurally unable to see.

One blogger's server logs, read as a data-quality audit

Tracy Durnell's AWStats data is the cleanest public look at what arrives at a small site. 90% of visits last under thirty seconds, and a single IP fetched 423,059 pages. Her caveat is also the limit: server-side stats give a rough human/bot estimate, nothing better. Google's AI Mode held query volume while external clicks fell 18.8 points, so composition moved where volume looked flat.

Decomposed by failure mode, it sharpens:

  • Label noise. Engagement, conversion-intent and dwell-time targets absorb agent sessions as human, and short-session dominance skews time-on-page features.
  • Covariate shift. Referrer and channel distribution moved; the production model's training window predates the move.
  • Measurement validity. With bot share unmeasured, an A/B primary metric carries an unknown mixture component, and mixture drift between arms inflates variance quietly.

Delegated-action agents worsen this in a specific way. Session depth, dwell, scroll and page-sequence features go degenerate when an agent transacts for the user, and attribution breaks at the referrer layer. That is live distribution shift against conversion, fraud and LTV models, and the aggregate volume chart stays flat through it, which is why the monitors miss it.

The corpus side of the same problem

Dormant, unmoderated corners of the web are the tail generic crawlers ingest happily and human editors no longer defend. With agents demonstrably writing to live third-party sites, the marginal document a 2026 crawl adds is meaningfully more likely to be machine-authored than a 2023 crawl's was, and it reads fluently enough to pass dedup and perplexity filters. Most pipelines carry no field for it.

The expensive version is an eval validity failure: the held-out set drifts toward machine text while production traffic stays human, so offline metrics stop predicting online performance. That correlation is the only reason the offline eval exists.

Artifact detection will not rescue either side. Auto-labelling of AI-generated video is bypassed by a real person reading an AI script at $26 per script and 24-hour turnaround, and clips persisted across dozens of accounts after 20 channels reaching 45M views were terminated. Generation artifacts are a dead end for a synthetic-content classifier, and the signal lives in behavioral and network-graph features.

The move: make the contamination a number

The cheap version is a verified-human holdout cohort, a refit of each engagement, propensity or session-based model on it, and the AUC or lift delta between contaminated and cleaned populations posted beside the drift monitors as a standing metric. That delta is the contamination estimate, and unlike a bot-detection heuristic it survives review. On the corpus side: a crawl-date field and domain-liveness filter on every web-derived ingestion path, a frozen pre-2026 human-verified eval slice as control, and 10K newly ingested documents sampled for machine-authored share. The threshold argument cannot start before that number exists.

If the held-out set drifts toward machine text while production traffic stays human, the offline metric has stopped predicting the only thing it was built to predict.

What to do

  1. Add an agent_channel dimension to the clickstream and feature schema now, and set PSI alarms on the session-behavior feature block for conversion, fraud and LTV models.

  2. Build a verified-human cohort this sprint, refit your primary engagement or propensity model on it, and publish the AUC delta versus the contaminated population as a standing data-quality metric.

  3. Freeze a pre-2026 human-verified eval slice as a control arm this quarter and sample 10K newly crawled documents to estimate the machine-authored share of each refresh.

The bottom line

Three separate stories resolve to the same repair, and none of it is a model change: the figures arriving from vendors and benchmarks all report a numerator, while the denominator — did the task actually finish, how many of those sessions were even human, which tier really served the response — exists only in telemetry you own. That inverts the usual sequencing, because the instrumentation is days of schema work and the model comparison it unlocks is weeks. Pick the one production number your team already acts on daily, give it a denominator this week, and let every routing and upgrade decision run against it.