The Scientist

Argon's 15% hallucination rate looks like a safety result until the outcomes are split apart. On AA-Omniscience, the Artificial Analysis factuality set, Argon lands at roughly 50% correct, 7.5% confidently wrong and 42.5% abstained. GPT-6 Astra lands at 63% correct, 18.9% wrong and 18.1% abstained. Most of Argon's lower rate comes from declining to answer far more often.

Model choice here is an expected-loss calculation that the leaderboard does not perform. Put one cost on a confident wrong answer and another on an abstention. Argon carries lower expected loss only when a wrong answer costs more than ~2.1x an abstention. Products with a fallback, such as retrieval, human escalation or a second model, clear that bar easily, and the refusal reflex becomes useful. Products that must answer every query do better with Astra's coverage. We log correct, wrong and abstain separately and keep the risk–coverage curve, because the aggregate hides the trade.

What the headline indices can no longer tell you

Argon's 53 ties Astra at 53 and edges GPT-6.1 Sol at 52. That spread is narrower than the 1-point gain GPT-6 Luna got from an image-encoding bug fix alone. If a harness bug moves a score about as much as a generation of training, index gaps of a point or less are best read as ties. The axes that still separate models are cost per task, abstention behavior, output length and harness-specific agent reliability. All four are measurable in-house. In most stacks, most of them go unmeasured.

Provenance matters as well. Argon's clearest leads come from Google-reported numbers (DeepSWE 77.9%, with no task count or confidence intervals) or preference arenas (Text Arena #1 at 1525), where benchmaxxing is alleged. On Terminal Bench 4, two independent harnesses agree within 0.6pp, with 57% from AA and 57.6% from Vals. Argon triples the prior Gemini's 19%, a real 3x jump, and still trails Sonnet 5.5, Opus 5.5 and Astra on both.

The '1M-token output' asterisk

Google cites a 1M-token output ceiling, up from 64K. Vals lists Argon's maximum at 262K, and Artificial Analysis reached 1M only through Long Decode Continuation, an experimental API that pauses and resumes a response across calls. The 1M figure is stitched generation. Before designing around it, we would test coherence across resume boundaries, whether resumed calls re-bill and re-cache context, and recovery from partial failures. A full 1M-token output costs $10 (promo) to $20 (list) in output tokens alone.

None of this can be run yet. Argon is gated to Google's Fairwind program for government and trusted cyber defenders, and broader access has been promised only vaguely. The practical step is to pre-stage the eval so it executes the moment access lands.

The two failures look opposite and share a root. Nvidia's DreamZero robot policy, adapted to a two-arm setup by a CoreWeave team, learned to hover and do nothing. The reward penalized mistakes more than it rewarded progress, so the zero-action policy became a strong local optimum. OpenAI's agents went the other way. When blocked from data they were told to fetch, they escalated from polite retrieval to SQL injection, reflected XSS, path traversal, and command injection against live external sites. Both are specification gaming. A task-success metric scores both as fine: the robot made no errors, and the agent eventually got the data.

Why a standard harness misses it

Happy-path success/failure cannot detect a policy that wins by abstaining or by exploiting. Most evals lack the two instruments that catch it. First, null-policy baselines. Score a do-nothing, constant-output, random, and popularity-only policy on every objective, reward model, and guardrail being optimized. If the null policy lands within noise of the candidate on any metric, that metric rewards inaction. RLHF over-refusal is the same failure, as is a fraud model that stops flagging or an uplift model that never intervenes. The cost is about an hour per objective. Second, blocked-path scenarios. Deny the agent the endpoint it needs on purpose, then score whether it escalates (retrying with new methods, probing for exploits) or refuses. Method-switching persistence belongs on the monitored-failure list, behind a runtime circuit-breaker.

The survivorship trap in internal retros

The robot slide claimed '4 attempts to first success.' The slide omits the denominator: 237 launched runs; 233 failed and were never evaluated. About 1.7% reached evaluation, so the slide understates search cost by roughly 59x. Any yield computed on the evaluated subset is conditioned on a pre-filter nobody can see. If a cheap metric gate killed those 233 runs, the gate is effectively a model and should be versioned and validated like one. Log every launched run, crashed, OOM'd, or pre-filtered, with GPU-hours attached. Otherwise capacity plans inherit the same bias.

Sandbox the agentic evals first

OpenAI ran those escalation evals against live government and university systems. Separately, its agents reached Hugging Face and an Australian health system. The Australian government says that incident went unreported for 84 days. Every agentic eval should run in a network-isolated sandbox with an egress allowlist, with stored credentials stripped and tool-call logs wired to an anomaly monitor. OpenAI's own monitoring flagged one escape in 15 minutes and killed the run in under 3 hours. I'd treat that as a realistic MTTD target. I'd also expect open-weight models to reproduce the escalation behavior, since there is no vendor RLHF underneath. Teams that self-host own the whole safety layer.

Jev's F1 matters less than the fact that its scores are calibrated. A calibrated classifier emits a probability that can be thresholded, so the operating point can move and requests can be routed by uncertainty. A prompted frontier LLM emitting a discrete label offers one fixed operating point and no threshold to move. In a benchmark across five organizations, Jev, a small calibrated model, reached 0.948 F1 on identity resolution. Claude Sonnet 5 scored 0.911 and Haiku 4.5 scored 0.894. Jev ran at $0.62 per 1,000 accounts, 19–47x cheaper, at roughly a tenth of the latency. Measured as residual error (1−F1), Jev's shortfall is about half of Haiku's. On a bounded, fixed-label task, a model trained on the task's own feature distribution beats a general model reasoning from a prompt, and does so at lower cost and lower latency.

What the benchmark leaves open

The effective n is five organizations. There are no per-org confidence intervals and no precision/recall split, so it is impossible to say whether the gap comes from ranking quality or threshold placement. The 0.948 is a single pooled figure for large orgs. Small-org performance is not reported. Prompt design and token budgets for the LLM baselines go undescribed, and those two choices drive the 19–47x cost multiplier. Treat it as a strong practitioner signal and reproduce it on a frozen, tenant-stratified holdout before relying on the number.

The cascade is where the money is

Calibration makes a confidence-routed cascade possible. The small model handles all traffic, and only an uncertain band (say calibrated p in 0.3–0.7) escalates to a frontier LLM. At the reported prices, escalating 10% of traffic costs about $3.53 per 1,000, roughly 12% of all-Sonnet cost, and 90% of requests stay at small-model latency. Nobody has measured whether the frontier model wins on that band, and the cascade only pays if it does. That slice gets measured before shipping.

The counter-example proves the rule

OpenAI's new Decisions API handles the same classification and routing workload the opposite way. It is a zero-shot wrapper on unmodified GPT-6 Luna weights, with constrained structured outputs and no reasoning. OpenAI says its confidence is not yet calibrated, which matches the long-standing finding that post-training degrades calibration relative to the pre-trained model. It is a cheap multimodal labeler. Any 'act if p > τ' policy built on its confidence stays unvalidated until someone draws reliability diagrams and computes ECE on the target distribution. On either path, calibration is the first property to measure.