The Cheapest Inference Call Is The One You Delete
Two of these cost levers sit inside decisions your team already owns, and the trap is booking either saving before you have measured it on your own traffic.
Route on fact class in code
The pitched version of this optimization is elegant: a confidence score decides when to retrieve, and the bill falls out on its own. The load-bearing detail in the Google Research and Technion work kills that version, and it is not the headline percentage. Google research scientist Nitay Calderon says models cannot reliably sense in advance when a direct answer will fail. Confidence-based routing is unshippable today, which leaves deterministic routing on fact class, written into code, as the only version that survives production.
| Query class | Retrieve? | Reason | Covered by the study? |
|---|---|---|---|
| Stable general facts | Probably not | 95-98% already encoded; extended thinking recovers 40-65% of recall misses | Directly measured |
| Freshness-critical (price, status, news) | Always | Weights carry a training cutoff | Not addressed |
| Private or customer data | Always | Never in the weights by definition | Not addressed |
| Citation or audit required | Always | Provenance is the requirement, not accuracy | Not addressed |
| Genuine reasoning | No — spend compute instead | Only 10-20% of facts land here | Directly measured |
Three of those five rows were never in scope, which bounds the prize honestly. This narrows retrieval spend without deleting the layer. The failure mode you are trading into is silent. A wrong stable fact generates no support ticket, so your ablation has to report an accuracy delta alongside the cost delta or it is not evidence.
The second lever is a default that ships set to high
Anthropic shipped its own discount switched off. Fable 5.1's effort dial has five levels from low to max and defaults to high, spending tokens whether or not the task needs them. Anthropic also shipped the tooling to find the waste and, per the release coverage, almost no users know it exists: /claude-api cost-optimize profiles real usage and ranks levers with nothing changing without your confirmation, and /claude-api prompt-audit strips the verification rituals, emphasis boosters and stale scaffolds that tax every single request until removed. There is also a migrate command, which is the tell. Moving to claude-fable-5-1 involves breaking parameter changes, not a model ID swap.
The reason the same release yields 25% on normal use and up to 45% on agentic work is mechanical rather than promotional. Agent loops re-read the same context on every iteration, so a cache-read discount compounds instead of landing once. Cache pricing, not headline per-token price, is now the dominant cost driver for anything that loops.
Where the two findings quietly disagree
Both push the same direction, and both have a seller behind them. Google's result moves the architectural story from retrieval toward parametric memory plus inference-time compute, which is exactly where Gemini's long-context positioning lives. Anthropic's CursorBench comparison sets 5.1 at low effort against Fable 5 at high effort, which is two configurations rather than two models. A team that genuinely needs high-effort reasoning may bank close to nothing, and dropping effort on non-routine work risks quality regressions that surface as churn long before they show on a cost dashboard. Score each workload on two axes before the sprint: does it loop, and are the facts it needs stable. Looping work on stable facts is where both levers actually pay.
Every saving in this cycle is a decision someone makes: delete the call, or open the config. Both require someone to open a file and change something.
The re-vote list
The same repricing reopens a graveyard. Gemini's agentic video understanding reportedly cuts tokens on long-form video by up to 88% while improving accuracy, using selective retrieval over the recording rather than brute-force frame ingestion. Features killed on unit economics, among them full-meeting analysis, lecture summarization and call review, deserve a re-vote with fresh numbers. One design requirement travels with all of it: weekly credit allowances are a hard usage ceiling, so ship automatic effort step-down, queueing and honest in-product messaging before a credit burn-through becomes a support escalation.
What to do
Run a retrieval-necessity ablation on 500-1,000 production queries before your next vector-database renewal: retrieval on, retrieval off, and retrieval off plus extended reasoning, reporting accuracy delta with p50/p95 latency and cost per call.
Publish a per-surface effort table this sprint — low or medium for routine surfaces, high or max only where reasoning depth is proven — and validate every downgrade against your golden eval set before it ships.
Re-score the two highest-value features you killed on inference cost by the end of this quarter, using cache-read and token-reduction figures you verified on your own workload rather than vendor benchmarks.