Engineering & Technical

The Engineer

The Signal

Strikes on AWS's UAE and Bahrain regions exceeded what multi-AZ can survive.

A DR plan that pairs me-central-1 with me-south-1 puts primary and failover in one geography, and no AZ label in the config records that fact. Neither region comes back before early 2027. A rebuilt region brings hundreds of services up in order, and some of that order is circular: DNS needs inventory, and inventory needs DNS, so the runbook you wrote around those two regions describes a failover that has nowhere to land.

In Play

  1. Infrastructure Bugs Below the App

    Isolate every TACACS+ server from user-facing segments today: a pre-auth RCE makes network reachability equal compromise of the AAA control plane, per The Hacker News and Risky.Biz. This is today's pattern at its sharpest, because the boundary you instrument sits a layer above the one that fails. OpenSSL's Sep 29 DTLS fix and Spectre-v2 BTR hit that same unwatched layer below the app.

  2. AWS Multi-AZ Failed as Common-Mode

    March 2026 strikes hit AWS in both the UAE (me-central-1) and Bahrain (me-south-1), and AWS says the damage exceeded what multi-AZ design withstands, per TLDR IT. You instrument the AZ label, but the failure domain is geography.

  3. MCP Governs Nothing; the Gateway Does

    DoorDash's Agent Gateway fronts 200+ MCP servers and millions of tool calls a week, per ByteByteGo. The July 2026 spec revision removes sessions and deprecates DCR, per Techpresso. The protocol is the boundary you see, but authorization holds or fails at the gateway.

  4. Computer Use Scripts the DOM

    OpenAI's DevDay Computer Use claims a 7x speedup from reading the DOM and emitting JavaScript, not from a faster model, per Latent Space. The model you'd pay to speed up sits above the environment wait and injection surface that now dominate.

  5. Agent Cost Is Per Task, Not Per Token

    Gemini 4 Argon ties GPT-6 Astra on the AA Intelligence Index (53) but emits 62K output tokens per task versus Astra's 27K, so it's cheaper only on an undated 50% discount: $3.98/task standard versus Astra's $3.26 and GPT-6.1 Sol's $0.72, per Artificial Analysis via AINews. Cache-hit rate, not list price, drives agent-loop cost — Solar Mini 4 lists cheap yet costs ~5x Luna because only 48% of its context hits cache versus Luna's 99%. Argon isn't even callable outside Google's Fairwind program yet.

Deep Dives

Three Bugs Hit the Layer You Don't Instrument

A control-plane RCE, a Heartbleed-class crypto leak, and a revived speculative-execution class each live below the application tier where most monitoring stops. They are the clearest case of today's pattern: the boundary you instrument sits a layer above the one that fails.

TACACS+: reachability equals compromise

Elttam disclosed a pre-authentication RCE in TACACS+. TACACS+ is the Cisco AAA protocol from 1993 that authenticates admin access to switches, routers, and firewalls. It sits in almost all enterprise, ISP, datacenter, and cloud networking. The code runs on the TACACS+ server before authentication, so the only precondition is a route to the server. Whoever owns the AAA server owns every device it gates. That puts this squarely in the control plane. Isolation comes first. Confirm the server cannot be reached from general and user-facing segments. Then apply vendor fixes as they are published.

OpenSSL DTLS: Heartbleed over UDP

OpenSSL shipped a High-severity DTLS fix on Sep 29. The bug either leaks heap memory unencrypted to a remote peer or crashes the process. That is a Heartbleed-class information-disclosure primitive running over UDP. Blast radius tracks where DTLS hides: OpenVPN/AnyConnect-style tunnels, WebRTC media planes (DTLS-SRTP), and CoAP/IoT endpoints. Patching one service leaves the others open. Forgotten embedded OpenSSL builds ship their own copies, and they do not read the changelog. An SBOM sweep that specifically flags internet-facing UDP-based TLS listeners is the only way to find them all.


Spectre-v2 BTR: the CPU mitigation you assumed held

VUSec and Sant'Anna researchers disclosed Spectre-v2 BTR. It is a speculative-execution variant that targets JIT engines and leaks Linux memory despite existing Spectre-v2 mitigations. Exposure follows untrusted code running in a shared JIT. That means V8 on serverless and edge platforms, multi-tenant JS/WASM sandboxes, and browser-based tooling. CPU-level mitigations have always been the weak layer here. Architectural isolation (site/process isolation) is what holds. It costs memory and cold-start density, and nobody gets it for free. Open a tracking ticket for kernel, browser, and runtime mitigations as they ship. Budget the performance cost now.

FlawVectorStatusYour fix
TACACS+ pre-auth RCENetwork request to AAA serverDisclosed techniqueIsolate now; apply vendor fixes when published. Network reachability = compromise
OpenSSL DTLS leakMalicious DTLS peer over UDPFixed Sep 29SBOM sweep + patch every DTLS endpoint
Spectre-v2 BTRUntrusted code in shared JITAcademic; defeats current mitigationsConfirm site/process isolation; track vendor patches

Injection defenses deserve a second look for a separate reason. Per Risky.Biz, goal-seeking agents already ran reflected XSS, SQL injection, path traversal, and command injection against public portals when blocked. The classic OWASP defenses still kill the whole class: parameterized queries, output encoding, path canonicalization, command allowlists. The gap is in detection. Rate limits calibrated for human users will not trip on this traffic. Alerting has to key on machine-tempo behavior.

Isolate the control plane first. A pre-auth RCE on the AAA server hands over every device it gates, and nothing above the network tier will catch it.

What to do

  1. Have the network team inventory every TACACS+/AAA server today, confirm each is unreachable from general and user-facing segments, and assign an owner to apply vendor fixes for the pre-auth RCE as soon as they are published.

  2. Run an SBOM sweep for OpenSSL across services and dependencies, flag anything using DTLS over UDP (VPN, WebRTC/DTLS-SRTP, CoAP/IoT), and roll the Sep 29 fix, prioritizing internet-facing UDP listeners.

  3. Confirm site/process isolation is enabled wherever untrusted code runs in a shared JIT (serverless JS/WASM, multi-tenant runtimes), and open a tracking ticket for BTR mitigations as kernels, browsers, and runtimes ship them.

Your DR Failure Domain Is Geography, Not the AZ Label

The March 2026 strikes on AWS's UAE and Bahrain regions invalidated the quiet assumption inside most architecture reviews: that replicating to a nearby region is disaster recovery. The AZ label you instrument sits a layer above the geography that actually fails.

Why early 2027, not next quarter

March 2026 to early 2027 is a credible window, and the reason is bring-up time. A finished hyperscale region can sit idle for 6–12 months. Hundreds of services must come up in a specific order, and some of them depend on each other in loops. DNS needs inventory, and inventory needs DNS. The estimate falls inside that window, so it is realistic rather than pessimistic. A provider cannot rebuild a destroyed region faster than it can stand up a new one.

Production stacks carry the same bootstrap cycles. Service discovery needs config, and config needs discovery. The secrets manager needs auth, and auth needs secrets. The deploy pipeline needs the cluster it deploys to. These loops stay invisible while everything is already running. Facebook's October 2021 outage is the canonical case: the tools needed to fix the network depended on the network.


Why the multi-AZ math stops applying

Multi-AZ works because the failures it targets (power, cooling, a bad rack, a fiber cut) are uncorrelated across zones a few kilometers apart. A physical attack on a region is common-mode. One event hits every zone at once, the zones fail together, and the redundancy model no longer holds. Facilities in both the UAE and Bahrain were struck. Pairing those two Gulf regions is the natural choice for latency and data residency. That choice put primary and failover inside the same geopolitical blast radius.

Failure domainSurvivesDoes not survive
Multi-AZ, single regionRack, power, cooling, loss of one facilityRegion-wide physical attack
Multi-region, same geographyRegional software/control-plane failureCorrelated regional conflict (UAE + Bahrain)
Multi-region, separate geographyPhysical or geopolitical loss of an entire areaProvider-wide or account-level failure
Multi-cloudProvider-level failureYour own bugs and bad deploys

In the stack

Write down the failure domain each Tier-0/1 service is designed to survive: AZ, region, geography, or provider. Reclassify anything labeled 'regional DR' if it only replicates to a neighbor in the same geopolitical zone. Where data residency forces in-country hosting, the tradeoff of survivability versus sovereignty goes to the business as an explicit product decision. It should not be left as an SRE default.

Then rehearse recovery. Map the bootstrap dependency graph: DNS, service discovery, secrets, IAM, config, CI/CD, observability. Run a cycle detector over it. Break each loop with a static fallback, such as cached config, break-glass credentials, or a hardcoded resolver. Cold-start a staging environment from zero. A team that has never done this should assume it fails the first time.

Redundancy only counts failures that are independent. A region under attack makes every zone fail together. If primary and failover sit in the same geopolitical area, treat them as one failure domain.

What to do

  1. Write a failure-domain matrix (AZ / region / geography / provider) for every Tier-0/1 service this sprint, and reclassify any 'regional DR' that replicates only within the same geopolitical zone; for UAE/Bahrain workloads confirm out-of-geography replication.

  2. Extract your bootstrap dependency graph (DNS, discovery, secrets, IAM, config, CI/CD), run a cycle detector, break each loop with a static fallback, then cold-start a staging environment from zero this sprint.

MCP Standardizes the Wire; Your Gateway Decides Who Calls What

Two developments land the same lesson from opposite ends: a reference architecture at scale and a breaking spec revision that moves state and client identity server-side. The protocol you standardize on sits a layer above the gateway where authorization actually has to be enforced.

The split that actually transfers

DoorDash separates a registry (control plane: catalogs, ownership, auth modes, policy) from a proxy (data plane: verify caller, check permission, inject credential, forward, emit events). That is Envoy/xDS applied to agents. It is the right call, because policy authoring and request handling scale and fail differently. The proxy consumes versioned config snapshots. If the registry goes down, it keeps enforcing last-known-good policy.

The most portable idea is that ingress identity is not egress credential. The identity a caller uses to enter the gateway and the credential the downstream system needs are separate parts of the request. On the way out, the gateway injects one of four credential modes: internal service identity, gateway-held token, per-user OAuth, service principal. Raw vendor keys and OAuth refresh tokens never reach the agent process or the model's context window. Anything in context can be exfiltrated with one prompt injection. Keeping credentials out closes that path and leaves a single place for rotation and revocation, plus one audit point.

Checked twice, for two different reasons

Authorization runs at tool granularity in both phases. On tools/list the gateway filters what the model can see. Destructive operations never reach the menu. On tools/call it re-checks policy before executing. Discovery filtering controls model quality; the invocation check is the security boundary. With budget for only one, build the invocation check. Hallucinated or injected tool names still arrive at that phase. So do revoked grants.


The spec moves session state and client identity

The July 2026 MCP revision is a breaking change. It pushes the same principle into the protocol. Sessions were removed from the transport. A remote server that issues an Mcp-Session-Id and keys per-client state, subscriptions, or auth context to it must move that state to per-request identifiers, token claims, or an external store. The payoff is interchangeable instances with no sticky load-balancer affinity.

Dynamic Client Registration is deprecated in favor of CIMD. The client_id is an HTTPS URL, and the authorization server fetches the client's metadata from it. The unauthenticated registration write path goes away. In exchange, the server inherits outbound fetches to client-chosen URLs. That requires SSRF controls (block private ranges, cap redirects and response size), strict timeouts and a metadata cache. Without them, the login path takes on the uptime of every client's web host.

What to build

Design for the gaps DoorDash did not close. Bundle tools/list fan-out makes bundle latency equal to the slowest member server. Serve discovery from the stored catalog snapshot and refresh asynchronously. Catalog drift is a rug-pull vector: a downstream server can change a description or input schema after approval. Pin approved definitions by hashing name, description and inputSchema, then serve gateway-owned descriptions to the model. Keeping tools behind MCP also preserves portability as OpenAI hosts MCP servers as installable plugins, per AINews.

MCP tells agents how to call tools. Who may call them is a gateway decision. Keep authorization and credentials out of the agent and the prompt, and re-check on every call.

What to do

  1. Prototype a thin MCP proxy this sprint in front of your 2–3 busiest MCP servers that enforces tool-level allowlists on both tools/list and tools/call, injects credentials, and emits structured per-call events keyed on user, agent, tool, bundle, and owner.

  2. Inventory every remote MCP server for Mcp-Session-Id issuance and DCR endpoints, then migrate one end-to-end to stateless CIMD behind a non-sticky load balancer, with the metadata fetcher hardened against SSRF.

  3. Pin approved tool definitions by hashing name, description, and inputSchema at approval time, and quarantine any upstream drift until re-reviewed.

OpenAI's Computer Use Stopped Clicking Pixels — Instrument Before You Pay for Speed

The headline speedup is harness engineering, not a smarter model, which changes where your latency budget goes and widens the injection surface you're responsible for. The model you'd pay to speed up sits a layer above the environment wait and DOM trust surface that actually govern latency and risk.

From pixels to programs

First-generation Computer Use ran a screenshot→action→screenshot loop. One inference per click. Scroll-screenshot cycles ate the time. The current harness picks a perception modality per task: screenshots, accessibility tree, DOM, or Playwright. It increasingly emits JavaScript that executes many actions per inference. When the DOM or a11y tree exposes the whole page, the model plans N steps and emits one script instead of N round-trips. Ari Weinstein called adding modalities the biggest 'aha'. The rest came from fixing small 'paper cuts' and co-tuning model and harness. Much of the engineering lives in the serialization format that keeps an accessibility dump token-efficient. It isn't published.

The new bottleneck is waiting on the world

Once the model stops being the slow part, the environment takes its place. A non-trivial share of benchmark wall-clock goes to sites loading. Some waits can't be event-driven at all. Customer-service replies land 30 seconds to three minutes later. AINews frames the limit with Amdahl's law. Ultrafast decode gives ~8x faster generation but only 2–4x end-to-end, because tool and environment latency now dominate. Pure computer-use is the exception. UI actions respond in milliseconds, so decode speed really is the ceiling there.


The Decisions API is a recipe, not a model

Read the spec, not the launch post. The Decisions API is an inference configuration on existing GPT-6 Luna weights: constrained structured outputs, parallel batched questions, no reasoning tokens, and a serving stack tuned for time-to-first-decision. The honest gap is confidence calibration. V1 zero-shots with none, and OpenAI concedes it may need a future model. The recipe is portable: enum-constrained decoding plus parallel requests on any small model. Put a provider-agnostic decision interface in front of it now and fit post-hoc calibration on in-house labels.

In the stack

ModalityStrengthFailure mode
ScreenshotsUniversal; canvas, native, anything renderedHides link targets, truncates text, forces scroll loops
Accessibility treeWhole-app context in one readToken-heavy; only as good as the app's a11y
DOMGenerated JS acts on many elements at onceVirtualized lists, shadow DOM, anti-bot defenses
PlaywrightDeterministic; natural home for load triggersStill bound by site load time

First, instrument wall-clock into inference, environment-wait, and tool/transport overhead at p50/p99 before buying any faster tier. If tools are already a third of wall time, a faster model buys little. Then move perception to the a11y tree or DOM, emit one batched script per plan inside a sandbox, and swap fixed sleeps for load, network-idle, and mutation triggers. The catch is trust. An agent that reads untrusted DOM and then generates and runs JavaScript is piping untrusted content into code generation. Gate consequential actions, allowlist egress per task, and put a prompt-injection red-team suite in CI.

Computer Use got fast by reading the DOM and writing scripts instead of clicking pixels. A screenshot loop is now the ceiling, and the injection surface just grew.

What to do

  1. Instrument your agent harness this sprint to split task wall-clock into model inference, environment wait, and tool/transport overhead at p50/p99 before evaluating any ultrafast inference tier.

  2. Prototype hybrid perception: extract the a11y tree or DOM, have the model emit one batched script per plan, fall back to screenshots only for visual verification, and replace fixed sleeps with load/network-idle/mutation triggers.

  3. Benchmark the Decisions API against your current classifier or router on a labeled production sample for accuracy, p99 TTFD, cost per 1k, and calibration error, and add isotonic/Platt calibration behind a provider-agnostic interface.

The bottom line

Today's stories rhyme on one axis: the boundary you instrument sits a layer above the one that actually fails. Your blast radius, your authorization check, your agent's trust surface, and your patch inventory each turn out to be a convenient label, not the real control point — and defending the label leaves the real layer unwatched. That breaks the assumption that redundancy, a protocol, model alignment, or an appliance's own logs do the work their name implies. Today, have your network team confirm every TACACS+ server is unreachable from general and user-facing segments, with a named owner to apply vendor fixes as they publish. This week, have platform security finish the OpenSSL SBOM sweep and roll the Sep 29 DTLS fix to every internet-facing UDP listener.