Rust for high-throughput agent runners — when it's worth it

The Rust-in-AI-stack conversation has shifted in the last year. Two years ago, Rust in this space meant ML framework internals (PyO3, candle) and the occasional vector store. Now it shows up in the agent-runner layer — the part of the stack that schedules agent steps, manages concurrent tool calls, and routes traffic to model backends.

The question worth asking, before anyone writes the Rust, is: is this layer actually the bottleneck? The short answer is “usually no, but sometimes yes, and the sometimes is worth knowing.” This piece is the survey of where the answer is yes.

What an agent runner does

A runner is the component that takes a structured agent specification (a graph or a state machine of agent steps), executes the steps, manages the model calls, handles tool execution, and persists state. The runner is the substrate. The agents are the workload.

The runner’s work is mostly:

• HTTP I/O to model providers (the dominant cost at the wall-clock level).
• HTTP I/O to tool endpoints (the second-dominant cost).
• JSON serialization, deserialization, and validation.
• Concurrency management — fanning out parallel tool calls, joining results.
• State management — checkpointing, recovering, resuming.

Most of those are I/O-bound. The HTTP I/O dominates wall-clock time. The compute work is small.

This is why the answer to “should I write the agent runner in Rust” is usually “no.” If the runner is spending 95% of its wall-clock time waiting on HTTP responses from a model API, the language of the runner is not the bottleneck. Python with asyncio handles this load fine.

Where Rust earns its keep

There are three exception conditions. They are narrow, but where they apply, the case is strong.

High-concurrency request fanout

A runner that fans out to 200+ concurrent tool calls, joins their results, makes decisions on the joined results, and continues, can hit the limits of Python’s asyncio runtime. Not because asyncio is slow, but because the Python interpreter’s overhead per coroutine is meaningful at scale. We have seen runners hit 30% CPU on the orchestration layer alone at high fanout. Rust’s tokio runtime is order-of-magnitude cheaper per concurrent task.

If you are running multi-thousand-tool-call agents (rare but real — search agents, simulation orchestrators, multi-tenant high-volume routers), Rust on the orchestration layer is worth the rewrite. We have seen 5-10x throughput improvements on the orchestration layer alone.

Strict latency SLOs on the runner overhead

If the application has a latency budget where the runner overhead is a meaningful fraction (e.g., a real-time application where every millisecond of orchestration overhead matters), Rust wins on the consistency of latency, not just the mean. Python’s GC pauses, in particular, can produce p99 latency spikes that are hard to design around.

The teams that pick Rust for this reason are typically working in the voice-AI or low-latency-completion space, where the model call itself is 50-100ms and the runner overhead has to stay under 5ms p99. Python can hit that mean; Python’s p99 is harder.

Heavy validation and transformation

If the runner does a lot of structured-output validation (JSON Schema validation on every tool call response, for example), the validation work itself can become meaningful. Rust’s serde and JSON Schema libraries are roughly 50x faster than Python’s equivalents on our benchmarks. For runners that process millions of structured tool responses per day, the validation layer is a real cost.

This is the most niche of the three. Most teams’ validation work is small enough that Python’s overhead is fine.

Where Rust is the wrong call

Three places we have seen Rust picked for the wrong reason:

“We want better types”

Python’s type-checking story is genuinely improved (mypy, pyright, pydantic). It is not Rust. The gap is real but it is much smaller than it was in 2020. If “we want types” is the primary motivation for picking Rust, the right answer is usually to invest more in your Python type discipline.

“Rust is the future”

Maybe. Probably for some layers of the stack. Not necessarily for yours. Pick the language against the workload, not against the trend.

“Hiring is easier in Rust”

This is provider-dependent and team-dependent and not actually true in most markets we have observed. Hiring senior Rust engineers is roughly as hard as hiring senior Python engineers, with the additional constraint that the candidate pool is smaller and concentrated in specific subcultures.

The hybrid pattern

The pattern that has emerged in production is hybrid: Rust on the runner, Python on the agent definitions and tools.

The runner is a Rust service that exposes a small API: “execute this agent graph, with these tools, against these models.” The agent definitions, the tools, the prompts, the eval harnesses — those stay in Python, where the iteration velocity is higher and the ecosystem is denser.

The handoff is via a structured RPC protocol (Cap’n Proto, gRPC, plain HTTP). The Rust runner does not have to understand the agent’s semantics; it only has to schedule the steps and route the model calls. The Python layer expresses what the agent does.

This pattern is also the cleanest migration path. Teams that have an existing Python runner can rewrite the orchestration layer in Rust without rewriting the agent definitions. The hybrid is most of the win for a fraction of the work.

What the candidate stack looks like

If you have decided the runner is the bottleneck and you are picking Rust, the production-ready candidates as of 2026:

• tokio for the async runtime. The default choice; it is unrivaled.
• reqwest or hyper for HTTP. reqwest is the friendlier surface; hyper is the lower-level option.
• serde + schemars for JSON serialization and JSON Schema. Battle-tested.
• tracing + the opentelemetry-otlp crate for observability. The OTel story in Rust is good.
• tonic if you are exposing a gRPC API to upstream services.

For state persistence, the standard cloud-native options (Postgres via sqlx, S3 via aws-sdk-rust) work fine. The Rust-native ecosystem is dense enough now that you do not have to write infrastructure plumbing.

What we would tell a team considering the rewrite

Three questions, in order:

1. Have you profiled the existing runner? If you do not know whether the orchestration layer is the bottleneck, the answer is “do not rewrite.” Profile first.
2. Is the bottleneck in a layer that Rust will actually help? I/O-bound work in Python is fast. CPU-bound coordination of high-concurrency I/O is where Rust helps. Validation and serialization is where Rust helps. The model call itself is not where Rust helps.
3. Can you afford the iteration-velocity tax? Rust development is slower per line than Python development. The trade is real and you should be honest about it. If your runner is going to need significant changes over the next year, the velocity tax adds up.

If you can answer “yes” to all three, the rewrite is justified. If any answer is “no,” stay in Python and revisit later.

The Rust path is the right call for a specific subset of agent-stack components. It is not the right call for most of them. The teams that get the win do so because they are honest about which subset they are in.

— The Editorial Team