k8s for AI workloads — node autoscaling for variable inference

The autoscaling story for Kubernetes was written for stateless web tiers in 2017. AI inference is not a stateless web tier. The autoscaler defaults will lose you money in one direction and lose you traffic in the other, and the most expensive lessons here are the ones you only learn after a real incident.

This piece is the playbook I wish I had when I was setting up our first multi-tenant inference cluster. None of it is novel. Most of it is documented somewhere in the cloud providers’ tutorials, the karpenter docs, and the cluster-autoscaler issues. The value is in seeing it in one place against the use case it was made for.

Why default autoscaling does not work for inference

A stateless web tier scales on CPU and request rate. The pods are interchangeable. The work is short. The metric-to-decision loop runs in seconds. When you misestimate demand, you over-provision by a fraction of a node and the cost is small.

Inference is different on every one of those axes. Pods are not interchangeable; a pod running a 70B model cannot move to a smaller GPU and a pod doing CPU-only embedding cannot use a GPU node. The work is bursty — long tail of inference latency, occasional batched calls that hold the GPU for tens of seconds — and the metric-to-decision loop has to be smarter than “average CPU over the last five minutes.” When you misestimate, you either over-provision an h100 (expensive) or you under-provision and your p99 inference latency moves to “the model has not loaded yet” territory (catastrophic).

The right mental model is: inference is a fleet of bin-packing problems against expensive nodes. The autoscaler’s job is to keep the bins fed without buying spare nodes. The defaults do neither.

Node groups

The first move is to stop running mixed inference and non-inference workloads on the same node group. Even within inference, split:

CPU embedding (sentence transformers, BGE, Cohere embed local). Cheap, dense, bin-packs well. Lives on c6i or c7i instances.
Small-model GPU inference (7B–13B class). Lives on g5.xlarge or g6.xlarge.
Mid-model GPU inference (30B–70B class). Lives on g5.12xlarge, g6.12xlarge, or p4d.24xlarge depending on memory.
Large-model GPU inference (100B+). Lives on h100-class. Frequently dedicated; cost discipline forces single-tenant.
Batch GPU work (overnight evals, fine-tuning, reindexing). Lives on spot-class nodes with aggressive eviction handling.

Each node group has its own taint, its own labels, and its own scaling policy. The pods carry the matching nodeSelector and toleration. The result is that the autoscaler never has to wonder whether to scale GPU nodes for a CPU pod or vice versa. The cost discipline is mechanical.

Karpenter or cluster-autoscaler

We have run both. Karpenter is the better answer for inference workloads as of 2025 for three reasons.

First, Karpenter does just-in-time node provisioning per-pod. The cluster-autoscaler watches node groups and bumps the count when pods are unscheduled. The Karpenter model is faster (often 30–60 seconds vs. 90–180 seconds for cluster-autoscaler in our measurements) and more flexible on instance type. For GPU workloads where node-up-time is real money, the speed matters.

Second, Karpenter’s bin-packing is smarter on heterogeneous fleets. If you have a g5.xlarge with 2 GiB of GPU memory left and a pod requesting 2 GiB GPU, Karpenter will place it. cluster-autoscaler is more likely to scale up.

Third, the Karpenter NodePool primitive maps cleanly to the “one node group per inference class” pattern above. cluster-autoscaler’s autoscaling groups make this work but with more YAML.

The argument for cluster-autoscaler is operational maturity. It has been in production at major shops for years. Karpenter is newer; if your team does not have someone willing to track upstream issues, the older tool may be the right call. Pick on team maturity, not raw capability.

GPU scheduling primitives

Three primitives are load-bearing for inference scheduling:

nvidia.com/gpu resource limits

Every GPU pod must declare its GPU requirement explicitly. The Kubernetes scheduler will not split a GPU between pods — even a fractional request is rounded up to one full GPU. This is the difference between “we have GPU capacity” and “we have schedulable GPU capacity” on a busy cluster.

MIG partitions for h100 / a100

On h100 and a100 nodes, MIG (Multi-Instance GPU) lets a single physical GPU present as multiple smaller logical GPUs. This is the only way to bin-pack small inference workloads onto large nodes without leaving 80% of the GPU idle. Configure MIG in your DaemonSet, expose the partitions as nvidia.com/mig-*, and your scheduler can place 7 pods on a single h100 instead of 1.

Time-slicing as a fallback

If MIG is not available, NVIDIA’s time-slicing GPU operator lets multiple pods share a GPU at the time-multiplexed level. This is much worse for inference latency than MIG (pods do contend), but it is sometimes the only option on a g5 node and is much better than leaving the GPU idle.

Scaling triggers

The HPA (Horizontal Pod Autoscaler) default is CPU. CPU is wrong for inference. The metrics that matter:

Queue depth. How many requests are waiting in the inference router’s queue. This is the leading indicator. Scale when queue depth exceeds 2x your healthy steady-state for 30 seconds.
In-flight requests per pod. How many concurrent inference calls a pod is holding. This bounds tail latency. Scale when median per-pod in-flight exceeds 80% of the configured concurrency.
GPU memory pressure. How close pods are to OOM. This is the safety net; if it triggers, you have already lost some traffic.

KEDA is the right tool for queue-depth and in-flight scaling. It plugs into the HPA but lets you scale on arbitrary metrics, including Prometheus queries against your router’s own counters. Karpenter then provisions the nodes the pods land on.

The scale-down problem

Scaling up is easy. Scaling down is where the money is.

GPU nodes are expensive. A g5.12xlarge runs around $5/hour. If you do not scale down aggressively, you pay for capacity you are not using. If you scale down too aggressively, you bounce up and down, and each scale-up costs you 30–60 seconds of latency on the affected workload.

The pattern that works:

Pod-level idle detection. A pod that has not served traffic for 5 minutes is a candidate for eviction. Implement an idle-check sidecar or a custom controller; the default HPA cannot do this.
Node-level consolidation. Karpenter’s disruption config can consolidate underutilized nodes. Configure it with a 10-minute “consolidate after” window and an empty-node threshold of 1 hour. Pod-disruption budgets prevent it from being destructive.
Time-of-day shaping. If your traffic is predictable, run a cron that scales the minimum replica count down at night and up in the morning. This sounds primitive; it is primitive; it works.

We have seen teams cut their inference compute bill by 40% by paying attention to the scale-down side of the loop. The scale-up patterns are well-documented. The scale-down patterns are mostly tribal knowledge.

What we are watching

Three threads worth tracking through the next few quarters:

NVIDIA Dynamic Resource Allocation (DRA) is the standardized successor to the device-plugin model. It is alpha in Kubernetes 1.31. When it lands stable, the MIG and time-slicing story gets a lot cleaner. We will write again when we have run it under load.
kueue for batch GPU work is maturing. It is the right primitive for fine-tuning and overnight eval pipelines. We have started using it for the batch class above.
The autoscaler-aware inference router is the missing piece in most stacks. The router knows queue depth; the autoscaler knows node-scaling-cost. Few projects bridge them well. The first one that does will be a big quality-of-life upgrade.

The setup is more complex than the web-tier autoscaling story. The savings are real and the latency improvements are real. The work is worth doing.

— Reza Mokhtari