29. 06. 2026
APM, Kubernetes
30. 06. 2026
Davide Sbetti
AI, Kubernetes
Load-balancing Requests to LLMs in Kubernetes: A KV-cache Approach with llm-d!
Hi everyone 😃
Today I’d like to walk you through some experiments we ran about load-balancing requests to LLMs in Kubernetes. Let’s dive deep into it!
Why Deploy LLMs in Kubernetes?
Well, Kubernetes has now established itself as a leading technology when it comes to workloads orchestration. And with time the increasing support for accelerators such as GPUs will make it easier to run workloads which can exploit the large parallelism those accelerators provide.
In addition, many projects under development to efficiently serve LLMs are also provided in the form of containers, ensuring an effortless experience when it comes to their deployments. Some examples include, but are not limited to:
- vLLM: One of the most famous inference engine, given its really good performance, which provides an official Helm Chart
- llama.cpp: The famous engine written in C/C++ with support for many accelerator combinations and also plain CPUs, which offers many documented Docker images
- Huggingface: The well-known platform which serves as repositories for models, challenges and recipes, offers many pre-built images for various purposes, such as inference or training
But these are probably not the main reasons to bring Kubernetes into the loop when it comes to LLMs. That’s because one of the main advantages that Kubernetes-like environment brings is the mentality to deploy workloads in a GitOps way, which ensures our application is consistent with its definition, allowing it to also re-deploy on a different machine or in the case where a re-installation is needed.
A Kube-native Approach: Kubeflow
In our case, we decided to try Kubeflow, a project aimed at providing users with separate Kubernetes namespaces where their workloads can run, with support for accelerators and a modular approach that allows the operator to decide which Kubeflow components to deploy, skipping for example some components that are not needed in case you’re not training your own models.
We deployed it using the official manifest but with a little trick, namely updating KServe (the component we can actually recognize as a full-fledged model serving platform) to version 0.17, which was not the default one at the time in Kubeflow.
Why 0.17? It all comes down to… llm-d!
Serving a Model with KServe and llm-d
Okay, when it comes to applications and serving requests efficiently, one of the main concepts adopted regardless of the type of application and architecture is having a cache for everything possible, trying to re-use it and maximize the number of cache hits. It turns out this can also play a role when it comes to LLMs, but to understand how, we need to explore a bit how the decode phase in modern LLMs works!
But First, How Does an LLM Actually Generate Text? A 2-minute Recap!
Generating text in an LLM works roughly like this:
- The input – which could be just a system prompt, or a full conversation – is broken into tokens by a tokenizer.
- Each token is projected, via weights learned during training, into a key and a value vector: The key represents what kind of information the token carries, the value represents the actual content of that information.
- The last token also produces a query vector. This query is compared (via dot product) against every key vector in the sequence, producing a score for each token. These scores are normalized into attention weights, which are used to combine the value vectors into a single output – effectively, “how much should I focus on each previous token right now?”
This is what an attention layer does. LLMs stack many of these layers, each followed by a small feed-forward network, refining the representation step by step. After the final layer, the result is projected onto the vocabulary to get a probability distribution over possible next tokens – one of which is sampled (or picked) as the output.
That new token gets appended to the sequence, and the whole process repeats – one token at a time – until a special end-of-sequence token is produced.
Given this, where could we fit an optimization under the form of a cache? Well, keys and values computed for the sequence until a certain step can also be re-used in the computation of the next step, without the need to recompute them!
This is exactly the concept behind the KV-Cache, which aims at reusing those already-computed values.
Where llm-d Comes into Play
Let’s imagine we’d like to deploy two instances of a model running on two different GPUs, how can we do this? Up until the latest version, KServe offered the InferenceService, which allowed us to easily serve an ML model, and not just limited to an LLM.
For example, to serve Qwen 2.5 on our two GPUs, we might do something like this:
Given that we have two instances, we could load balance users’ requests among the two instances using some standard techniques such as round robin, or using metrics such as the load. However, this wouldn’t allow us to really exploit, at this stage, the concept of the KV cache we just saw. That’s why in recent versions KServe also started offering a more specialized LLMInferenceService that among the other features, integrates llm-d, which also provides a KV Cache-aware router (both approximated or precise, exploiting for example vLLM metrics pushed through ZeroMQ).
We could for example design the following configuration for the EndpointPicker, namely the llm-d component designed to choose among the available instances (endpoints), in order to use different metrics with different weights, including the precise prefix cache scorer, to maximize the hits, also in this case in an extreme way 😃
Given that we’re using the precise prefix cache we need then to configure our inference runtime, such as vllm, to pass the information of the cache content back to the EndpointPicker, to make it able to take great routing decisions!
And in this way we’re able to serve two instances of a model and have a service able to route requests also based on the content of the KV Cache, to maximize performances and reduce the time to the first token!
Conclusions
Today we saw some of the benefits of using Kubernetes for our ML workflows, and some interesting tools to be able to efficiently and easily serve LLMs, with an eye on KV Cache-aware routing!
See you soon 😉
These Solutions are Engineered by Humans
Did you find this article interesting? Are you an “under the hood” kind of person? We’re really big on automation and we’re always looking for people in a similar vein to fill roles like this one as well as other roles here at Würth IT Italy.
Davide Sbetti
Hi! I'm Davide and I'm a Software Developer with the R&D Team in the "IT System & Service Management Solutions" group here at Würth IT Italy. IT has been a passion for me ever since I was a child, and so the direction of my studies was...never in any doubt! Lately, my interests have focused in particular on data science techniques and the training of machine learning models.
Author
Latest posts by Davide Sbetti
30. 03. 2026
APM, Log Management, Log-SIEM, NetEye
Sending OTel Data to Elasticsearch: Tenant Segregation through OAuth