Why Kubernetes Is Bad at Serving LLMs (and Why That's Not Kubernetes' Fault)

Part 1 of my llm-d series. Writing this down as I learn it, so both of us understand it properly.

You have seen it. You ask Claude or ChatGPT something, the answer starts streaming nicely, word by word… and then it freezes. Mid-sentence. Two seconds of nothing, then it continues like nothing happened.

Ever wondered why that happens?

By the end of this post, you will know exactly why. And you will also know why Kubernetes, the way we normally use it, cannot fix it.

The setup we would all build

You know how to serve an application on Kubernetes. All of us do. A Deployment running your app, a Service in front, round-robin load balancing, HPA scaling on CPU. Done, right?

So let’s do the same with an LLM. Put vLLM in a Deployment, Service in front, ship it.

Here’s the thing: it works. Pods come up, requests get answers, everything looks green. It’s just quietly terrible. To understand why, we need to understand what an LLM request actually is, because it behaves nothing like the HTTP requests we’ve been serving all these years.

A normal request vs an LLM request

Think about a normal HTTP application. Its behaviour is predictable. You know the rough round-trip time, you can see requests per second, and from CPU and memory telemetry you can predict the resources you need. Every request costs roughly the same as the next one. That assumption is baked into everything: round-robin load balancing, CPU-based autoscaling, all of it.

Now look at an LLM request. The input is not equal to the output. You might ask a question in 1,000 words and get back 5,000. Someone sends a 20-token prompt, the next person sends a 30,000-token document. And here is the worst part for anyone doing capacity planning: you don’t know the output length until the model finishes writing it. One request can genuinely be 1000x heavier than the next.

Round-robin assumes requests are interchangeable. Here, they are anything but. “Fair” distribution creates unfair hotspots, with one pod drowning in a giant request while its neighbours sit idle.

Before we go behind the scenes, two quick fundamentals: what an LLM actually does, and what a token is. Two minutes, then straight back to Kubernetes.

Okay, but what is an LLM actually doing?

Strip away the hype, and an LLM is a prediction machine. It predicts the next word based on everything that came before it. That’s it. That’s the whole trick.

You already use a tiny version of this every day: your phone keyboard suggesting the next word. An LLM (Large Language Model, as the name says) is that same idea scaled up billions of times, trained on so much text that its “next word” suggestions chain into entire essays.

And how did it learn to predict? Training. We feed the model enormous amounts of text: books, websites, forums, everything you can name. But here is the important part: it is not memorising this data. It is learning the patterns in it. Think of a student who has read the entire library. He hasn’t kept every book word-by-word in his head. What he has absorbed is how books are written: how a recipe flows, how a legal document argues, how a science answer is structured. That is why “you are a chef, give me a recipe” works. The model has read thousands of cookbooks, and your instruction steers its predictions toward the pattern of chef-writing.

(One technical honesty note: the model doesn’t predict the next “word”. It predicts the next token. So let’s clear that up.)

What exactly is a token?

A token is a chunk of text. Sometimes a full word, sometimes a piece of one. Every model has a fixed vocabulary, a list of roughly 50,000 to 100,000 chunks, built by scanning the training data and keeping the most frequent sequences. Common words earn their own token. Rare words get assembled from pieces.

Some examples:

• “How are you” is 3 tokens: How + are + you. Notice the space travels with the following word; are with its leading space is one token.
• A rough rule of thumb: 1 token is about three-fourths of an English word. So 1,000 words is around 1,300 tokens.
• A long word like “Kubernetes” might split into Kub + ernetes.

Now, something close to home. Tokenizers are trained mostly on English text, so Malayalam gets punished. “How are you” is 3 tokens, but “സുഖമാണോ”, one single word carrying the same meaning, can explode into 5 to 8 tokens. Same question, triple the cost. Every context limit, every API bill, every throughput number you will ever see is measured in tokens, so this matters more than it looks.

(Try it yourself: paste some text into an online tokenizer playground and watch how it splits.)

Fundamentals done. Back to serving.

Every LLM request has two phases

Every single request an LLM serves goes through two very different phases: prefill and decode.

Let me put it in exam terms. You are sitting for an exam. Malayalam, science, whichever subject you like.

Prefill is reading the question paper. The moment you get it, you skim through the whole thing in one go. All the questions are already known, nothing depends on anything, so you take it all in as a single burst. The GPU does exactly this with your prompt: all 5,000 tokens processed in parallel, one massive burst of computation. This is compute-heavy work. Muscle work. The GPU’s math units run at full blast.

Decode is writing the answers. And this can only happen one word at a time. You cannot write word 52 without having written words 1 to 51, because each word depends on everything before it. No shortcut, no parallelism. And here is the surprising part: to produce even one token, the GPU must re-read the entire model weights plus its memory of the conversation so far. The math per token is tiny; the reading is enormous, and it repeats for every single token. This is memory-bandwidth-heavy work. Reading work. The math units mostly sit idle, waiting for data to arrive.

Two phases, opposite appetites. One is limited by muscle, the other by reading speed.

These two phases also give us the two metrics that define LLM serving quality:

• TTFT (Time To First Token): how long until the first word appears. This is prefill’s report card. Big prompt means a long stare at a blank screen.
• TPOT (Time Per Output Token): once streaming starts, how smoothly the words keep coming. This is decode’s report card.

Why can’t we just mix them on the same GPU?

Remember the freeze from the beginning of this post? Here is where the mystery gets solved.

Think about a dosa stall. Five customers have ordered. Their dosas are on the pan, and the chef is in rhythm: flip, flip, flip. That is decode, everyone’s answer streaming out token by token, steady and smooth.

Suddenly a wedding party walks in with a bulk order. The chef has to stop flipping and go run the big grinder for a fresh batch of batter. That is a giant prefill arriving. And what happens to the five dosas already on the pan? They sit there. Everyone’s order is delayed.

That is your freeze. Your answer was streaming (decode, flip flip flip), and someone else’s massive prompt landed on the same GPU (a giant prefill). The chef walked away from the pan. Their muscle work paused your reading work. Two seconds later, he is back, and your stream continues like nothing happened.

The KV cache: the state nobody talks about

Back to our exam student one more time. While writing an answer, what does he need to hold in his mind? The question he read. Plus every sentence of his own answer written so far, because sentence 5 must follow from sentences 1 to 4.

The model needs exactly the same thing, and it is called the KV cache (key-value cache). During prefill, the GPU computes attention state for every prompt token and keeps it in GPU memory. During decode, every generated token gets appended to that cache. So the cache is not just your prompt; it grows as the answer grows. A long conversation means a big cache sitting in expensive GPU memory.

So GPU memory is holding two things: the model weights (fixed) and the KV cache (growing with every request and every token).

And here is the nuance that changes everything: the KV cache is expensive to build, but cheap to reuse. Requests share prefixes all the time: the same system prompt, the same chat history resent with every message. A pod that still holds the cache for that shared prefix can skip most of the prefill entirely and start writing almost immediately.

It is like being a regular at your local tea shop. You walk in, and the chettan at the counter already knows your order; he is moving before you even sit down. That is a cache hit. Now walk into a brand-new shop: nobody knows you, everything starts from zero. That is what round-robin does to every single request. It sends you to a random shop every time, even when your regular shop is right there with your chai already half-made.

Not all workloads are shaped the same

Let’s apply all this to two real systems.

A RAG system. You feed it huge documents (release notes, org wikis, entire manuals) and ask a question. It answers in three lines. Huge in, tiny out. Which phase dominates? You already know: prefill-heavy. Massive muscle work, barely any reading work.

A chatbot. “Write me an essay on Kerala’s monsoon.” Tiny question, enormous answer. Tiny in, huge out: decode-heavy. Barely any muscle work, endless reading work.

Same model. Same GPUs on paper. Completely opposite capacity needs: one wants raw compute for prefill, the other wants memory bandwidth for decode. And if both phases live inside one Deployment, you cannot scale one without dragging the other along.

The point of all this

So let’s land the plane. Kubernetes is not broken. It is balancing and scaling on the signals it was built for: CPU, memory, connection counts. But for LLM serving, those are the wrong signals. The real signals are KV cache state, queue depth, and the shape of the prompt. Round-robin throws away cache hits. CPU-based HPA is blind to what is actually saturating the GPU. One Deployment cannot separate muscle work from reading work.

We need something that routes and scales based on what LLM inference actually cares about.

That something is llm-d, a CNCF project built exactly for this. In the next post, we will open it up and follow a single request through its architecture.

If any part of this didn’t click, tell me in the comments. This series exists because I am learning it too.