Serving models at scale with vLLM

LLMs might be fantastic at prediction, but they weren’t designed to handle bajillions of requests efficiently. After all, they’re models, not servers. On top of that, a model running one request at a time uses shockingly little of the GPU’s compute, leaving tons of processing power sitting idle. Given how expensive GPUs are, that’s quite a waste!

Running inference at scale is a problem folks have been chipping away at for years, and one project in particular has become the go-to answer for open-weight models: vLLM.

So: let’s take a look at why high-throughput inference is so hard and why vLLM kicks ass at it.

Inference 101

Why exactly are models so bad at utilizing GPU compute? To understand this, we need to break down how inference works.

First, your prompt gets broken up into tokens, which you can just think of as numbers that represent words. Is this an oversimplification? Absolutely. Do we care? Not today.

These tokens are then funneled through a series of layers, where each one gets multiplied, added, and generally mathed against the model’s weights. The output of one layer becomes the input to the next, and the final layer’s output is used to generate the next token. Each sweep through the layers is called a forward pass. Reading your prompt takes one. Generating the response takes one for each new token. Those two phases have names:

1. Prefill: the first pass over your prompt, which produces the first token of the response
2. Decode: every pass after that, one per new token

Click through the demo to see inference in action:

So: prefill is when the LLM tries to understand your prompt, and decoding is when it generates the response. And when it comes to serving models at scale, the biggest challenge comes from decoding. Why is that?

Decoding wastes compute

When a model generates text, it does so one token at a time, and each token requires the GPU to haul the model’s entire weights out of memory just to do a little bit of math. Okay, maybe more than a little, but nowhere near as much as it could do. After all, these are GPUs we’re talking about. They could bench press a trillion operations, and we’re asking them to open a pickle jar.

To be clear, this isn’t a flaw in how LLMs are designed. It’s just how they work. The job of running large-scale inference servers is to find workarounds to these limitations.

Worse, while the GPU is barely breaking a sweat, every other request has to wait in line. I don’t know about you, but I don’t want Claude telling me, “Please hold. Your call is important to us.”

We’d get a lot more bang for our buck if we batched requests so one read of the model’s weights allowed us to generate tokens for multiple requests.

Static batching

The simplest way to batch requests is through a method called static batching, in which a scheduler triggers a forward pass containing tokens from multiple requests.

The columns in the diagram below show how tokens across requests can be processed in a single forward pass. This is much more efficient than running one request at a time because we generate tokens for multiple requests at once, even during decoding.

However, the length of any given LLM response can vary wildly, and with static batching the next round of requests can’t start until the last request from the first batch wraps up. Each finished request leaves an empty seat, so every forward pass does less useful work, while new requests wait in line.

Well, instead of batching requests, what if we batched tokens?

Continuous batching

The method most production inference servers use today is called continuous batching. It fills those seats right away: the moment a request finishes, a waiting request’s tokens join the very next forward pass, keeping the GPU as busy as possible.

Hooray! We’re now squeezing a lot more juice out of our GPUs, which means our AI service is ready for production, right?

Not quite.

KV cache

Every new token needs to understand every token before it: your prompt + everything generated thus far. Why? Because context impacts the meaning of any given word. For example, by looking at the previous tokens, the model can deduce that “dog” means something different in “guard dog” than in “hot dog”. This context also influences the next predicted token:

In practical terms, we need the calculations from all previous tokens to generate the next token. Now, we could just rerun all that math, but no one does this in practice because that would mean redoing the math for your entire conversation for every new token. Instead, we use something called a KV cache. KV stands for key and value, two pieces of information calculated for each token.

This cache grows with every token, in every layer, for every request, and all of it lives in GPU memory. And this is where we run into our next big problem.

KV caches are over-allocated

It’s impossible to know how long an LLM’s generated response will be, and as such, it’s impossible to know how big each request’s KV cache will get. To play it safe, some inference frameworks reserve enough memory for every request’s longest possible response. Most of that memory sits empty. In the example below, there’s enough unused space to fit Request 4 almost twice, but most of it’s promised to the other requests.

That’s bad news for continuous batching. A waiting request can only take an open seat if there’s room in memory for its KV cache, and when every request hogs a worst-case chunk of memory, space runs out before the batch is big enough to keep the GPU busy.

Thankfully, there’s a better way, and now we meet our protagonist: vLLM.

vLLM

vLLM is an inference server and framework built specifically for high-throughput environments. It started in UC Berkeley’s Sky Computing Lab, where the team introduced PagedAttention (more on that later) in a 2023 paper, and it’s now maintained by a large open-source community. Here’s the structure:

1. API server: a user’s prompt hits an OpenAI-compatible endpoint and gets tokenized. On the way out, the model’s new tokens get detokenized back into text.
2. Scheduler: determines which requests’ tokens are added to the next forward pass.
3. KV cache manager: allocates blocks of memory where each request’s KV cache is stored.
4. Workers on the GPU: run the actual forward passes.

Walk through the demo to see the lifecycle of a request on vLLM:

To understand the genius of vLLM, we’re going to look a little closer at three of these parts: the KV cache manager, the scheduler, and the workers.

KV cache manager

When vLLM starts up, it loads the model’s weights and then claims most of the remaining GPU memory as one big pool for KV caches. The KV cache manager decides where each request’s KV data goes.

Now, as we saw earlier, we can’t know how much room will be needed for each request’s KV cache, and since continuous batching tries to fit as many requests as possible into each forward pass, every wasted byte means fewer requests per batch. So, what can be done?

“Hold my beer,” says vLLM.

PagedAttention

vLLM gets a lot of stuff right when it comes to serving LLMs at scale, but its crown jewel is a technique called PagedAttention. Instead of reserving a giant hunk o’ memory per request, vLLM divvies up the memory into small blocks, each with room for the KV data of a fixed number of tokens (typically 16).

When a new request comes in, its KV cache can go in any free blocks, wherever they are. And instead of claiming everything up front, the request grabs another block only when it fills up its last one:

Each request uses only as much memory as it needs right now, so there’s no need to over-allocate. The KV cache manager then keeps tabs on which blocks belong to which request through block tables. Each request’s block table is a list of which blocks hold its tokens, in order. In the image above, Req 2’s block table points to blocks 2, 5, and 8.

Less wasted memory means more open seats for continuous batching to fill. Heck yeah!

But the memory optimizations don’t stop there.

Prefix caching

Let’s say all of our requests start with a shared system prompt:

You are a helpful customer service agent.

Any prompt that begins with the same tokens will use the same initial calculations. Rather than recomputing them for each request and storing them in separate blocks, wouldn’t it be nice if they could be shared? This is called prefix caching, which saves on both compute and memory.

The word “prefix” is key here. Matching words aren’t enough: they have to be the same tokens in the same position. Take these two system prompts:

You are a helpful customer service agent.

You are a rude customer service agent.

They both contain “customer service agent”, but the only shared prefix is “You are a”. A token’s KV data depends on every token before it, so the word “agent” after “rude” has different KV data than “agent” after “helpful.” Once two prompts differ, nothing after that point can be shared, even if the words match again.

Since a block table can only point to whole blocks, vLLM matches prefixes one block at a time. To do that, every time a block fills up, vLLM gives it a hash (a fingerprint) based on the tokens inside it plus the hash of the block before. So block 3’s hash depends on its tokens and block 2’s hash, block 2’s hash depends on its tokens and block 1’s hash, and so on.

Click through the demo to see it in action:

Just as a token’s KV data depends on every token before it, a block’s hash depends on every block before it.

When a new request comes in, vLLM splits the prompt’s tokens into block-sized chunks (but no KV blocks are reserved yet). It then hashes those chunks of tokens and looks for identical hashes of existing KV blocks. The new request’s block table simply points to the existing block. No new memory, no recomputing. For simplicity’s sake, the blocks in the demo only hold four tokens, but 16 is more typical.

Give it a whirl!

There’s one more clever bit. When a request finishes, its blocks get freed, but vLLM doesn’t erase them. They go back into the pool (the overall KV cache memory) with their KV data intact. The KV cache manager stops associating those blocks with a specific request but still remembers their fingerprints. If a new request comes along with a matching prefix, it can claim those blocks as-is, no recomputing needed. If not, they’ll eventually get recycled for new data, least recently used first.

This is a huge deal for chatbots. Every time you send a new message, the entire conversation gets sent back to the model. With prefix caching, most of the conversation can already be sitting in memory.

But before any token starts making its way downtown through the layers, something has to decide which requests get a seat in the next forward pass. This is handled by the…

Scheduler

Before every forward pass, something has to decide which request’s tokens are included in the batch. That’s the scheduler’s job, and it makes that decision fresh every single step.

Each batch has a limit to how many tokens it can use in the forward pass, so the scheduler needs to decide how to divvy up the requests. Some of the requests are in the middle of running (decoding), whereas others haven’t started. Here you can see requests A, B, and C are in progress, whereas requests D and E are not.

The scheduler prioritizes requests that are already running, and if there’s space left over in the token budget, it fills those seats with any prefill requests in line.

Walk through the demo to see it in action:

Here you can see the three running requests are each processing one token right now (because they’re decoding). Since there’s still space left over, request D’s prefill tokens are also added. That leaves seven tokens, and request E’s prompt needs 10, so E has to wait in the queue. Note: this is without chunked prefill, which we’ll get to in a moment.

The token budget isn’t the only thing that limits a forward pass. The scheduler can also run out of space for KV data. Let’s say it’s building up a new batch and reaches a running request that needs a new KV block, but there are none left.

In this case, it preempts the most recently admitted running request, which usually isn’t the one that came up short. It frees up all of that request’s KV blocks and sends it to the front of the waiting line, and the request that needed a block gets one. The preempted request gets recomputed from scratch, prompt and everything it had generated so far, once enough KV cache memory is available again.

Now, what if a new request comes in with a prompt of 50,000 tokens? Our token budget is only 16. Ruh-roh. What can we do?

Chunked prefill

Well, we could increase our token budget to 50,000, but every other request in the batch would still have to wait for it to finish before getting its next token. If you were halfway through a response, you’d watch it freeze mid-sentence while someone else’s novel gets processed. How rude.

The fix is chunked prefill. Instead of processing a giant prompt all at once, the scheduler breaks it into chunks that fit in whatever budget is left, and spreads them across several forward passes.

As you walk through the demo, you’ll see that the decoding tokens get added first, followed by each prefill chunk for Request F:

Now, if you recall, each token’s KV data is impacted by the tokens that came before (e.g., “hot dog” vs “guard dog”). If we split the prompt up, how do the tokens in the second chunk account for the ones in the first?

The KV cache! Step through the demo and see: for a given request, the first chunk of tokens gets processed, and its KV data is stored. When the second chunk comes through, it reads chunk 1’s KV data straight from the cache. And since a token’s KV data only depends on the tokens before it, chunk 1’s data is already final by the time chunk 2 needs it. This continues until the prompt is fully processed and the model generates the first response token.

Meanwhile, since running requests still get their seats first, everyone else keeps getting their next token every single pass. No more freezing mid-sentence. The giant prompt just takes a few passes to get through.

There’s a bonus, too. Prefill is heavy on math, while decoding is heavy on memory reads, so mixing them in the same pass keeps more of the GPU busy. And chunked prefill is on by default in current versions of vLLM, so you get all of this for free. Rad.

Now it’s worth noting that there is a trade-off for chunked prefill: spreading a prompt across several passes means that the request waits a bit longer for its first token. A smaller token budget keeps everyone else’s responses smooth, while a bigger one gets long prompts through faster. Operators tune the budget to strike the balance that fits their traffic.

So: we’ve talked about what manages the KV cache and how requests get scheduled and optimized. Now let’s talk about the workhorses of our inference framework: the GPU workers.

Workers

vLLM’s workers each drive a GPU and do the sweaty labor of mathing over every token. They’re responsible for loading the model weights and running the forward pass the scheduler hands them.

Now, remember: our model is made up of layers and layers of mathematical operations. Every function that runs one of these operations is called a kernel, doing stuff like “multiply these two matrices” or “compute attention for this layer.”

The worker, running on the CPU, tells the GPU which kernel to run next (called launching it). But there’s a problem: launching each kernel takes a few microseconds. If the GPU finishes before the CPU can launch the next kernel, the GPU sits idle, waiting on the CPU. That’s no bueno.

Luckily, there’s a fix: CUDA graphs.

CUDA graphs

The first time you make a new recipe, you check the cookbook after every step. That slows you down, but hey, you’ve never made it before! After a few rounds, though, you know the steps by heart and can move from one to the next without stopping. That’s the idea behind CUDA graphs, which are “memorized” sequences of kernels that the CPU can kick off with a single launch.

At startup, vLLM runs the forward pass once for each supported batch size and records its kernels into a CUDA graph. From then on, each forward pass can then replay the matching graph, and the GPU runs its kernels back to back without waiting on the CPU in between.

This pays off most in decode passes. Their kernels are tiny, so launch gaps eat up a big share of the time. Prefill kernels are big enough to keep the GPU busy while the CPU launches the next one, so they gain less.

With CUDA graphs, the GPU is able to do significantly more work from just a single launch.

CUDA graphs get rid of the gaps within a forward pass, but they don’t change what each pass produces: one token per request. If we want to speed things up further, we need to generate more than one token at a time.

Wait, can models even do that??

Yes. Sort of.

Speculative decoding

Speculative decoding is a pattern of using something cheap, often a smaller “draft” model, to take guesses at the next several tokens. Then a larger model checks all of those guesses in a single forward pass, like so:

Running two models on the same tokens might seem like extra work for no reason, but in practice it can make a big difference. The small draft model can decode really quickly, and the big model checks all of its guesses in a single forward pass, almost like a mini prefill. Remember, a decode pass is bottlenecked by hauling the weights out of memory, not by the math, so running four tokens through the big model costs about the same as running one.

The big model keeps every guess up until the first one it disagrees with, then swaps in its own token there. If the draft model guessed four tokens and got three right, we just got four tokens out of one expensive forward pass instead of one. And if all four guesses are right, the big model tacks on its own next token too, for five.

The best part: the output is just as good as the big model’s on its own. The draft model never impacts the quality of the answer, just how fast you get it.

There’s one catch: those extra tokens are only “free” when the GPU has spare compute. Under heavy load, big batches already keep the GPU busy, so speculative decoding gives a smaller boost and can even slow things down. Like a lot of things in serving, it’s a dial to tune, not a switch to flip on.

We’ve covered tons of ways to squeeze as much as we can out of each forward pass, but at a certain point, you can’t squeeze any more: you just gotta add more GPUs.

Scaling GPUs

As your project grows, you may run into one of two issues:

1. Your model doesn’t fit on one GPU (or just barely). A 70B-parameter model needs around 140 GB for its weights alone at standard precision, which is more than most GPUs have (an H100 has 80 GB). Even on a GPU big enough to hold that, there’s hardly any room left for the KV cache. And the biggest open-weight models today have trillions of parameters, which won’t fit on any single GPU.
2. One copy of your model can’t handle the traffic. Smaller models can run fine on a single GPU, but even with all the fancy features that vLLM provides, at a certain point you’re going to need to run more copies and put a load balancer in front.

So, what can you do? Let’s start with the first problem.

Your model is too big

The simplest solution is to make your model smaller using our good ol’ friend: quantization. Quantization is a technique for using weights and KV data at lower precision (like 8-bit instead of 16-bit). Less precision = fewer bits required = smaller model. vLLM ships with specialized GPU code for most formats, which makes it great for quantized models.

Another approach is to split a model’s weights across multiple GPUs rather than loading the entire model onto one GPU. This is called tensor parallelism.

Here you can see how each GPU is responsible for a set of weights (tensors). The results across GPUs for layer 1 are then combined before moving on to layer 2, etc.

The alternative is to let each GPU handle specific layers. This is called pipeline parallelism:

For really big models, you’d use both tensor and pipeline parallelism. In these scenarios, you’ll have multiple machines with multiple GPUs. Each machine will be responsible for certain layers, and each GPU in that machine will process their chunk of tensors (weights) for said layer.

There’s one more layer of optimization that can come from splitting up the model weights across multiple GPUs, and that’s when using mixture-of-experts or MoE.

Earlier we learned that each layer of the model is just a big math equation. Well, it’s more like two math equations. The first is attention, where each token calculates its KV data and then combines it with the KV data of every token before it. The next is the feed-forward block. Unlike with attention, the feed-forward step operates independent of other tokens. Here’s the basic structure of the layer:

Extremely large models produce excellent results, but folks have discovered that you can reduce the amount of work per token by only using certain weights for certain tokens––that’s MoE. Let’s pretend the token in question is a medical term. Using MoE, that token can be processed by the weights that “know” about said medical term.

In reality, these “experts” don’t actually map to specific subjects. They’re simply groups of weights that the model has discovered, through training, are useful in predicting the next token.

Similar to tensor parallelism, vLLM offers a way to have certain GPUs only house the weights for select experts. When a token hits a GPU, it first goes through the full attention step, and then gets routed to the GPU containing the experts needed for that token. This feature is called expert parallelism.

As you step through the demo, you’ll see how the token for “actually” is processed first with the attention weights, then routed to the appropriate experts on different GPUs. The results are then blended and passed to the next layer.

Your model can’t handle the traffic

Data parallelism replicates the model across multiple GPUs, each processing different batches of requests. With vLLM’s built-in data parallelism, there’s still a single entry point: one API server hands requests to GPUs on the same machine or on other machines. You could even have fully separate deployments, as in, distinct vLLM servers each with its own endpoint, with an external router in front of them all.

There’s a catch, though: each copy has its own KV cache. Remember how prefix caching keeps your chat history in memory? That only helps if your next message lands on the same copy. That’s why many deployments use a cache-aware router, which sends each request to the copy most likely to already have its prefix cached.

One wrinkle with data parallelism: large MoE models might not fit on a single GPU, as we saw earlier. That’s where expert parallelism comes in. Each GPU still handles its own requests and keeps a full copy of the attention weights, but holds only a slice of the experts. When a token needs an expert that lives on another GPU, it takes a quick round trip there and back:

But wait, there’s more

We’ve covered the big ideas today, but vLLM has a lot more tricks up its sleeve. A few worth knowing about include multi-LoRA, attention kernels, and a wide range of hardware support.

Additionally, despite this article raving about vLLM, plenty of these features aren’t vLLM-specific. Another big player in the inference server world is SGLang, which has a lot of overlap. While vLLM is more common, SGLang excels at workloads where requests share long prefixes, like agent loops and multi-turn chat. vLLM, on the other hand, supports a wider range of models, runs on a wider range of hardware, and has a larger community.

Waste nothing

Let’s step back. Our goal is to serve as many requests as possible, as fast as possible, without breaking the bank. Not enough GPUs is an obvious problem, but the problems that inference servers handle best are:

1. Wasted compute
2. Wasted memory

vLLM has solutions to both.

When it comes to compute, continuous batching and the scheduler keep every seat in the forward pass filled, chunked prefill keeps a giant prompt from stalling everyone else, and CUDA graphs cut out the idle gaps between kernels. Speculative decoding squeezes out even more by turning spare compute into extra tokens. Ka-ching.

When it comes to memory, PagedAttention gives each request’s KV cache one block at a time instead of reserving the worst case. Prefix caching stores shared prompts once, and quantization shrinks the weights and KV data themselves.

And when you really don’t have enough GPUs, parallelism spreads the model, or the traffic, across more of them.

The best part: most of this is on by default. Continuous batching, PagedAttention, prefix caching, chunked prefill, and CUDA graphs all work the moment you start the server. Speculative decoding, quantization, and parallelism are the settings you choose, based on your model and your hardware.

While there’s no one-size-fits-all solution for scaling open-weight models, understanding the challenges you’ll face along with the tools at your disposal will help you construct a well-oiled server that’s ready for anything that’s thrown at it.