Scaling Vllm Without Oom

Go Big or Go OOM: The Art of Scaling vLLM

Skip to Main Menu

Skip to Main Content

Skip to Footer

Back to Blog

-->

Back to Blog

TL;DR

Sharing LLM-as-a-Judge (JLM) deployments across multiple concurrent training jobs can reduce GPU underutilization, but makes the deployment vulnerable to buckling under load. We mitigated this by applying a two-pronged approach: optimizing single-node performance and scaling multi-node deployment, arriving at a strategy applicable to any high-throughput inference deployment facing variable load.

The challenge: multi-node LLM deployments that don’t buckle under load

During the final training stage of our Jamba model family , we introduced a custom implementation of online-RL (GRPO). Online RL training requires a reward function that steers model behavior in the right direction. For verifiable tasks such as math, the reward can be easily defined with a deterministic function. However, freeform tasks (e.g., question-answering and instruction-following) require more complex, model-based rewards – which we implement using LLMs-as-a-Judge (JLMs).

In a classic online RL setting, JLMs are used only during the reward phase and sit idle for the rest of each GRPO training step, leading to GPU underutilization. One way to mitigate this is a fully async approach, which maximizes GPU utilization per training job but introduces other complexities and constraints. Another solution is to share JLM deployments across multiple training jobs – when one job is busy with a policy update, others can use the otherwise-idle JLMs.

This improved GPU utilization, but also exposed the JLMs to unpredictable bursts of traffic from multiple sources. We needed to make the JLM deployments robust enough to handle multiple concurrent training jobs without buckling under load. In this post, we share the process we went through and the lessons we learned.

While our use case focused on JLM serving for GRPO training, these principles apply to any high-throughput vLLM deployment facing variable load, whether you’re serving chat applications with traffic spikes, running batch inference, or supporting ML training pipelines like ours.

Let’s dive in.

The two-angle approach: the key to optimizing multi-node LLM deployments

Like many classic system architecture challenges, we chose to tackle this problem by introducing optimizations from two angles: the vertical (single-node performance) and the horizontal (multi-node scaling).

The vertical angle: optimizing single-node performance

Since we use vLLM as the inference framework for our JLMs, we started by examining vLLM’s configuration. The vLLM engine exposes numerous arguments controlling its execution, from batching strategy and batch size to memory allocation and utilization. By tuning these parameters for our specific sequence lengths, load patterns, model architecture, and hardware, we could significantly improve performance and GPU utilization.

We used OpenShift’s Auto-Tune vLLM , which leverages GuideLLM for benchmarking and Optuna for multi-objective hyperparameter optimization.

Step 1: Analyzing our workload

Before running Auto-Tune, we needed to make decisions about several key parameters:

Sequence length distribution : Minimum, maximum, and average input/output lengths. These numbers guide how synthetic test data is generated and distributed to match real traffic patterns.

Traffic pattern : Are requests arriving in bursts or as a constant stream? This affects vLLM’s batching configuration – for instance --max-num-seqs . For bursty traffic (like ours), vLLM needs a high value to absorb large request spikes into a single batch. For constant-rate traffic, a lower value can reduce individual request latency.

Optimization target : Are we optimizing for throughput or latency? In other words, do we care more about individual request duration or total batch processing time?

Since our JLM’s clients are our own training runs, we could answer these questions by examining our evaluation patterns:

Step 2: Configuring the parameters for the optimization run

With our workload characterized, we configured the optimization run. We used Auto-Tune vLLM with NSGA-II strategy, a multi-objective genetic algorithm, to explore the parameter space and find Pareto-optimal configurations.

Search space

This is the search space we defined for the algorithm – vLLM engine arguments we wanted to optimize and the range of values to be tested for each:

Benchmark configuration

Each candidate configuration was tested with synthetic traffic matching our workload:

2K input tokens (varying up to 8K)

100–1K output tokens

Rate of 2k concurrent requests

5-minute timeout per trial

Optimization objectives

We defined the optimization metrics according to which each candidate configuration should be evaluated:

Output tokens/second (maximize)

Requests/second (maximize)

Request latency (minimize)

Time-to-first-token (minimize)

Inter-token latency (TPOT) (minimize)

Step 3: Running the trials

The optimization ran 300 trials, of which 125 completed successfully. Some configurations encountered OOM errors or timed out – but this is expected and handled gracefully by the framework.

Step 4: Picking the winning configuration

Since we optimized for five metrics simultaneously, we were left with 24 Pareto-optimal configurations representing different tradeoffs between the list we outlined above. Since our goal was maximizing throughput, because we needed the complete reward computation to finish as quickly as possible before proceeding to the policy update phase, we selected the configuration with the highest token throughput.

This is the resulting vLLM config:

Model: 32B dense attention model Hardware: H100 SXM 80GB

Looking at the results, you might think – of course the tuned config produced better throughput than the old one – it used 4 times more GPU resources!

The answer to that is rooted in the way we evaluate the configurations:

When comparing configurations, we evaluate throughput at the deployment level. Given a fixed GPU budget, how should we distribute them across vLLM processes for best performance?

The equation is:

num_instances = total_gpus / tensor_parallel_size

total_throughput = num_instances × throughput_per_instance

Since we tested tp ∈ {1, 2, 4, 8} , we can normalize to 8 total GPUs for comparison. The fact that tp=4 produced the optimal...