The landscape of large language model (LLM) serving is evolving rapidly, shifting from just generating text to managing entire systems. Key factors like the choice of inference stack now play a crucial role in determining performance metrics such as tokens processed per second, latency, and overall costs associated with GPU fleets.
A recent comparison highlights four prominent inference stacks that are currently leading the field: vLLM, NVIDIA TensorRT-LLM, Hugging Face Text Generation Inference (TGI v3), and LMDeploy. Each of these solutions brings unique strengths to the table, catering to different needs in the market.
vLLM stands out for its innovative use of PagedAttention, which treats the key-value (KV) cache like a paged memory system. Instead of using one large area for each request, it divides the cache into smaller blocks. This method reduces memory waste and allows for more efficient use of GPU resources. vLLM claims to boost throughput by 2 to 4 times compared to older systems while keeping latency low, especially for longer sequences. It also integrates seamlessly with orchestration tools like Ray Serve.
On the other hand, NVIDIA’s TensorRT-LLM is optimized specifically for its own GPUs. This library leverages custom kernels and advanced techniques like quantization to maximize performance. In tests, it has demonstrated the ability to process over 10,000 tokens per second with low latency, outperforming previous models significantly. This stack is particularly effective for scenarios requiring high throughput and low time to first token.
Hugging Face’s TGI v3 focuses on handling long prompts efficiently. It employs techniques like chunking and prefix caching to enhance performance. In benchmarks, TGI v3 has shown remarkable speed improvements, processing long prompts in a fraction of the time taken by vLLM. This makes it an attractive option for applications that need to manage extensive contexts, such as retrieval-augmented generation (RAG) tasks.
Lastly, LMDeploy, part of the InternLM ecosystem, aims to compress and serve LLMs effectively. It emphasizes high throughput and features like blocked KV caching and aggressive quantization, which can significantly reduce memory usage. LMDeploy claims to achieve up to 1.8 times the request throughput of vLLM, making it suitable for larger models on mid-range GPUs.
As organizations weigh their options, the choice of inference stack will depend on specific needs. For maximum throughput on NVIDIA hardware, TensorRT-LLM is the go-to option. For applications centered around lengthy prompts, TGI v3 is a strong candidate. vLLM serves as a reliable standard for general use, while LMDeploy is ideal for those working with open models and needing efficient multi-model serving.
In practice, many development teams are likely to mix these systems to optimize performance across various workloads. The key takeaway is that understanding the strengths and limitations of each stack is essential for aligning them with specific operational needs and cost considerations.