⬢github Python · 95.1K ★ +880 since we first saw it · pushed 23 min ago · Apache-2.0
vllm-project/vllm
A high-throughput and memory-efficient inference and serving engine for LLMs
vLLM is an open-source Python library for fast, memory-efficient inference and serving of large language models. It uses PagedAttention, continuous batching, prefix caching, quantization, speculative decoding, and distributed parallelism to serve 200+ Hugging Face model architectures across NVIDIA, AMD, Intel GPUs, TPUs, CPUs, and other accelerators, with an OpenAI-compatible API server.
Why now: not clear from the sources
Who it is for: Developers and teams who want to self-host or serve LLMs at high throughput in production.
amdblackwellcudadeepseekdeepseek-v3gptgpt-ossinferencekimillama
Stars over our 184 snapshots: 94.2K to 95.1K, since 1 d ago.
Where people talked about it
- ⬢github vllm-project/vllm 23 min ago
API: https://socialmediatrends-api.osmike.com/v1/repos/vllm-project/vllm