Vllm

vLLM is the inference server many production LLM deployments sit behind, and until recently it had no answer on a Mac beyond a source build of its CPU backend. The vllm-metal plugin changed that: it keeps vLLM’s engine, scheduler and OpenAI-compatible API, and swaps the compute layer for MLX, Apple’s array framework, running on the GPU through Metal. By the end of this article you will have that server running on your Mac, answering on the same OpenAI-compatible endpoint you would deploy on a CUDA box, sized so it leaves the machine usable. You will then measure the number that decides whether vLLM is the right server for your workload at all: how much aggregate throughput continuous batching buys you, and how much per-request latency it costs. Continuous batching is vLLM’s core trick, running many requests through the model together and admitting new ones as others finish. ...