vLLM is the inference server many production LLM deployments sit behind, and until recently it had no answer on a Mac beyond a source build of its CPU backend. The vllm-metal plugin changed that: it keeps vLLM’s engine, scheduler and OpenAI-compatible API, and swaps the compute layer for MLX, Apple’s array framework, running on the GPU through Metal.

By the end of this article you will have that server running on your Mac, answering on the same OpenAI-compatible endpoint you would deploy on a CUDA box, sized so it leaves the machine usable. You will then measure the number that decides whether vLLM is the right server for your workload at all: how much aggregate throughput continuous batching buys you, and how much per-request latency it costs. Continuous batching is vLLM’s core trick, running many requests through the model together and admitting new ones as others finish.

On the machine used here the answer was 3.87 times the output token throughput (tokens being the pieces text is split into before a model sees it, and the unit throughput is counted in throughout) at 32 concurrent requests, paid for with per-token latency 8.3 times worse. Those two numbers, measured on your own hardware and your own model, decide it.

This is not the fastest way to talk to a model on a Mac. For one conversation at a time (single stream, in the benchmark vocabulary this article uses later), llama.cpp is simpler to install and quicker off the mark, and Serve a Local OpenAI-Compatible Endpoint with llama.cpp on macOS covers that path end to end. Docker, who built vllm-metal for the vLLM project before contributing it to them, publish a single-stream benchmark where llama.cpp stays ahead. Run vLLM on a Mac when you want the thing llama.cpp is not: the same server, the same flags and the same request semantics you run in production, on hardware you already own.

Versions used throughout: vllm-metal 0.29.0 (vLLM 0.29.0, MLX 0.32.1) on macOS 26.6.2 (arm64), Apple M5 Max with 128 GB, Homebrew 7.0.3, uv 0.11.26, openai 3.14.1, pytest 9.1.1. Throughput numbers are from this machine and will differ on yours; the commands that produce them will not.

What you end up with

  • vLLM serving an MLX-format model on your Mac’s GPU, on the standard OpenAI-compatible routes.
  • A sizing recipe that fits the server to your Mac’s memory instead of letting it claim nearly all of it.
  • client.py — an OpenAI SDK client whose only Mac-specific parts are two defaults you can override. That is the point.
  • sweep.py — a concurrency sweep that turns vllm bench serve into one readable trade-off table.
  • A pytest suite that runs with no server and no model, and a Makefile whose default target prints help.

Prerequisites

  • Apple Silicon and macOS 15 (Sequoia) or later. The plugin’s Homebrew formula refuses to install otherwise. Validated here on macOS 26.6.2.
  • Homebrew. brew --version; 7.0.3 here.
  • About 4 GB of free disk for the plugin and its private Python environment, plus whatever your model weighs. The model used here is 4-bit and small.
  • uv 0.11.26 or newer — uv --version, or brew install uv. Only Steps 5 onward need it.
  • Enough unified memory for the model you pick. Unified memory is the pool Apple Silicon shares between CPU and GPU, so a model’s weights and its cache come out of the same RAM your editor is using. The model here is small enough that it should fit an 8 GB Mac with the Step 3 flags, though it was validated only on the machine above; Step 3 is how you keep a bigger one from taking the machine down with it.
  • Familiarity with the OpenAI chat completions API is assumed. If you want it explained first, the llama.cpp article linked above does that.

Step 1: Install the plugin

The plugin ships as a Homebrew formula that carries its own Python 3.12, vLLM, MLX and prebuilt Metal kernels. Nothing is compiled on your machine and no virtual environment has to be activated afterwards. A formula is Homebrew’s build recipe for one package, and a tap is a repository of them, which is why installing takes two commands rather than one. Do not reach for pip install vllm-metal: the project’s install docs say in as many words that it is not supported, and the pip install vllm you use on Linux has no Metal path either.

brew tap vllm-project/vllm-metal https://github.com/vllm-project/vllm-metal
brew install vllm-project/vllm-metal/vllm-metal

Homebrew reported 1 minute 8 seconds for the install step itself, with the wheel downloads before it on top of that. The last lines are the ones to read:

/opt/homebrew/Cellar/vllm-metal/0.29.0: 60,144 files, 1.9GB, built in 1 minute 8 seconds
==> Caveats
==> vllm-metal
Start a server with:
  vllm serve <model>

Python and the vLLM Metal plugin are installed in a private environment.
No virtual environment activation is needed.

Confirm what landed:

vllm --version

0.29.0+cpu

That version string is the first thing about this stack that will mislead you. It says cpu, the plugin is not using the CPU for inference, and the reason is packaging: the build is a stock vLLM wheel (a prebuilt Python package) whose platform label upstream has no GPU variant of for macOS. The plugin registers itself separately, which the command’s own log lines show:

INFO Available plugins for group vllm.platform_plugins:
INFO - metal -> vllm_metal:register
INFO Platform plugin metal is activated

Platform plugin metal is activated is the line that matters. Step 2 shows the second place this stack says CPU while running on the GPU.

Step 2: Serve a model and read the startup log

vLLM on Metal serves MLX-format models: weight files converted for MLX and published mostly under the mlx-community organization on Hugging Face. A stock Hugging Face checkpoint (one published upload of a model’s weights) works for some model families too, but starting with an MLX build avoids a conversion step on first load. The model here is Qwen3 at 0.6B parameters and 4-bit quantization, meaning its weights are stored at reduced precision to cut memory and bandwidth. It downloads in seconds and makes the later measurements quick to repeat.

vllm serve mlx-community/Qwen3-0.6B-4bit

First run downloads the weights. Startup ends with:

INFO [launcher.py:70] Route: /v1/chat/completions, Methods: POST
INFO:     Started server process [13721]
INFO:     Waiting for application startup.
INFO:     Application startup complete.

Before sending it anything, read four lines further up the log. They are the difference between assuming this works and knowing it does.

INFO [worker.py:127] MLX device set to: Device(gpu, 0)
INFO [platform.py:739] Metal: chunked prefill enabled (paged attention), max_num_batched_tokens=8192
INFO [platform.py:860] Metal memory: 137.4GB total, 119.6GB available
INFO [kv_cache_utils.py:2032] GPU KV cache size: 915,088 tokens, Maximum concurrency for 40,960 tokens per request: 22.34x

What each line is telling you

  • MLX device set to: Device(gpu, 0) is the proof that inference is on the GPU. Elsewhere in the same log the engine config prints device_config=cpu, which is the same upstream-tag artifact as the version string in Step 1. The MLX device line is the one that reflects where the arithmetic happens.
  • chunked prefill enabled (paged attention) names the two mechanisms the rest of this article depends on. Prefill is the pass that reads your prompt, and decode is the phase after it that generates tokens one at a time; chunking prefill lets a long prompt be processed in pieces of at most max_num_batched_tokens instead of blocking everything else. Paged attention stores each request’s KV cache (the per-token attention state a model keeps for the text so far) in fixed-size blocks rather than one contiguous slab, so admitting and evicting requests mid-flight stays cheap.
  • Metal memory: 137.4GB total, 119.6GB available is the pool the server believes it can draw from. On a 16 GB Mac this line reads very differently, which is Step 3’s subject.
  • GPU KV cache size: 915,088 tokens, Maximum concurrency ... 22.34x is the server telling you how many requests of full context length fit in the cache at once. Left at its defaults on this machine it reserved about 105 GB for that cache, far more than this model needs and more than most Macs have.

Leave the server running in this terminal and open a second one for the rest of the article.

Step 3: Size the server for your Mac

The defaults in Step 2 are written for a machine whose GPU memory is its own. On Apple Silicon the GPU is drawing from the same pool as everything else you have open, so a server that reserves 92% of it for a cache is a server that makes your Mac unpleasant to use. Three flags fix that, and they are the same three flags you would set on a CUDA deployment, the first concrete payoff of running the real thing rather than a lookalike.

Stop the server with Ctrl-C and start it again with limits:

vllm serve mlx-community/Qwen3-0.6B-4bit \
  --max-model-len 4096 \
  --gpu-memory-utilization 0.35 \
  --max-num-seqs 32

The memory arithmetic is now printed in full:

INFO [cache_policy.py:1160] Paged attention memory breakdown: metal_limit=115.45GB, fraction=0.35, usable_metal=40.41GB, model_memory=0.34GB, overhead=0.93GB, kv_budget=39.14GB, per_block_bytes=1835008, num_blocks=21331, max_tokens_cached=341296
INFO [kv_cache_utils.py:2032] GPU KV cache size: 341,296 tokens, Maximum concurrency for 4,096 tokens per request: 83.32x

What the three flags do

  • --max-model-len 4096 caps the context length of a single request. It is the biggest lever on cache size, because the cache has to be able to hold a request’s whole context. Cutting it tenfold from the model’s default 40,960 would on its own push concurrency to roughly 223x.
  • --gpu-memory-utilization 0.35 is the fraction of metal_limit the server may use, where metal_limit is the working set Metal recommends as a maximum, 115.45 GB on this 128 GB Mac. It is a fraction rather than a byte count, so the same number scales across machines. Pulling it from vLLM’s default of 0.92 down to 0.35 is what brings that 223x back to the 83.32x printed above: the two flags multiply. Start low, and watch the fraction= and kv_budget= fields.
  • --max-num-seqs 32 caps how many requests the scheduler runs together. The KV cache said 83 requests would fit; this says batch at most 32 of them. The lower of the two wins, and Step 8 measures why you might not want the higher number even when it fits.

Reading the memory breakdown

Every field in that line is derived, so you can check the server’s arithmetic rather than trusting it. metal_limit is what Metal reports as its recommended maximum working set, which is why it sits below the 119.6GB available from Step 2: they come from different measurements, and this is the one the sizing math uses. usable_metal is metal_limit times fraction (115.45 × 0.35 = 40.41 GB). kv_budget is what remains after the weights and overhead come out (40.41 − 0.34 − 0.93 = 39.14 GB). num_blocks is that budget divided by per_block_bytes, where a block is the fixed-size unit paged attention allocates, 16 tokens here, giving 21,331 blocks and 341,296 cached tokens. Your own division will land a block or two off, because the printed kv_budget is rounded to two decimals. These GB are decimal, not binary, which is also why a 128 GB Mac reports 137.4GB total.

Divide cached tokens by --max-model-len and you have the concurrency figure the next line prints: 341,296 ÷ 4,096 = 83.3. If you want to serve N requests of full length at once, that quotient is the number to keep above N.

On a 16 GB Mac, start at --gpu-memory-utilization 0.3 and --max-model-len 2048 with a model this size, then raise them while watching kv_budget. The failure mode of going too high is not an error message, it is your Mac swapping.

Step 4: Talk to it, and meet the reasoning trap

The server speaks the OpenAI chat completions API, so curl is enough to prove it works. The first chat request below returns something that looks like a bug and is not, the fastest way to learn the one model-side gotcha in this stack.

curl -s http://localhost:8000/v1/models | python3 -m json.tool | head -12

{
    "object": "list",
    "data": [
        {
            "id": "mlx-community/Qwen3-0.6B-4bit",
            "object": "model",
            "created": 1789618234,
            "owned_by": "vllm",
            "root": "mlx-community/Qwen3-0.6B-4bit",
            "parent": null,
            "max_model_len": 4096,
            "permission": [

max_model_len is 4096 rather than the model’s own 40,960, which confirms the flag from Step 3 reached the server that is answering.

Now ask it something with a checkable answer, and cap the reply at 200 tokens:

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"mlx-community/Qwen3-0.6B-4bit",
       "messages":[{"role":"user","content":"What is 17 * 23? Answer with just the number."}],
       "max_tokens":200,"temperature":0}' \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print(repr(d['choices'][0]['message']['content'][:120])); print('finish:', d['choices'][0]['finish_reason'], 'tokens:', d['usage']['completion_tokens'])"

'<think>\nOkay, so I need to figure out what 17 multiplied by 23 is. Let me start by recalling how multiplicat'
finish: length tokens: 200

No answer, 200 tokens spent, and finish_reason: length because the budget ran out mid-thought. Qwen3 is a reasoning model: left alone it opens a <think> block and reasons before answering, and on a small model with a small token budget the answer never arrives. The fix is a per-request flag that vLLM passes through to the model’s chat template, the model-supplied template that turns a list of messages into the single string the model actually sees:

curl -s http://localhost:8000/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"mlx-community/Qwen3-0.6B-4bit",
       "messages":[{"role":"user","content":"What is 17 * 23? Answer with just the number."}],
       "max_tokens":200,"temperature":0,
       "chat_template_kwargs":{"enable_thinking":false}}' \
  | python3 -c "import json,sys; d=json.load(sys.stdin); print(repr(d['choices'][0]['message']['content'])); print('finish:', d['choices'][0]['finish_reason'], 'tokens:', d['usage']['completion_tokens'])"

'17 * 23 = 391.'
finish: stop tokens: 13

Thirteen tokens instead of two hundred, and the arithmetic is right. Two things to take from this. The first is that chat_template_kwargs is a vLLM extension to the OpenAI request body, so a client that validates strictly against the OpenAI schema needs it passed as an extra field, which Step 6 shows. The second is what reasoning costs. Raise the budget to 3,000 tokens and the same question finishes in 994, against 13 with reasoning off: 76 times the tokens for the same answer, and every benchmark number later in this article would move if you left it on.

Step 5: Create the project

Everything so far has been the server. The rest of the article is the client side and the measurements, which need a Python project of their own. The .gitignore comes first, because the moment you tee a server log or pass --save-result to the benchmark you have machine-specific output in the tree, and a project that gets its ignores late commits it.

Create the file

mkdir -p ~/projects/vllm-metal-client
cd ~/projects/vllm-metal-client
touch .gitignore

Add the code: .gitignore

# Python
__pycache__/
*.py[cod]

# uv
.venv/

# pytest
.pytest_cache/

# Benchmark output
results/
*.log

# macOS
.DS_Store

Detailed breakdown

  • results/ and *.log are where benchmark output lands if you add --save-result or redirect a server log. Nothing in this article writes them, and the entries are here so that the first time you do, the result stays out of the repository.
  • .venv/ is created by uv on the first uv run. The vLLM server does not use it: the server lives in Homebrew’s private environment from Step 1, and this project only holds the client.

Now the project file. The client depends on the OpenAI SDK and nothing else.

Create the file

touch pyproject.toml
mkdir -p tests

Add the code: pyproject.toml

[project]
name = "vllm-metal-client"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = [
    "openai>=3.6.0",
]

[dependency-groups]
dev = [
    "pytest>=9.1.1",
]

[tool.pytest.ini_options]
pythonpath = ["."]
testpaths = ["tests"]

Detailed breakdown

  • openai is the only runtime dependency, and it is here to make a point that Step 6 spends a listing on: the client for a Mac-hosted vLLM server is the stock OpenAI client.
  • pytest in the dev group runs the Step 9 suite, which needs neither the server nor a model.
  • pythonpath = ["."] lets the tests import sweep.py from the project root without installing anything.

Install:

uv sync

Step 6: Point a real client at it

The reason to run vLLM rather than something simpler is that your application code barely has to know it is talking to a Mac. This step is the check on that claim: a client written against the OpenAI SDK whose Mac-specific parts are two defaults and one request field, all three overridable.

Create the file

touch client.py

Add the code: client.py

"""Talk to the local vLLM server with the OpenAI SDK.

Two defaults name a Mac-specific model and one request field is a vLLM
extension; everything else is stock OpenAI SDK. Point `--base-url` and
`--model` at a CUDA host running the same `vllm serve` command and the rest of
the file does not notice.
"""

from __future__ import annotations

import argparse
import os

from openai import OpenAI, APIConnectionError

DEFAULT_BASE_URL = os.environ.get("VLLM_BASE_URL", "http://localhost:8000/v1")
DEFAULT_MODEL = os.environ.get("VLLM_MODEL", "mlx-community/Qwen3-0.6B-4bit")


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(description="Send one prompt to a vLLM server.")
    parser.add_argument("prompt", nargs="+", help="The prompt to send.")
    parser.add_argument("--base-url", default=DEFAULT_BASE_URL)
    parser.add_argument("--model", default=DEFAULT_MODEL)
    parser.add_argument("--max-tokens", type=int, default=256)
    parser.add_argument(
        "--think",
        action="store_true",
        help="Leave Qwen3 reasoning on (off by default, see Step 4).",
    )
    return parser


def main(argv: list[str] | None = None) -> int:
    args = build_parser().parse_args(argv)
    # vLLM ignores the key, but the SDK refuses to start without one.
    client = OpenAI(base_url=args.base_url, api_key="not-used")

    extra_body = {} if args.think else {"chat_template_kwargs": {"enable_thinking": False}}
    try:
        completion = client.chat.completions.create(
            model=args.model,
            messages=[{"role": "user", "content": " ".join(args.prompt)}],
            max_tokens=args.max_tokens,
            temperature=0,
            extra_body=extra_body,
        )
    except APIConnectionError:
        print(f"error: nothing answering at {args.base_url}; is `vllm serve` running?")
        return 1

    choice = completion.choices[0]
    usage = completion.usage
    print(choice.message.content.strip())
    print(
        f"\n[{usage.completion_tokens} completion tokens, "
        f"{usage.prompt_tokens} prompt tokens, finish_reason={choice.finish_reason}]"
    )
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Detailed breakdown

  • OpenAI(base_url=..., api_key="not-used") is the whole connection setup. vLLM does not check the key by default, and the SDK refuses to construct a client without one, so it needs a placeholder: an empty string fails the same way None does. A server that checks no key is fine on loopback and is not something to expose beyond it.
  • extra_body is how the SDK passes a field the OpenAI schema does not have. This is where the chat_template_kwargs from Step 4 goes, and the reason reasoning is off unless you ask for it with --think.
  • APIConnectionError is caught because the most common failure here is a server that is not running, and the SDK’s own traceback buries that under a stack of retry frames.
  • VLLM_BASE_URL and VLLM_MODEL mean the same file runs against a CUDA deployment by changing an environment variable. Nothing else in the file knows what hardware is on the other end.

Ask it something:

uv run python client.py "What is 17 * 23? Answer with just the number."

17 * 23 = 391.

[13 completion tokens, 28 prompt tokens, finish_reason=stop]

The same prompt with reasoning left on shows what Step 4 measured, from the client side:

uv run python client.py --think --max-tokens 60 "What is 17 * 23?"

<think>
Okay, so I need to figure out what 17 multiplied by 23 is. Let me start by recalling multiplication. I know that 17 and 23 are both prime numbers, right? Wait, 17 is a prime number, and 23 is

[60 completion tokens, 18 prompt tokens, finish_reason=length]

The <think> line is the tell from Step 4, arriving through the SDK exactly as it did through curl.

Step 7: Measure one request at a time

vLLM ships its own benchmark client, so there is no need to write a load generator. Run it at concurrency 1 to get the single-stream baseline that every “how fast is it?” comparison reports, and that Step 8 will show is the least interesting number available. Run it more than once: the first benchmark against a freshly started server reads high on this machine and settles after a run or two.

vllm bench serve \
  --model mlx-community/Qwen3-0.6B-4bit \
  --dataset-name random \
  --random-input-len 256 \
  --random-output-len 128 \
  --num-prompts 16 \
  --max-concurrency 1

============ Serving Benchmark Result ============
Successful requests:                     16        
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  12.12     
Total input tokens:                      4096      
Total generated tokens:                  2048      
Request throughput (req/s):              1.32      
Output token throughput (tok/s):         168.92    
Peak output token throughput (tok/s):    179.00    
Peak concurrent requests:                3.00      
Total token throughput (tok/s):          506.75    
---------------Time to First Token----------------
Mean TTFT (ms):                          32.18     
Median TTFT (ms):                        20.41     
P99 TTFT (ms):                           185.11    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          5.71      
Median TPOT (ms):                        5.77      
P99 TPOT (ms):                           5.85      
==================================================

--dataset-name random generates prompts of the requested token length instead of downloading a corpus, which keeps the benchmark offline and repeatable, and it forces the output length, so reasoning cannot skew the result. --num-prompts 16 is the total number of requests the run sends, not a rate.

The four numbers to keep

  • Output token throughput is tokens generated per second across the whole run, 169 here. This is the number people quote.
  • Request throughput is completed requests per second, 1.32 here. It is output throughput divided by the 128 output tokens each request was told to produce, and it is the column that matters when a request rather than a token is your unit of work.
  • TTFT, time to first token, is how long a user waits before anything appears. At concurrency 1 it is essentially prefill time.
  • TPOT, time per output token, is the gap between tokens once generation starts, 5.71 ms here, which is about 175 tokens per second for the one request in flight.

The rest of the block is derived from these. The transcript above drops the inter-token latency (ITL) rows, which repeat TPOT from a different angle. One oddity to expect rather than chase: Peak concurrent requests reads 3.00 on a run capped at 1. That counter comes from the benchmark client’s own accounting, not from the server’s scheduler, and Maximum request concurrency: 1 near the top of the same block is the value that was actually enforced.

Run it twice before you believe it

The first benchmark after vllm serve starts is not representative. Three runs back to back against a server that had just started, changing nothing between them:

run 1: Output token throughput (tok/s): 189.66   Mean TPOT (ms): 4.99
run 2: Output token throughput (tok/s): 170.72   Mean TPOT (ms): 5.67
run 3: Output token throughput (tok/s): 167.16   Mean TPOT (ms): 5.80

That is a 13% drop from the first measurement to the third, and five further runs stayed between 162 and 168. A cool GPU boosts higher than it sustains, so the first number off an idle machine flatters itself. Take the settled value, and give every configuration you compare the same treatment.

Step 8: Sweep concurrency and read the trade-off

One benchmark run answers a question you already had answered. The question that decides whether a batching server earns its place is what happens when requests arrive together, and answering it takes the same benchmark at several concurrency levels with the results side by side. Concurrency here always means requests in flight at the same moment, not requests per second. This step writes the script that does it. vLLM also ships vllm bench sweep for parameter sweeps; this script exists because it prints one small table, and because its parser is the part of the project worth testing in Step 9.

Create the file

touch sweep.py

Add the code: sweep.py

"""Run `vllm bench serve` at several concurrency levels and tabulate the result.

One run of the benchmark answers "how fast is this?", which is the question
single-stream comparisons already answer. A sweep answers the question that
decides whether a batching server is worth running at all: how much aggregate
throughput you buy for how much per-request latency, on your hardware, with
your model.
"""

from __future__ import annotations

import argparse
import re
import subprocess
import sys

# Label printed by `vllm bench serve` -> the column it becomes here.
METRICS = {
    "Output token throughput (tok/s):": "output_tok_s",
    "Request throughput (req/s):": "req_s",
    "Mean TTFT (ms):": "ttft_ms",
    "Mean TPOT (ms):": "tpot_ms",
    # Not a column. `vllm bench serve` exits 0 against a dead port and prints a
    # complete block of zeros, so this is the field that tells a real result
    # from a total failure.
    "Successful requests:": "successful",
}

NUMBER = re.compile(r"([0-9]+\.?[0-9]*)\s*$")


def parse(output: str) -> dict[str, float]:
    """Pull the four numbers this sweep cares about out of a benchmark block."""
    found: dict[str, float] = {}
    for line in output.splitlines():
        for label, column in METRICS.items():
            if line.startswith(label):
                match = NUMBER.search(line)
                if match:
                    found[column] = float(match.group(1))
    missing = set(METRICS.values()) - set(found)
    if missing:
        raise ValueError(f"benchmark output missing {sorted(missing)}")
    if found.pop("successful") == 0:
        raise ValueError("benchmark completed with zero successful requests")
    return found


def run(concurrency: int, args: argparse.Namespace) -> dict[str, float]:
    """Run one benchmark at one concurrency level."""
    command = [
        "vllm", "bench", "serve",
        "--model", args.model,
        "--base-url", args.base_url,
        "--dataset-name", "random",
        "--random-input-len", str(args.input_len),
        "--random-output-len", str(args.output_len),
        "--num-prompts", str(args.num_prompts),
        "--max-concurrency", str(concurrency),
    ]
    try:
        completed = subprocess.run(command, capture_output=True, text=True, timeout=900)
    except subprocess.TimeoutExpired:
        raise RuntimeError("vllm bench serve did not finish within 15 minutes") from None
    if completed.returncode != 0:
        raise RuntimeError(completed.stderr.strip()[-500:] or "vllm bench serve failed")
    return parse(completed.stdout)


def table(rows: list[tuple[int, dict[str, float]]]) -> str:
    """Format the sweep as a table, with throughput relative to the first row."""
    baseline = rows[0][1]["output_tok_s"]
    against = f"vs c={rows[0][0]}"
    lines = [
        f"{'concurrency':>11}  {'out tok/s':>9}  {against:>6}  {'req/s':>6}  {'TTFT ms':>8}  {'TPOT ms':>8}",
    ]
    for concurrency, metrics in rows:
        lines.append(
            f"{concurrency:>11}  {metrics['output_tok_s']:>9.1f}  "
            f"{metrics['output_tok_s'] / baseline:>5.2f}x  "
            f"{metrics['req_s']:>6.2f}  {metrics['ttft_ms']:>8.1f}  {metrics['tpot_ms']:>8.2f}"
        )
    return "\n".join(lines)


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(description="Sweep vLLM concurrency and tabulate.")
    parser.add_argument("--model", default="mlx-community/Qwen3-0.6B-4bit")
    parser.add_argument("--base-url", default="http://localhost:8000")
    parser.add_argument("--levels", default="1,8,16,32", help="Comma-separated concurrency levels.")
    parser.add_argument("--num-prompts", type=int, default=64)
    parser.add_argument("--input-len", type=int, default=256)
    parser.add_argument("--output-len", type=int, default=128)
    args = parser.parse_args(argv)

    rows = []
    for level in [int(value) for value in args.levels.split(",")]:
        print(f"running concurrency {level} ...", file=sys.stderr)
        try:
            rows.append((level, run(level, args)))
        except (RuntimeError, ValueError) as exc:
            print(f"error at concurrency {level}: {exc}", file=sys.stderr)
            return 1

    print(table(rows))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Detailed breakdown

  • METRICS maps vLLM’s printed labels to columns. Matching on the printed label rather than a line number is what keeps the parser working when vLLM adds a row, which it does between releases.
  • line.startswith(label) anchors the match at column 0. The benchmark prints a near-twin of one label, Peak output token throughput (tok/s):. Today the capital O keeps the two apart on its own; anchoring is what keeps them apart if a future label ever contains another as a substring.
  • parse raises on a missing metric, and on a run where nothing succeeded. The second guard is the one that earns its place: vllm bench serve exits 0 against a port with nothing on it and prints a complete block of zeros, so without the Successful requests: check the sweep would divide by a zero baseline and crash, or worse, print a row of plausible zeros.
  • run shells out to vllm bench serve rather than importing it. The benchmark lives in Homebrew’s private Python environment and this project has its own, so a subprocess is the honest boundary between them.
  • table normalizes throughput against the first row, because the absolute numbers are hardware-specific and the ratio is the part that transfers. The vs c=1 column is throughput relative to the first level swept, and the header is built from that level, so sweeping 8,16,32 labels it vs c=8.

Run the sweep against the server from Step 3:

uv run python sweep.py --levels 1,8,16,32

running concurrency 1 ...
running concurrency 8 ...
running concurrency 16 ...
running concurrency 32 ...
concurrency  out tok/s  vs c=1   req/s   TTFT ms   TPOT ms
          1      184.5   1.00x    1.44      30.6      5.22
          8      498.7   2.70x    3.90      72.8     15.59
         16      626.6   3.40x    4.90     121.1     24.78
         32      713.2   3.87x    5.57     206.9     43.57

The whole sweep took 1 minute 43 seconds.

The c=1 row reads 184.5 tokens per second where Step 7’s settled runs sat around 168. Same server, same concurrency, and the sweep’s own first level is the run that follows an idle gap, which Step 7 showed reads high. That is why sweep.py normalizes against its own first row rather than against a number from an earlier session, and why the ratios below are the portable part.

Reading the table

The server generated 3.87 times as many tokens per second at 32 concurrent requests as it did serving one at a time. That is continuous batching working: the same weights are read once per step and applied to every request in the batch, which --max-num-seqs caps, so the memory bandwidth that dominates single-stream decoding is amortized across all of them.

The cost is in the last column. TPOT went from 5.22 ms to 43.57 ms, so each individual user’s tokens arrive 8.3 times slower, and TTFT went from 31 ms to 207 ms. Nothing here is free; the throughput was bought with latency.

Which way that trade should go is a property of your workload, not of the server. A chat interface with one person typing wants the top row. A batch job scoring ten thousand documents wants the bottom row and would happily go further. An API serving many users at once wants the highest row whose TPOT is still under whatever your users will tolerate, which is exactly the measurement this table gives you and a single-stream benchmark cannot.

Two details matter when you read your own numbers. Returns are already diminishing here: doubling concurrency from 16 to 32 bought 14% more throughput while adding 76% to TPOT. And the sweep cannot exceed the --max-num-seqs 32 from Step 3: at concurrency 64 against this server the extra requests queue instead of joining the batch, which on a measured run bought 1% more throughput while TTFT went from 345 ms to 3.1 seconds. Keep your LEVELS at or below SEQS unless measuring the queue is the point, because a sweep that runs past it prints a flat throughput column and climbing TTFT with no error to explain why.

Step 9: Test the parser with no server

The parser is the only part of this project that can be wrong silently. A benchmark that fails is loud, and a server that is down is louder, but a parser that picks up the peak throughput instead of the mean produces a table that looks right and is not. These tests run with no server, no model and no network, so they also work on a machine that cannot run vLLM at all.

Create the file

touch tests/test_sweep.py

Add the code: tests/test_sweep.py

"""Tests for the benchmark parser and table. No server, no model, no network.

The fixture below is real `vllm bench serve` output, pasted verbatim, because a
parser tested against output invented by its own author tests nothing.
"""

from __future__ import annotations

import pytest

import sweep

BENCH_OUTPUT = """============ Serving Benchmark Result ============
Successful requests:                     16        
Failed requests:                         0         
Maximum request concurrency:             1         
Benchmark duration (s):                  12.12     
Total input tokens:                      4096      
Total generated tokens:                  2048      
Request throughput (req/s):              1.32      
Output token throughput (tok/s):         168.92    
Peak output token throughput (tok/s):    179.00    
Peak concurrent requests:                3.00      
Total token throughput (tok/s):          506.75    
---------------Time to First Token----------------
Mean TTFT (ms):                          32.18     
Median TTFT (ms):                        20.41     
P99 TTFT (ms):                           185.11    
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          5.71      
Median TPOT (ms):                        5.77      
P99 TPOT (ms):                           5.85      
---------------Inter-token Latency----------------
Mean ITL (ms):                           5.76      
Median ITL (ms):                         5.73      
P99 ITL (ms):                            6.99      
==================================================
"""


def test_parse_pulls_the_four_metrics():
    parsed = sweep.parse(BENCH_OUTPUT)

    assert parsed == {
        "output_tok_s": 168.92,
        "req_s": 1.32,
        "ttft_ms": 32.18,
        "tpot_ms": 5.71,
    }


def test_parse_takes_mean_not_median_or_p99():
    """Median TTFT is 20.41 and P99 is 185.11; only the mean may be picked up."""
    assert sweep.parse(BENCH_OUTPUT)["ttft_ms"] == 32.18


def test_parse_ignores_the_peak_throughput_line():
    """`Peak output token throughput` (179.00) must not shadow the plain line."""
    assert sweep.parse(BENCH_OUTPUT)["output_tok_s"] == 168.92


ALL_FAILED = """============ Serving Benchmark Result ============
Successful requests:                     0         
Failed requests:                         4         
Maximum request concurrency:             1         
Benchmark duration (s):                  0.31      
Total input tokens:                      0         
Total generated tokens:                  0         
Request throughput (req/s):              0.00      
Output token throughput (tok/s):         0.00      
Peak output token throughput (tok/s):    0.00      
Peak concurrent requests:                0.00      
Total token throughput (tok/s):          0.00      
---------------Time to First Token----------------
Mean TTFT (ms):                          0.00      
Median TTFT (ms):                        0.00      
P99 TTFT (ms):                           0.00      
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          0.00      
Median TPOT (ms):                        0.00      
P99 TPOT (ms):                           0.00      
==================================================
"""


def test_parse_rejects_a_run_where_every_request_failed():
    """`vllm bench serve` exits 0 against a dead port and prints zeros."""
    with pytest.raises(ValueError, match="zero successful requests"):
        sweep.parse(ALL_FAILED)


def test_parse_rejects_truncated_output():
    with pytest.raises(ValueError, match="missing"):
        sweep.parse("Request throughput (req/s):              1.32\n")


def test_table_header_names_the_row_it_normalizes_against():
    rows = [
        (8, {"output_tok_s": 494.4, "req_s": 3.86, "ttft_ms": 76.4, "tpot_ms": 15.71}),
        (16, {"output_tok_s": 617.5, "req_s": 4.82, "ttft_ms": 121.9, "tpot_ms": 25.15}),
    ]

    assert "vs c=8" in sweep.table(rows).splitlines()[0]


def test_table_reports_throughput_relative_to_the_first_row():
    rows = [
        (1, {"output_tok_s": 184.5, "req_s": 1.44, "ttft_ms": 30.6, "tpot_ms": 5.22}),
        (32, {"output_tok_s": 713.2, "req_s": 5.57, "ttft_ms": 206.9, "tpot_ms": 43.57}),
    ]

    rendered = sweep.table(rows).splitlines()

    assert rendered[1].split()[2] == "1.00x"
    assert rendered[2].split()[2] == "3.87x"

Detailed breakdown

  • The fixture is real output, pasted from the Step 7 run.
  • test_parse_takes_mean_not_median_or_p99 pins the one ambiguity in the block: three TTFT lines reading 32.18, 20.41 and 185.11, and only the mean belongs in the table.
  • test_parse_ignores_the_peak_throughput_line pins the plain line against its near-twin, so a loosened match gets caught here rather than in a table.
  • test_parse_rejects_a_run_where_every_request_failed covers the failure the exit status does not report: a benchmark against a dead port, which returns 0 and a block of zeros.
  • test_parse_rejects_truncated_output covers the half-finished benchmark, the shape a Ctrl-C or an out-of-memory kill leaves behind.
  • test_table_reports_throughput_relative_to_the_first_row checks the arithmetic that makes the table portable, using the sweep’s own two end rows: 713.2 ÷ 184.5 = 3.87, so the column reads 3.87x.
  • test_table_header_names_the_row_it_normalizes_against covers the vs c=8 case, because a header that always says c=1 while the maths uses whatever came first is a quiet way to mislabel a table.
uv run pytest -q

.......                                                                  [100%]
7 passed in 0.00s

Step 10: Wrap it in a Makefile

The serve command from Step 3 is four lines long and the sizing flags are the part you will want to change most often, which makes them exactly the wrong thing to keep retyping. The default target prints help, so the numbers you chose for this Mac are visible without reading a recipe.

Create the file

touch Makefile

Add the code: Makefile

MODEL  ?= mlx-community/Qwen3-0.6B-4bit
PORT   ?= 8000
MAXLEN ?= 4096
MEM    ?= 0.35
SEQS   ?= 32
LEVELS ?= 1,8,16,32
Q      ?= What is 17 * 23? Answer with just the number.

.DEFAULT_GOAL := help

.PHONY: help install serve ask bench sweep models test clean

help:
	@echo "vLLM on Apple Silicon"
	@echo ""
	@echo "Targets:"
	@echo "  help      Show this help screen (default)"
	@echo "  install   Install the Python client dependencies with uv"
	@echo "  serve     Start vllm serve, sized for this Mac"
	@echo "  models    List what the running server is serving"
	@echo "  ask       Send one prompt through the OpenAI SDK client"
	@echo "  bench     Run one vllm bench serve at concurrency 1"
	@echo "  sweep     Sweep concurrency levels and print the trade-off table"
	@echo "  test      Run the pytest suite (no server needed)"
	@echo "  clean     Remove caches and benchmark output"
	@echo ""
	@echo "Variables:"
	@echo "  MODEL     Model to serve (default: $(MODEL))"
	@echo "  PORT      Port for the server (default: $(PORT))"
	@echo "  MAXLEN    Context length per request (default: $(MAXLEN))"
	@echo "  MEM       Fraction of unified memory for KV cache (default: $(MEM))"
	@echo "  SEQS      Max requests batched at once (default: $(SEQS))"
	@echo "  LEVELS    Concurrency levels for 'sweep' (default: $(LEVELS))"
	@echo "  Q         Prompt for 'ask' (default: $(Q))"

install:
	uv sync

serve:
	vllm serve $(MODEL) \
	  --port $(PORT) \
	  --max-model-len $(MAXLEN) \
	  --gpu-memory-utilization $(MEM) \
	  --max-num-seqs $(SEQS)

models:
	@models=$$(curl -sf http://localhost:$(PORT)/v1/models) \
	  && echo "$$models" | python3 -c "import json,sys; [print(m['id'], m['max_model_len']) for m in json.load(sys.stdin)['data']]" \
	  || echo "no server answering on port $(PORT)"

ask:
	uv run python client.py --model $(MODEL) --base-url http://localhost:$(PORT)/v1 "$(Q)"

bench:
	vllm bench serve --model $(MODEL) --base-url http://localhost:$(PORT) \
	  --dataset-name random --random-input-len 256 --random-output-len 128 \
	  --num-prompts 16 --max-concurrency 1

sweep:
	uv run python sweep.py --model $(MODEL) --base-url http://localhost:$(PORT) --levels $(LEVELS)

test:
	uv run pytest -q

clean:
	rm -rf .pytest_cache results *.log
	find . -path ./.venv -prune -o -name __pycache__ -type d -prune -exec rm -rf {} +

Detailed breakdown

  • .DEFAULT_GOAL := help makes a bare make print the target list rather than starting a server, which is the friendlier accident.
  • MAXLEN, MEM and SEQS are the three flags from Step 3, hoisted into variables. make serve MEM=0.2 is how you retune for a smaller Mac without editing the file.
  • models uses curl and python3, not the project’s venv, so it answers even before uv sync has run. It is the quickest check that a server is up and serving what you think, and it prints one line rather than a traceback when nothing is listening.
  • bench and sweep both target --base-url, because vllm bench serve otherwise assumes port 8000 on localhost and will silently benchmark a different server if you are running two.
  • clean removes results/ and *.log, matching the .gitignore from Step 5.

Confirm the default target:

make

vLLM on Apple Silicon

Targets:
  help      Show this help screen (default)
  install   Install the Python client dependencies with uv
  serve     Start vllm serve, sized for this Mac
  models    List what the running server is serving
  ask       Send one prompt through the OpenAI SDK client
  bench     Run one vllm bench serve at concurrency 1
  sweep     Sweep concurrency levels and print the trade-off table
  test      Run the pytest suite (no server needed)
  clean     Remove caches and benchmark output

Variables:
  MODEL     Model to serve (default: mlx-community/Qwen3-0.6B-4bit)
  PORT      Port for the server (default: 8000)
  MAXLEN    Context length per request (default: 4096)
  MEM       Fraction of unified memory for KV cache (default: 0.35)
  SEQS      Max requests batched at once (default: 32)
  LEVELS    Concurrency levels for 'sweep' (default: 1,8,16,32)
  Q         Prompt for 'ask' (default: What is 17 * 23? Answer with just the number.)

And the two targets that need no server:

make test

uv run pytest -q
.......                                                                  [100%]
7 passed in 0.00s

Troubleshooting

OSError: [Errno 48] Address already in use. A server is already on that port, often one you started in another terminal and forgot. Either stop it or pass --port 8011, and remember that vllm bench serve defaults to port 8000 regardless of what you served on.

httpx.HTTPStatusError: Client error '401 Unauthorized' on startup. This looks like an authentication problem and is usually a typo in the model name. Hugging Face answers 401 rather than 404 for a repository you cannot see, so a misspelled public model and a real private one produce the same error. Check the name against mlx-community first, and only then go looking for a token.

The model answers with <think> and never gets to the point. Step 4. Send "chat_template_kwargs": {"enable_thinking": false}, or raise max_tokens enough to let it finish reasoning.

WARNING Found ulimit of 2048 and failed to automatically increase. The server is telling you it could hit OSError: [Errno 24] Too many open files under load. ulimit -n 8192 in the shell you launch it from clears it.

The sweep prints the same throughput at every level. Your LEVELS are above the server’s --max-num-seqs, so the extra requests are queueing rather than batching. Raise SEQS or lower LEVELS; TTFT climbing while throughput stays flat is the signature.

Your Mac gets slow while the server runs. The defaults reserve a large fraction of unified memory for the KV cache, and on Apple Silicon that is the same memory everything else is using. Lower --gpu-memory-utilization and --max-model-len as in Step 3, then check the kv_budget= field to see what you actually reserved.

The version string says cpu. Expected, and so is device_config=cpu in the engine config. Look for MLX device set to: Device(gpu, 0) and Platform plugin metal is activated instead.

Throughput numbers differ from the ones here. They will. Chip, memory bandwidth, model, quantization and macOS version all move them, and so does how long the server has been under load: see Step 7 on discarding the first run. The shape of the curve in Step 8 is the claim, not the absolute values.

Recap

You installed vLLM on a Mac with one brew install, served an MLX model on the GPU, sized the server so it leaves the machine usable, and pointed a stock OpenAI client at it. Then you measured what settles whether this server belongs in your stack:

  • Single stream, this model, this Mac: 169 output tokens per second settled, 32 ms to first token, 5.71 ms per token after that. The sweep’s own c=1 baseline reads 185, because it runs after an idle gap.
  • Thirty-two concurrent requests: 713 tokens per second, 3.87x the sweep’s baseline, with TTFT at 207 ms and TPOT at 43.6 ms.
  • The trade is the product. Aggregate throughput and per-request latency move in opposite directions, and where you want to sit on that curve is a fact about your workload that no benchmark headline can tell you.

The reason to do this on a Mac is parity. The flags in Step 3, the API in Step 4, the client in Step 6 and the benchmark in Step 7 are the same ones you use against a CUDA deployment, so a config you tune here is a config you can ship. When you are done, Ctrl-C the server: it is holding the kv_budget you gave it until you do. For one conversation at a time, llama.cpp remains the simpler and faster local path, and Getting Started with llama.cpp on macOS is where that starts.

Where to go next, roughly in order of payoff:

  • Serve a model you would actually deploy. mlx-community/Qwen3-0.6B-4bit is small enough to make the measurements quick and too small to tell you much else. A model of 8 billion parameters at 4-bit is a realistic local target; re-run Step 8 against it, because a larger model shifts where the bandwidth goes and flattens the batching curve differently.
  • Sweep the sizing flags, not just concurrency. make sweep varies load against a fixed server. The other half of the space is varying --max-num-seqs and --max-model-len against fixed load.
  • Put a harness in front of it. The Model Is Read-Only: Build a Glass-Box LLM Harness in Python builds a tool-calling loop against exactly this API shape, and the base URL is the only line that changes.
  • Check the model matrix before committing to an architecture. The plugin’s supported models table marks which families are verified, which are experimental, and which have no Metal attention kernel yet.