Run Qwen3.8-27B Locally with llama.cpp on macOS

Qwen3.8-27B is a 27-billion-parameter dense model from the Qwen team, released August 2026 under Apache 2.0. It reads images and video as well as text, ships a multi-token-prediction head for faster decoding, and claims 262,144 tokens of native context. Quantized to Q4_K_M it is 17.7 GiB on disk.

The context number is the interesting one. A conventional 27B model with 64 attention layers would need roughly 64 GiB of KV cache to hold 262,144 tokens, which is more memory than most machines have for the cache alone. Qwen3.8 gives only every fourth layer a real attention cache and runs the other 48 layers on a fixed-size recurrent state, so the same context costs 16 GiB. Step 6 reads those numbers straight out of the loader.

This article installs the model, runs it four ways (CLI, OpenAI-compatible server, image input, speculative decoding), and ends with a small Python project that measures the two knobs worth measuring: what each reasoning_effort level costs, and what the MTP head is actually worth.

Three things are worth knowing before you start:

• Homebrew is new enough. Unlike some recent releases, this one needs no source build. The qwen35 architecture has been registered since at least b10330, which is what Homebrew shipped in August 2026.
• reasoning_effort accepts low, medium, and xhigh. Not high. Sending high the documented way returns a 500 from the chat template. Sending it the OpenAI way returns a normal answer and changes nothing. Step 4 covers both.
• The MTP head is worth turning on. Measured here it took decoding from 25.9 to 47.6 tokens/sec, about 1.84x, at a 65% draft acceptance rate. That is a much larger win than a separate drafter model usually delivers.

Prerequisites

• macOS on Apple Silicon with at least 32 GB of unified memory. Written and validated on macOS 26.5.2 (build 25F84), arm64, Apple M5 Max with 128 GB. The weights occupy about 18.3 GiB once resident, and Step 6 shows how fast the cache grows if you ask for long context.
• About 20 GiB of free disk. The three files total 19.8 GiB, and the Hugging Face cache keeps one copy of each.
• llama.cpp b10330 or newer — brew install llama.cpp. Step 1 verifies the build understands the architecture.
• uv 0.11.26 or newer for the Python project in Steps 9–17 — check with uv --version, install with brew install uv.
• make, from the Xcode command line tools (xcode-select --install).

No Hugging Face account or token is needed. The model is Apache 2.0 and the GGUF conversions are public.

Step 1: Confirm your build knows the qwen35 architecture

Twenty gigabytes of download is a long way to go to discover your llama.cpp cannot open the file. Two things decide that, and both are checkable in a second: which build you have, and whether the architecture this model declares is compiled into it. The second is the one that actually matters.

llama version

b10330-687e77892

The build number alone does not tell you whether an architecture is compiled in, so check the shared library for the architecture string the GGUF declares:

strings "$(brew --prefix)/lib/libllama.dylib" | grep -c '^qwen35$'

1

Detailed breakdown

• qwen35 is the architecture name, not qwen38. Qwen3.8 is built on the Qwen3.5 architecture, and llama.cpp names architectures after the family rather than the release. The GGUF’s general.architecture key holds the literal string qwen35, so that is what has to be registered. Searching your build for qwen38 finds nothing and proves nothing.
• 1 means supported, 0 means the model will be rejected at load. llama.cpp registers every architecture name in a table compiled into libllama, so the string is present exactly when the loader can dispatch on it. A build without it fails on the model file rather than producing bad output.
• This is a pre-flight check, not a guarantee of correctness. It proves the architecture is registered. It does not prove every kernel is optimal on your hardware. The real confirmation is Step 3 returning a right answer.
• Homebrew was ahead of the requirement here, and that is not always true. This machine had b10330 installed while the formula had already moved to 10470. Both work. If your grep -c returns 0, brew upgrade llama.cpp is the fix, and a source build is not necessary.

Step 2: Download the model, the MTP head, and the projector

This model is three files, and only the first is required. The other two are the multi-token-prediction head that Step 8 uses for speculative decoding, and the vision encoder that Step 7 needs for image input. Each is requested differently, and the projector in particular does not arrive the way the documentation suggests it should.

llama download -hf ggml-org/Qwen3.8-27B-GGUF:Q4_K_M --mtp
llama download -hf ggml-org/Qwen3.8-27B-GGUF -hff mmproj-Qwen3.8-27B-Q8_0.gguf

Confirm all three landed:

SNAP=$(find ~/.cache/huggingface/hub/models--ggml-org--Qwen3.8-27B-GGUF/snapshots \
       -maxdepth 1 -mindepth 1 -type d | head -1)
ls -lhL "$SNAP"

total 41617984
-rw-r--r--  1 you  staff   600M Aug 18 11:29 mmproj-Qwen3.8-27B-Q8_0.gguf
-rw-r--r--  1 you  staff   1.6G Aug 18 11:23 mtp-Qwen3.8-27B-Q4_0.gguf
-rw-r--r--  1 you  staff    18G Aug 18 11:28 Qwen3.8-27B-Q4_K_M.gguf

Detailed breakdown

• What each file is. Qwen3.8-27B-Q4_K_M.gguf (18,973,870,432 bytes) is the model and the only required file. mtp-Qwen3.8-27B-Q4_0.gguf (1,680,271,648 bytes) is the multi-token-prediction head used for speculative decoding in Step 8. mmproj-Qwen3.8-27B-Q8_0.gguf (629,247,008 bytes) is the vision encoder, needed for Step 7 and useless on its own.
• --mtp is what fetches the second file, and it matches quantizations for you. Asking for :Q4_K_M pulled mtp-...-Q4_0.gguf rather than the BF16 or Q8_0 heads also in the repo. The flag is off by default, so a plain -hf leaves you with no drafter and no error explaining why Step 8 cannot run.
• The projector needs its own command, despite --mmproj-auto being on by default. This is the trap in this step. The default is documented as fetching a matching mmproj alongside the model, and against this repository it does not: re-running the first command with the model already cached printed the model path and downloaded nothing. The projector filenames here carry the model name (mmproj-Qwen3.8-27B-Q8_0.gguf) rather than the bare mmproj.gguf the auto-matcher looks for, so naming the file with -hff is the reliable route.
• ls -lhL follows the symlinks, and plain ls -lh does not. The snapshot directory holds symlinks into blobs/, so without -L every file reports as 76 bytes of link.
• SNAP is resolved with find, not a glob. The snapshot directory is named after a commit hash, so it cannot be typed from memory. Writing SNAP=.../snapshots/*/ looks tidier and breaks under zsh, which does not expand a glob stored in a variable, so a literal * reaches the command. The find form behaves the same in zsh and bash.
• Q4_K_M is a choice. The same repository carries Q8_0 (26.6 GiB) and BF16 (50.1 GiB). Everything below works with any of them; substitute the filename.

Step 3: A one-shot answer from the CLI

With the weights on disk, the shortest route to a working model is one question answered at the command line. This also fixes the number everything later is measured against: single-stream generation speed with nothing else loaded.

SNAP=$(find ~/.cache/huggingface/hub/models--ggml-org--Qwen3.8-27B-GGUF/snapshots \
       -maxdepth 1 -mindepth 1 -type d | head -1)
llama cli -m "$SNAP"/Qwen3.8-27B-Q4_K_M.gguf \
    -c 8192 \
    --temp 1.0 --top-p 0.95 --top-k 20 --min-p 0.0 \
    -st -p "What is 17 * 23? Reply with just the number."

build      : b10330-687e77892
model      : /Users/you/.cache/huggingface/hub/models--ggml-org--Qwen3.8-27B-GGUF/snapshots/0669b98607d47046c7c2b3f801011d54a08cfccf/Qwen3.8-27B-Q4_K_M.gguf
ftype      : Q4_K - Medium
modalities : text

> What is 17 * 23? Reply with just the number.

[Start thinking]

We need answer user's request: "What is 17 * 23? Reply with just the number." Need final only number. Compute 17*23 = 391. Ensure no extra.
[End thinking]

391

[ Prompt: 131.9 t/s | Generation: 26.1 t/s ]

Detailed breakdown

• -st makes it answer and exit. Short for --single-turn. Without it llama cli prints the answer and waits at a prompt for your next message, which inside a script is indistinguishable from a hang.
• The sampling flags come from the model card’s thinking-mode recommendation: temperature 1.0, top-p 0.95, top-k 20, min-p 0.0. Three of them are also baked into the GGUF as general.sampling.* keys (top_k = 20, top_p = 0.95, temp = 1.0), which Step 6 reads out of the file. Passing them explicitly is what makes the run reproducible for a reader on a different build.
• --jinja is not needed on this build. It defaults to enabled, so the chat template inside the GGUF is used without asking. Older articles that pass it explicitly are not wrong, just redundant here.
• modalities : text confirms the projector was not loaded. That is correct for this step; Step 7 loads it.
• The thinking trace is inline, bracketed by [Start thinking] and [End thinking]. Getting the two channels as separate fields is the server’s job, covered in Step 5.
• 26.1 tok/s is single-stream generation with nothing else loaded. Treat it as this machine’s baseline; Step 8 roughly doubles it.

Step 4: reasoning_effort is a three-value enum, and high is not one of them

Thinking mode has a depth control called reasoning_effort. It is not a llama.cpp flag, so there is no --reasoning-effort to pass: the value travels into the chat template as a variable, which is what --chat-template-kwargs carries. The value most people reach for first is the one this template refuses, so the run below is meant to fail, and the error it produces is the useful part.

SNAP=$(find ~/.cache/huggingface/hub/models--ggml-org--Qwen3.8-27B-GGUF/snapshots \
       -maxdepth 1 -mindepth 1 -type d | head -1)
llama cli -m "$SNAP"/Qwen3.8-27B-Q4_K_M.gguf -c 8192 \
    --chat-template-kwargs '{"reasoning_effort":"high"}' \
    -st -p "hi"

...', 'low') %}↵        {{- raise_exception('Unexpected reasoning effort ' ~ reason...
                                           ^
Error: Jinja Exception: Unexpected reasoning effort high. Supported types are xhigh (default), medium, and low.

Sent the OpenAI way instead, the same value is accepted and does nothing. This needs the server from Step 5, so come back to it if you are following in order:

for v in high banana; do
  echo "--- top-level reasoning_effort: $v ---"
  curl -s http://127.0.0.1:8080/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d "{\"messages\":[{\"role\":\"user\",\"content\":\"Capital of France? One word.\"}],
         \"reasoning_effort\":\"$v\",\"max_tokens\":128}" \
    | python3 -c "import sys,json; d=json.load(sys.stdin); \
        print(d['choices'][0]['message']['content'] if 'choices' in d else d)"
done

--- top-level reasoning_effort: high ---
Paris
--- top-level reasoning_effort: banana ---
Paris

A supported value works, and enable_thinking turns the trace off entirely:

llama cli -m "$SNAP"/Qwen3.8-27B-Q4_K_M.gguf -c 8192 \
    --temp 0.7 --top-p 0.80 --top-k 20 --presence-penalty 1.5 \
    --chat-template-kwargs '{"enable_thinking":false}' \
    -st -p "Is 91 a prime number? Answer yes or no, then give the reason in one sentence."

> Is 91 a prime number? Answer yes or no, then give the reason in one sentence.
No, because 91 is divisible by 7 and 13 (since $7 \times 13 = 91$), making it a composite number rather than a prime.

[ Prompt: 140.4 t/s | Generation: 25.7 t/s ]

Detailed breakdown

• The three legal values are low, medium, and xhigh, and the default is xhigh. The chat template validates the string and calls raise_exception on anything else, so the failure happens at template-render time, before a single token is generated. The error text names the legal set, which makes this one of the friendlier failure modes in local inference.
• high is the trap because it is right everywhere else. Every other reasoning model in common use takes low/medium/high, so high is the value that gets typed from habit and the one value this template rejects. Some third-party GGUF repackagings patch their template to silently map high to xhigh; the ggml-org conversion used here does not.
• --chat-template-kwargs takes a JSON object string, and the shell quoting matters: single quotes on the outside, double quotes inside. A malformed object is a parse error rather than a silent no-op.
• The top-level reasoning_effort field is the worse trap, because it fails quietly. It is the field OpenAI’s own API defines, so it is what an SDK will send, and llama.cpp accepts it without complaint. banana proves it never reaches the template: a value the template would reject on sight returns a normal answer. Anything routed that way is choosing an effort level it does not get. The two behaviours are opposite in the worst way, since the documented route fails loudly on high while the habitual route stays silent on anything.
• enable_thinking: false is a separate switch from effort. It emits a closed empty <think> block in the generation prompt, so the model starts on the answer. The sampling flags change with it: the model card recommends temperature 0.7, top-p 0.80 and a presence penalty of 1.5 for non-thinking mode, against 1.0/0.95/0.0 for thinking.
• The effort dial does not bite on easy questions. Asked whether 91 is prime, the three levels produced traces within noise of each other, and low sometimes ran longer than medium. Step 12 measures it on a prompt hard enough to separate them.

Step 5: Serve an OpenAI-compatible endpoint

The CLI is fine for one question and awkward for anything programmatic, not least because it prints the model’s reasoning inline with its answer. Running the same weights behind llama.cpp’s HTTP server separates the two into distinct JSON fields and lets any OpenAI-compatible client connect with a base-URL change. The rest of the article talks to this server.

SNAP=$(find ~/.cache/huggingface/hub/models--ggml-org--Qwen3.8-27B-GGUF/snapshots \
       -maxdepth 1 -mindepth 1 -type d | head -1)
llama serve \
    -m  "$SNAP"/Qwen3.8-27B-Q4_K_M.gguf \
    -mm "$SNAP"/mmproj-Qwen3.8-27B-Q8_0.gguf \
    -c 32768 --host 127.0.0.1 --port 8080 \
    --reasoning-preserve

srv    load_model: initializing, n_slots = 4, n_ctx_slot = 32768, kv_unified = 'true'
srv  llama_server: model loaded
srv  llama_server: listening on http://127.0.0.1:8080

From a second terminal:

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "messages": [{"role": "user", "content": "What is the capital of France? One word."}],
    "chat_template_kwargs": {"reasoning_effort": "low"},
    "max_tokens": 512
  }' | python3 -c "
import sys, json
d = json.load(sys.stdin)
m = d['choices'][0]['message']
print('content     :', repr(m.get('content')))
print('reasoning   :', repr(m.get('reasoning_content')))
print('usage       :', json.dumps(d['usage']))
"

content     : 'Paris'
reasoning   : 'The user is asking for the capital of France and wants a one-word answer.\n'
usage       : {"completion_tokens": 21, "prompt_tokens": 50, "total_tokens": 71, "prompt_tokens_details": {"cached_tokens": 0}}

Detailed breakdown

• The server splits reasoning from the answer. message.content holds Paris and message.reasoning_content holds the trace. Any OpenAI client that reads only content therefore works unmodified, which is not true of the CLI output in Step 3.
• n_ctx_slot = 32768 with n_slots = 4 means each slot gets the whole 32,768. kv_unified = 'true' puts all four slots in one cache rather than dividing -c between them. Builds that split the context report a smaller n_ctx_slot than the -c you asked for, so read this line rather than assuming.
• --reasoning-preserve keeps the trace in the conversation history, not just on the last assistant turn. The server suggests it at startup for this model because its template advertises the capability. Leave it off and a multi-turn agent loses the reasoning from earlier turns.
• -mm is the projector, and it is loaded here so Step 7 can reuse this server. It costs 600 MB and nothing else if you never send an image.
• --host 127.0.0.1 keeps it on the loopback interface. llama.cpp warns at startup that CORS is open and no API key is set; that is acceptable only because nothing outside the machine can reach the port.

Step 6: What 262,144 tokens of context actually costs

A context length is only as real as the memory behind it, and this is where the hybrid attention layout stops being a spec-sheet detail. The loader reports exactly what it allocated, in two separate pools, if you turn the verbosity up far enough to see them.

SNAP=$(find ~/.cache/huggingface/hub/models--ggml-org--Qwen3.8-27B-GGUF/snapshots \
       -maxdepth 1 -mindepth 1 -type d | head -1)
llama serve -m "$SNAP"/Qwen3.8-27B-Q4_K_M.gguf -c 262144 --port 8099 -lv 5 \
  > /tmp/qwen-ctx.log 2>&1 &
SRV=$!
until grep -q 'listening on' /tmp/qwen-ctx.log; do sleep 1; done
kill $SRV
grep -E 'llama_kv_cache: size|llama_memory_recurrent: size' /tmp/qwen-ctx.log | tail -2

llama_kv_cache: size = 16384.00 MiB (262144 cells,  16 layers,  4/1 seqs), K (f16): 8192.00 MiB, V (f16): 8192.00 MiB
llama_memory_recurrent: size =  598.50 MiB (     4 cells,  64 layers,  4 seqs  0 rs_seq), R (f32):   22.50 MiB, S (f32):  576.00 MiB

The same log holds the model’s own declaration of the numbers the rest of this step reasons about, so there is no need to take them on trust:

grep -E 'general\.architecture|general\.sampling|qwen35\.block_count|qwen35\.attention\.head_count|qwen35\.attention\.key_length|qwen35\.full_attention_interval' \
  /tmp/qwen-ctx.log | sed 's/^[0-9.]* I //' | sort -u

llama_model_loader: - kv   0:                       general.architecture str              = qwen35
llama_model_loader: - kv   2:                     general.sampling.top_k i32              = 20
llama_model_loader: - kv   3:                     general.sampling.top_p f32              = 0.950000
llama_model_loader: - kv   4:                      general.sampling.temp f32              = 1.000000
llama_model_loader: - kv   9:                         qwen35.block_count u32              = 64
llama_model_loader: - kv  13:                qwen35.attention.head_count u32              = 24
llama_model_loader: - kv  14:             qwen35.attention.head_count_kv u32              = 4
llama_model_loader: - kv  18:                qwen35.attention.key_length u32              = 256
llama_model_loader: - kv  25:             qwen35.full_attention_interval u32              = 4

Repeating that at four context sizes gives the whole picture:

Detailed breakdown

• 16 layers, not 64, is the whole point. The model has 64 layers, and the loader logs llama_kv_cache: layer N: filtered for three out of every four, keeping a cache only on layers 3, 7, 11, and so on. That is the full_attention_interval = 4 value in the dump above, against a block_count of 64: one gated-attention layer after every three Gated DeltaNet layers. Note that head_count = 24 is the query-head count, and the KV cache is sized by head_count_kv = 4, which is the number that matters here.

• The arithmetic checks out exactly, and every input to it is in the dump above. Two tensors (K and V) x 4 KV heads (head_count_kv) x 256 head dimension (key_length) x 2 bytes for f16 is 4,096 bytes per layer per token. Times 16 layers times 262,144 tokens is 17,179,869,184 bytes, which is the 16,384.00 MiB reported. Run the same formula over all 64 layers and you get 65,536 MiB, so the hybrid layout is saving 48 GiB at full context.

• The recurrent state does not grow with context. It is 598.50 MiB at 8,192 tokens and 598.50 MiB at 262,144, because a linear-attention layer carries a fixed-size state rather than a per-token history. The 4 cells are the four sequence slots, not tokens.

• llama serve is a server, so this runs it in the background and kills it. Piping it straight into grep prints the two lines and then hangs, because nothing ever closes the pipe: the process is still sitting there listening. Starting it with &, waiting for listening on, and killing it is what makes this a command that returns. Stop it before Step 7 either way; port 8099 is used here only to keep it clear of the Step 5 server on 8080.

• tail -2 is there because the model is loaded twice. llama.cpp runs a fit-params pass to size the allocation against free memory before the real load, and each pass logs its own copy of both lines. Without the tail you get four lines and a reasonable suspicion that something is wrong. The two passes report identical figures here.

• -lv 5 is required to see any of this. At default verbosity the server prints three lines and none of them mention memory. The flag also produces a few thousand lines of metadata dump, hence the grep.

• The 64 layers on the recurrent line is not a contradiction, and it is the most confusing number in this output. It is the layer span the memory module covers, not the count of layers holding a state. The per-layer lines in the same log settle it: llama_memory_recurrent: layer N: skipped appears 16 times per load, on exactly the layers the KV cache claims, leaving 48 with a state. The state sizes confirm it independently, since 576.00 MiB over 48 layers and 4 sequences is 3 MiB each, which is exactly the 48 value heads x 128 head dimension x 128 state x 4 bytes an f32 DeltaNet state needs.

• Budget the weights plus the table above. The same log carries the other three figures, under load_tensors: and sched_reserve::

  load_tensors:   CPU_Mapped model buffer size =   682.03 MiB
  load_tensors:  MTL0_Mapped model buffer size = 18084.41 MiB
  sched_reserve:       MTL0 compute buffer size =   440.48 MiB
  sched_reserve:        CPU compute buffer size =   276.02 MiB
  

  That is 18.3 GiB of weights plus 716.50 MiB of compute buffers, so a loaded server starts at about 19.0 GiB before any cache. Add the 262,144-token row and you are at roughly 35.6 GiB, past what a 32 GB machine has; the 131,072 row lands near 27.6 GiB, which fits but leaves little headroom.

Step 7: Describe an image

This step loads its own copy of the model rather than using the Step 5 server, so stop that server first or give this one a different terminal.

It also needs an image, and the point of drawing one rather than downloading one is that you know the right answer before you ask. Write it first, from the current directory:

python3 - <<'PY'
import struct, zlib
W, H, BARS = 480, 320, (60, 110, 180, 250)
rows = [[(255, 255, 255)] * W for _ in range(H)]
for i, h in enumerate(BARS):
    for x in range(40 + i * 100, 100 + i * 100):
        for y in range(H - 30 - h, H - 30):
            rows[y][x] = (30, 90, 200)
raw = b"".join(b"\x00" + bytes(c for p in r for c in p) for r in rows)
chunk = lambda t, d: struct.pack(">I", len(d)) + t + d + struct.pack(">I", zlib.crc32(t + d))
open("chart.png", "wb").write(
    b"\x89PNG\r\n\x1a\n"
    + chunk(b"IHDR", struct.pack(">IIBBBBB", W, H, 8, 2, 0, 0, 0))
    + chunk(b"IDAT", zlib.compress(raw, 9))
    + chunk(b"IEND", b""))
PY

Four blue bars, strictly increasing. Now ask the model what it sees:

SNAP=$(find ~/.cache/huggingface/hub/models--ggml-org--Qwen3.8-27B-GGUF/snapshots \
       -maxdepth 1 -mindepth 1 -type d | head -1)
llama cli -m "$SNAP"/Qwen3.8-27B-Q4_K_M.gguf \
    -mm "$SNAP"/mmproj-Qwen3.8-27B-Q8_0.gguf \
    -c 8192 --chat-template-kwargs '{"enable_thinking":false}' \
    --image chart.png -st \
    -p "How many bars are in this chart, what colour are they, and are they increasing or decreasing left to right?"

Loaded media from 'chart.png'

> How many bars are in this chart, what colour are they, and are they increasing or decreasing left to right?
There are **4 bars** in this chart.

They are **blue** in colour.

They are **increasing** in height from left to right — each bar is taller than the one before it, forming a stepped upward pattern.

[ Prompt: 255.0 t/s | Generation: 25.9 t/s ]

Detailed breakdown

• Do not skip the generator and hope a stray image is lying around. A missing --image file does not stop the run: llama.cpp prints Error: file does not exist or cannot be opened: 'chart.png' and then answers anyway, with no image, in the same confident format. The error scrolls past behind the model banner and the reply looks like a reply. That is the worst possible failure for a vision check, and it costs a full model load to reach.
• Checking a vision model against an image whose correct description you already know is the only way to tell a right answer from a fluent one. The snippet above is the throwaway version; Step 10 promotes the same drawing into the project as a tested module with named constants, which is what the rest of the article uses.
• --image takes the file directly, and accepts comma-separated paths for several images. The same flag doubles as --video; the server reports "video": true in its modalities, and "audio": false.
• -mm names the projector explicitly. With -hf it can be omitted, but this run loads from a local path, so nothing is auto-detected.
• Prompt throughput jumps to 255 t/s because the image expands into a large block of prefill tokens that batch well. Generation speed is unchanged at 25.9 t/s, which is expected: the encoder runs once, then decoding is ordinary.
• enable_thinking: false keeps the answer short. With thinking on, the model reasons about the image first, which is useful for a hard chart and wasteful for a bar count.

Step 8: Turn on the MTP head

The MTP head pulled down in Step 2 has been sitting unused. Loading it as a speculative drafter costs two flags and buys more than anything else in this article, without changing the weights or the answers. The comparison below is one sample each, to make the flags concrete; Step 17 measures it properly.

SNAP=$(find ~/.cache/huggingface/hub/models--ggml-org--Qwen3.8-27B-GGUF/snapshots \
       -maxdepth 1 -mindepth 1 -type d | head -1)
llama cli -m "$SNAP"/Qwen3.8-27B-Q4_K_M.gguf -c 8192 \
    --chat-template-kwargs '{"enable_thinking":false}' \
    -st -p "Write a Python function that reverses a linked list. Code only."

[ Prompt: 113.9 t/s | Generation: 25.9 t/s ]

Same prompt, with the MTP head loaded as a speculative drafter:

llama cli -m "$SNAP"/Qwen3.8-27B-Q4_K_M.gguf -c 8192 \
    -md "$SNAP"/mtp-Qwen3.8-27B-Q4_0.gguf --spec-type draft-mtp \
    --chat-template-kwargs '{"enable_thinking":false}' \
    -st -p "Write a Python function that reverses a linked list. Code only."

[ Prompt: 76.7 t/s | Generation: 55.6 t/s ]

Detailed breakdown

• Two flags are needed, and neither works alone. -md names the draft model and --spec-type draft-mtp selects the MTP algorithm. Passing -md with no --spec-type leaves speculative decoding off, because the default is none.
• draft-mtp is one of several strategies. The same flag accepts draft-simple for a separate small model, draft-eagle3, draft-dflash, and a family of ngram-* options that need no second model at all. Only draft-mtp matches the head published alongside this model.
• Generation roughly doubled: 25.9 to 55.6 tok/s. Speculative decoding wins when the drafter’s guesses are usually accepted, and an MTP head trained alongside the model it drafts for guesses well. Step 17 measures acceptance directly and gets 65%.
• Prompt processing got slower here, and by much less than this one sample suggests. Four samples per side put baseline at about 109 t/s and the drafted run at about 100 t/s, roughly 9%, and one of those pairs had the drafted run ahead. Prefill on a twenty-token prompt is noisy enough that a single reading like the 76.7 above should not be read as the size of the effect. The benchmark in Step 13 fixes the output length instead of timing whole requests for the same reason.
• A single sample is not a measurement. These two numbers come from one run each and are shown to make the flags concrete. The repeated version in Step 17 is the one to quote.

Step 9: Create the companion project

Everything so far has been one command at a time. The rest of the article builds a small Python project that measures the two things the model card leaves open on this runtime: what each reasoning_effort level costs, and what the MTP head is worth. It runs against the server from Step 5, so nothing below reloads the weights.

Create the file

mkdir -p ~/src/qwen38-probe
cd ~/src/qwen38-probe
touch .gitignore

# ~/src/qwen38-probe/.gitignore
# Python
__pycache__/
*.py[cod]
.venv/
*.egg-info/

# uv
uv.lock

# pytest
.pytest_cache/
.coverage

# Model weights, projectors, MTP heads — never commit these
*.gguf

# Server logs and measurement transcripts
*.log
*.jsonl

# Sample images generated at run time
*.png

# Editor / OS
.DS_Store
.idea/
.vscode/

Then initialize the project:

cd ~/src/qwen38-probe
mkdir -p src/qwen38_probe tests images
touch src/qwen38_probe/__init__.py

Detailed breakdown

• .gitignore comes first, before any other file. A single *.gguf staged by accident is 18 GB in a repository that will not forget it.
• *.png is ignored because the chart is generated, not authored. Step 10 writes images/chart.png from a script, so the file is build output.
• uv.lock is ignored here because this project is a measurement harness rather than a deployed application, and the article pins versions in its validation section instead. Committing the lock file is the better default for anything you deploy.
• The src/ layout keeps the package importable only after install, which is what catches a test that accidentally depends on the working directory.

Step 10: Draw a test chart with the standard library

Step 7 wrote a throwaway chart inline. This is the same drawing promoted into the project as a module with named constants, which is what makes it testable: a test in Step 16 asserts the bars really do increase, because the question put to the vision model has no right answer otherwise.

Create the file

cd ~/src/qwen38-probe
touch src/qwen38_probe/make_chart.py

# ~/src/qwen38-probe/src/qwen38_probe/make_chart.py
"""Write a small bar chart PNG using nothing but the standard library.

The vision steps need an image whose correct description is known in advance, so
that a wrong answer is recognisable. Encoding the PNG by hand keeps the project
free of an image dependency: `zlib` and `struct` are enough for a truecolour PNG.
"""

from __future__ import annotations

import struct
import zlib
from pathlib import Path

WIDTH, HEIGHT = 480, 320
BARS = (60, 110, 180, 250)  # four bars, strictly increasing left to right
BAR_COLOR = (30, 90, 200)  # a blue the model should call "blue"
BAR_WIDTH = 60
BAR_GAP = 100
LEFT_MARGIN = 40
BASELINE = 30


def render() -> bytes:
    """Return the bytes of a PNG showing four increasing blue bars."""
    rows = [[(255, 255, 255)] * WIDTH for _ in range(HEIGHT)]

    for index, height in enumerate(BARS):
        x0 = LEFT_MARGIN + index * BAR_GAP
        for x in range(x0, x0 + BAR_WIDTH):
            for y in range(HEIGHT - BASELINE - height, HEIGHT - BASELINE):
                rows[y][x] = BAR_COLOR

    for x in range(20, WIDTH - 20):  # axis line under the bars
        rows[HEIGHT - BASELINE][x] = (0, 0, 0)

    # PNG scanlines are prefixed with a filter byte; 0 means "no filter".
    raw = b"".join(
        b"\x00" + bytes(channel for pixel in row for channel in pixel) for row in rows
    )

    def chunk(tag: bytes, data: bytes) -> bytes:
        body = tag + data
        return struct.pack(">I", len(data)) + body + struct.pack(">I", zlib.crc32(body))

    return (
        b"\x89PNG\r\n\x1a\n"
        + chunk(b"IHDR", struct.pack(">IIBBBBB", WIDTH, HEIGHT, 8, 2, 0, 0, 0))
        + chunk(b"IDAT", zlib.compress(raw, 9))
        + chunk(b"IEND", b"")
    )


def write(path: str | Path = "images/chart.png") -> Path:
    target = Path(path)
    target.parent.mkdir(parents=True, exist_ok=True)
    target.write_bytes(render())
    return target


if __name__ == "__main__":
    written = write()
    print(f"wrote {written} ({written.stat().st_size} bytes)")

Detailed breakdown

• BARS is the answer key. Four values, strictly increasing, so “four blue bars, increasing” is checkable rather than plausible. A test in Step 16 asserts the tuple really is sorted and distinct, because the question has no right answer otherwise.
• IHDR declares colour type 2, truecolour RGB with 8 bits per channel. That is the simplest format that gets a recognisable blue without a palette chunk.
• Every scanline is prefixed with a zero byte. PNG filters each row and stores the filter type inline; 0 means “store the row as-is”. Omitting the byte produces a file that passes a header check and decodes to noise.
• zlib.crc32 over tag plus data is what makes each chunk valid. The length prefix covers the data only, while the CRC covers the tag as well, and getting that boundary wrong is the usual reason a hand-built PNG fails to open.
• The whole file is about 1.9 KB, which matters because Step 12 sends it inline as a base64 data URL.

Step 11: Write the client

Two details of this model’s API do not survive contact with a stock OpenAI client: reasoning_effort has to travel inside chat_template_kwargs, and the speculative-decoding counters live in a non-standard timings object that an official SDK drops. This module keeps both in one place so the measurement code never has to think about them.

Create the file

cd ~/src/qwen38-probe
touch src/qwen38_probe/client.py

# ~/src/qwen38-probe/src/qwen38_probe/client.py
"""A thin client over the llama.cpp OpenAI-compatible endpoint.

What this adds over calling `httpx` directly is that it keeps the two
Qwen3.8-specific details in one place: `reasoning_effort` has to travel in
`chat_template_kwargs` rather than the top-level OpenAI field, and the
speculative-decoding counters live in the non-standard `timings` object.
"""

from __future__ import annotations

import base64
from dataclasses import dataclass
from pathlib import Path

import httpx

# The chat template raises a Jinja exception on anything outside this set. Note
# that "high" is NOT a member: the template accepts xhigh, medium and low only.
EFFORTS = ("low", "medium", "xhigh")


@dataclass(frozen=True)
class Reply:
    """One completion, with the fields this project measures pulled out."""

    content: str
    reasoning: str
    prompt_tokens: int
    completion_tokens: int
    predicted_per_second: float
    draft_n: int
    draft_accepted: int

    @property
    def acceptance(self) -> float | None:
        """Fraction of drafted tokens the target model kept, or None if no drafter.

        With `--spec-type draft-mtp` the server reports how many tokens the MTP
        head proposed and how many survived verification. Without a drafter
        `draft_n` is 0 and the ratio is undefined rather than zero.
        """
        if self.draft_n == 0:
            return None
        return self.draft_accepted / self.draft_n


def encode_image(path: str | Path) -> str:
    """Return a PNG as a data URL, which is what the chat API accepts inline."""
    data = base64.b64encode(Path(path).read_bytes()).decode("ascii")
    return f"data:image/png;base64,{data}"


class QwenClient:
    def __init__(self, base_url: str = "http://127.0.0.1:8080", timeout: float = 600.0):
        self.base_url = base_url.rstrip("/")
        self._client = httpx.Client(timeout=timeout)

    def close(self) -> None:
        self._client.close()

    def __enter__(self) -> "QwenClient":
        return self

    def __exit__(self, *exc: object) -> None:
        self.close()

    def props(self) -> dict:
        """Server-side properties, including the per-slot context actually granted."""
        r = self._client.get(f"{self.base_url}/props")
        r.raise_for_status()
        return r.json()

    def chat(
        self,
        prompt: str,
        *,
        effort: str | None = None,
        thinking: bool = True,
        max_tokens: int = 512,
        image: str | Path | None = None,
    ) -> Reply:
        """Send one chat turn and return the parsed reply.

        `effort` and `thinking` are routed through `chat_template_kwargs`. Passing
        `reasoning_effort` as a top-level field instead is accepted by the server
        and never reaches the template, so it silently does nothing.
        """
        if effort is not None and effort not in EFFORTS:
            raise ValueError(f"effort must be one of {EFFORTS}, got {effort!r}")

        content: object = prompt
        if image is not None:
            content = [
                {"type": "text", "text": prompt},
                {"type": "image_url", "image_url": {"url": encode_image(image)}},
            ]

        kwargs: dict[str, object] = {}
        if effort is not None:
            kwargs["reasoning_effort"] = effort
        if not thinking:
            kwargs["enable_thinking"] = False

        payload: dict[str, object] = {
            "messages": [{"role": "user", "content": content}],
            "max_tokens": max_tokens,
        }
        if kwargs:
            payload["chat_template_kwargs"] = kwargs

        r = self._client.post(f"{self.base_url}/v1/chat/completions", json=payload)
        r.raise_for_status()
        return _parse(r.json())


def _parse(body: dict) -> Reply:
    message = body["choices"][0]["message"]
    usage = body.get("usage", {})
    timings = body.get("timings", {})
    return Reply(
        content=message.get("content") or "",
        reasoning=message.get("reasoning_content") or "",
        prompt_tokens=usage.get("prompt_tokens", 0),
        completion_tokens=usage.get("completion_tokens", 0),
        predicted_per_second=timings.get("predicted_per_second", 0.0),
        draft_n=timings.get("draft_n", 0),
        draft_accepted=timings.get("draft_n_accepted", 0),
    )

Detailed breakdown

• Rejecting a bad effort locally is the point of the EFFORTS check. The server’s answer to high is a 500 carrying a Jinja stack trace, which is unpleasant to debug from inside a benchmark loop. Failing before the request leaves is cheaper and names the problem.
• The top-level reasoning_effort field is a silent no-op, which is why this client does not use it. Sending {"reasoning_effort": "high"} at the top level returns a normal answer, and so does {"reasoning_effort": "banana"}. Neither value reaches the template, which is provable precisely because the template rejects both when they arrive through chat_template_kwargs.
• acceptance returns None rather than 0.0 with no drafter. A server started without --spec-type reports draft_n = 0, and a zero there would read as “the drafter proposed tokens and none were kept” instead of “there is no drafter”.
• content and reasoning_content are coalesced with or "". A reasoning model can return content: null when the whole token budget went into the trace, and a bare body["choices"][0]["message"]["content"] then yields None where every caller expects a string.
• draft_n and draft_n_accepted are llama.cpp extensions. They live in the timings object, which OpenAI’s schema has no equivalent of, so an official SDK will drop them. That is the reason this project talks to the endpoint with httpx rather than the openai package.
• The 600-second timeout is deliberate. At xhigh effort with a 2,048-token budget, a single request on this hardware can run past a client library’s default of 5 or 10 seconds.

Step 12: Measure the reasoning-effort dial

The model card presents the three effort levels as a dial without saying what turning it costs. This module answers that by running one prompt at each level and reporting the size of the reasoning trace. It deliberately does not assume an ordering, which turns out to matter: Step 17 shows the levels landing out of order on a small sample.

Create the file

cd ~/src/qwen38-probe
touch src/qwen38_probe/effort.py

# ~/src/qwen38-probe/src/qwen38_probe/effort.py
"""Measure what each `reasoning_effort` level actually costs.

The model card presents xhigh / medium / low as a dial. It is a dial over the
*thinking* trace, not over the answer, and on an easy question the three levels
can land within noise of each other. This module runs the same prompt at each
level, repeated, and reports the reasoning tokens rather than asserting an
ordering up front.

`repeats` defaults to 8 for a reason: at temperature 1.0 a three-sample mean is
small enough that the low/medium/xhigh ordering can come out backwards, which was
observed on this prompt before the default was raised.
"""

from __future__ import annotations

import statistics
from dataclasses import dataclass

from .client import EFFORTS, QwenClient


@dataclass(frozen=True)
class EffortResult:
    effort: str
    reasoning_chars: list[int]
    completion_tokens: list[int]

    @property
    def mean_reasoning(self) -> float:
        return statistics.mean(self.reasoning_chars)

    @property
    def mean_completion(self) -> float:
        return statistics.mean(self.completion_tokens)


def sweep(
    client: QwenClient,
    prompt: str,
    *,
    repeats: int = 8,
    max_tokens: int = 2048,
) -> list[EffortResult]:
    """Run `prompt` at every supported effort level and collect trace sizes."""
    results: list[EffortResult] = []
    for effort in EFFORTS:
        chars: list[int] = []
        tokens: list[int] = []
        for _ in range(repeats):
            reply = client.chat(prompt, effort=effort, max_tokens=max_tokens)
            chars.append(len(reply.reasoning))
            tokens.append(reply.completion_tokens)
        results.append(EffortResult(effort, chars, tokens))
    return results


def format_table(results: list[EffortResult]) -> str:
    lines = [
        f"{'effort':<8} {'mean reasoning chars':>21} {'mean completion tokens':>23}",
        "-" * 54,
    ]
    for r in results:
        lines.append(f"{r.effort:<8} {r.mean_reasoning:>21.1f} {r.mean_completion:>23.1f}")
    return "\n".join(lines)

Detailed breakdown

• The trace is measured in characters, not tokens, because the API reports it that way. usage.completion_tokens covers the answer and the trace together, so it cannot isolate the trace. len(reply.reasoning) is a proxy, and it is consistent across effort levels, which is all a comparison needs.
• Both numbers are collected because they answer different questions. Reasoning length says how hard the model thought; completion tokens says what the request cost you. Step 17 shows a case where the first rises and the second does not.
• repeats defaults to 8, and it started at 3. Temperature 1.0 is the model card’s recommendation for thinking mode, so a single sample per level measures the sampler more than the dial. Three turned out to be too few as well: one three-sample run on the Step 17 prompt put low above medium and xhigh, reversing the effect entirely. Eight is where the ordering became stable on this prompt, which is a property of this prompt rather than a universal constant.
• Iterating EFFORTS rather than a local list means the client’s validation and this sweep can never disagree about which levels exist.
• max_tokens defaults to 2,048 so that xhigh is not truncated. A trace cut off by the budget makes every level look the same.

Step 13: Measure the MTP head

Step 8 showed the MTP flags with one sample each, which is enough to see the flags work and not enough to quote. This module fixes the output length, runs several prompts of different shapes, and reads the acceptance counters the server reports, so the speedup can be stated as a measurement rather than an impression.

Create the file

cd ~/src/qwen38-probe
touch src/qwen38_probe/bench.py

# ~/src/qwen38-probe/src/qwen38_probe/bench.py
"""Measure decode throughput and, when the MTP head is loaded, its acceptance rate.

Qwen3.8 ships a multi-token-prediction head that llama.cpp can use as a
speculative drafter (`--spec-type draft-mtp`). Because the head was trained
alongside the model rather than being a separate small model, its acceptance rate
is high enough that the speedup is worth measuring rather than assuming.
"""

from __future__ import annotations

import statistics
from dataclasses import dataclass

from .client import QwenClient

PROMPTS = (
    "Write a Python function that reverses a singly linked list. Code only.",
    "Explain what a GGUF file is, in one paragraph.",
    "List the first 12 prime numbers, comma separated.",
)


@dataclass(frozen=True)
class BenchResult:
    label: str
    tokens_per_second: list[float]
    draft_n: int
    draft_accepted: int

    @property
    def mean_tps(self) -> float:
        return statistics.mean(self.tokens_per_second)

    @property
    def acceptance(self) -> float | None:
        if self.draft_n == 0:
            return None
        return self.draft_accepted / self.draft_n


def run(
    client: QwenClient,
    *,
    label: str,
    max_tokens: int = 256,
    repeats: int = 2,
) -> BenchResult:
    """Decode `max_tokens` from each prompt and average the reported rate.

    Thinking is disabled so that the token budget is spent on the answer. That
    keeps the comparison between a drafted and an undrafted server honest: a
    reasoning trace of a different length would change the sample size.
    """
    rates: list[float] = []
    drafted = 0
    accepted = 0
    for _ in range(repeats):
        for prompt in PROMPTS:
            reply = client.chat(prompt, thinking=False, max_tokens=max_tokens)
            rates.append(reply.predicted_per_second)
            drafted += reply.draft_n
            accepted += reply.draft_accepted
    return BenchResult(label, rates, drafted, accepted)


def format_result(result: BenchResult) -> str:
    lines = [
        f"label            : {result.label}",
        f"samples          : {len(result.tokens_per_second)}",
        f"mean tok/s       : {result.mean_tps:.2f}",
        f"min/max tok/s    : {min(result.tokens_per_second):.2f} / {max(result.tokens_per_second):.2f}",
    ]
    if result.acceptance is None:
        lines.append("drafter          : not loaded (draft_n = 0)")
    else:
        lines.append(f"drafted tokens   : {result.draft_n}")
        lines.append(f"accepted tokens  : {result.draft_accepted}")
        lines.append(f"acceptance rate  : {result.acceptance * 100:.1f}%")
    return "\n".join(lines)

Detailed breakdown

• predicted_per_second comes from the server, not a stopwatch on the client. Timing the HTTP round trip would fold in prefill, queueing, and JSON handling, none of which speculative decoding touches.
• Thinking is off for the benchmark. With it on, the two configurations would generate traces of different lengths and the comparison would be between two different amounts of work.
• Three prompts of different shapes. Code, prose, and a list. Acceptance rates vary with how predictable the text is, and a code-only benchmark would flatter the drafter.
• min/max is printed alongside the mean because the drafted numbers are far more variable than the undrafted ones. Reporting a mean alone would hide that.
• Drafted and accepted counts are summed across all samples rather than averaged per request, which is what makes the acceptance rate a token-weighted figure instead of an average of ratios.

Step 14: Wire up a command line

The three modules so far have no entry point. This adds one, with a subcommand per measurement plus a props command that answers the question the server logs only at startup: how much context a request actually gets.

Create the file

cd ~/src/qwen38-probe
touch src/qwen38_probe/__main__.py

# ~/src/qwen38-probe/src/qwen38_probe/__main__.py
"""Command line for the probe: ask, sweep, bench, image, props."""

from __future__ import annotations

import argparse
import json
import sys

from .bench import format_result, run as run_bench
from .client import QwenClient
from .effort import format_table, sweep


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        prog="qwen38-probe",
        description="Probe a Qwen3.8-27B model served by llama.cpp.",
    )
    parser.add_argument(
        "--base-url",
        default="http://127.0.0.1:8080",
        help="llama-server base URL (default: %(default)s)",
    )
    sub = parser.add_subparsers(dest="command", required=True)

    p_ask = sub.add_parser("ask", help="Send one prompt and print the answer")
    p_ask.add_argument("prompt")
    p_ask.add_argument("-n", "--max-tokens", type=int, default=512)
    p_ask.add_argument("-e", "--effort", choices=("low", "medium", "xhigh"))
    p_ask.add_argument(
        "--no-thinking", action="store_true", help="Suppress the reasoning trace"
    )
    p_ask.add_argument("--show-reasoning", action="store_true")

    p_sweep = sub.add_parser("sweep", help="Compare reasoning trace size by effort")
    p_sweep.add_argument("prompt")
    p_sweep.add_argument("-n", "--max-tokens", type=int, default=2048)
    p_sweep.add_argument("-r", "--repeats", type=int, default=8)

    p_bench = sub.add_parser("bench", help="Measure decode throughput and MTP acceptance")
    p_bench.add_argument("--label", default="baseline")
    p_bench.add_argument("-n", "--max-tokens", type=int, default=256)
    p_bench.add_argument("-r", "--repeats", type=int, default=2)

    p_image = sub.add_parser("image", help="Describe a PNG through the vision encoder")
    p_image.add_argument("path")
    p_image.add_argument(
        "-p",
        "--prompt",
        default="How many bars are in this chart, what colour are they, "
        "and are they increasing or decreasing left to right?",
    )
    p_image.add_argument("-n", "--max-tokens", type=int, default=512)

    sub.add_parser("props", help="Print the context the server actually granted")
    return parser


def main(argv: list[str] | None = None) -> int:
    args = build_parser().parse_args(argv)

    with QwenClient(args.base_url) as client:
        if args.command == "ask":
            reply = client.chat(
                args.prompt,
                effort=args.effort,
                thinking=not args.no_thinking,
                max_tokens=args.max_tokens,
            )
            if args.show_reasoning and reply.reasoning:
                print(f"[reasoning, {len(reply.reasoning)} chars]\n{reply.reasoning}\n")
            print(reply.content)
            return 0

        if args.command == "sweep":
            results = sweep(
                client, args.prompt, repeats=args.repeats, max_tokens=args.max_tokens
            )
            print(format_table(results))
            return 0

        if args.command == "bench":
            result = run_bench(
                client,
                label=args.label,
                max_tokens=args.max_tokens,
                repeats=args.repeats,
            )
            print(format_result(result))
            return 0

        if args.command == "image":
            reply = client.chat(
                args.prompt,
                thinking=False,
                max_tokens=args.max_tokens,
                image=args.path,
            )
            print(reply.content)
            return 0

        if args.command == "props":
            props = client.props()
            # n_ctx is not a top-level key: it is reported inside
            # default_generation_settings, which is where the context a request
            # actually gets is recorded.
            settings = props.get("default_generation_settings", {})
            print(
                json.dumps(
                    {
                        "build_info": props.get("build_info"),
                        "n_ctx": settings.get("n_ctx"),
                        "total_slots": props.get("total_slots"),
                        "modalities": props.get("modalities"),
                        "model_path": props.get("model_path"),
                    },
                    indent=2,
                )
            )
            return 0

    return 1


if __name__ == "__main__":
    sys.exit(main())

Now the packaging. The readme field below names a file, and hatchling checks it exists at build time, so create both:

Create the file

cd ~/src/qwen38-probe
printf '# qwen38-probe\n\nMeasurement harness for Qwen3.8-27B on llama.cpp.\n' > README.md
touch pyproject.toml

# ~/src/qwen38-probe/pyproject.toml
[project]
name = "qwen38-probe"
version = "0.1.0"
description = "A client and measurement harness for Qwen3.8-27B served by llama.cpp"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "httpx>=0.28.1",
]

[project.scripts]
qwen38-probe = "qwen38_probe.__main__:main"

[dependency-groups]
dev = [
    "pytest>=9.1.1",
]

[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"

[tool.hatch.build.targets.wheel]
packages = ["src/qwen38_probe"]

Detailed breakdown

• props reads n_ctx out of default_generation_settings, not the top level. The top level of /props carries total_slots, model_path, build_info, modalities, and the entire chat template, but the context length is nested one level down. Looking for props["n_ctx"] returns nothing and gives no hint why.
• modalities is worth printing because it is the server’s own answer to “did the projector load”. It reports vision: true, video: true, audio: false when -mm was passed, and vision false when it was not.
• -e/--effort uses choices=, so argparse rejects high with a usage message before any network call. That duplicates the client’s check on purpose: one guards the library, the other guards the CLI.
• run as run_bench avoids shadowing. The module exports a function called run, and main already has enough locals named after what they do.
• main(argv=None) takes an argument list so the tests can drive the parser without touching sys.argv.
• requires-python = ">=3.12" matches the X | None syntax used in the type hints, which needs 3.10 at minimum; 3.12 is what this was validated on.
• README.md is not decoration, it is a build dependency. readme = "README.md" makes hatchling validate the path while building the wheel, so without the file uv sync in Step 16 fails with OSError: Readme file does not exist: README.md and every later step is unreachable. Either create it or drop the readme line; leaving the line and not the file is the one combination that does not work.

Step 15: Add the Makefile

The commands in this project are long, and two of them differ by exactly the two flags that turn speculative decoding on. Putting them behind named targets is what keeps the Step 17 comparison honest, since the two servers then differ only where they are supposed to.

Create the file

cd ~/src/qwen38-probe
touch Makefile

# ~/src/qwen38-probe/Makefile
.DEFAULT_GOAL := help

# Homebrew's llama.cpp already registers the `qwen35` architecture, so these are
# the Homebrew binaries. Point LLAMA_BIN at a source build if you have one.
LLAMA_BIN ?= $(shell dirname $$(command -v llama))

REPO      ?= ggml-org/Qwen3.8-27B-GGUF
SNAPSHOT  ?= $(shell find $(HOME)/.cache/huggingface/hub/models--ggml-org--Qwen3.8-27B-GGUF/snapshots -maxdepth 1 -mindepth 1 -type d 2>/dev/null | head -1)

MODEL     ?= $(SNAPSHOT)/Qwen3.8-27B-Q4_K_M.gguf
MMPROJ    ?= $(SNAPSHOT)/mmproj-Qwen3.8-27B-Q8_0.gguf
MTP       ?= $(SNAPSHOT)/mtp-Qwen3.8-27B-Q4_0.gguf

CTX       ?= 32768
PORT      ?= 8080
BASE      ?= http://127.0.0.1:$(PORT)
PROMPT    ?= What is 17 * 23? Reply with just the number.
TOKENS    ?= 512
IMAGE     ?= images/chart.png

.PHONY: help install download serve serve-mtp props ask sweep bench bench-mtp \
        chart image test test-live clean

help: ## Show this help screen
	@echo "Qwen3.8-27B on llama.cpp"
	@echo ""
	@echo "Targets:"
	@grep -E '^[a-zA-Z_-]+:.*?## ' $(MAKEFILE_LIST) \
		| awk 'BEGIN {FS = ":.*?## "}; {printf "  %-12s %s\n", $$1, $$2}'
	@echo ""
	@echo "Variables:"
	@echo "  LLAMA_BIN=$(LLAMA_BIN)"
	@echo "  SNAPSHOT=$(SNAPSHOT)"
	@echo "  CTX=$(CTX) PORT=$(PORT)"

install: ## Sync Python dependencies with uv
	uv sync

download: ## Fetch the model, the MTP head, and the vision projector
	$(LLAMA_BIN)/llama download -hf $(REPO):Q4_K_M --mtp
	$(LLAMA_BIN)/llama download -hf $(REPO) -hff mmproj-Qwen3.8-27B-Q8_0.gguf

serve: ## Run the server with vision, no speculative decoding
	$(LLAMA_BIN)/llama-server \
		-m  $(MODEL) \
		-mm $(MMPROJ) \
		-c $(CTX) --host 127.0.0.1 --port $(PORT) \
		--reasoning-preserve

serve-mtp: ## Run the server with the MTP head as a speculative drafter
	$(LLAMA_BIN)/llama-server \
		-m  $(MODEL) \
		-mm $(MMPROJ) \
		-md $(MTP) --spec-type draft-mtp \
		-c $(CTX) --host 127.0.0.1 --port $(PORT) \
		--reasoning-preserve

props: ## Show the context the server actually granted
	uv run qwen38-probe --base-url $(BASE) props

ask: ## Ask PROMPT and print the reply
	uv run qwen38-probe --base-url $(BASE) ask "$(PROMPT)" -n $(TOKENS)

sweep: ## Compare reasoning trace size at low / medium / xhigh
	uv run qwen38-probe --base-url $(BASE) sweep "$(PROMPT)" -n $(TOKENS)

bench: ## Measure decode throughput (run against `make serve`)
	uv run qwen38-probe --base-url $(BASE) bench --label baseline

bench-mtp: ## Measure throughput and acceptance (run against `make serve-mtp`)
	uv run qwen38-probe --base-url $(BASE) bench --label mtp

chart: ## Generate the test chart PNG (stdlib only)
	uv run python -m qwen38_probe.make_chart

image: chart ## Describe IMAGE through the vision encoder
	uv run qwen38-probe --base-url $(BASE) image $(IMAGE)

test: ## Run the unit tests (no server needed)
	uv run pytest -q

test-live: ## Run the tests including the one that needs a running server
	QWEN_LIVE_TEST=1 uv run pytest -q

clean: ## Remove Python build and test artifacts
	rm -rf .pytest_cache src/qwen38_probe/__pycache__ tests/__pycache__

Running make with no target prints the help screen:

cd ~/src/qwen38-probe
make

Qwen3.8-27B on llama.cpp

Targets:
  help         Show this help screen
  install      Sync Python dependencies with uv
  download     Fetch the model, the MTP head, and the vision projector
  serve        Run the server with vision, no speculative decoding
  serve-mtp    Run the server with the MTP head as a speculative drafter
  props        Show the context the server actually granted
  ask          Ask PROMPT and print the reply
  sweep        Compare reasoning trace size at low / medium / xhigh
  bench        Measure decode throughput (run against `make serve`)
  bench-mtp    Measure throughput and acceptance (run against `make serve-mtp`)
  chart        Generate the test chart PNG (stdlib only)
  image        Describe IMAGE through the vision encoder
  test         Run the unit tests (no server needed)
  test-live    Run the tests including the one that needs a running server
  clean        Remove Python build and test artifacts

Variables:
  LLAMA_BIN=/opt/homebrew/bin
  SNAPSHOT=/Users/you/.cache/huggingface/hub/models--ggml-org--Qwen3.8-27B-GGUF/snapshots/0669b98607d47046c7c2b3f801011d54a08cfccf
  CTX=32768 PORT=8080

Detailed breakdown

• .DEFAULT_GOAL := help is what makes a bare make explain itself instead of building the first target it finds, which here would be install.
• The help screen is generated from the ## comments, so a target added without a comment silently vanishes from the listing. That is the usual failure and the reason the .PHONY line lists every target explicitly.
• LLAMA_BIN is derived from command -v llama, so it follows a Homebrew install or a source build already on PATH without editing. The $$ escapes the shell’s $ through make’s own expansion.
• SNAPSHOT swallows errors with 2>/dev/null. Before the download runs, the directory does not exist, and find would otherwise print an error every time make parses the file, including for make help.
• serve and serve-mtp differ by two flags and are separate targets because the benchmark needs the server restarted between measurements, not reconfigured in place.
• image depends on chart so the PNG cannot be stale or missing when the vision request is sent.

Step 16: Test it

Most of what can break here breaks without a model: a null content field, an acceptance rate divided by zero drafts, an effort value the template will reject. Those are worth catching in under a second rather than after a 19 GiB load, so the suite runs without a server and gates the one test that needs one.

Create the file

cd ~/src/qwen38-probe
touch tests/test_client.py

# ~/src/qwen38-probe/tests/test_client.py
"""Unit tests for the parts that do not need a running server, plus one that does."""

from __future__ import annotations

import base64
import os

import pytest

from qwen38_probe.client import EFFORTS, QwenClient, Reply, _parse, encode_image
from qwen38_probe.make_chart import BARS, render, write


def _reply(**overrides) -> Reply:
    base = dict(
        content="391",
        reasoning="thinking",
        prompt_tokens=50,
        completion_tokens=21,
        predicted_per_second=51.1,
        draft_n=18,
        draft_accepted=16,
    )
    base.update(overrides)
    return Reply(**base)


class TestReply:
    def test_acceptance_is_the_kept_fraction(self):
        assert _reply().acceptance == pytest.approx(16 / 18)

    def test_acceptance_is_none_without_a_drafter(self):
        # A server started without --spec-type reports draft_n = 0. Returning 0.0
        # here would read as "the drafter is useless" rather than "absent".
        assert _reply(draft_n=0, draft_accepted=0).acceptance is None


class TestParse:
    def test_pulls_both_channels_apart(self):
        body = {
            "choices": [
                {"message": {"content": "Paris", "reasoning_content": "short trace"}}
            ],
            "usage": {"prompt_tokens": 50, "completion_tokens": 21},
            "timings": {
                "predicted_per_second": 51.13,
                "draft_n": 18,
                "draft_n_accepted": 16,
            },
        }
        reply = _parse(body)
        assert reply.content == "Paris"
        assert reply.reasoning == "short trace"
        assert reply.draft_accepted == 16

    def test_survives_a_null_content_field(self):
        # A reasoning model can return content: null when the whole budget went
        # into the trace, which would make a bare `message["content"]` a None.
        body = {
            "choices": [{"message": {"content": None, "reasoning_content": None}}],
            "usage": {},
            "timings": {},
        }
        reply = _parse(body)
        assert reply.content == ""
        assert reply.reasoning == ""
        assert reply.acceptance is None


class TestEffortValidation:
    def test_high_is_rejected_before_the_request_leaves(self):
        # "high" is the value every other reasoning model takes, and it is the one
        # value this template raises on. Failing locally beats a server 500.
        client = QwenClient("http://127.0.0.1:9")
        with pytest.raises(ValueError, match="high"):
            client.chat("hi", effort="high")
        client.close()

    def test_the_three_supported_levels(self):
        assert EFFORTS == ("low", "medium", "xhigh")


class TestImageEncoding:
    def test_data_url_round_trips(self, tmp_path):
        png = tmp_path / "chart.png"
        png.write_bytes(render())
        url = encode_image(png)
        assert url.startswith("data:image/png;base64,")
        assert base64.b64decode(url.split(",", 1)[1]) == png.read_bytes()


class TestChart:
    def test_is_a_png(self):
        assert render().startswith(b"\x89PNG\r\n\x1a\n")

    def test_bars_are_strictly_increasing(self):
        # The vision check asks the model whether the bars increase. That question
        # only has a right answer if the generator actually makes them increase.
        assert list(BARS) == sorted(BARS)
        assert len(set(BARS)) == len(BARS)

    def test_write_creates_the_parent_directory(self, tmp_path):
        target = write(tmp_path / "nested" / "chart.png")
        assert target.exists()
        assert target.stat().st_size > 0


@pytest.mark.skipif(
    os.environ.get("QWEN_LIVE_TEST") != "1",
    reason="needs a running llama-server; set QWEN_LIVE_TEST=1",
)
class TestLive:
    def test_the_model_answers_and_splits_the_channels(self):
        with QwenClient(os.environ.get("QWEN_BASE_URL", "http://127.0.0.1:8080")) as c:
            reply = c.chat("What is 17 * 23? Reply with just the number.", effort="low")
        assert "391" in reply.content
        assert reply.completion_tokens > 0

Install and run:

cd ~/src/qwen38-probe
uv sync
make test

..........s                                                              [100%]
10 passed, 1 skipped in 0.12s

With the Step 5 server running, the skipped test runs too:

make test-live

...........                                                              [100%]
11 passed in 1.50s

Detailed breakdown

• The suite runs in 0.12 seconds without a server, which is the point of separating _parse from the HTTP call. Ten of the eleven tests never open a socket.
• test_high_is_rejected_before_the_request_leaves points at port 9 on purpose. If validation ever regressed and the request were actually sent, the test would fail with a connection error rather than passing quietly.
• test_survives_a_null_content_field is the regression test for a real API behaviour, not a hypothetical. A reasoning model that spends its entire budget in the trace returns content: null, and the or "" in _parse is what keeps that from propagating a None into every caller.
• test_bars_are_strictly_increasing tests the fixture, not the code. It exists because the vision assertion in Step 7 is only meaningful if the image really shows what the prompt asks about.
• _parse is imported despite the leading underscore. It is the only piece of response handling worth testing in isolation, and testing it through a mocked httpx client would test the mock.
• The live test is skipped by default, gated on QWEN_LIVE_TEST=1, so make test stays useful on a machine with no model downloaded.

Step 17: Measure both servers

Start the server without the drafter and benchmark it:

cd ~/src/qwen38-probe
make serve        # terminal one
make bench        # terminal two

label            : baseline
samples          : 6
mean tok/s       : 25.92
min/max tok/s    : 25.80 / 26.17
drafter          : not loaded (draft_n = 0)

Restart with the MTP head and repeat:

make serve-mtp    # terminal one
make bench-mtp    # terminal two

label            : mtp
samples          : 6
mean tok/s       : 47.64
min/max tok/s    : 31.58 / 56.89
drafted tokens   : 525
accepted tokens  : 342
acceptance rate  : 65.1%

Six samples is not many for a figure this variable. Re-running both sides with -r 5 (15 samples each) gives:

baseline   mean 25.50 tok/s   min/max 24.51 / 26.17
mtp        mean 44.37 tok/s   min/max 27.41 / 55.73   acceptance 61.5% (885/1440)

And the effort dial, on a prompt hard enough to separate the levels:

make sweep PROMPT="A farmer has 17 sheep. All but 9 run away. He then buys 3 times as many sheep as he has left, and sells 11. How many sheep does he have? Show the arithmetic." TOKENS=2048

effort    mean reasoning chars  mean completion tokens
------------------------------------------------------
low                      380.1                   287.5
medium                   429.2                   325.8
xhigh                    485.6                   276.1

Detailed breakdown

• Call it 1.7x to 1.8x, not 1.84x. Six samples each gave 1.84x and fifteen gave 1.74x. The baseline barely moved between the two runs (25.92 to 25.50); the drafted side is what wanders, which is the same variability the min/max column shows within a single run. Quote the range, and re-measure on your own hardware before planning around either number.
• Acceptance lands around 61-65%, 342 of 525 on the first run and 885 of 1,440 on the longer one, and it is what explains the speedup. Rejected drafts are not free, so a drafter below roughly 30% acceptance can make decoding slower rather than faster.
• The baseline is far steadier than the drafted run: 25.80 to 26.17 against 31.58 to 56.89. Speculative decoding makes throughput depend on how predictable each particular passage is, so a run that lands on prose will look worse than one that lands on code. Quote the mean over several prompts, not a single sample.
• The effort dial is real but small next to the noise, and this is the number in the article that took the most work to trust. The table above is eight samples per level and shows about +28% from low to xhigh. Two earlier three-sample runs of the identical command gave +47% once and, the second time, low 389.3 / medium 373.7 / xhigh 381.0 — the ordering inverted outright. At temperature 1.0 a three-sample mean is not enough to see an effect this size. If you run this and get a flat or backwards result, raise -r before concluding the dial does nothing.
• Completion tokens do not track effort. They came out 287.5 / 325.8 / 276.1, with xhigh the lowest of the three. The dial acts on the thinking trace, and whatever it does to the length of the final answer is lost in the variance.
• All three levels got the puzzle right (9 sheep remain, plus 27 bought, minus 11 sold, so 25). On this prompt the extra reasoning bought confidence rather than correctness. That is a single prompt on one machine, not a claim about the effort levels in general.
• Restarting the server between the two benchmarks is required, not tidiness. Speculative decoding is set up when the model loads, so there is no way to toggle it per request.

Where to go next

• Give it the context it can use. -c 262144 costs 16 GiB of KV cache on top of the weights, per Step 6. On a 64 GB machine that fits comfortably and is the main reason to pick this model over a conventional 27B.
• Point an agent at it. The endpoint is OpenAI-compatible and the template supports tool calls, so anything speaking that protocol connects with a base-URL change. Route reasoning_effort through chat_template_kwargs or it will be ignored.
• Try the larger quantizations. The same repository has Q8_0 at 26.6 GiB and BF16 at 50.1 GiB. The Q8_0 MTP head is there too, and --mtp picks a matching one.
• Feed it video. The server reports "video": true, and --video takes a file the same way --image does. Nothing in this article exercises that path.

Validation

Every command, output, and figure in this article was executed on the machine described below. Nothing is quoted from the model card without being re-run.

Environment: macOS 26.5.2 (build 25F84), arm64, Apple M5 Max, 128 GB unified memory. llama.cpp b10330 (687e77892), installed with Homebrew. Model files from ggml-org/Qwen3.8-27B-GGUF at snapshot 0669b98607d47046c7c2b3f801011d54a08cfccf. Python 3.12, uv 0.11.26, httpx 0.28.1, pytest 9.1.1. Validated 2026-08-18.

Verified by running:

Not verified: the model card’s benchmark scores, its 1,000,000-token extended context via YaRN (llama.cpp is not among the frameworks the card documents for that), and video input. The throughput figures are six samples over three prompts on one machine and compare two configurations of the same build; they are not a general performance claim. The Q8_0 and BF16 weights are listed from the repository’s file sizes and were not downloaded or run.