Run Muse Glimmer-30B Locally with llama.cpp on macOS

Muse Glimmer is a 30-billion-parameter agentic model from Meta Superintelligence Lab, released in August 2026 with a perception encoder for image input and a speculative-decoding drafter. Quantized to roughly 4 bits it fits in about 17 GB, which puts a capable agentic model inside the memory budget of a single consumer machine.

This article installs a llama.cpp new enough to load it, pulls the three GGUF files, and runs the model four ways: a one-shot CLI answer, an OpenAI-compatible server, an image description, and a speculative-decoding run. It ends with a small Python project that measures decode throughput, because the numbers published for Apple Silicon were measured with a different runtime and do not transfer to this one.

Two things are worth knowing before you start, and both cost time if you meet them the hard way:

• Homebrew’s llama.cpp is too old. Muse Glimmer support landed in build b10353. The current Homebrew formula is b10330, which does not register the architecture at all. You need a source build.
• The advertised Apple Silicon speedup is not a llama.cpp number. The model card’s 1.8x figure for an M5 Max was measured with ExecuTorch. Measured through llama.cpp on the same class of machine, the drafter is worth about 3%. Step 16 shows the measurement.

Prerequisites

• macOS on Apple Silicon with at least 24 GB of unified memory. Written and validated on macOS 26.5.2, arm64, on an Apple M5 Max with 128 GB. The weights alone are about 17 GB and the working context adds a few more.
• About 20 GB of free disk. The three GGUF files total 18 GB, and Hugging Face’s cache keeps one copy.
• CMake and a C++ toolchain — Xcode command line tools (xcode-select --install) plus brew install cmake.
• uv 0.11.26 or newer for the Python project in Steps 10–16 — uv --version. Install with brew install uv.
• Git. Step 2 explains why the clone depth matters more than usual here.

No Hugging Face account or token is needed. The model is Apache 2.0 and publicly downloadable.

Step 1: Confirm Homebrew’s build is too old

If you already have llama.cpp from Homebrew, check what you have before doing anything else:

llama version

b10330-687e77892

brew info --json=v2 llama.cpp | python3 -c "import json,sys; print(json.load(sys.stdin)['formulae'][0]['versions']['stable'])"

10330

Detailed breakdown

• b10330 is below the b10353 floor, so this install cannot load Muse Glimmer. The failure is not subtle — builds at or before b10344 do not register the muse-glimmer architecture, so the model file is rejected at load rather than producing bad output.
• The second command checks the formula, not just your install. Running brew upgrade llama.cpp does not help while the stable formula itself is 10330. Confirming that saves an upgrade-and-retry cycle.
• llama update is not the way out either. The dispatcher ships an update subcommand, but on a Homebrew install it refuses: Updates are available only when installed from https://llama.app. Homebrew owns the files, so the self-updater stands aside.
• You can leave the Homebrew install in place. The source build in Step 2 goes in its own directory and is invoked by path, so the two do not collide. Nothing below depends on the Homebrew binaries.

Step 2: Clone llama.cpp with full history

git clone https://github.com/ggml-org/llama.cpp ~/src/llama.cpp
cd ~/src/llama.cpp
git checkout b10362

Confirm the checkout actually contains Muse Glimmer support:

grep -c LLM_ARCH_MUSE_GLIMMER src/llama-arch.cpp

1

Detailed breakdown

• Do not shallow-clone this repository. --depth is the usual reflex for a large repo, and here it quietly breaks the one check the model card tells you to run. llama.cpp derives its build number from git rev-list --count HEAD, so a shallow clone of the correct tag reports at most the clone depth, less however far the tag trails the default branch. A --depth 50 clone followed by git checkout b10362 reported b41 with master nine commits ahead, and b40 once master moved on again. Either reads as failing the >= 10353 requirement even though the source is right.
• grep is the reliable check, and the version string is not. The grep asks whether the architecture is registered in the source you are about to compile, which is the question that actually matters. 0 means your checkout predates Muse Glimmer support.
• b10362 is what this article was validated against. Any tag at b10353 or above works, and master works. The project tags most days, so pin whatever you build and record it.
• If you already made a shallow clone, git fetch --unshallow repairs the history — but see Step 3, because that alone does not fix the binary.

Step 3: Build with Metal

cd ~/src/llama.cpp
cmake -B build -DCMAKE_BUILD_TYPE=Release -DLLAMA_CURL=ON
cmake --build build --config Release -j $(sysctl -n hw.ncpu)

Check the build number the binary reports:

./build/bin/llama-cli --version

version: 10362 (4801e3c56)
built with AppleClang 21.0.0.21000101 for Darwin arm64

Detailed breakdown

• Metal is on by default on Apple Silicon. There is no flag to add. The model card’s -DGGML_CUDA=ON is for NVIDIA hardware; including it here does nothing useful.
• -DLLAMA_CURL=ON is what makes Step 4 work. It links the HTTPS downloader, so llama download and the -hf flag can fetch from Hugging Face directly. Without it you would fetch the weights by other means.
• The build number is computed at configure time, not build time. cmake/build-info.cmake runs git rev-list --count HEAD when you run cmake -B build. If you shallow-cloned and then ran git fetch --unshallow, rebuilding is not enough — the cached value survives and the binary keeps reporting the old number. Re-run the cmake -B build line to regenerate it. This was verified: after unshallowing, a plain rebuild still reported version: 41; re-running the configure step produced version: 10362.
• -j $(sysctl -n hw.ncpu) uses every core. The build takes a few minutes on an M-series machine.

Step 4: Download the three GGUF files

cd ~/src/llama.cpp
./build/bin/llama download -hf meta-models/Muse-Glimmer-30B-GGUF -hff muse-glimmer-30B-kquant-17gb.gguf
./build/bin/llama download -hf meta-models/Muse-Glimmer-30B-GGUF -hff mmproj-kquant.gguf
./build/bin/llama download -hf meta-models/Muse-Glimmer-30B-GGUF -hff dflash-kquant.gguf

Confirm all three landed:

ls -lh ~/.cache/huggingface/hub/models--meta-models--Muse-Glimmer-30B-GGUF/snapshots/*/

lrwxr-xr-x  1 you  staff    76B Aug 11 09:51 dflash-kquant.gguf -> ../../blobs/27d9a805fa29b943cfb6ad4843367cd4eaaaf06bd452d8cc3e00a2cd18a677bc
lrwxr-xr-x  1 you  staff    76B Aug 11 09:50 mmproj-kquant.gguf -> ../../blobs/f48b452316f9b213758e8659444029b961a24a07f99a1abb2a9f88b06f7c00c6
lrwxr-xr-x  1 you  staff    76B Aug 11 09:49 muse-glimmer-30B-kquant-17gb.gguf -> ../../blobs/7e9b74b7c8875e9e265695df9613bf6290f2392e479ce740495a129019c488d8

Detailed breakdown

• What each file is. muse-glimmer-30B-kquant-17gb.gguf (16.8 GB) is the text model and the only required file. mmproj-kquant.gguf (1.4 GB) is the perception encoder, required for image input and useless on its own. dflash-kquant.gguf (1.6 GB) is the speculative-decoding drafter, optional. Total on disk: 18 GB.
• -hff suppresses the automatic projector download. -hf normally fetches a matching mmproj alongside the model, and --mmproj-auto is on by default. The moment you pass -hff to name an exact file, that convenience stops applying — the first command above downloads only the 16.8 GB text model. This is why all three files are fetched explicitly rather than relying on the default.
• -hff is needed at all because the filenames are non-standard. -hf picks a file by quantization label and defaults to Q4_K_M. These files are named kquant-17gb and kquant-dynamic, which match no label, so the exact filename is the only reliable selector.
• The files land in the shared Hugging Face cache, not a llama.cpp-private directory, and the snapshot directory holds symlinks into blobs/. Set LLAMA_CACHE or HF_HUB_CACHE to relocate them.
• The other text build is a choice, not an upgrade. muse-glimmer-30B-kquant-dynamic.gguf (19.7 GB) targets 32 GB machines and the model card puts it at 0.2% quality degradation against 1.0% for the 17 GB build. Everything below works with either; swap the filename.

Step 5: A one-shot answer from the CLI

cd ~/src/llama.cpp
SNAP=$(find ~/.cache/huggingface/hub/models--meta-models--Muse-Glimmer-30B-GGUF/snapshots \
       -maxdepth 1 -mindepth 1 -type d | head -1)
./build/bin/llama-cli \
    -m "$SNAP"/muse-glimmer-30B-kquant-17gb.gguf \
    -c 32768 \
    --jinja \
    --temp 1.0 --top-p 0.95 --top-k 64 \
    -st -p "What is 17 * 23? Reply with just the number."

build      : b10362-4801e3c56
model      : /Users/you/.cache/huggingface/hub/models--meta-models--Muse-Glimmer-30B-GGUF/snapshots/a0532f7263ee67f1e0a5f5c5fdcd50dd62fc9aa4/muse-glimmer-30B-kquant-17gb.gguf
ftype      : Q4_K - Medium
modalities : text

> What is 17 * 23? Reply with just the number.

[Start thinking]

17 * 23 = 391

Reply with just the number.

Probably just "391". No extra text.
[End thinking]

391

[ Prompt: 153.0 t/s | Generation: 32.3 t/s ]

Detailed breakdown

• --jinja is not optional. The chat template ships inside the GGUF, and --jinja is what makes llama.cpp use it. There is no separate template file to pass. Without the flag the multimodal CLI in Step 8 aborts outright with this custom template is not supported, try using --jinja.
• -st makes it answer and exit. Short for --single-turn. Without it llama-cli prints the answer and then waits at a prompt for your next message, which in a script looks exactly like a hang.
• The sampling flags are the model card’s recommended settings — temperature 1.0, top-p 0.95, top-k 64. They are not llama.cpp defaults, so they have to be passed every time.
• SNAP is resolved with find, not a glob. The snapshot directory is named after a commit hash, so it cannot be typed from memory. Writing SNAP=.../snapshots/*/ looks tidier and breaks on macOS: zsh does not expand globs stored in a variable, so the path arrives at llama-cli with a literal * in it. (bash does expand it, which is what makes the mistake easy to ship.) The find form works in both shells.
• modalities : text confirms the projector was not loaded. That is correct here; Step 6 loads it for the server, and Step 8 needs it for image input.
• The thinking trace is inline, bracketed by [Start thinking] and [End thinking]. The CLI interleaves the model’s reasoning with its answer. If you want them as separate fields, that is the server’s job — Step 6.
• 32.3 tok/s is single-stream generation with nothing else loaded. Treat it as the ceiling for this machine; the server figures in Step 16 are lower because four slots are allocated.

Step 6: Serve an OpenAI-compatible endpoint

cd ~/src/llama.cpp
SNAP=$(find ~/.cache/huggingface/hub/models--meta-models--Muse-Glimmer-30B-GGUF/snapshots \
       -maxdepth 1 -mindepth 1 -type d | head -1)
./build/bin/llama-server \
    -m       "$SNAP"/muse-glimmer-30B-kquant-17gb.gguf \
    --mmproj "$SNAP"/mmproj-kquant.gguf \
    -a muse-glimmer-30B \
    -c 131072 -np 4 \
    --host 127.0.0.1 --port 8080 \
    --jinja \
    --temp 1.0 --top-p 0.95 --top-k 64 2>&1 | tee server.log

In a second terminal:

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"muse-glimmer-30B",
       "messages":[{"role":"user","content":"What is 17 * 23? Reply with just the number."}]}' \
| python3 -c "import json,sys; m=json.load(sys.stdin)['choices'][0]['message']; \
print('content  :', repr(m['content'])); print('reasoning:', len(m.get('reasoning_content') or ''), 'chars')"

content  : '391'
reasoning: 177 chars

Detailed breakdown

• The server splits the two channels for you. The answer arrives in content and the thinking in reasoning_content, instead of the interleaved trace the CLI printed. Any OpenAI-compatible client reading content gets a clean 391.
• The trace length moves between runs. Sampling is on at temperature 1.0, so 391 comes back every time but the reasoning is a different length on each request; four runs of this exact call spanned 177 to 317 characters. Every count and captured reply in this article is one observation, not a fixed value.
• -a muse-glimmer-30B sets the model id clients must send. Without it the id is derived from the filename, and requests naming muse-glimmer-30B would be rejected.
• --mmproj here costs nothing if you never send an image. Loading it adds 1.4 GB and enables image input on the endpoint. Drop the flag on a text-only deployment.
• If content comes back starting with to=self<|message|>, your build is too old. That is the raw channel marker leaking through because the chat parser does not recognise the format — the symptom of skipping Steps 1–3.

Step 7: Find out how much context a request actually gets

Step 6 piped the server through tee, so its startup banner is in server.log:

grep n_ctx_slot ~/src/llama.cpp/server.log

0.01.362.557 I srv    load_model: initializing, n_slots = 4, n_ctx_slot = 32768, kv_unified = 'false'

The running server will also tell you directly:

curl -s http://127.0.0.1:8080/slots \
  | python3 -c 'import json,sys; s=json.load(sys.stdin); print(len(s), "slots,", s[0]["n_ctx"], "tokens each")'

4 slots, 32768 tokens each

Detailed breakdown

• tee is why there is a log to grep. llama-server runs in the foreground and writes its banner to the terminal; without the 2>&1 | tee server.log in Step 6 there is no file, and this step fails with No such file or directory.
• /slots is the check that works on a server you did not start, which is the common case when the endpoint is someone else’s. It reports the live value rather than what the startup banner said.
• -c is a pool that -np divides. Step 6 asked for -c 131072 with -np 4, and each slot got 32,768 tokens. A single request is bounded by n_ctx_slot, not by the number you passed to -c.
• Running out of context is silent. Nothing errors. The request simply produces no answer, and content comes back empty. On a model that reasons at length this is easy to hit, and in an evaluation it reads as a wrong answer rather than as an infrastructure failure — a lower score with nothing in the logs to explain it.
• To give each slot the full trained context, scale -c with -np. -c 524288 -np 4 gives 131,072 per slot while staying 4-way concurrent. The KV cache stays affordable because the model uses grouped-query attention with 2 KV heads and sliding-window attention on three of every four layers. Measured on this machine, that pool costs 6,656 MiB across the 13 full-context layers plus 390 MiB for the sliding-window 39, against 2,054 MiB at the Step 6 setting. Without the sliding window the same request pool would want roughly 26 GiB.
• n_ctx_slot is the number to record when you report a benchmark or an eval configuration. -c on its own does not describe what any single request saw.

Step 8: Describe an image

Image input needs llama-mtmd-cli, which is a different binary from llama-cli. The image below is the chart Steps 10 and 11 generate, so either run those two steps first or point --image at a PNG you already have. The binary loads all 17 GB before it opens the file, then exits with Unable to open file /Users/you/src/glimmer-probe/images/chart.png: No such file or directory, so a missing image costs a full model load to discover.

cd ~/src/llama.cpp
SNAP=$(find ~/.cache/huggingface/hub/models--meta-models--Muse-Glimmer-30B-GGUF/snapshots \
       -maxdepth 1 -mindepth 1 -type d | head -1)
./build/bin/llama-mtmd-cli \
    -m       "$SNAP"/muse-glimmer-30B-kquant-17gb.gguf \
    --mmproj "$SNAP"/mmproj-kquant.gguf \
    -c 32768 \
    --jinja \
    --temp 1.0 --top-p 0.95 --top-k 64 \
    --image ~/src/glimmer-probe/images/chart.png \
    -p "Describe this image. How many bars, what colour, and what is the trend?"

Against the four-bar chart generated in Step 11:

mtmd batch encoding done in 365 ms

 to=self<|message|>Describe this image. How many bars, what colour, and what is the trend?

Four blue bars. Trend increasing from left to right.

<|eom|><|start|>assistant to=user<|message|>The picture is a very simple vertical bar chart.

* **Bars:** There are four rectangular bars sitting on a common horizontal baseline with a single vertical axis on the left.
* **Colour:** All four bars are the same solid bright/blue colour.
* **Trend / shape:** The bars get progressively taller from left to right. The first bar on the left is the shortest, the second is taller, the third is taller again, and the fourth on the right is the tallest.

Detailed breakdown

• The description is correct. Four bars, blue, monotonically increasing — which is exactly what Step 11’s generator draws. Using an image whose contents you generated yourself turns “did it read the chart” into a checkable question.
• llama-cli cannot do this. The multimodal path lives in llama-mtmd-cli, and the consolidated llama dispatcher does not expose it as a subcommand either — llama mtmd returns error: unknown command 'mtmd'. The legacy binary is the only route.
• The raw channel markers are visible, and that is expected here rather than a bug. to=self<|message|> opens the thinking channel and <|eom|><|start|>assistant to=user<|message|> switches to the reply. The CLI prints the stream as it comes; the server parses it into content and reasoning_content. Send images to the server endpoint if you want them separated.
• <|eom|> means end of message, not end of turn. If you write your own client or add custom stop strings, leave it alone — stopping on it truncates the turn and collapses parallel tool calling.
• 365 ms to encode the image is the perception encoder running once, before any text is generated. Larger images cost more; the model accepts up to 4,096 visual tokens per image.

Step 9: Reasoning strength is the knob that works

Muse Glimmer always thinks. The usual switches do not turn it off:

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"muse-glimmer-30B",
       "messages":[{"role":"user","content":"Name the capital of France."}],
       "reasoning_effort":"none","max_tokens":256}' \
| python3 -c "import json,sys; m=json.load(sys.stdin)['choices'][0]['message']; \
print('content:', repr(m['content'])); print('reasoning chars:', len(m.get('reasoning_content') or ''))"

content: 'Paris'
reasoning chars: 440

What does work is reasoning_strength, passed through to the model’s own template:

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"muse-glimmer-30B",
       "messages":[{"role":"user","content":"Name the capital of France."}],
       "chat_template_kwargs":{"reasoning_strength":"low"},"max_tokens":256}'

Detailed breakdown

• reasoning_effort: "none" produced 440 characters of reasoning anyway. The template opens the thinking channel unconditionally, so the OpenAI-standard knob has nothing to switch. --reasoning off on the server has the same non-effect.
• reasoning_strength is a template variable, not an API field, which is why it travels in chat_template_kwargs. It takes low, medium, high, or xhigh, and defaults to high.
• Set it server-wide with --chat-template-kwargs '{"reasoning_strength":"low"}' if every request should use the same budget.
• --reasoning-budget N is the hard cap when you need a token ceiling rather than a strength hint.

What the four strengths cost, asking the same word problem at each level with a 2,048-token ceiling. One sample per strength, so read the ordering rather than the exact figures:

• The scaling is real and monotonic, and the default is already near the top. Moving from high to low cut wall-clock time by 57% on this prompt.
• The returns flatten at the top end. high to xhigh bought 9% more reasoning for 17% more time, so xhigh is worth reserving for problems where the extra depth changes the answer rather than applying it by default.

Step 10: Create the companion project

The rest of the article uses a small Python project that measures the endpoint. Create it, starting with the .gitignore so no 17 GB weight file can ever be staged:

mkdir -p ~/src/glimmer-probe/src/glimmer_probe ~/src/glimmer-probe/tests
cd ~/src/glimmer-probe
touch .gitignore

# ~/src/glimmer-probe/.gitignore

# Python
__pycache__/
*.py[cod]
.venv/
*.egg-info/

# uv
.uv/

# pytest
.pytest_cache/
.coverage

# Models, projectors, drafters — never commit weights
*.gguf
models/

# Server logs and benchmark transcripts
server.log
bench/*.json

# Sample images fetched at run time
images/*.png
images/*.jpg

# Editor / OS
.DS_Store
.idea/
.vscode/

Detailed breakdown

• *.gguf is the entry that matters. The three model files total 18 GB. One accidental git add -A in the wrong directory is a very slow mistake to undo.
• images/*.png is ignored because the chart is generated, not authored. The script that draws it is committed instead, which keeps the repository small and the image reproducible.
• bench/*.json keeps measurement output out of git. Throughput numbers are specific to one machine on one day; the harness that produces them is the durable artifact.

Then initialize the project:

cd ~/src/glimmer-probe
uv init --package .
uv add httpx
uv add --dev pytest

Step 11: Draw a test image with the standard library

Create the file

touch ~/src/glimmer-probe/src/glimmer_probe/make_chart.py

# ~/src/glimmer-probe/src/glimmer_probe/make_chart.py

"""Generate the test image for the vision path, using only the standard library.

A self-generated chart is used deliberately: it is safe to redistribute, it is
reproducible byte-for-byte, and the ground truth is known, so "did the model read
the chart" is a question with a checkable answer rather than a judgement call.
"""

from __future__ import annotations

import struct
import zlib
from pathlib import Path

WIDTH, HEIGHT = 480, 320
BACKGROUND = (255, 255, 255)
AXIS = (40, 40, 40)
BAR = (37, 99, 235)  # a clearly-nameable blue

#: Bar heights in chart units. Strictly increasing, so a model that reads the
#: image at all should describe an upward trend.
BARS: tuple[int, ...] = (40, 90, 150, 210)

_MARGIN = 40
_BASELINE = HEIGHT - _MARGIN


def _canvas() -> list[list[tuple[int, int, int]]]:
    return [[BACKGROUND for _ in range(WIDTH)] for _ in range(HEIGHT)]


def _fill(
    pixels: list[list[tuple[int, int, int]]],
    x0: int,
    y0: int,
    x1: int,
    y1: int,
    color: tuple[int, int, int],
) -> None:
    """Fill an axis-aligned rectangle, clipped to the canvas."""
    for y in range(max(0, y0), min(HEIGHT, y1)):
        row = pixels[y]
        for x in range(max(0, x0), min(WIDTH, x1)):
            row[x] = color


def render() -> bytes:
    """Draw the chart and encode it as a PNG."""
    pixels = _canvas()

    # Axes: a two-pixel L along the left and bottom edges.
    _fill(pixels, _MARGIN, _MARGIN, _MARGIN + 2, _BASELINE + 2, AXIS)
    _fill(pixels, _MARGIN, _BASELINE, WIDTH - _MARGIN, _BASELINE + 2, AXIS)

    slot = (WIDTH - 2 * _MARGIN) // len(BARS)
    bar_width = slot // 2
    for index, height in enumerate(BARS):
        left = _MARGIN + index * slot + (slot - bar_width) // 2
        _fill(pixels, left, _BASELINE - height, left + bar_width, _BASELINE, BAR)

    return _encode_png(pixels)


def _encode_png(pixels: list[list[tuple[int, int, int]]]) -> bytes:
    """Encode RGB rows as a PNG, each scanline prefixed with filter type 0."""
    raw = bytearray()
    for row in pixels:
        raw.append(0)
        for r, g, b in row:
            raw += bytes((r, g, b))

    def chunk(tag: bytes, data: bytes) -> bytes:
        body = tag + data
        return struct.pack(">I", len(data)) + body + struct.pack(
            ">I", zlib.crc32(body) & 0xFFFFFFFF
        )

    header = struct.pack(">IIBBBBB", WIDTH, HEIGHT, 8, 2, 0, 0, 0)
    return (
        b"\x89PNG\r\n\x1a\n"
        + chunk(b"IHDR", header)
        + chunk(b"IDAT", zlib.compress(bytes(raw), 9))
        + chunk(b"IEND", b"")
    )


def write(path: Path) -> Path:
    """Write the chart to ``path``, creating parent directories as needed."""
    path.parent.mkdir(parents=True, exist_ok=True)
    path.write_bytes(render())
    return path


if __name__ == "__main__":
    written = write(Path("images/chart.png"))
    print(f"{written} ({written.stat().st_size} bytes)")

Run it:

cd ~/src/glimmer-probe
uv run python -m glimmer_probe.make_chart

images/chart.png (1803 bytes)

Detailed breakdown

• No image library is involved. zlib and struct are enough to write a valid PNG: an 8-bit RGB IHDR, a zlib-compressed IDAT where every scanline is prefixed with a filter-type byte of 0, and an empty IEND. Each chunk carries its own CRC32 over tag and payload. Adding Pillow for four rectangles would be the larger cost.
• The image is generated rather than downloaded, which matters for three reasons: it is safe to redistribute, it is byte-for-byte reproducible, and its ground truth is known. Testing vision against a photo you found online gives you no way to grade the answer.
• BARS is strictly increasing on purpose. “Four bars, blue, increasing” is a claim you can check against the constant, so Step 8’s output is verifiable rather than plausible.
• The filter-type byte is the part people get wrong. Every scanline in the raw stream needs a leading 0 before the RGB triples, or the PNG decodes as noise.

Step 12: Write the client

Create the file

touch ~/src/glimmer-probe/src/glimmer_probe/client.py

# ~/src/glimmer-probe/src/glimmer_probe/client.py

"""A thin client over llama.cpp's OpenAI-compatible endpoint, shaped for Muse Glimmer.

Muse Glimmer always opens a thinking channel, so an answer arrives split across two
fields: the visible reply in ``content`` and the trace in ``reasoning_content``. The
helpers here keep those separate rather than concatenating them, because the whole
point of the server route is that it does the splitting for you.
"""

from __future__ import annotations

from dataclasses import dataclass
from typing import Any, Iterator

import httpx

DEFAULT_BASE_URL = "http://127.0.0.1:8080"

#: The four reasoning strengths the chat template accepts, weakest first.
REASONING_STRENGTHS = ("low", "medium", "high", "xhigh")

#: Sampling defaults from the model card's Best Practices section.
DEFAULT_TEMPERATURE = 1.0
DEFAULT_TOP_P = 0.95
DEFAULT_TOP_K = 64


@dataclass(frozen=True)
class Answer:
    """One completion, with the thinking trace kept apart from the reply."""

    content: str
    reasoning: str
    completion_tokens: int
    prompt_tokens: int
    seconds: float

    @property
    def tokens_per_second(self) -> float:
        """Decode throughput, or 0.0 when nothing was generated."""
        if self.seconds <= 0 or self.completion_tokens <= 0:
            return 0.0
        return self.completion_tokens / self.seconds

    @property
    def is_empty(self) -> bool:
        """True when the model produced no visible reply.

        This is the failure mode that matters on a reasoning model: a generation that
        runs out of context returns an empty ``content`` and no error at all.
        """
        return not self.content.strip()


@dataclass(frozen=True)
class SlotInfo:
    """What a single server slot actually gets, as opposed to what ``-c`` requested."""

    count: int
    context_per_slot: int

    @property
    def total_context(self) -> int:
        """The pool ``-c`` asked for, reconstructed from the per-slot figure."""
        return self.count * self.context_per_slot


def build_payload(
    prompt: str,
    *,
    model: str = "muse-glimmer-30B",
    max_tokens: int = 512,
    reasoning_strength: str | None = None,
    temperature: float = DEFAULT_TEMPERATURE,
    top_p: float = DEFAULT_TOP_P,
    top_k: int = DEFAULT_TOP_K,
) -> dict[str, Any]:
    """Assemble a ``/v1/chat/completions`` request body.

    ``reasoning_strength`` rides in ``chat_template_kwargs`` because it is a variable
    the model's own Jinja template reads. It is not an OpenAI field, and the standard
    ``reasoning_effort`` knob does nothing on this model.
    """
    if reasoning_strength is not None and reasoning_strength not in REASONING_STRENGTHS:
        raise ValueError(
            f"reasoning_strength must be one of {REASONING_STRENGTHS}, "
            f"got {reasoning_strength!r}"
        )

    payload: dict[str, Any] = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": max_tokens,
        "temperature": temperature,
        "top_p": top_p,
        "top_k": top_k,
    }
    if reasoning_strength is not None:
        payload["chat_template_kwargs"] = {"reasoning_strength": reasoning_strength}
    return payload


def parse_answer(body: dict[str, Any], seconds: float) -> Answer:
    """Turn a completion response into an :class:`Answer`.

    Both ``content`` and ``reasoning_content`` are coerced from ``None`` to ``""``:
    llama.cpp omits or nulls whichever one is unused, and callers should not have to
    care which.
    """
    message = body["choices"][0]["message"]
    usage = body.get("usage") or {}
    return Answer(
        content=message.get("content") or "",
        reasoning=message.get("reasoning_content") or "",
        completion_tokens=int(usage.get("completion_tokens") or 0),
        prompt_tokens=int(usage.get("prompt_tokens") or 0),
        seconds=seconds,
    )


class GlimmerClient:
    """Talks to one ``llama-server`` instance."""

    def __init__(
        self,
        base_url: str = DEFAULT_BASE_URL,
        *,
        timeout: float = 600.0,
        api_key: str | None = None,
    ) -> None:
        self.base_url = base_url.rstrip("/")
        headers = {"Authorization": f"Bearer {api_key}"} if api_key else {}
        self._http = httpx.Client(timeout=timeout, headers=headers)

    def __enter__(self) -> GlimmerClient:
        return self

    def __exit__(self, *exc_info: object) -> None:
        self.close()

    def close(self) -> None:
        self._http.close()

    def ask(self, prompt: str, **kwargs: Any) -> Answer:
        """Send one prompt and time the round trip."""
        payload = build_payload(prompt, **kwargs)
        started = _now()
        response = self._http.post(
            f"{self.base_url}/v1/chat/completions", json=payload
        )
        response.raise_for_status()
        return parse_answer(response.json(), _now() - started)

    def slots(self) -> SlotInfo:
        """Report how many slots exist and how much context each one really has."""
        response = self._http.get(f"{self.base_url}/slots")
        response.raise_for_status()
        slots = response.json()
        return SlotInfo(count=len(slots), context_per_slot=int(slots[0]["n_ctx"]))

    def health(self) -> bool:
        """True when the server answers ``/health``."""
        try:
            return self._http.get(f"{self.base_url}/health").status_code == 200
        except httpx.HTTPError:
            return False


def _now() -> float:
    from time import perf_counter

    return perf_counter()


def sweep_strengths(
    client: GlimmerClient, prompt: str, **kwargs: Any
) -> Iterator[tuple[str, Answer]]:
    """Ask the same prompt at each reasoning strength, weakest first."""
    for strength in REASONING_STRENGTHS:
        yield strength, client.ask(prompt, reasoning_strength=strength, **kwargs)

Detailed breakdown

• build_payload puts reasoning_strength in chat_template_kwargs. It is a variable the model’s Jinja template reads, not an OpenAI field, which is why it cannot go at the top level next to temperature. Step 9 showed reasoning_effort being ignored; this is the knob that works.
• parse_answer coerces both channels from None to "". llama.cpp omits or nulls whichever field is unused, so a client that assumes a string crashes on the case it was written to handle.
• Answer.is_empty names the failure mode from Step 7. A generation that exhausts its slot returns an empty content and no error, so the only way to notice is to check for it explicitly.
• SlotInfo.total_context reconstructs what -c asked for by multiplying the per-slot figure back up, which makes the division visible rather than implied.
• The sampling defaults are module constants, so the model card’s recommended values live in one place instead of being retyped per call.

Step 13: Write the benchmark harness

Create the file

touch ~/src/glimmer-probe/src/glimmer_probe/bench.py

# ~/src/glimmer-probe/src/glimmer_probe/bench.py

"""Measure decode throughput against a running server.

The model card publishes Apple M5 Max figures measured with **ExecuTorch**, not
llama.cpp, so those numbers do not transfer to this stack. This module produces the
llama.cpp equivalent: run the same prompts against a server started with and without
the DFlash drafter, and compare.
"""

from __future__ import annotations

import statistics
from dataclasses import asdict, dataclass
from typing import Sequence

from glimmer_probe.client import Answer, GlimmerClient

#: Prompts chosen to force real decoding rather than a one-word reply.
DEFAULT_PROMPTS: tuple[str, ...] = (
    "Explain what a GGUF file is and how it differs from a safetensors checkpoint.",
    "Write a Python function that merges two sorted lists, then explain its complexity.",
    "Summarize the trade-offs between speculative decoding and a larger batch size.",
)


@dataclass(frozen=True)
class BenchResult:
    """Aggregate throughput over a prompt set."""

    label: str
    runs: int
    tokens: int
    seconds: float
    per_prompt_tps: tuple[float, ...]

    @property
    def mean_tps(self) -> float:
        """Mean of the per-prompt rates.

        Averaging the per-prompt rates rather than dividing totals keeps one long
        answer from dominating, which is what the model card's "average across a
        diverse prompt set" means.
        """
        if not self.per_prompt_tps:
            return 0.0
        return statistics.fmean(self.per_prompt_tps)

    @property
    def aggregate_tps(self) -> float:
        """Total tokens over total seconds."""
        if self.seconds <= 0:
            return 0.0
        return self.tokens / self.seconds

    def as_dict(self) -> dict[str, object]:
        return asdict(self) | {
            "mean_tps": round(self.mean_tps, 2),
            "aggregate_tps": round(self.aggregate_tps, 2),
        }


def summarize(label: str, answers: Sequence[Answer]) -> BenchResult:
    """Fold a list of timed answers into one result."""
    return BenchResult(
        label=label,
        runs=len(answers),
        tokens=sum(a.completion_tokens for a in answers),
        seconds=round(sum(a.seconds for a in answers), 3),
        per_prompt_tps=tuple(round(a.tokens_per_second, 2) for a in answers),
    )


def run_bench(
    client: GlimmerClient,
    label: str,
    *,
    prompts: Sequence[str] = DEFAULT_PROMPTS,
    max_tokens: int = 256,
    reasoning_strength: str = "low",
    warmup: bool = True,
) -> BenchResult:
    """Time each prompt in turn and summarize.

    A warmup request is sent first and discarded. Without it the first timed prompt
    absorbs model warmup and Metal shader compilation, which shows up as an
    artificially low rate for whichever configuration happens to run first.
    """
    if warmup:
        client.ask("Say OK.", max_tokens=16, reasoning_strength="low")

    answers = [
        client.ask(
            prompt, max_tokens=max_tokens, reasoning_strength=reasoning_strength
        )
        for prompt in prompts
    ]
    return summarize(label, answers)


def format_table(results: Sequence[BenchResult]) -> str:
    """Render results as a fixed-width table, fastest configuration last."""
    header = f"{'configuration':<24} {'runs':>5} {'tokens':>7} {'mean tok/s':>11}"
    lines = [header, "-" * len(header)]
    for result in results:
        lines.append(
            f"{result.label:<24} {result.runs:>5} {result.tokens:>7} "
            f"{result.mean_tps:>11.2f}"
        )
    if len(results) == 2:
        base, other = results
        if base.mean_tps > 0:
            lines.append("")
            lines.append(f"speedup: {other.mean_tps / base.mean_tps:.2f}x")
    return "\n".join(lines)

Detailed breakdown

• The warmup request is the reason this measures anything useful. The first request after a server start absorbs model warmup and Metal shader compilation. Step 16 shows what happens without it: readings about 3 tok/s high, in whichever configuration happened to run first.
• mean_tps averages the per-prompt rates rather than dividing totals, so one long answer cannot dominate the result. aggregate_tps keeps the other definition available, and the two agreeing is a weak check that nothing is skewed.
• reasoning_strength defaults to low here on purpose. The benchmark is measuring decode speed, and a long thinking trace at high spends most of the wall clock generating tokens nobody reads.
• format_table only prints a speedup for exactly two results, which keeps it honest: a speedup is a comparison between two configurations, not a property of a single run.

Step 14: Wire up a command line

Create the file

touch ~/src/glimmer-probe/src/glimmer_probe/__main__.py

# ~/src/glimmer-probe/src/glimmer_probe/__main__.py

"""Command-line entry point for the probe."""

from __future__ import annotations

import argparse
import json
import sys

from glimmer_probe.bench import DEFAULT_PROMPTS, format_table, run_bench
from glimmer_probe.client import (
    DEFAULT_BASE_URL,
    REASONING_STRENGTHS,
    GlimmerClient,
    sweep_strengths,
)


def build_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(
        prog="glimmer-probe",
        description="Query and measure Muse Glimmer served by llama.cpp.",
    )
    parser.add_argument(
        "--base-url", default=DEFAULT_BASE_URL, help=f"default: {DEFAULT_BASE_URL}"
    )
    sub = parser.add_subparsers(dest="command", required=True)

    ask = sub.add_parser("ask", help="Send one prompt and print both channels")
    ask.add_argument("prompt")
    ask.add_argument("-n", "--max-tokens", type=int, default=512)
    ask.add_argument("-r", "--reasoning-strength", choices=REASONING_STRENGTHS)
    ask.add_argument(
        "--show-reasoning",
        action="store_true",
        help="Print the thinking trace as well as the reply",
    )

    sub.add_parser("slots", help="Report the real per-slot context size")

    sweep = sub.add_parser(
        "sweep", help="Ask one prompt at every reasoning strength"
    )
    sweep.add_argument("prompt")
    sweep.add_argument("-n", "--max-tokens", type=int, default=512)

    bench = sub.add_parser("bench", help="Measure decode throughput")
    bench.add_argument(
        "--label", default="baseline", help="Name for this configuration"
    )
    bench.add_argument("-n", "--max-tokens", type=int, default=256)
    bench.add_argument(
        "--json", action="store_true", help="Emit JSON instead of a table"
    )

    return parser


def main(argv: list[str] | None = None) -> int:
    args = build_parser().parse_args(argv)

    with GlimmerClient(args.base_url) as client:
        if not client.health():
            print(
                f"no server on {args.base_url} — start one with: make serve",
                file=sys.stderr,
            )
            return 1

        if args.command == "ask":
            answer = client.ask(
                args.prompt,
                max_tokens=args.max_tokens,
                reasoning_strength=args.reasoning_strength,
            )
            if args.show_reasoning and answer.reasoning:
                print(f"--- reasoning ({len(answer.reasoning)} chars) ---")
                print(answer.reasoning)
                print("--- reply ---")
            print(answer.content)
            print(
                f"\n[{answer.completion_tokens} tokens, "
                f"{answer.tokens_per_second:.1f} tok/s, "
                f"{len(answer.reasoning)} chars of reasoning]",
                file=sys.stderr,
            )
            if answer.is_empty:
                print(
                    "warning: empty reply — the generation likely ran out of "
                    "context. Check the per-slot size with: glimmer-probe slots",
                    file=sys.stderr,
                )
                return 2
            return 0

        if args.command == "slots":
            info = client.slots()
            print(
                f"{info.count} slots x {info.context_per_slot} tokens "
                f"= {info.total_context} total"
            )
            return 0

        if args.command == "sweep":
            for strength, answer in sweep_strengths(
                client, args.prompt, max_tokens=args.max_tokens
            ):
                print(
                    f"{strength:<6} {len(answer.reasoning):>7} chars reasoning  "
                    f"{answer.completion_tokens:>4} tokens  "
                    f"{answer.seconds:>6.1f}s"
                )
            return 0

        if args.command == "bench":
            result = run_bench(client, args.label, max_tokens=args.max_tokens)
            if args.json:
                print(json.dumps(result.as_dict(), indent=2))
            else:
                print(format_table([result]))
            return 0

    return 1


if __name__ == "__main__":
    raise SystemExit(main())

uv init --package already wrote a [project.scripts] table pointing at glimmer_probe:main. Repoint that existing line at __main__ rather than adding a second table, which would be a duplicate key and fail to parse:

# ~/src/glimmer-probe/pyproject.toml

[project.scripts]
glimmer-probe = "glimmer_probe.__main__:main"

Detailed breakdown

• The health check runs before every subcommand, so a stopped server produces one clear line instead of a connection traceback. The hint names a make serve target this article never creates, so following the steps by hand, restart the server with the Step 6 command instead.
• ask exits 2 on an empty reply and says why. An empty content is the silent context-exhaustion case from Step 7, and a non-zero exit is what makes it visible to a script or a CI job.
• Token counts and rates go to stderr, the answer to stdout. That keeps glimmer-probe ask ... > answer.txt clean while the diagnostics stay on the terminal.
• sweep is what produced the Step 9 table, asking one prompt at each of the four strengths and reporting trace length against wall clock.

Step 15: Test it

Create the file

touch ~/src/glimmer-probe/tests/test_client.py

# ~/src/glimmer-probe/tests/test_client.py

"""Unit tests for the request builder and response parser.

Everything here runs without a server. The one test that needs a live Muse Glimmer
is guarded by ``GLIMMER_LIVE_TEST`` so ``make test`` stays fast and offline.
"""

from __future__ import annotations

import os

import pytest

from glimmer_probe.bench import format_table, summarize
from glimmer_probe.client import (
    DEFAULT_TOP_K,
    REASONING_STRENGTHS,
    Answer,
    GlimmerClient,
    SlotInfo,
    build_payload,
    parse_answer,
)


def test_payload_carries_sampling_defaults() -> None:
    payload = build_payload("hello")
    assert payload["temperature"] == 1.0
    assert payload["top_p"] == 0.95
    assert payload["top_k"] == DEFAULT_TOP_K


def test_payload_omits_template_kwargs_when_strength_unset() -> None:
    assert "chat_template_kwargs" not in build_payload("hello")


@pytest.mark.parametrize("strength", REASONING_STRENGTHS)
def test_reasoning_strength_rides_in_template_kwargs(strength: str) -> None:
    payload = build_payload("hello", reasoning_strength=strength)
    assert payload["chat_template_kwargs"] == {"reasoning_strength": strength}


def test_reasoning_strength_rejects_openai_style_value() -> None:
    """``reasoning_effort: none`` is the OpenAI knob, and it does nothing here."""
    with pytest.raises(ValueError, match="reasoning_strength must be one of"):
        build_payload("hello", reasoning_strength="none")


def test_parse_answer_splits_the_two_channels() -> None:
    answer = parse_answer(
        {
            "choices": [
                {
                    "message": {
                        "content": "391",
                        "reasoning_content": "17 * 23 = 391",
                    }
                }
            ],
            "usage": {"completion_tokens": 3, "prompt_tokens": 20},
        },
        seconds=0.5,
    )
    assert answer.content == "391"
    assert answer.reasoning == "17 * 23 = 391"
    assert answer.completion_tokens == 3
    assert answer.tokens_per_second == pytest.approx(6.0)


def test_parse_answer_coerces_null_reasoning_to_empty_string() -> None:
    answer = parse_answer(
        {"choices": [{"message": {"content": "hi", "reasoning_content": None}}]},
        seconds=1.0,
    )
    assert answer.reasoning == ""
    assert not answer.is_empty


def test_empty_content_is_flagged() -> None:
    """A context-exhausted generation returns an empty reply and no error."""
    answer = parse_answer(
        {"choices": [{"message": {"content": "", "reasoning_content": "..."}}]},
        seconds=1.0,
    )
    assert answer.is_empty


def test_tokens_per_second_is_zero_when_nothing_generated() -> None:
    assert Answer("", "", 0, 0, 1.0).tokens_per_second == 0.0


def test_slot_info_reconstructs_the_requested_pool() -> None:
    """``-c 131072 -np 4`` gives four slots of 32768, not four of 131072."""
    info = SlotInfo(count=4, context_per_slot=32768)
    assert info.total_context == 131072


def test_summarize_averages_per_prompt_rates() -> None:
    answers = [Answer("a", "", 100, 10, 5.0), Answer("b", "", 300, 10, 10.0)]
    result = summarize("baseline", answers)
    assert result.tokens == 400
    assert result.per_prompt_tps == (20.0, 30.0)
    assert result.mean_tps == pytest.approx(25.0)
    assert result.aggregate_tps == pytest.approx(400 / 15.0)


def test_format_table_reports_speedup_for_a_pair() -> None:
    base = summarize("baseline", [Answer("a", "", 100, 10, 10.0)])
    fast = summarize("dflash", [Answer("a", "", 200, 10, 10.0)])
    table = format_table([base, fast])
    assert "speedup: 2.00x" in table


@pytest.mark.skipif(
    not os.environ.get("GLIMMER_LIVE_TEST"),
    reason="set GLIMMER_LIVE_TEST=1 with a server running on :8080",
)
def test_live_server_answers_and_splits_channels() -> None:
    with GlimmerClient() as client:
        assert client.health()
        answer = client.ask(
            "What is 17 * 23? Reply with just the number.", max_tokens=256
        )
        assert "391" in answer.content
        assert not answer.is_empty

Run them:

cd ~/src/glimmer-probe
uv run pytest -q

..............s                                                          [100%]
14 passed, 1 skipped in 0.05s

With a server running, the skipped test runs too:

GLIMMER_LIVE_TEST=1 uv run pytest -q

...............                                                          [100%]
15 passed in 1.66s

Detailed breakdown

• Everything except one test runs with no server. Payload construction and response parsing are pure functions, so the suite stays fast and works offline; only the last test needs 17 GB of weights loaded.
• test_reasoning_strength_rejects_openai_style_value encodes the Step 9 finding. Passing "none" is the natural mistake for anyone coming from the OpenAI API, and failing loudly beats silently sending a value the template ignores.
• test_slot_info_reconstructs_the_requested_pool pins the arithmetic that Step 7 describes, so the 4 x 32,768 relationship is asserted rather than left in prose.
• test_empty_content_is_flagged covers the failure that produces no error. It is the only way the context-exhaustion case shows up in a test suite.
• The s in the first run is the skip, not a failure. GLIMMER_LIVE_TEST=1 turns it into the fifteenth pass.

Step 16: Measure the drafter honestly

With the server from Step 6 still running, take a baseline. The numbers below are settled readings — run the command a few times and use the ones that stop moving, for the reason in the breakdown:

cd ~/src/glimmer-probe
uv run glimmer-probe bench --label baseline

configuration             runs  tokens  mean tok/s
--------------------------------------------------
baseline                     3     768       26.94

Stop that server, then start one with the drafter added:

cd ~/src/llama.cpp
SNAP=$(find ~/.cache/huggingface/hub/models--meta-models--Muse-Glimmer-30B-GGUF/snapshots \
       -maxdepth 1 -mindepth 1 -type d | head -1)
./build/bin/llama-server \
    -m       "$SNAP"/muse-glimmer-30B-kquant-17gb.gguf \
    --mmproj "$SNAP"/mmproj-kquant.gguf \
    -md      "$SNAP"/dflash-kquant.gguf \
    -a muse-glimmer-30B \
    -c 131072 -np 4 \
    --host 127.0.0.1 --port 8080 \
    --jinja \
    --temp 1.0 --top-p 0.95 --top-k 64 2>&1 | tee server-draft.log

cd ~/src/glimmer-probe
uv run glimmer-probe bench --label dflash

configuration             runs  tokens  mean tok/s
--------------------------------------------------
dflash                       3     768       27.45

Repeating each four times, at 256 tokens per prompt, on an M5 Max:

Speedup: 1.03x.

Detailed breakdown

• The model card advertises 1.8x on an M5 Max, and that is not this number. The card is explicit in its footnote that the M4 and M5 measurements were taken with ExecuTorch, and that only the RTX 5090 row used llama.cpp. So the two figures do not contradict each other — but anyone reading the Apple row as a llama.cpp expectation will be disappointed, and the gap is large enough to change whether loading the drafter is worth 1.6 GB.
• Discard the first measurement after a server start. An early baseline sample came in at 29.99 tok/s against a settled figure of about 26.9, and an early drafter sample at 30.25 against a settled 27.7. Both were warmup artifacts. The harness sends and discards one request before timing, and even that is not always enough — take several samples and look at the spread rather than trusting one run.
• A single number here would have been wrong in either direction. Comparing the first baseline sample (27.48) against the first drafter sample (30.25) suggests 1.10x; comparing settled figures gives 1.03x. The settled comparison is the honest one because both configurations were measured in the same thermal state.
• [spec] failed to measure draft model memory: failed to create llama_context from model appears at startup and is harmless. The drafter loads and serves normally afterwards — common_speculative_init_result: loading draft model a second later confirms it.
• Prompt processing is unaffected by the drafter. Speculative decoding targets token generation, so a workload dominated by long prompts and short answers gains nothing at all.
• Skip the drafter on this stack for now. 3% does not pay for 1.6 GB of memory and a second model to keep in step, unless memory is abundant. Re-measure when llama.cpp’s block-diffusion support matures — DFlash proposes 16-token blocks in one forward pass, and the generic speculative path is not built around that shape.

Where to go next

• Give each slot the context it needs. -c 524288 -np 4 if you are running evaluations, so a long reasoning trace cannot silently exhaust a slot.
• Point an agent at it. The endpoint is OpenAI-compatible, so anything that speaks that protocol connects with a base-URL change.
• Try the dynamic build if you have 32 GB. The model card puts it at 0.2% degradation against 1.0% for the 17 GB build, for 3 GB more memory.
• Re-check the Homebrew formula. Once it passes b10353, Steps 2 and 3 reduce to brew install llama.cpp and the source build stops being necessary.

Validation

Every command, output, and figure in this article was executed on the machine described below. Nothing is quoted from the model card without being re-run.

Environment: macOS 26.5.2 (build 25F84), arm64, Apple M5 Max, 128 GB unified memory. llama.cpp b10362 (4801e3c56), built from source with AppleClang 21.0.0. Model files from meta-models/Muse-Glimmer-30B-GGUF at snapshot a0532f7263ee67f1e0a5f5c5fdcd50dd62fc9aa4. Python 3.12, uv 0.11.26, pytest 9.1.1. Validated 2026-08-11.

Verified by running:

Not verified: the muse-glimmer-30B-kquant-dynamic.gguf build’s quality figures. The file itself was downloaded and run here (19.7 GB on disk, 2.7 GiB more resident memory than the 17 GB build in the same server configuration), but the 0.2% and 1.0% degradation numbers come from 15 benchmarks on the model card and cannot be checked from one machine. The throughput benchmark covers three prompts at 256 tokens on one machine; it is a comparison between two configurations, not a general performance claim.