llama.cpp runs large language models directly on your machine, with no Python runtime, no server process you did not start, and no account. On Apple Silicon it uses Metal for GPU work and the unified memory architecture means a model does not have to be copied to a separate card before it can run.

This article installs llama.cpp with Homebrew, pulls a model from Hugging Face, runs it two ways, and shows exactly where the weights land on disk. It closes with a small Python project that reads a GGUF file’s header with nothing but the standard library, so the model file stops being an opaque blob you downloaded and becomes something you can inspect and assert against.

Two things worth knowing before you start, because most of the llama.cpp tutorials currently online predate them:

  • The binaries were consolidated. Older material calls llama-cli and llama-server. Current builds dispatch through a single llama command with subcommands: llama cli, llama serve, llama bench.
  • GPU offload is automatic. The advice to “pass -ngl 99 to use the GPU” is stale. -ngl now defaults to auto, and the flag’s job today is mostly to let you turn offload off for comparison.

Prerequisites

  • macOS on Apple Silicon. Written and validated on macOS 26.5.2, arm64. Intel Macs work but have no Metal offload worth measuring, so the timings in Step 6 will not reproduce.
  • Homebrew 6.0.15 or newer — brew --version.
  • uv 0.11.26 or newer for the Python project in Steps 7–11 — uv --version. Install with brew install uv.
  • About 2 GB of free disk. The model used here is 795 MiB at Q8_0, and Hugging Face’s cache keeps one copy.
  • llama.cpp b10330 or newer. The project tags a release most days; pin the build number you actually ran, because flag names move.

No Hugging Face account or token is needed. Every model in this article is publicly downloadable.

Step 1: Install llama.cpp

brew install llama.cpp

Verify the install:

llama version

b10330-687e77892

Detailed breakdown

  • That one line is the whole version output: the build number and the commit it was built from. Older material shows a two-line version: … / built with … banner, which was the standalone llama-cli format; the dispatcher prints this instead.
  • The formula pulls four dependencies: ggml, openssl@3, ca-certificates, and libomp. ggml is the tensor library underneath llama.cpp, split into its own formula so other tools can share it. openssl@3 and ca-certificates are there because llama.cpp downloads models over HTTPS itself in Step 3; without TLS the -hf flag would not work. libomp provides OpenMP for CPU threading.
  • The build number (10330) is the upstream release tag, not a semantic version. There is no 1.x to reason about, so record the number. When a command in this article stops matching what you see, comparing build numbers is the fastest way to find out why.
  • Homebrew ships a bottle, so this is a download rather than a compile. If you want a source build with different flags, brew install --HEAD llama.cpp compiles from the tip of master instead.

Step 2: One binary, several subcommands

Run llama with no arguments:

llama

Usage: llama <command> [options]

Available commands:
  serve           HTTP API server
  cli             Command-line interactive interface
  download        Download a model
  version         Show version
  licenses        Show third-party licenses
  help            Show available commands

Run 'llama help all' to show additional commands.
Run 'llama <command> --help' for command-specific usage.

Several more subcommands exist but are hidden from the default listing:

llama help all

That adds update, completion, bench, batched-bench, fit-params, quantize, and perplexity.

Detailed breakdown

  • The subcommands you will use most. cli is the interactive chat loop. serve starts an OpenAI-compatible HTTP server. completion is the one to reach for in scripts and tests — but only with -no-cnv, which is what makes it read a prompt, write an answer, and exit. Step 3 covers why.
  • Aliases are real but no longer advertised. The listing prints one name per command, yet llama server, llama client, llama get, and llama credits all still resolve to serve, cli, download, and licenses. If you are adapting a script written against older material, server still works.
  • Hidden does not mean unsupported. bench, quantize, and perplexity are fully supported; they are hidden to keep the top-level help short for people who only want to chat with a model. Each takes --help of its own, and llama bench --help is a different flag set from llama cli --help.
  • On older tutorials. If you find a command like llama-cli -m model.gguf, translate it to llama cli -m model.gguf. The flags after the subcommand have mostly survived the consolidation; the binary name is the part that changed. Some builds also install standalone llama-<tool> executables alongside the dispatcher, but the subcommand form is the one that is always present.

Step 3: Run a model from Hugging Face

The -hf flag takes a Hugging Face repository and handles the download itself:

llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF:Q8_0 -rea off

The first run downloads the weights and then drops into an interactive prompt. Type a question, press Return, and press Ctrl-C when you want out.

-rea off is not optional here, and leaving it out is the single most likely way for this article to waste your afternoon. Qwen3.5-0.8B is a reasoning model: it opens a thinking block before answering, and at 0.8B parameters it is not strong enough to close one. Ask it mary had a without -rea off and it argues with itself forever —

*   Wait, let me look closer at the prompt: "mary had a". It's missing a word.
*   Wait, is it asking for the completion? Or is it a prompt to generate content?
*   No, that's not it.
*   Wait, there's a specific line: "Mary had a little lamb".

— and never returns your prompt, because nothing stops it. This build ships --repeat-penalty 1.00, which means repetition penalties are disabled, so the loop has no brake. -rea off skips the thinking block entirely and the same prompt answers immediately and terminates.

If you want to keep thinking enabled, --repeat-penalty 1.1 --dry-multiplier 0.8 breaks the verbatim loop but the model still circles for hundreds of tokens, and -n 500 only caps the damage. The real answer at this size is to turn reasoning off; a 4B or 8B model handles a thinking loop far better if you want one.

For anything scripted, use completion with -no-cnv, which prints one answer and exits:

llama completion -hf ggml-org/Qwen3.5-0.8B-GGUF:Q8_0 \
  -p "In one sentence, what is a GGUF file?" \
  -n 64 -no-cnv

Without -no-cnv that command answers and then sits at a > prompt waiting for your next turn, which is a hang in any script or Makefile target.

Getting just the answer

That command works, but “works” and “usable in a pipeline” are different things. It writes 38 lines of log to stderr, echoes your prompt back before answering, and — because this is a reasoning model — often opens a <think> block first.

How you avoid the thinking depends on what you are asking for, and the difference is bigger than it looks.

Continuing text is reliable. completion is a text-continuation tool, and a prompt that reads like the start of a sentence gets treated as one. Across five runs of this, a <think> block appeared zero times:

llama completion -hf ggml-org/Qwen3.5-0.8B-GGUF:Q8_0 \
  -p "A GGUF file is" \
  -n 48 -no-cnv --no-display-prompt -lv 0 \
  | sed -e 's/ *\[end of text\]//' -e '/^[[:space:]]*$/d'

 a binary format used for the conversion of large text files into a smaller size.

Asking a question is not reliable, on this model. Phrase the prompt as a question and the model reaches for its chat behavior even with -no-cnv — a <think> block showed up in three of five runs. Going the other way and asking properly through the chat template with reasoning disabled is the honest version:

llama completion -hf ggml-org/Qwen3.5-0.8B-GGUF:Q8_0 \
  -p "In one sentence, what is a GGUF file?" \
  -n 64 -st -rea off --no-display-prompt -lv 0 \
  | awk '/<think>/ { t = 1; next } /<\/think>/ { t = 0; next } !t { gsub(/ *\[end of text\]/, ""); if ($0 ~ /[^[:space:]]/) { print; got = 1 } } END { if (!got) { print "no answer: the model never finished thinking. Retry, or use a larger model." > "/dev/stderr"; exit 1 } }'

A GGUF file is a compressed format used to store large models for fast inference.

…when it answers at all. In eight runs of that command, five printed no answer, because the model spent all 64 tokens thinking and never emitted a closing </think>. That is the same defect Step 3 warns about, and -rea off reduces it without eliminating it. Raising -n does not help — at -n 200 and -n 400 the model simply thinks for longer.

Detailed breakdown

  • -lv 0 sets the log verbosity threshold to zero, which silences all 38 stderr lines — backend init, the unused tensor warnings, load timings, and the common_perf_print block — without touching the generated text.
  • Do not reach for --log-disable here. It looks like the obvious flag and it suppresses the model’s output along with the logs, so you get an empty result and a zero exit code. -lv 0 is the one that keeps the answer.
  • --no-display-prompt stops the prompt being echoed before the completion. The default is to print it, which is helpful interactively and noise everywhere else.
  • -st -rea off replaces -no-cnv for the question form. -rea off only works through the chat template, and -no-cnv discards the template — so the two do not combine. -st (--single-turn) keeps the template, runs exactly one turn, and exits.
  • The filter reports failure instead of hiding it, and that is the whole point. The obvious version of this is sed '/<think>/,/<\/think>/d', and it is a trap: sed ranges run to end of file when the closing pattern never arrives, so on every run where the model does not finish thinking it deletes the entire output and prints nothing. A command that silently produces empty output on more than half its runs is worse than one that is merely noisy. The awk version drops the block the same way, strips the trailing [end of text], removes the blank lines left behind, and — if nothing survived — writes a diagnosis to stderr and exits 1.
  • No flag will remove the tags for you. Even with -rea off the model emits the <think> pair, and --reasoning-format only changes how thoughts are reported through the server API, not whether they appear in CLI output. --reasoning-budget 0 sounds like the answer and still left one run in five unclosed.
  • If you need dependable one-shot answers, this is the wrong model. 0.8B is too small to run a reasoning loop, which is the same conclusion Step 3 reaches from the other direction. Use the continuation form here, or move to a 4B or 8B model where the question form actually returns.

Step 4: Find where the model landed

llama.cpp does not invent its own model directory. It writes into the standard Hugging Face hub cache:

ls ~/.cache/huggingface/hub/

CACHEDIR.TAG
models--ggml-org--Qwen3.5-0.8B-GGUF

Look inside one:

find ~/.cache/huggingface/hub/models--ggml-org--Qwen3.5-0.8B-GGUF -maxdepth 2

.../models--ggml-org--Qwen3.5-0.8B-GGUF
.../models--ggml-org--Qwen3.5-0.8B-GGUF/blobs
.../models--ggml-org--Qwen3.5-0.8B-GGUF/snapshots
.../models--ggml-org--Qwen3.5-0.8B-GGUF/refs

And check what it cost you:

du -sh ~/.cache/huggingface/hub/*

Detailed breakdown

  • The layout is Hugging Face’s, not llama.cpp’s. A repository user/name becomes the directory models--user--name. Actual file contents live in blobs/ keyed by object id; snapshots/<commit>/ holds human-named symlinks pointing at those blobs. That indirection is why two revisions of a repo can share unchanged files instead of storing them twice.
  • The practical consequence is that the cache is shared. If you already use Python’s huggingface_hub, llama.cpp reads and writes the same directory. A model pulled by one is visible to the other, and clearing the cache affects both.
  • Six environment variables can move it, checked in this order: LLAMA_CACHE, HF_HUB_CACHE, HUGGINGFACE_HUB_CACHE, then HF_HOME (with hub appended), then XDG_CACHE_HOME (with huggingface/hub appended), and finally $HOME/.cache/huggingface/hub. LLAMA_CACHE wins over all of them, which is the one to set if you want llama.cpp’s models on an external disk without moving anyone else’s.
  • To delete a model, delete its models--… directory. There is no llama uninstall. Removing the directory is the supported way to reclaim the space, and the next -hf for that repo re-downloads it.

Step 5: Download without running

llama download fetches weights and stops, which is useful when you want the network work to happen before a demo rather than during it:

llama download -hf ggml-org/Qwen3.5-0.8B-GGUF:Q8_0

The -hf flag is required here. llama download <repo> with the repository as a bare positional argument fails with error: invalid argument.

Afterwards, llama completion -hf ggml-org/Qwen3.5-0.8B-GGUF:Q8_0 … starts immediately, because the file is already in the cache.

You can also skip -hf entirely and point at a file you have on disk:

llama completion -m /path/to/model.gguf -p "Hello" -n 32 -no-cnv

Detailed breakdown

  • -hf and -m are alternatives: the first resolves a repository through the cache, the second takes a filesystem path. -hf is really “download if needed, then set -m to the cached path.”
  • Separating the download matters for anything timed. A benchmark that includes a 900 MB download in its first iteration produces a meaningless first number.
  • -m is also how you run a model that was never on Hugging Face, including one you quantized yourself.

Step 6: Control GPU offload

By default llama.cpp offloads as much of the model to the GPU as fits. The -ngl flag overrides that:

# CPU only — no Metal offload
time llama completion -hf ggml-org/Qwen3.5-0.8B-GGUF:Q8_0 \
  -p "Count from one to ten." -n 64 -ngl 0 -no-cnv

# Every layer on the GPU
time llama completion -hf ggml-org/Qwen3.5-0.8B-GGUF:Q8_0 \
  -p "Count from one to ten." -n 64 -ngl all -no-cnv

Detailed breakdown

  • -ngl accepts an exact layer count, auto, or all, and its long forms are --gpu-layers and --n-gpu-layers. The default is auto.
  • auto is why the old advice is obsolete. Tutorials that tell you to pass -ngl 99 were written when the default was zero and you had to opt in. Passing a large number today is harmless but redundant.
  • The interesting direction is now downward. -ngl 0 forces CPU-only execution, which is the honest baseline to compare against and a useful diagnostic: if a model produces correct output at -ngl 0 and garbage at auto, you have a backend problem, not a model problem.
  • Expect the gap to be smaller than you think on a model this size. A 0.8B model at Q8_0 is small enough that CPU inference on Apple Silicon is already quick, and a short 64-token generation is dominated by startup. The offload difference typically widens with model size and prompt length. Measure on your own hardware rather than trusting a number from an article; llama bench exists for exactly this and is the subject of its own piece.
  • Unified memory changes the trade-off. On a discrete GPU, offload is bounded by VRAM and exceeding it is catastrophic. On Apple Silicon the GPU addresses the same memory as the CPU, so auto has far more room and the failure mode is gradual rather than a hard wall.

Step 7: Scaffold the inspection project

The rest of the article builds a small tool that reads a GGUF file’s header. The point is not that you need it to run a model — you do not — but that it turns the model file into something you can assert against, which is what makes the setup testable.

Create the .gitignore before anything else, so no generated file is ever untracked-by-accident:

Create the files

mkdir -p ~/projects/gguf-inspector
cd ~/projects/gguf-inspector
touch .gitignore

Add the code: .gitignore

# Python
__pycache__/
*.py[cod]
.venv/
*.egg-info/

# uv
.uv/

# pytest
.pytest_cache/
.coverage

# Models — never commit weights
*.gguf
models/

# Editor / OS
.DS_Store
.idea/
.vscode/

Detailed breakdown

  • *.gguf and models/ are the entries that matter here. A quantized model is hundreds of megabytes to tens of gigabytes. Committing one, even once, is difficult to undo because git keeps it in history. The pattern goes in before the first git add, not after.
  • .venv/ is excluded because uv creates the environment locally and it is rebuilt from pyproject.toml plus the lockfile. The lockfile is what belongs in version control.
  • .pytest_cache/ and __pycache__/ are regenerated on every run and carry no information worth sharing.

Step 8: Initialize with uv

Create the file

cd ~/projects/gguf-inspector
uv init --package --name gguf-inspector
uv add --dev pytest

Add the code: pyproject.toml (generated, shown for reference)

[project]
name = "gguf-inspector"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = []

[build-system]
requires = ["uv_build>=0.11.26,<0.12.0"]
build-backend = "uv_build"

[dependency-groups]
dev = [
    "pytest>=9.1.1",
]

Detailed breakdown

  • --package is what makes python -m gguf_inspector work. A plain uv init creates an application: no [build-system], nothing installed into the environment, and src/ never on the import path — so the tests and the CLI both die with ModuleNotFoundError: No module named 'gguf_inspector'. With --package, uv writes the build backend shown above, creates src/gguf_inspector/, and uv sync installs the project so the module resolves.
  • Two things to delete from what uv generates. It adds a [project.scripts] entry pointing at gguf_inspector:main, which will not exist once Step 9 replaces __init__.py — remove that block. It also fills in authors from your git config; that is your name and email, so decide deliberately whether it belongs in a file you publish.
  • dependencies is empty and stays empty. The parser uses struct, pathlib, dataclasses, and argparse, all standard library. There is a gguf package on PyPI that does this and more; the point of writing it by hand is that the format is small enough to understand in one sitting, and a dependency-free reader is easy to drop into a test suite or a CI check.
  • uv add --dev pytest puts pytest in the dev dependency group, so it is available for uv run pytest but not part of what the project would ship.
  • uv init --package also writes a README.md and a stub src/gguf_inspector/__init__.py containing a main() function. Step 9 empties that file. Unlike a plain uv init, there is no top-level main.py to remove.

Step 9: Write the GGUF header parser

Create the file

# src/gguf_inspector/ already exists from `uv init --package`; empty the stub
# __init__.py it wrote, since the entry point lives in __main__.py instead.
: > src/gguf_inspector/__init__.py
touch src/gguf_inspector/reader.py

Add the code: src/gguf_inspector/reader.py

"""Read GGUF file headers and metadata using only the standard library.

The format is documented at https://github.com/ggml-org/ggml/blob/master/docs/gguf.md
Every integer is little-endian by default.
"""

from __future__ import annotations

import struct
from dataclasses import dataclass, field
from pathlib import Path
from typing import Any, BinaryIO

GGUF_MAGIC = b"GGUF"

# gguf_metadata_value_type, straight from the specification.
UINT8, INT8 = 0, 1
UINT16, INT16 = 2, 3
UINT32, INT32 = 4, 5
FLOAT32, BOOL = 6, 7
STRING, ARRAY = 8, 9
UINT64, INT64 = 10, 11
FLOAT64 = 12

# value type -> (struct format, byte width)
_SCALARS: dict[int, tuple[str, int]] = {
    UINT8: ("<B", 1),
    INT8: ("<b", 1),
    UINT16: ("<H", 2),
    INT16: ("<h", 2),
    UINT32: ("<I", 4),
    INT32: ("<i", 4),
    FLOAT32: ("<f", 4),
    BOOL: ("<?", 1),
    UINT64: ("<Q", 8),
    INT64: ("<q", 8),
    FLOAT64: ("<d", 8),
}


class GGUFError(ValueError):
    """Raised when a file is not valid GGUF or ends unexpectedly."""


@dataclass(frozen=True)
class GGUFFile:
    """The header and metadata of a GGUF file. Tensor data is not read."""

    path: Path
    version: int
    tensor_count: int
    metadata: dict[str, Any] = field(default_factory=dict)

    @property
    def architecture(self) -> str | None:
        return self.metadata.get("general.architecture")

    @property
    def name(self) -> str | None:
        return self.metadata.get("general.name")

    @property
    def quantization(self) -> int | None:
        return self.metadata.get("general.file_type")

    @property
    def context_length(self) -> int | None:
        arch = self.architecture
        if arch is None:
            return None
        return self.metadata.get(f"{arch}.context_length")


def _read_exact(fh: BinaryIO, count: int) -> bytes:
    data = fh.read(count)
    if len(data) != count:
        raise GGUFError(f"file ended early: wanted {count} bytes, got {len(data)}")
    return data


def _read_u32(fh: BinaryIO) -> int:
    return struct.unpack("<I", _read_exact(fh, 4))[0]


def _read_u64(fh: BinaryIO) -> int:
    return struct.unpack("<Q", _read_exact(fh, 8))[0]


def _read_string(fh: BinaryIO) -> str:
    length = _read_u64(fh)
    return _read_exact(fh, length).decode("utf-8", errors="replace")


def _read_value(fh: BinaryIO, value_type: int, max_array: int) -> Any:
    if value_type == STRING:
        return _read_string(fh)

    if value_type == ARRAY:
        element_type = _read_u32(fh)
        length = _read_u64(fh)
        kept: list[Any] = []
        for index in range(length):
            value = _read_value(fh, element_type, max_array)
            if index < max_array:
                kept.append(value)
        if length > max_array:
            kept.append(f"... {length - max_array} more")
        return kept

    if value_type in _SCALARS:
        fmt, width = _SCALARS[value_type]
        return struct.unpack(fmt, _read_exact(fh, width))[0]

    raise GGUFError(f"unknown metadata value type: {value_type}")


def read_gguf(path: str | Path, *, max_array: int = 8) -> GGUFFile:
    """Parse the header and metadata block of a GGUF file.

    Only the header is read; the tensor data that follows is left untouched,
    so this is fast even on a multi-gigabyte model.
    """
    path = Path(path)
    with path.open("rb") as fh:
        magic = _read_exact(fh, 4)
        if magic != GGUF_MAGIC:
            raise GGUFError(f"not a GGUF file: magic was {magic!r}, expected {GGUF_MAGIC!r}")

        version = _read_u32(fh)
        tensor_count = _read_u64(fh)
        kv_count = _read_u64(fh)

        metadata: dict[str, Any] = {}
        for _ in range(kv_count):
            key = _read_string(fh)
            value_type = _read_u32(fh)
            metadata[key] = _read_value(fh, value_type, max_array)

    return GGUFFile(
        path=path,
        version=version,
        tensor_count=tensor_count,
        metadata=metadata,
    )

Detailed breakdown

  • The header is fixed and tiny: the four magic bytes GGUF, a uint32 version (currently 3), then two uint64 counts for tensors and metadata entries. Everything after that is variable-length, which is why the parser is a sequential read rather than a set of offsets.
  • Checking the magic first is the whole validity test. A truncated download or an HTML error page saved with a .gguf extension both fail here, immediately, with a message naming what was actually found. That is the single most useful thing this tool does.
  • Strings are uint64 length plus raw UTF-8, with no terminator. The 8-byte length prefix is generous for a format whose longest strings are chat templates, but it means you never have to scan for a null byte.
  • errors="replace" on decode is deliberate. Tokenizer vocabularies contain byte sequences that are not valid UTF-8 on their own, because a token can be a fragment of a multi-byte character. Strict decoding raises on real, well-formed model files.
  • Arrays must be consumed even when you do not want them. max_array limits what is kept, not what is read: the loop still parses every element, because entries are variable-length and there is no size field to skip past. A vocabulary array holds a hundred thousand-plus strings, and storing them all would dominate memory for no benefit — but skipping the reads would desynchronize the stream and corrupt every key that follows.
  • _read_exact exists so truncation is an error, not silence. A bare fh.read(n) returns short at end of file, and struct.unpack would then raise something opaque about buffer length. Failing here names the problem.
  • The properties encode the naming convention. Generic keys are prefixed general.; architecture-specific ones are prefixed with the architecture name itself, so context length lives under qwen35.context_length for the model in this article and llama.context_length for another. context_length reads general.architecture first and builds the key from it, which is why it can return None twice over.
  • Tensor data is never touched. The parser stops at the end of the metadata block, so inspecting a 40 GB model costs the same as inspecting a 1 MB one.

Step 10: Add the command-line entry point

Create the file

touch src/gguf_inspector/__main__.py

Add the code: src/gguf_inspector/__main__.py

"""Print a summary of a GGUF file: `uv run python -m gguf_inspector <path>`."""

from __future__ import annotations

import argparse
import json
import sys
from pathlib import Path

from .reader import GGUFError, read_gguf


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(
        prog="gguf-inspector",
        description="Read the header and metadata of a GGUF model file.",
    )
    parser.add_argument("path", type=Path, help="path to a .gguf file")
    parser.add_argument(
        "--json", action="store_true", help="print all metadata as JSON"
    )
    parser.add_argument(
        "--max-array",
        type=int,
        default=8,
        help="how many elements of each array to keep (default: 8)",
    )
    args = parser.parse_args(argv)

    try:
        model = read_gguf(args.path, max_array=args.max_array)
    except GGUFError as exc:
        print(f"error: {exc}", file=sys.stderr)
        return 1
    except OSError as exc:
        print(f"error: cannot read {args.path}: {exc}", file=sys.stderr)
        return 1

    if args.json:
        print(json.dumps(model.metadata, indent=2, default=str))
        return 0

    size_mb = args.path.stat().st_size / (1024 * 1024)
    print(f"file          {model.path.name}")
    print(f"size          {size_mb:.1f} MiB")
    print(f"gguf version  {model.version}")
    print(f"tensors       {model.tensor_count}")
    print(f"metadata keys {len(model.metadata)}")
    print(f"architecture  {model.architecture or 'unknown'}")
    print(f"name          {model.name or 'unknown'}")
    print(f"context       {model.context_length or 'unknown'}")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Detailed breakdown

  • __main__.py makes the package runnable as python -m gguf_inspector, with no console-script entry point and no install step. Under uv run that resolves against the project environment automatically.
  • Both failure paths return 1 and write to stderr. GGUFError covers “this is not a GGUF file”, OSError covers “there is no file there at all”. Separating them means the message tells you which mistake you made, and a non-zero exit makes the tool usable in a shell conditional or a Makefile.
  • --json dumps metadata rather than the summary, with default=str so any value that is not JSON-serializable degrades to its string form instead of raising. That keeps the flag useful on unusual models rather than fragile.
  • --max-array is exposed because the default hides things. At 8, a vocabulary shows its first eight tokens. Raising it is how you inspect a chat template or a full token list when you actually need one.
  • The file-size line comes from stat(), not from the parser, because the parser deliberately never reads to the end of the file.

Step 11: Add the tests

Create the file

mkdir -p tests
touch tests/test_reader.py

Add the code: tests/test_reader.py

"""Tests for the GGUF reader.

The fixtures build GGUF files byte by byte, so the suite runs without
downloading a model. One test reads a real model if it is present and skips
otherwise.
"""

from __future__ import annotations

import os
import struct
from pathlib import Path

import pytest

from gguf_inspector.reader import (
    ARRAY,
    GGUF_MAGIC,
    STRING,
    UINT32,
    GGUFError,
    read_gguf,
)


def _string(text: str) -> bytes:
    """Encode a gguf_string_t: uint64 length, then UTF-8 bytes."""
    raw = text.encode("utf-8")
    return struct.pack("<Q", len(raw)) + raw


def _kv_string(key: str, value: str) -> bytes:
    return _string(key) + struct.pack("<I", STRING) + _string(value)


def _kv_u32(key: str, value: int) -> bytes:
    return _string(key) + struct.pack("<I", UINT32) + struct.pack("<I", value)


def _kv_string_array(key: str, values: list[str]) -> bytes:
    body = _string(key) + struct.pack("<I", ARRAY)
    body += struct.pack("<I", STRING) + struct.pack("<Q", len(values))
    for value in values:
        body += _string(value)
    return body


def _build_gguf(path: Path, *, kv_blocks: list[bytes], tensor_count: int = 0) -> Path:
    """Write a minimal but specification-valid GGUF header to `path`."""
    header = GGUF_MAGIC + struct.pack("<I", 3)
    header += struct.pack("<Q", tensor_count)
    header += struct.pack("<Q", len(kv_blocks))
    path.write_bytes(header + b"".join(kv_blocks))
    return path


@pytest.fixture
def sample_model(tmp_path: Path) -> Path:
    return _build_gguf(
        tmp_path / "sample.gguf",
        kv_blocks=[
            _kv_string("general.architecture", "llama"),
            _kv_string("general.name", "Sample Model"),
            _kv_u32("llama.context_length", 4096),
        ],
        tensor_count=12,
    )


def test_reads_header_fields(sample_model: Path) -> None:
    model = read_gguf(sample_model)
    assert model.version == 3
    assert model.tensor_count == 12
    assert len(model.metadata) == 3


def test_exposes_convenience_properties(sample_model: Path) -> None:
    model = read_gguf(sample_model)
    assert model.architecture == "llama"
    assert model.name == "Sample Model"
    assert model.context_length == 4096


def test_context_length_key_follows_architecture(tmp_path: Path) -> None:
    """The context key is prefixed with the architecture, not a fixed string."""
    path = _build_gguf(
        tmp_path / "qwen.gguf",
        kv_blocks=[
            _kv_string("general.architecture", "qwen3"),
            _kv_u32("qwen3.context_length", 32768),
            _kv_u32("llama.context_length", 2048),
        ],
    )
    assert read_gguf(path).context_length == 32768


def test_missing_metadata_returns_none(tmp_path: Path) -> None:
    path = _build_gguf(tmp_path / "bare.gguf", kv_blocks=[])
    model = read_gguf(path)
    assert model.architecture is None
    assert model.context_length is None


def test_rejects_a_file_that_is_not_gguf(tmp_path: Path) -> None:
    path = tmp_path / "not-a-model.gguf"
    path.write_bytes(b"<html>404 Not Found</html>")
    with pytest.raises(GGUFError, match="not a GGUF file"):
        read_gguf(path)


def test_rejects_a_truncated_file(tmp_path: Path) -> None:
    """A download cut short must fail loudly rather than parse as empty."""
    path = tmp_path / "truncated.gguf"
    path.write_bytes(GGUF_MAGIC + struct.pack("<I", 3) + b"\x00\x00")
    with pytest.raises(GGUFError, match="file ended early"):
        read_gguf(path)


def test_array_is_truncated_but_fully_consumed(tmp_path: Path) -> None:
    """Truncation must not desynchronize the keys that follow the array."""
    path = _build_gguf(
        tmp_path / "vocab.gguf",
        kv_blocks=[
            _kv_string_array("tokenizer.ggml.tokens", [f"tok{n}" for n in range(50)]),
            _kv_string("general.architecture", "llama"),
        ],
    )
    model = read_gguf(path, max_array=3)

    tokens = model.metadata["tokenizer.ggml.tokens"]
    assert tokens[:3] == ["tok0", "tok1", "tok2"]
    assert tokens[-1] == "... 47 more"
    # The key after the array is the real proof: the stream stayed aligned.
    assert model.architecture == "llama"


def test_max_array_can_keep_everything(tmp_path: Path) -> None:
    path = _build_gguf(
        tmp_path / "small-vocab.gguf",
        kv_blocks=[_kv_string_array("tokenizer.ggml.tokens", ["a", "b"])],
    )
    model = read_gguf(path, max_array=100)
    assert model.metadata["tokenizer.ggml.tokens"] == ["a", "b"]


@pytest.mark.skipif(
    not os.environ.get("GGUF_MODEL_PATH"),
    reason="set GGUF_MODEL_PATH to a downloaded .gguf to run this test",
)
def test_reads_a_real_model() -> None:
    model = read_gguf(os.environ["GGUF_MODEL_PATH"])
    assert model.version >= 2
    assert model.tensor_count > 0
    assert model.architecture

Detailed breakdown

  • The suite builds its own GGUF files. _build_gguf writes a specification-valid header from struct.pack calls, so the tests are fast, deterministic, and runnable on a machine that has never downloaded a model. This matters more than it sounds: a test suite that depends on a 900 MB artifact is a test suite people skip.
  • Writing the fixtures is a second implementation of the format. The parser reads what the fixtures write; if either misunderstands the layout, the tests disagree. Hand-rolling both directions is the cheapest available check on a binary format.
  • test_array_is_truncated_but_fully_consumed is the important test. It puts a 50-element array first and a scalar key second, then asserts on the second key. If the parser ever “optimized” truncation by skipping reads, the array assertion would still pass and the architecture assertion would fail. That is the bug this design invites, so it gets a dedicated test.
  • test_context_length_key_follows_architecture pins the prefix rule by writing both a qwen3. and a llama. context length into one file and requiring the reader to pick the one matching general.architecture. A parser that hardcoded llama. would pass every other test in the file.
  • test_rejects_a_file_that_is_not_gguf uses an HTML error page rather than random bytes, because that is the realistic failure: a redirect or a rate-limit response saved under a .gguf name.
  • The real-model test is opt-in via GGUF_MODEL_PATH. It skips by default so the suite stays hermetic, but it is there so you can point the parser at an actual download and confirm the fixtures were not lying.

Step 12: Add the Makefile

Create the file

touch Makefile

Add the code: Makefile

.DEFAULT_GOAL := help

MODEL_REPO ?= ggml-org/Qwen3.5-0.8B-GGUF:Q8_0
PROMPT     ?= In one sentence, what is a GGUF file?
TEXT_PROMPT ?= A GGUF file is
TOKENS     ?= 64
CACHE      ?= $(HOME)/.cache/huggingface/hub

.PHONY: help install test check model run ask text cpu inspect cache clean

help: ## Show this help screen
	@echo "GGUF inspector — llama.cpp getting started"
	@echo ""
	@echo "Targets:"
	@grep -E '^[a-zA-Z_-]+:.*?## ' $(MAKEFILE_LIST) \
		| awk 'BEGIN {FS = ":.*?## "}; {printf "  %-10s %s\n", $$1, $$2}'
	@echo ""
	@echo "Variables: MODEL_REPO=$(MODEL_REPO) TOKENS=$(TOKENS)"

install: ## Sync dependencies with uv
	uv sync

test: ## Run the test suite
	uv run pytest -q

check: ## Verify llama.cpp is installed and print its build
	@command -v llama >/dev/null || { echo "llama not found: brew install llama.cpp"; exit 1; }
	@llama version

model: ## Download MODEL_REPO without running it
	llama download -hf $(MODEL_REPO)

run: ## Generate TOKENS tokens with default GPU offload
	llama completion -hf $(MODEL_REPO) -p "$(PROMPT)" -n $(TOKENS) -no-cnv

ask: ## Print only the answer -- reports if the model never stops thinking
	@llama completion -hf $(MODEL_REPO) -p "$(PROMPT)" -n $(TOKENS) \
		-st -rea off --no-display-prompt -lv 0 \
		| awk '/<think>/ { t = 1; next } /<\/think>/ { t = 0; next } !t { gsub(/ *\[end of text\]/, ""); if ($$0 ~ /[^[:space:]]/) { print; got = 1 } } END { if (!got) { print "no answer: the model never finished thinking. Retry, or use a larger model." > "/dev/stderr"; exit 1 } }'

text: ## Continue TEXT_PROMPT as plain text -- no chat template, no thinking
	@llama completion -hf $(MODEL_REPO) -p "$(TEXT_PROMPT)" -n $(TOKENS) \
		-no-cnv --no-display-prompt -lv 0 \
		| sed -e 's/ *\[end of text\]//' -e '/^[[:space:]]*$$/d'

cpu: ## Same generation with offload disabled (-ngl 0)
	llama completion -hf $(MODEL_REPO) -p "$(PROMPT)" -n $(TOKENS) -ngl 0 -no-cnv

inspect: ## Summarize every cached GGUF file
	@find $(CACHE) -name '*.gguf' -print0 \
		| xargs -0 -I{} uv run python -m gguf_inspector {}

cache: ## Show what the model cache is costing you
	@du -sh $(CACHE)/* 2>/dev/null || echo "cache is empty: $(CACHE)"

clean: ## Remove Python build and test artifacts (keeps models)
	rm -rf .pytest_cache .venv **/__pycache__ src/**/__pycache__

Detailed breakdown

  • .DEFAULT_GOAL := help means a bare make prints the help screen, which the house style requires. The grep/awk pair reads the ## comments off the target lines, so the help text cannot drift from the targets: adding a target with a ## comment adds a help line automatically.
  • check fails fast with an actionable message. command -v llama costs nothing and turns “some later target exploded confusingly” into “llama not found: brew install llama.cpp”.
  • run and cpu differ only in -ngl 0, which makes the comparison from Step 6 a two-command exercise rather than a copy-paste of a long command line. Both carry -no-cnv; without it these targets answer and then hang waiting for a second turn. They can still emit a <think> block — -no-cnv drops the chat template but not the model’s habit of producing thought tags — and what keeps that bounded here is -n $(TOKENS), not the absence of thinking. Use the -st -rea off form from Step 3 when you want the answer alone.
  • The ## help comments name variables without $(...). Make does not expand inside those comments, so $(MODEL_REPO) would print literally on the help screen. The expanded values are shown by the Variables: line instead.
  • inspect uses -print0/xargs -0 so paths with spaces survive. The Hugging Face cache nests models under directories derived from repo names, and while those are usually tame, a snapshot path is not something to assume about.
  • Three generation targets, because they fail differently. run shows what llama.cpp actually prints, logs and all, which is what you want when something is wrong. ask asks PROMPT as a question and prints only the answer — and exits 1 with a diagnosis on the runs where the model never stops thinking, which is most of them on this model. text continues TEXT_PROMPT as plain text, which is the form that does not trigger thinking at all.
  • Two Makefile-specific details in those recipes. Each is prefixed with @ so Make does not echo the long pipeline before running it, and every $ inside the awk and sed programs is written $$ — otherwise Make expands $0 and $/ as variables and hands the filter a broken script.
  • clean deliberately leaves models alone. Deleting hundreds of megabytes of weights is not the sort of thing a clean target should do silently; the cache target shows you the cost and you remove directories yourself.
  • The variables are overridable at the command line: make run TOKENS=200 or make run MODEL_REPO=ggml-org/tiny-llamas works without editing the file.

Step 13: Run everything

make            # help screen, because .DEFAULT_GOAL is help
make check      # llama.cpp present, with its build number
make install    # sync the Python environment
make test       # the parser suite, no model needed
make model      # download the weights
make inspect    # parse every cached GGUF header
make run        # generate text, logs and all
make ask        # ask PROMPT, answer only (may report no answer)
make text       # continue TEXT_PROMPT, reliably clean

Expected shape of make inspect against the downloaded model:

file          Qwen3.5-0.8B-Q8_0.gguf
size          795.0 MiB
gguf version  3
tensors       335
metadata keys 41
architecture  qwen35
name          Qwen3.5-0.8B
context       262144

To run the opt-in real-model test:

GGUF_MODEL_PATH=$(find ~/.cache/huggingface/hub -name '*.gguf' | head -1) \
  uv run pytest -q

Detailed breakdown

  • Running make first is not a formality: it is the check that the help rule works, which is easy to break by editing the awk format string.
  • make test before make model is intentional ordering. The suite passing without any model on disk is the proof that the fixtures are self-contained.
  • The exact size, tensors, and metadata keys values depend on which quantization you pulled; the architecture and context lines come from the model’s own metadata and are the ones to sanity-check against the repository’s model card.

Troubleshooting

llama: command not found after brew install. Homebrew’s bin directory is not on your PATH. brew --prefix prints the install root; make sure $(brew --prefix)/bin is on PATH in your shell profile.

unknown argument: -hf. You are running a build old enough to predate the flag, or a subcommand that does not accept it. Check llama version and llama <subcommand> --help.

The download fails with a TLS or certificate error. llama.cpp does the HTTPS fetch itself through openssl@3. brew reinstall openssl@3 is the first thing to try; a corporate TLS-inspecting proxy is the second thing to suspect.

A :quant tag reports no GGUF files found in repository. The message is misleading: the repository has GGUF files, just not one matching the tag you asked for. Open the repo’s file listing on Hugging Face and use a tag that matches an actual filename — quantization names are not standardized across repositories.

make inspect prints “not a GGUF file”. A previous download was interrupted and left a partial or error-page file in the cache. Delete the corresponding models--… directory and re-download. This is exactly the case the magic-byte check exists to catch.

Generation is much slower than expected. Confirm you are not stuck at -ngl 0 from an earlier experiment, and check whether the model actually fits in memory — once macOS starts swapping, throughput collapses in a way that looks like a llama.cpp problem but is not.

Tests fail with ModuleNotFoundError: No module named 'gguf_inspector'. The project was created with a plain uv init rather than uv init --package, so it is an application with no [build-system] and nothing installs src/ onto the import path. Adding __init__.py and re-running uv sync does not fix it. Add the build backend from Step 8 to pyproject.toml and run uv sync again, or start over with uv init --package.

The model never stops talking, or fills the screen with repeated half-thoughts. You are in a reasoning loop; see Step 3. Add -rea off. This build disables repetition penalties by default (--repeat-penalty 1.00), so nothing interrupts the loop on its own.

Recap

  • llama.cpp installs from Homebrew as a single llama command with subcommands. cli is interactive, completion is the scriptable one-shot, serve is the HTTP server, and llama help all reveals the rest.
  • -hf <user>/<model>[:quant] downloads from Hugging Face on demand, and llama download does it without running anything.
  • Models live in the shared Hugging Face hub cache at ~/.cache/huggingface/hub/models--<user>--<repo>/. LLAMA_CACHE overrides the location and takes priority over the HF_* variables.
  • GPU offload defaults to auto; -ngl is now mainly useful for turning offload off to get a CPU baseline.
  • A GGUF file’s header is four magic bytes, a version, two counts, and a run of typed key/value pairs — enough structure to validate a download and read a model’s architecture and context length in about 120 lines of standard-library Python.

Next improvements

  • Serve the same model over an OpenAI-compatible HTTP endpoint with llama serve, which lets existing OpenAI-SDK code run locally with a base-URL change.
  • Constrain output to a grammar so the model cannot emit malformed JSON, rather than validating and retrying after the fact.
  • Quantize a model yourself and measure what each level costs in size, speed, and output quality.
  • Benchmark prompt processing separately from token generation with llama bench, across quantizations and -ngl values.