If you already have Python code that talks to a local model through openai.OpenAI, there is a second way to run that model: import it directly. The llama-cpp-python package binds llama.cpp into your own process, so the weights load inside your program and generation is a function call rather than an HTTP request.

By the end of this article you will have taken a working OpenAI-SDK client and rewritten it around the Llama constructor. No server to start, no base_url, no API key, and no port to keep free. The finished module is about forty lines and comes with a test suite that runs without loading a model at all.

This is a porting article. It covers the swap and what the swap changes, and deliberately stops there — the bindings expose a great deal more (custom sampling, token probabilities, direct access to the context), and none of it is needed to move a script across.

Versions used throughout: llama-cpp-python 0.3.35 on macOS 26.6.2 (arm64, Apple M5 Max), uv 0.11.26, Python 3.12.9, with the same ggml-org/Qwen3.5-0.8B-GGUF:Q8_0 model as Getting Started with llama.cpp on macOS.

What you end up with

A package called local-llm-direct with:

  • load() — builds a Llama from either a file path or a Hugging Face repo id.
  • ask() — sends one chat turn and returns a small Reply value.
  • A pytest suite that exercises the parsing with a stub, plus one live test that actually loads the weights.
  • A Makefile whose bare make prints the targets.

Prerequisites

  • macOS on Apple Silicon. Written and validated on macOS 26.6.2, arm64. Metal, Apple’s GPU programming API, is compiled in automatically; you do not pass a flag for it.
  • The Xcode Command Line Tools. Installing the package compiles C++, so a working toolchain is required — xcode-select -p should print a path. If it does not, run xcode-select --install.
  • uv 0.11.26 or newer — uv --version.
  • A GGUF model. A GGUF is the single-file format llama.cpp loads weights from. This article uses ggml-org/Qwen3.5-0.8B-GGUF:Q8_0, which is 795 MiB. You do not need to download it separately (Step 4 pulls it), but if you followed Getting Started with llama.cpp on macOS it is already cached and will be reused.
  • Optional, for the comparison: the client from Serve a Local OpenAI-Compatible Endpoint with llama.cpp on macOS. This article rewrites that article’s client, so having it open makes the diff concrete. Nothing here depends on it.

Step 1: Create the project and its .gitignore

Start with the ignore file, before there is anything to ignore. A local-model project accumulates two things git should never see: multi-hundred-megabyte weight files, and a virtual environment holding a compiled copy of llama.cpp. Writing the rules first means neither ever reaches the index.

Create the file

mkdir -p ~/projects/local-llm-direct
cd ~/projects/local-llm-direct
touch .gitignore

Add the code: .gitignore

# Python
.venv/
__pycache__/
*.py[cod]
.pytest_cache/

# Model weights — never commit these
*.gguf
models/

# Local overrides
.env

Detailed breakdown

  • *.gguf and models/ are the entries that matter most here. The model in this article is 795 MiB, and git keeps a committed blob forever even after a later commit deletes it. The default cache lives outside the project, but a model_path= experiment that copies weights alongside the code is an easy mistake to make once.
  • .venv/ matters here for a reason specific to this package. In most projects the virtual environment is just downloaded wheels; here it also contains the llama.cpp shared libraries built during install. It is regenerated from the lockfile, so it carries nothing worth sharing, but it is considerably larger than usual.
  • .env is habit rather than necessity. There is no API key in this project (that is rather the point), but the environment overrides in Step 5 are the kind of thing that ends up in a .env later.

Step 2: Initialize the project with uv

With the ignore rules in place, create the package skeleton. uv init --package produces a src/ layout and a pyproject.toml, which is what lets the tests import the module by name rather than by relative path.

Create the file

cd ~/projects/local-llm-direct
uv init --package --name local-llm-direct
uv python pin 3.12

Detailed breakdown

  • uv init --package creates src/local_llm_direct/__init__.py and a pyproject.toml declaring the project. The hyphenated project name becomes an underscored module name.
  • uv python pin 3.12 writes a .python-version file. This is not ceremony: llama-cpp-python compiles against the Python it is installed for, and pinning means a later uv sync on a machine with a newer system Python rebuilds against the same target instead of silently choosing another.

Step 3: Install the bindings, and watch it compile

This is the step that behaves unlike a normal dependency, so it gets its own place in the article rather than a mention in the prerequisites.

PyPI ships llama-cpp-python as a source distribution only — a .tar.gz of source code rather than a prebuilt .whl binary. There is no wheel to download for any platform. Installing it therefore runs CMake and compiles llama.cpp on your machine the first time you ask for it, which is why the Xcode Command Line Tools are a prerequisite and why that first install takes visibly longer than the ones around it.

Create the file

cd ~/projects/local-llm-direct
uv add llama-cpp-python huggingface-hub
uv add --dev pytest

Detailed breakdown

  • The compile is quick, despite the reputation. On the test machine the first uv add llama-cpp-python took 23.94 seconds of wall time at 974% CPU — CMake parallelizes across cores. Expect longer on fewer cores, but not the ordeal that “compiles from source” usually implies.
  • You pay that once per uv cache, not once per project. uv keeps the wheel it builds locally (distinct from the published wheel, which does not exist), so a second project asking for the same version reuses it. Installing this same version into a second project on the test machine resolved in 0.21 seconds, with no compiler involved. If you see a long build a second time, the version changed, the cache was cleared, or you are a different user on the same machine.
  • You do not need CMAKE_ARGS for Metal. Guides written for other platforms tell you to pass -DGGML_METAL=on. On Apple Silicon it is already the default; the build picks up Metal, and the loaded library reports MTL : EMBED_LIBRARY = 1. Setting the flag by hand changes nothing.
  • huggingface-hub is a second, separate dependency. It is not pulled in automatically, and it is what Llama.from_pretrained() in Step 4 needs to resolve a repo id. Install it now and that method works; skip it and you get a ModuleNotFoundError at the point of use.
  • The compiled libraries land in site-packages/lib/ as well as in site-packages/llama_cpp/lib/ — libllama.dylib, libggml-metal.dylib and friends, installed to both places as separate copies. Useful when working out which llama.cpp a given environment is actually running.

What version of llama.cpp did you just build?

The package vendors its own copy of llama.cpp, pinned to whatever the release was cut against. That copy is independent of any llama binary Homebrew installed, and it is usually older. Check that the vendored build recognizes the model you intend to load, because an unsupported architecture fails at load time with an unhelpful message.

Create the file

cd ~/projects/local-llm-direct
strings .venv/lib/python3.12/site-packages/lib/libllama.dylib | grep -cx qwen35

Detailed breakdown

  • The command prints 1, meaning this build registers qwen35 — the architecture name the Qwen3.5 family declares in its GGUF metadata. The model in this article will load.
  • Substitute your own architecture if you are pointing this at different weights. The value comes from the GGUF’s general.architecture key, and a 0 here means the vendored llama.cpp predates support for that family. The fix is a newer llama-cpp-python, not a newer Homebrew llama, because the two do not share a library.
  • grep -cx matches the whole line exactly and counts, so a substring like qwen35moe does not produce a false positive.

Step 4: Swap the constructor

Here is the actual subject of the article. The OpenAI client is constructed around a network location; the Llama object is constructed around a file. Everything else in this step follows from that one difference.

The client from Serve a Local OpenAI-Compatible Endpoint connects like this, with a llama serve process already running in another terminal:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:8080/v1",
    api_key="no-key-required",
)

The in-process equivalent takes no URL and no key, because there is nothing to authenticate to:

Create the file

cd ~/projects/local-llm-direct
touch first_run.py

Add the code: first_run.py

"""Smallest possible in-process generation, to prove the install works."""

from llama_cpp import Llama

llm = Llama.from_pretrained(
    repo_id="ggml-org/Qwen3.5-0.8B-GGUF",
    filename="Qwen3.5-0.8B-Q8_0.gguf",
    n_ctx=2048,
    n_gpu_layers=-1,
    verbose=False,
)

out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Name the capital of France. One word."}],
    max_tokens=32,
    temperature=0.0,
)

print(repr(out["choices"][0]["message"]["content"]))

Detailed breakdown

  • from_pretrained takes the place of base_url. It resolves a Hugging Face repo id and filename to a local file, downloading it once and caching it. It reuses the same cache llama download -hf fills, so if you followed the getting-started article the 795 MiB is already on disk and nothing is re-fetched. Measured on the test machine, the first from_pretrained in a process takes 0.62 seconds and later ones 0.25 seconds, the difference being the Hugging Face metadata round trip; the cache stayed at a single copy throughout.
  • n_gpu_layers=-1 means “offload every layer to the GPU.” This is the in-process spelling of llama.cpp’s -ngl flag. The default is 0, which is CPU-only and much slower, so this is the one argument you should not leave out on Apple Silicon.
  • n_ctx sets the context window: the number of tokens of prompt plus reply the model can hold at once. Unlike the server’s -c it is not a pool divided between parallel slots. There is one context here, and it is yours. The server article’s warning that -c 8192 -np 2 gives each request 4096 has no counterpart in this route.
  • verbose=False silences llama.cpp’s own logging, and thoroughly: with it set, llama.cpp writes zero bytes to stderr during a load, as does a bare import llama_cpp. It does not muzzle other libraries, which is why the huggingface_hub warning in the run below still gets through. Set it to True when you want the Metal device report and the per-tensor offload table.
  • The reply is a plain dictionary. out["choices"][0]["message"]["content"] is the direct translation of the SDK’s resp.choices[0].message.content. This is the change most likely to break ported code, because attribute access on a dict raises AttributeError somewhere well downstream of the call that produced it.

Run it:

cd ~/projects/local-llm-direct
uv run python first_run.py

.../huggingface_hub/utils/_validators.py:205: UserWarning: The `local_dir_use_symlinks`
argument is deprecated and ignored in `hf_hub_download`. Downloading to a local
directory does not use symlinks anymore.
  warnings.warn(
'Paris'

The UserWarning comes from inside huggingface_hub, not from anything you wrote — from_pretrained passes a deprecated argument on its way to the cache. It is noise, it appears on every run that resolves a repo id, and Step 5’s model_path route avoids it entirely.

That is the whole swap. A model loaded from disk, a chat completion, and an answer, with no server process anywhere.

What did not carry over

Three details from the OpenAI route have no equivalent here, and expecting them is the main way a port goes wrong.

  • There is no model= argument. The SDK requires one, and a llama serve process has nothing to select with it because it serves the single model it was started with. Here the model is the object you called the method on, so the parameter does not exist.
  • There is no reasoning_content. This is the interesting one. The server article’s headline finding is that a reasoning model can answer with content set to the empty string and the real text in a separate reasoning_content field, handing an SDK client an empty string and no error. That does not happen in-process. Asked a question written to induce step-by-step working, this route returned a message with the keys ['content', 'role'] and nothing else, with all 291 tokens of working inside content. A chat template is the formatting rule a model file carries for turning a list of messages into one token sequence; the binding renders the GGUF’s own in Python (llm.chat_format reports chat_template.default) and never separates a reasoning channel out.
  • Errors arrive as exceptions, not status codes. There is no connection to refuse and no APIConnectionError to catch. A bad path raises at construction time instead.

Step 5: Write the module

The one-file version proves the install. This step turns it into something worth importing: a loader that takes either a path or a repo id, and a function that returns a small typed value instead of a raw dictionary.

The Reply dataclass exists to solve the dictionary problem once. Ported code that reaches into out["choices"][0] in a dozen places will break in a dozen places the first time a response shape changes; converting at the boundary means only one function knows what the response looks like.

Create the file

cd ~/projects/local-llm-direct
touch src/local_llm_direct/client.py

Add the code: src/local_llm_direct/client.py

"""Run a local GGUF model in this process, with no server in the way.

The OpenAI SDK builds a client around a URL; this module builds one around a
file. The only shape worth remembering is that llama-cpp-python returns plain
dictionaries where the SDK returns objects, so `ask` converts once, here, and
callers never index into a response.
"""

from __future__ import annotations

import os
from dataclasses import dataclass

from llama_cpp import Llama

DEFAULT_REPO_ID = "ggml-org/Qwen3.5-0.8B-GGUF"
DEFAULT_FILENAME = "Qwen3.5-0.8B-Q8_0.gguf"


@dataclass(frozen=True)
class Reply:
    """One assistant turn, converted out of the response dictionary."""

    text: str
    finish_reason: str

    @property
    def truncated(self) -> bool:
        """True when generation stopped at the token cap rather than finishing."""
        return self.finish_reason == "length"


def load(
    *,
    model_path: str | None = None,
    n_ctx: int = 2048,
    n_gpu_layers: int = -1,
    verbose: bool = False,
) -> Llama:
    """Load a model, preferring an explicit path over a Hugging Face repo id.

    `LLAMA_MODEL_PATH` selects a local file and works offline. Without it, the
    model is resolved from `LLAMA_REPO_ID` and `LLAMA_FILENAME`, which needs
    network access even when the file is already cached.
    """
    path = model_path or os.environ.get("LLAMA_MODEL_PATH")
    if path:
        return Llama(
            model_path=path,
            n_ctx=n_ctx,
            n_gpu_layers=n_gpu_layers,
            verbose=verbose,
        )
    return Llama.from_pretrained(
        repo_id=os.environ.get("LLAMA_REPO_ID", DEFAULT_REPO_ID),
        filename=os.environ.get("LLAMA_FILENAME", DEFAULT_FILENAME),
        n_ctx=n_ctx,
        n_gpu_layers=n_gpu_layers,
        verbose=verbose,
    )


def ask(llm: Llama, prompt: str, *, max_tokens: int = 256) -> Reply:
    """Send one user turn and convert the response dictionary into a `Reply`."""
    out = llm.create_chat_completion(
        messages=[{"role": "user", "content": prompt}],
        max_tokens=max_tokens,
        temperature=0.0,
    )
    choice = out["choices"][0]
    return Reply(
        text=choice["message"]["content"] or "",
        finish_reason=choice["finish_reason"],
    )

Detailed breakdown

  • load() prefers model_path for a concrete reason. from_pretrained contacts the Hugging Face API to resolve repo metadata before it checks the cache, so it fails with huggingface_hub.errors.OfflineModeIsEnabled under HF_HUB_OFFLINE=1 even when the weights are sitting on disk. Llama(model_path=...) never touches the network. Offer both and the caller picks.
  • ask() takes the Llama as an argument rather than building one, because loading is the expensive part. The model stays in memory between calls; a function that constructed its own would pay the load on every question and hold two copies of the weights if called twice.
  • choice["message"]["content"] or "" guards a real case. The content key can hold None rather than a string when a turn produces no text, and or "" keeps Reply.text typed as str so callers are not forced to handle None.
  • truncated reads finish_reason, which carries the same values as the OpenAI SDK: "stop" for a natural ending, "length" when max_tokens cut it off. This is one of the few pieces of the response shape that ports unchanged.
  • The environment variables mean the same code runs against different weights without an edit, which is what makes the live test in Step 6 skippable on a machine that has no model.

Step 6: Test it without loading a model

Most of what a port can get wrong is in the conversion, not the inference — and the conversion can be tested in milliseconds. A stub object with a create_chat_completion method satisfies everything ask() touches, so the suite runs with no weights, no GPU, and no network.

One test does load the real model, because a suite that only ever sees a stub cannot tell you the response shape is still what you assumed. It skips itself when the weights are not available.

Create the file

cd ~/projects/local-llm-direct
mkdir -p tests
touch tests/test_client.py

Add the code: tests/test_client.py

"""Tests for the response conversion, with one live check against real weights."""

from __future__ import annotations

import os

import pytest

from local_llm_direct.client import Reply, ask, load


class StubLlama:
    """Stands in for `Llama`, recording the call and returning a fixed response."""

    def __init__(self, content, finish_reason="stop"):
        self._content = content
        self._finish_reason = finish_reason
        self.calls = []

    def create_chat_completion(self, **kwargs):
        self.calls.append(kwargs)
        return {
            "choices": [
                {
                    "message": {"role": "assistant", "content": self._content},
                    "finish_reason": self._finish_reason,
                }
            ]
        }


def test_ask_extracts_text():
    reply = ask(StubLlama("Paris"), "Capital of France?")
    assert reply.text == "Paris"
    assert reply.finish_reason == "stop"


def test_ask_converts_none_content_to_empty_string():
    assert ask(StubLlama(None), "anything").text == ""


def test_ask_sends_one_user_message():
    stub = StubLlama("ok")
    ask(stub, "hello", max_tokens=17)
    sent = stub.calls[0]
    assert sent["messages"] == [{"role": "user", "content": "hello"}]
    assert sent["max_tokens"] == 17


def test_reply_is_truncated_at_the_token_cap():
    assert Reply(text="...", finish_reason="length").truncated is True


def test_reply_is_not_truncated_on_a_natural_stop():
    assert Reply(text="done", finish_reason="stop").truncated is False


@pytest.mark.skipif(
    not os.environ.get("LLAMA_LIVE"),
    reason="set LLAMA_LIVE=1 to load the real model",
)
def test_live_model_answers():
    llm = load()
    reply = ask(llm, "Name the capital of France. One word.", max_tokens=32)
    assert "paris" in reply.text.lower()

Detailed breakdown

  • StubLlama duck-types the one method ask() calls. No mocking library and no subclassing of Llama, which would load a model just to construct. This is the payoff of having ask() accept the model rather than create it.
  • test_ask_converts_none_content_to_empty_string is the test that would have caught the None case in review rather than in production. It is the kind of thing a stub tests well and a live model tests badly, because you cannot reliably make a model return nothing.
  • test_ask_sends_one_user_message asserts on what was sent, not what came back. It fails if someone changes the message shape while porting, which is the other half of the OpenAI-to-bindings translation.
  • The live test is opt-in via LLAMA_LIVE, so make test stays fast and works on a machine with no weights. Run it deliberately when you want to confirm the real response shape.

Run the fast suite:

cd ~/projects/local-llm-direct
uv run pytest -q

.....s                                                                   [100%]
5 passed, 1 skipped in 0.04s

Then run the live one, which loads the weights:

LLAMA_LIVE=1 uv run pytest -q

tests/test_client.py::test_live_model_answers
  .../huggingface_hub/utils/_validators.py:205: UserWarning: The
  `local_dir_use_symlinks` argument is deprecated and ignored in `hf_hub_download`.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
6 passed, 1 warning in 0.68s

Six passed rather than five and a skip, and this run took 0.68 seconds, nearly all of it the single live load; 795 MiB of weights map in fast once the operating system has the file cached in memory. Treat the timings in both transcripts as captured examples rather than fixed values — repeat runs on the same machine landed between 0.04 and 0.10 seconds for the fast suite and 0.68 and 0.90 for the live one. The warning is the same huggingface_hub deprecation from Step 4; pytest collects warnings and reports them at the end rather than inline.

Step 7: Add a Makefile

The commands so far are short but easy to misremember — particularly the LLAMA_LIVE=1 prefix. A Makefile records them, and a bare make lists what is available rather than doing something unexpected.

Create the file

cd ~/projects/local-llm-direct
touch Makefile

Add the code: Makefile

.DEFAULT_GOAL := help

.PHONY: help install test test-live run clean

help:  ## Show this help
	@grep -E '^[a-zA-Z_-]+:.*?## ' $(MAKEFILE_LIST) \
		| awk 'BEGIN {FS = ":.*?## "}; {printf "  \033[36m%-12s\033[0m %s\n", $$1, $$2}'

install:  ## Sync dependencies (compiles llama.cpp on first run)
	uv sync

test:  ## Run the fast suite (no model loaded)
	uv run pytest -q

test-live:  ## Run every test, including the one that loads real weights
	LLAMA_LIVE=1 uv run pytest -q

run:  ## Ask the model one question
	uv run python first_run.py

clean:  ## Remove caches (keeps the venv and its compiled libraries)
	rm -rf .pytest_cache
	find . -type d -name __pycache__ -prune -exec rm -rf {} +

Detailed breakdown

  • .DEFAULT_GOAL := help makes a bare make print the target list. Without it, make runs the first target in the file, which is rarely what someone exploring the project wants.
  • The help target parses its own Makefile. Each target’s description comes from the ## comment on its line, so adding a target documents it automatically and a target without a comment stays hidden.
  • clean deliberately leaves .venv/ alone. Deleting it here would throw away the compiled llama.cpp and make the next make install recompile. That is a 24-second penalty for no benefit, which is exactly the kind of thing a clean target should not do by surprise.

Confirm it:

cd ~/projects/local-llm-direct
make

  help         Show this help
  install      Sync dependencies (compiles llama.cpp on first run)
  test         Run the fast suite (no model loaded)
  test-live    Run every test, including the one that loads real weights
  run          Ask the model one question
  clean        Remove caches (keeps the venv and its compiled libraries)

Troubleshooting

ModuleNotFoundError: No module named 'huggingface_hub' — from_pretrained needs it and llama-cpp-python does not depend on it. Run uv add huggingface-hub, or switch to Llama(model_path=...), which has no such requirement.

OfflineModeIsEnabled: Cannot reach https://huggingface.co/... — you set HF_HUB_OFFLINE=1. from_pretrained resolves repo metadata before it looks in the cache, so it needs the network even for a model you already have. Use LLAMA_MODEL_PATH to point at the file directly.

UserWarning: The 'local_dir_use_symlinks' argument is deprecated — raised inside huggingface_hub by from_pretrained, not by your code, and safe to ignore. Loading with model_path= does not go through that helper and does not warn.

Warning: You are sending unauthenticated requests to the HF Hub — harmless for a public repo. Set HF_TOKEN to raise the rate limit if you are resolving models frequently.

The load is slow and generation is slower still — n_gpu_layers defaulted to 0, leaving everything on the CPU. Pass n_gpu_layers=-1.

A ResourceWarning about an unclosed /dev/null — this comes from the verbose=False suppression path inside the package. It is cosmetic, but it surfaces under pytest -W error and in strict warning configurations.

The model loads but answers badly — a model problem, not a port problem. The 0.8B model here is small enough to fumble simple instructions; asked to “Count: one two three” it replied “I’m ready to help! What would you like to count or do?” That is the model, not the bindings, and pointing LLAMA_REPO_ID at larger weights is the fix.

Recap

You ported a client from the OpenAI SDK to the llama.cpp bindings, and the change came down to four things:

  1. OpenAI(base_url=..., api_key=...) became Llama.from_pretrained(...) or Llama(model_path=...). A client built around a URL became one built around a file.
  2. client.chat.completions.create(...) became llm.create_chat_completion(...), and the model= argument disappeared, because the model is the object.
  3. The response became a plain dictionary. resp.choices[0].message.content became out["choices"][0]["message"]["content"], which is why ask() converts once at the boundary and hands back a Reply.
  4. The server went away entirely — with it the port, the API key, the connection errors, and the reasoning_content field that can hand an SDK client an empty string.

What you gained is a program with no moving parts outside itself. What you gave up is the thing a server is for: one loaded copy of the weights serving many clients, in many languages, possibly on another machine. A script, a test suite, or a batch job is a good fit for the bindings. Several processes that each want the same model are still a job for llama serve.