Serve a Local OpenAI-Compatible Endpoint with llama.cpp on macOS

One command turns a downloaded GGUF file into an HTTP server that speaks the OpenAI API. Code written against openai.OpenAI runs against it with one line changed — the base_url — and nothing leaves your machine.

This article starts that server, reads what its startup log is telling you, drives it with curl, and then builds a small Python client on the official OpenAI SDK. The interesting part is not the happy path, which takes about ninety seconds. It is the two places where “OpenAI-compatible” stops being the whole story: a reasoning model can hand your SDK an empty content field, and the -c you pass is not the context each request gets.

Versions used throughout: llama.cpp b10330 on macOS 26.5.2 (arm64), with the same ggml-org/Qwen3.5-0.8B-GGUF:Q8_0 model as Getting Started with llama.cpp on macOS.

Prerequisites

llama.cpp installed and a model in the cache. Getting Started with llama.cpp on macOS covers brew install llama.cpp and pulling the weights; this article assumes llama version works and llama download -hf ggml-org/Qwen3.5-0.8B-GGUF:Q8_0 has been run. The model is 795 MiB.
macOS on Apple Silicon. Written and validated on macOS 26.5.2, arm64.
uv 0.11.26 or newer — uv --version.
A free port. The server defaults to 8080; --port moves it.

Step 1: Start the server

llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF:Q8_0 -c 8192 -np 2

The last lines of the startup log are the ones worth reading:

0.00.805.859 I srv    load_model: initializing, n_slots = 2, n_ctx_slot = 4096, kv_unified = 'false'
0.00.808.082 I srv  llama_server: model loaded
0.00.808.085 I srv  llama_server: listening on http://127.0.0.1:8080
0.00.808.085 W srv  llama_server: NOTICE: server default port will be changed to :9931 in a future release
0.00.808.085 W srv  llama_server:         ref: https://github.com/ggml-org/llama.cpp/pull/26508

In another terminal, confirm it is up:

curl -s http://localhost:8080/health

{"status":"ok"}

Detailed breakdown

n_ctx_slot = 4096 is the number that matters, and it is not the 8192 you asked for. Step 5 covers why.
The port is changing. This build warns that the default moves from 8080 to 9931 in a future release. Pass --port 8080 explicitly in anything you intend to keep working across upgrades, rather than relying on the default.
listening on http://127.0.0.1:8080 means loopback only. Nothing outside your Mac can reach it, which is the right default. --host 0.0.0.0 opens it to the network, and you should not do that without reading Step 13 first.
-hf resolves through the same Hugging Face cache the CLI uses, so a model already downloaded starts instantly. -m /path/to/model.gguf works too.
Startup depends on whether the weights are in the page cache: about two seconds on the first start after a reboot, and closer to half a second on a repeat start of the same model. Either way the server is not ready the instant the command returns, which is why the health check exists and why scripts should poll it rather than sleep. While it loads, the protected endpoints answer 503 rather than refusing the connection.

Step 2: The built-in web UI

Open http://localhost:8080 in a browser and you get a chat interface, served by the same process. It is useful for a quick sanity check without writing any code.

Fetching it with curl is more interesting than it sounds:

curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080/

curl -s --compressed -o /dev/null -w "%{http_code}\n" http://localhost:8080/

The UI hits the thinking problem first

Type In one sentence, what is a GGUF file? into that chat box and, on this model, you will watch it generate for a while and then stop with nothing to show. The same request through the API explains why:

finish_reason : length
content       : ''
reasoning len : 16015
completion_tk : 4075

It produced 4,075 tokens — sixteen kilobytes of thinking — filled the slot’s context, and never wrote an answer. Step 3 covers the mechanism. The reason it bites here first is that the web UI sends no max_tokens and does not know about the per-request field that disables thinking, so there is nothing to cap it.

Fix it where the UI can benefit, on the server:

llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF:Q8_0 -c 8192 -np 2 \
  --chat-template-kwargs '{"enable_thinking":false}'

The same question now answers in 24 to 35 tokens with finish_reason of stop, in the UI and over the API alike.

Detailed breakdown

--chat-template-kwargs is the server-wide twin of the per-request field in Step 4. Set it at startup and every client benefits, including ones you do not control — the web UI, a plugin, anything written before reasoning models existed.
It is the right default for a server whose clients you do not own, and the wrong one for a shared server where some caller legitimately wants the reasoning. Per-request wins where you control the caller; server-wide wins for the UI.
The UI is stored gzip-compressed and served as-is. A client that does not advertise Accept-Encoding: gzip gets a 415 with the body Error: gzip is not supported by this browser, which reads like a bug and is not one. curl --compressed sends the header; every real browser already does.
This applies only to the static UI. The JSON endpoints below need no such flag.
--no-webui disables the interface if you are running this as a service and would rather not serve a chat page at all. GET / then returns 404 while /v1/models still answers 200 — the API is untouched.

Step 3: The first request

The endpoint is /v1/chat/completions, and the request body is OpenAI’s:

curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "In one sentence, what is a GGUF file?"}],
    "max_tokens": 80
  }' | python3 -m json.tool

What comes back is where this article earns its keep:

{
    "choices": [
        {
            "finish_reason": "length",
            "index": 0,
            "message": {
                "role": "assistant",
                "content": "",
                "reasoning_content": "Thinking Process:\n\n1.  **Analyze the Request:**\n    *   Target: A GGUF file.\n    *   Constraint: One sentence.\n    *   Context: Large Language Models (LLMs) / Quantization / Frameworks.\n\n2.  **Define \"GGUF\":**\n    *   GGUF stands for Google's GGML (Google Model Language"
            }
        }
    ],
    "model": "ggml-org/Qwen3.5-0.8B-GGUF:Q8_0",
    "system_fingerprint": "b10330-687e77892",
    "object": "chat.completion",
    "usage": {
        "completion_tokens": 80,
        "prompt_tokens": 21,
        "total_tokens": 101
    }
}

content is empty. The model spent all 80 tokens thinking, the thinking landed in reasoning_content, and finish_reason is length rather than stop. An OpenAI SDK client that reads .choices[0].message.content — which is every OpenAI example ever written — gets an empty string and no error.

Detailed breakdown

reasoning_content is llama.cpp’s extension, not part of the OpenAI schema. It exists so a reasoning model’s thinking can be returned without polluting the answer, which is the right design and also the reason a naive client sees nothing.
This is the same failure as the CLI’s, described in the getting-started article: Qwen3.5-0.8B opens a thinking block it is not large enough to close. Over HTTP it is quieter, because you get a well-formed 200 response containing nothing useful.
system_fingerprint carries the build number, b10330-687e77892 here. That is genuinely useful for bug reports, and it is the only place the server reports its version over the API.
usage counts what you were charged in tokens even though nothing is billed. Watching prompt_tokens grow across a conversation is how you notice a context problem before it becomes the error in Step 5.
timings is also in the response (trimmed above): prompt and generation token rates per request, which beats guessing at throughput.

Step 4: Turn the thinking off

The fix is a per-request field:

curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "messages": [{"role": "user", "content": "In one sentence, what is a GGUF file?"}],
    "max_tokens": 80,
    "chat_template_kwargs": {"enable_thinking": false}
  }' | python3 -c 'import json,sys; d=json.load(sys.stdin); print(d["choices"][0]["finish_reason"]); print(d["choices"][0]["message"]["content"])'

stop
A GGUF file is a compressed and optimized format for running large language models (LLMs) efficiently on hardware like GPUs.

Detailed breakdown

chat_template_kwargs passes values into the model’s chat template, which is a llama.cpp extension to the OpenAI body. enable_thinking is the switch Qwen’s template reads; other reasoning families use different names, so check the template rather than assuming this key is universal.
finish_reason flipping from length to stop is the signal to watch. It means the model finished a thought rather than running out of budget.
The server-wide equivalent is --chat-template-kwargs, which takes the same JSON and applies it to every request — that is what Step 2 uses to make the web UI usable. --reasoning-budget 0 and --reasoning-format are adjacent knobs, but the kwargs flag is the exact twin of this field. Per-request is better where you control the caller: someone who wants the thinking can ask.
This is where a local endpoint stops being a drop-in. The body is OpenAI-shaped, but a field OpenAI never defined is what makes the reply usable. Any client you write against a reasoning model needs to send it, which is exactly what the Python client in Step 8 does by default.

Step 5: Context size and slots

-c 8192 -np 2 did not give each request 8192 tokens. Ask the server:

curl -s http://localhost:8080/slots \
  | python3 -c 'import json,sys; s=json.load(sys.stdin); print(len(s), "slots,", s[0]["n_ctx"], "tokens each")'

2 slots, 4096 tokens each

Exceed one and the error says so exactly:

{"error":{"code":400,"message":"request (4010 tokens) exceeds the available context size (2048 tokens), try increasing it","type":"exceed_context_size_error","n_prompt_tokens":4010,"n_ctx":2048}}

Detailed breakdown

-c is the total KV cache, divided by -np. Two slots at -c 8192 means 4096 tokens each; four slots would mean 2048. The message above came from a server started with -c 4096 -np 2, which is why it names 2048.
Slots are concurrency. Each one serves a request at a time, so -np 2 handles two callers simultaneously and queues the third. -np -1, the default, lets llama.cpp choose.
The trade is real and unavoidable: KV cache is memory, and you are dividing a fixed pool. More concurrency means a smaller window per request. Size -c for the longest single request you intend to serve, multiplied by -np.
The 400 is a good error. It names the request size, the limit, and the remedy, and it arrives before any generation happens.
/slots is enabled by default and is worth checking whenever throughput looks wrong; --no-slots turns it off for a public-facing server.

Step 6: Scaffold the client project

The rest of the article builds a small Python client so the endpoint is usable from code rather than from curl, with the Step 4 fix applied by default.

Create the .gitignore before anything else. This project sits next to an API key and a server log, and the ignore file is doing real work.

Create the files

mkdir -p ~/projects/local-llm-client
cd ~/projects/local-llm-client
touch .gitignore

Add the code: `.gitignore`

# Python
__pycache__/
*.py[cod]
.venv/
*.egg-info/

# uv
.uv/

# pytest
.pytest_cache/
.coverage

# Models and server logs — never commit weights or transcripts
*.gguf
models/
server.log

# Secrets — the API key the server is started with
.env

# Editor / OS
.DS_Store
.idea/
.vscode/

Detailed breakdown

.env and server.log are the two entries specific to this project. The first would hold the API key from Step 13; the second is a transcript of every prompt the server has seen, which is not something to commit by accident.
*.gguf and models/ are here for the same reason as in the getting-started article: a stray copy of a model in the working tree is hundreds of megabytes that git will keep forever.
.venv/, __pycache__/, and .pytest_cache/ are all regenerated from the lockfile and the source, so they carry nothing worth sharing.

Step 7: Initialize with uv

Create the file

cd ~/projects/local-llm-client
uv init --package --name local-llm-client
uv add openai
uv add --dev pytest

Add the code: `pyproject.toml` (generated, shown for reference)

[project]
name = "local-llm-client"
version = "0.1.0"
description = "Add your description here"
readme = "README.md"
requires-python = ">=3.12"
dependencies = [
    "openai>=2.53.0",
]

[build-system]
requires = ["uv_build>=0.11.26,<0.12.0"]
build-backend = "uv_build"

[dependency-groups]
dev = [
    "pytest>=9.1.1",
]

Detailed breakdown

--package is not optional. A plain uv init creates an application with no [build-system], nothing gets installed into the environment, and src/ never lands on the import path — so python -m local_llm_client and every test fail with ModuleNotFoundError. With --package, uv sync installs the project and the module resolves.
Delete the [project.scripts] block uv generates. It points at a main in __init__.py that this project does not define; the entry point lives in __main__.py instead. uv also fills in authors from your git config, which is your name and email in a file you may publish.
The one runtime dependency is the official OpenAI SDK. That is the point of the exercise: the same library you would use against api.openai.com, aimed somewhere else. No llama.cpp-specific client is needed or exists.
requires-python = ">=3.12" is what uv chose from the interpreter on this machine. Nothing here needs 3.12 specifically.

Step 8: The client module

Create the file

: > src/local_llm_client/__init__.py
touch src/local_llm_client/client.py

Add the code: `src/local_llm_client/client.py`

"""Talk to a local llama.cpp server through the OpenAI SDK.

The endpoint is OpenAI-compatible, so the only thing that changes versus talking
to OpenAI is `base_url`. What is not the same is what a reasoning model puts in
the response: the answer can arrive in `reasoning_content` with `content` empty,
which is the failure this module exists to make impossible to miss.
"""

from __future__ import annotations

import os
from dataclasses import dataclass
from typing import Any

from openai import OpenAI

DEFAULT_BASE_URL = "http://localhost:8080/v1"
# llama.cpp ignores the key unless it was started with --api-key, but the SDK
# refuses to construct a client without one.
DEFAULT_API_KEY = "no-key-required"


class EmptyReplyError(RuntimeError):
    """The model produced no answer — usually it spent the budget thinking."""


@dataclass(frozen=True)
class Reply:
    """One assistant turn, with thinking kept separate from the answer."""

    text: str
    reasoning: str
    finish_reason: str

    @property
    def truncated(self) -> bool:
        """True when generation stopped at the token cap rather than finishing."""
        return self.finish_reason == "length"


def connect(
    base_url: str | None = None, api_key: str | None = None
) -> OpenAI:
    """Build an OpenAI client pointed at a local server.

    `LLAMA_BASE_URL` and `LLAMA_API_KEY` override the defaults, so the same code
    runs against a remote host without an edit.
    """
    return OpenAI(
        base_url=base_url or os.environ.get("LLAMA_BASE_URL", DEFAULT_BASE_URL),
        api_key=api_key or os.environ.get("LLAMA_API_KEY", DEFAULT_API_KEY),
    )


def build_request(
    prompt: str,
    *,
    model: str = "local",
    max_tokens: int = 256,
    thinking: bool = False,
    stream: bool = False,
) -> dict[str, Any]:
    """Assemble the request body, disabling thinking unless it is asked for.

    `chat_template_kwargs` is a llama.cpp extension passed through to the chat
    template. For a Qwen-family reasoning model, `enable_thinking: false` is what
    keeps the answer in `content`.
    """
    body: dict[str, Any] = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "max_tokens": max_tokens,
        "stream": stream,
    }
    if not thinking:
        body["extra_body"] = {"chat_template_kwargs": {"enable_thinking": False}}
    return body


def parse_choice(choice: Any) -> Reply:
    """Turn one response choice into a Reply, tolerating a null `content`.

    The OpenAI SDK types `content` as `str | None`, and llama.cpp really does
    return null when a reasoning model used every token on its thinking block.
    """
    message = choice.message
    text = (getattr(message, "content", None) or "").strip()
    reasoning = (getattr(message, "reasoning_content", None) or "").strip()
    return Reply(
        text=text,
        reasoning=reasoning,
        finish_reason=getattr(choice, "finish_reason", "") or "",
    )


def ask(
    client: OpenAI,
    prompt: str,
    *,
    max_tokens: int = 256,
    thinking: bool = False,
) -> Reply:
    """Send one prompt and return the reply, raising when there is no answer."""
    request = build_request(prompt, max_tokens=max_tokens, thinking=thinking)
    response = client.chat.completions.create(**request)
    reply = parse_choice(response.choices[0])
    if not reply.text:
        raise EmptyReplyError(
            "the model returned no content"
            + (
                f" but did produce {len(reply.reasoning)} characters of thinking; "
                "raise max_tokens or keep thinking disabled"
                if reply.reasoning
                else ""
            )
        )
    return reply


def stream_text(
    client: OpenAI,
    prompt: str,
    *,
    max_tokens: int = 256,
    thinking: bool = False,
):
    """Yield answer fragments as they arrive.

    Deltas carry `content: null` on the opening chunk and on any chunk that is
    part of a thinking block, so empty fragments are skipped rather than printed.
    """
    request = build_request(
        prompt, max_tokens=max_tokens, thinking=thinking, stream=True
    )
    for chunk in client.chat.completions.create(**request):
        if not chunk.choices:
            continue
        piece = getattr(chunk.choices[0].delta, "content", None)
        if piece:
            yield piece

Detailed breakdown

connect() is the entire “port an OpenAI app to local” story. Same class, same methods; base_url points at your Mac. The API key is required by the SDK constructor even though the server ignores it unless started with --api-key, which is why there is a placeholder default rather than None.
build_request disables thinking by default, which encodes Step 4 as behavior instead of documentation. thinking=True opts back in for a caller who wants the reasoning.
extra_body is how the OpenAI SDK sends fields it does not know about. chat_template_kwargs is not in the SDK’s typed request model, and passing it as a normal keyword argument raises a TypeError. This is the one place the SDK’s strictness gets in the way, and extra_body is the documented escape hatch.
parse_choice treats content: None as normal, because it is. The SDK types the field as str | None, and a reasoning model that ran out of budget really does return null. Coercing to "" here means every caller downstream handles one type.
ask raises rather than returning an empty string. The whole failure mode in Step 3 is that nothing looks wrong. EmptyReplyError names what happened and includes how much thinking was produced, which tells you whether to raise the cap or turn thinking off.
stream_text skips empty deltas. The first chunk of a stream carries content: null with the role, and thinking chunks do the same, so a naive print(chunk.choices[0].delta.content) prints None before any text.

Step 9: The command-line entry point

Create the file

touch src/local_llm_client/__main__.py

Add the code: `src/local_llm_client/main.py`

"""Ask the local server a question: `uv run python -m local_llm_client "..."`."""

from __future__ import annotations

import argparse
import sys

from openai import APIConnectionError, APIStatusError

from .client import EmptyReplyError, ask, connect, stream_text


def main(argv: list[str] | None = None) -> int:
    parser = argparse.ArgumentParser(
        prog="local-llm-client",
        description="Send a prompt to a local llama.cpp server.",
    )
    parser.add_argument("prompt", help="the question to ask")
    parser.add_argument(
        "-n", "--max-tokens", type=int, default=256, help="token cap (default: 256)"
    )
    parser.add_argument(
        "--stream", action="store_true", help="print fragments as they arrive"
    )
    parser.add_argument(
        "--thinking",
        action="store_true",
        help="leave the model's thinking block enabled (off by default)",
    )
    args = parser.parse_args(argv)

    client = connect()

    try:
        if args.stream:
            for piece in stream_text(
                client,
                args.prompt,
                max_tokens=args.max_tokens,
                thinking=args.thinking,
            ):
                print(piece, end="", flush=True)
            print()
            return 0

        reply = ask(
            client, args.prompt, max_tokens=args.max_tokens, thinking=args.thinking
        )
    except APIConnectionError:
        print(
            "error: no server at that address — start one with `make serve`",
            file=sys.stderr,
        )
        return 1
    except APIStatusError as exc:
        print(f"error: server returned {exc.status_code}: {exc.message}", file=sys.stderr)
        return 1
    except EmptyReplyError as exc:
        print(f"error: {exc}", file=sys.stderr)
        return 1

    print(reply.text)
    if reply.truncated:
        print("(stopped at the token cap — raise -n for more)", file=sys.stderr)
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Detailed breakdown

__main__.py makes the package runnable as python -m local_llm_client with no console script and no install step beyond uv sync.
The three error paths are separated on purpose, because they need different fixes: APIConnectionError means no server, APIStatusError means the server said no (a 401 from Step 13, or the 400 from Step 5), and EmptyReplyError means the model answered with nothing. Each returns 1, so the command is usable in a shell conditional.
The truncation notice goes to stderr, so make ask > answer.txt captures the answer alone while the warning still reaches the terminal.
--thinking exists to make the failure reproducible. Running with it and a low -n is the fastest way to see EmptyReplyError on demand.

Step 10: The tests

Create the file

mkdir -p tests
touch tests/test_client.py

Add the code: `tests/test_client.py`

"""Tests for the request building and response parsing.

Nothing here needs a running server: the parts worth pinning down are the ones
that decide whether a reply is usable. One opt-in test talks to a live server
when LLAMA_LIVE_TEST is set.
"""

from __future__ import annotations

import os
from types import SimpleNamespace

import pytest

from local_llm_client.client import (
    EmptyReplyError,
    Reply,
    ask,
    build_request,
    connect,
    parse_choice,
)


def _choice(content, reasoning=None, finish_reason="stop"):
    """A stand-in for one OpenAI SDK response choice."""
    message = SimpleNamespace(content=content, reasoning_content=reasoning)
    return SimpleNamespace(message=message, finish_reason=finish_reason)


def test_thinking_is_disabled_by_default():
    body = build_request("hello")
    assert body["extra_body"]["chat_template_kwargs"]["enable_thinking"] is False


def test_thinking_can_be_asked_for():
    body = build_request("hello", thinking=True)
    assert "extra_body" not in body


def test_request_carries_the_prompt_and_cap():
    body = build_request("why is the sky blue?", max_tokens=42)
    assert body["messages"] == [{"role": "user", "content": "why is the sky blue?"}]
    assert body["max_tokens"] == 42
    assert body["stream"] is False


def test_parse_choice_reads_a_normal_reply():
    reply = parse_choice(_choice("  A GGUF file is a container.  "))
    assert reply.text == "A GGUF file is a container."
    assert reply.reasoning == ""
    assert reply.truncated is False


def test_parse_choice_survives_a_null_content():
    """llama.cpp returns content: null when the model only produced thinking."""
    reply = parse_choice(_choice(None, reasoning="Thinking Process: ...", finish_reason="length"))
    assert reply.text == ""
    assert reply.reasoning == "Thinking Process: ..."
    assert reply.truncated is True


def test_truncated_is_only_true_for_a_length_stop():
    assert parse_choice(_choice("done", finish_reason="stop")).truncated is False
    assert parse_choice(_choice("cut", finish_reason="length")).truncated is True


def test_ask_raises_when_the_model_only_thought():
    """The empty-content case must fail loudly, not return an empty string."""

    class FakeCompletions:
        def create(self, **_):
            return SimpleNamespace(
                choices=[_choice(None, reasoning="x" * 120, finish_reason="length")]
            )

    fake = SimpleNamespace(chat=SimpleNamespace(completions=FakeCompletions()))

    with pytest.raises(EmptyReplyError, match="120 characters of thinking"):
        ask(fake, "anything")


def test_ask_returns_the_text_when_there_is_one():
    class FakeCompletions:
        def create(self, **_):
            return SimpleNamespace(choices=[_choice("Bananas.")])

    fake = SimpleNamespace(chat=SimpleNamespace(completions=FakeCompletions()))

    assert ask(fake, "name a fruit").text == "Bananas."


def test_reply_is_immutable():
    reply = Reply(text="a", reasoning="", finish_reason="stop")
    with pytest.raises(AttributeError):
        reply.text = "b"  # type: ignore[misc]


def test_connect_honors_the_environment(monkeypatch):
    monkeypatch.setenv("LLAMA_BASE_URL", "http://example.invalid:9931/v1")
    monkeypatch.setenv("LLAMA_API_KEY", "from-env")
    client = connect()
    assert str(client.base_url).rstrip("/") == "http://example.invalid:9931/v1"


@pytest.mark.skipif(
    not os.environ.get("LLAMA_LIVE_TEST"),
    reason="set LLAMA_LIVE_TEST=1 with a server running to exercise the real endpoint",
)
def test_live_server_answers():
    reply = ask(connect(), "Name one fruit.", max_tokens=32)
    assert reply.text

Detailed breakdown

No test needs a server. SimpleNamespace stands in for the SDK’s response objects, which is enough because the code under test only reads attributes. A suite that requires a running server and a loaded model is a suite that gets skipped.
test_parse_choice_survives_a_null_content is the important one. It pins the exact shape from Step 3 — null content, thinking present, finish_reason of length — so a future refactor cannot quietly reintroduce the empty-string bug.
test_ask_raises_when_the_model_only_thought asserts on the message, not just the exception type. The character count is the part that tells a user which knob to turn, so it is worth pinning.
test_thinking_is_disabled_by_default is a one-line test guarding a decision that is invisible at the call site. If someone removes the default, this is what says so.
The live test is opt-in through LLAMA_LIVE_TEST. It skips by default, so make test stays fast and hermetic, and make test-live exercises the real endpoint when a server is running.

Step 11: The Makefile

Create the file

touch Makefile

Add the code: `Makefile`

.DEFAULT_GOAL := help

MODEL_REPO ?= ggml-org/Qwen3.5-0.8B-GGUF:Q8_0
CTX        ?= 8192
SLOTS      ?= 2
PORT       ?= 8080
THINK      ?= on
PROMPT     ?= In one sentence, what is a GGUF file?
TOKENS     ?= 128
BASE       ?= http://localhost:$(PORT)

.PHONY: help install serve health slots ask stream models test test-live clean

help: ## Show this help screen
	@echo "Local OpenAI-compatible endpoint — llama.cpp"
	@echo ""
	@echo "Targets:"
	@grep -E '^[a-zA-Z_-]+:.*?## ' $(MAKEFILE_LIST) \
		| awk 'BEGIN {FS = ":.*?## "}; {printf "  %-10s %s\n", $$1, $$2}'
	@echo ""
	@echo "Variables: MODEL_REPO=$(MODEL_REPO) CTX=$(CTX) SLOTS=$(SLOTS) PORT=$(PORT) THINK=$(THINK)"

install: ## Sync dependencies with uv
	uv sync

serve: ## Run the server in the foreground (THINK=off fixes the web UI)
	llama serve -hf $(MODEL_REPO) -c $(CTX) -np $(SLOTS) --port $(PORT) \
		$(if $(filter off,$(THINK)),--chat-template-kwargs '{"enable_thinking":false}')

health: ## Check the server is up
	@curl -sf $(BASE)/health && echo "" || { echo "no server on $(BASE) — run: make serve"; exit 1; }

slots: ## Show each slot's own context size
	@curl -sf $(BASE)/slots \
		| python3 -c 'import json,sys; s=json.load(sys.stdin); print(len(s), "slots,", s[0]["n_ctx"], "tokens of context each")'

models: ## List what the server is serving
	@curl -sf $(BASE)/v1/models \
		| python3 -c 'import json,sys; [print(m["id"]) for m in json.load(sys.stdin)["data"]]'

ask: ## Ask PROMPT through the Python client
	uv run python -m local_llm_client "$(PROMPT)" -n $(TOKENS)

stream: ## Ask PROMPT and print fragments as they arrive
	uv run python -m local_llm_client "$(PROMPT)" -n $(TOKENS) --stream

test: ## Run the unit tests (no server needed)
	uv run pytest -q

test-live: ## Run the tests including the one that needs a running server
	LLAMA_LIVE_TEST=1 uv run pytest -q

clean: ## Remove Python build and test artifacts
	rm -rf .pytest_cache **/__pycache__ src/**/__pycache__

Detailed breakdown

.DEFAULT_GOAL := help means a bare make prints the target list, which matters more than usual here: serve blocks in the foreground, so an accidental default that started a server would be genuinely annoying.
health fails with an actionable message rather than printing nothing when the server is down. curl -sf returns non-zero on an HTTP error, which is what makes the || branch fire.
THINK=off adds --chat-template-kwargs to the server, which is what makes the web UI usable on a reasoning model. It is off by default so that Steps 3 to 5 reproduce as written — those steps depend on seeing the empty content the plain server returns. make serve THINK=off is the one to run when you actually want to use the UI.
serve runs in the foreground on purpose. You want the log visible while you are learning what the server does. Backgrounding it is a service-manager job, not a Makefile one.
test and test-live are separate targets so the fast, hermetic suite is the default and the one needing a live server is a deliberate choice.
Every variable is overridable: make serve CTX=16384 SLOTS=1 or make ask PROMPT="Name three fruits." TOKENS=40 work without editing the file.

Step 12: Run everything

With the server from Step 1 running in another terminal:

make            # help screen
make install    # sync the environment
make test       # 10 unit tests, no server needed
make health     # {"status":"ok"}
make models     # what the server is serving
make slots      # each slot's own context
make ask        # ask through the Python client
make stream     # the same, printed as it arrives
make test-live  # includes the test that needs the server

Real output from that sequence:

$ make slots
2 slots, 4096 tokens of context each

$ make models
ggml-org/Qwen3.5-0.8B-GGUF:Q8_0

$ make ask
A GGUF file is a compressed, lightweight, model-based format for serving deep learning models as a pre-trained language model (PLM) in the context of large language models (LLMs).

$ make test-live
11 passed

And the two failure paths, which are the ones worth seeing before you need them:

$ uv run python -m local_llm_client "In one sentence, what is a GGUF file?" -n 30 --thinking
error: the model returned no content but did produce 96 characters of thinking; raise max_tokens or keep thinking disabled

$ LLAMA_BASE_URL=http://localhost:9999/v1 uv run python -m local_llm_client "hi"
error: no server at that address — start one with `make serve`

Detailed breakdown

make test before make health is deliberate ordering. The suite passing with no server running is the proof that the tests are self-contained.
The answers vary between runs and are frequently wrong on the facts. This is a 0.8B model; it is here because it is fast and small, not because it knows what GGUF is. Swap MODEL_REPO for something larger and every command in this article is unchanged.
The --thinking failure above is reproducible on demand, which is why it is in the article rather than left as a surprise.

Step 13: Lock it down before it leaves your machine

The server has no authentication by default, and --host 0.0.0.0 would put an unauthenticated model endpoint with shell-adjacent capabilities on your network.

llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF:Q8_0 -c 8192 -np 2 --api-key demo-key-123

Without the key:

{"error":{"message":"Invalid API Key","type":"authentication_error","code":401}}

With it, the client needs one line:

LLAMA_API_KEY=demo-key-123 uv run python -m local_llm_client "Name one fruit."

Detailed breakdown

--api-key is the minimum, not a security model. It is a shared secret in a process argument, visible in ps. --api-key-file keeps it out of the command line, which is why that flag exists.
/health and /v1/models both stay open. Neither requires the key. /health answering 200 unauthenticated is what makes it usable as a load-balancer probe; /v1/models being open means an unauthenticated caller can also enumerate which model you are serving. The key does cover the endpoints where the capability and the real information live:

Endpoint	Without the key
`/health`	200
`/v1/models`	200
`/props`	401
`/slots`	401
`/v1/chat/completions`	401

The client already supports it. connect() reads LLAMA_API_KEY, so the same code works against an authenticated server without an edit.
Prefer a tunnel or a reverse proxy over --host 0.0.0.0 for anything beyond your own machine. The loopback default is the safest thing about this setup.

Troubleshooting

Symptom	Cause and fix
`content` is empty, `finish_reason` is `length`	A reasoning model used the whole budget thinking. Send `chat_template_kwargs: {"enable_thinking": false}`, or raise `max_tokens`.
The web UI generates for ages and shows nothing	Same cause, and the UI sends no `max_tokens` to cap it. Restart the server with `--chat-template-kwargs '{"enable_thinking":false}'` (`make serve THINK=off`).
`415` and `Error: gzip is not supported by this browser`	You fetched the web UI without gzip support. Use `curl --compressed`, or a browser.
`exceed_context_size_error`	The prompt is larger than one slot. Raise `-c`, or lower `-np` — the context is divided between slots.
`Invalid API Key` (401)	The server was started with `--api-key`. Set `LLAMA_API_KEY`, or pass `Authorization: Bearer <key>`.
Connection refused on 8080	The server is not up yet, or a future build moved the default port to 9931. Pass `--port` explicitly.
`ModuleNotFoundError: local_llm_client`	The project was created with `uv init` instead of `uv init --package`. See Step 7.
`TypeError` passing `chat_template_kwargs`	It is not a typed SDK parameter. Send it inside `extra_body`.

Recap

llama serve -hf <repo> -c <ctx> -np <slots> is the whole server. It speaks /v1/chat/completions, serves a web UI, and listens on loopback by default.
The response is OpenAI-shaped, but a reasoning model puts its output in reasoning_content and leaves content empty. Disabling thinking per request with chat_template_kwargs is what makes an OpenAI SDK client work unchanged.
-c is a pool divided by -np. Two slots at -c 8192 means 4096 tokens per request, and exceeding it is a 400 that names both numbers.
Pointing the official OpenAI SDK at http://localhost:8080/v1 is the entire porting effort. Fields the SDK does not know about travel in extra_body.
A client that returns an empty string when the model produced nothing is worse than one that raises. The failure is silent by default; make it loud.

Next improvements

Constrain the output with a GBNF grammar so the model cannot emit malformed JSON, rather than validating and retrying after the fact.
Put the endpoint behind a reverse proxy with TLS and a real credential store, which is the step --api-key is standing in for.
Measure the concurrency ceiling with llama bench and a load generator, and find where slot count stops helping on your hardware.
Serve two models from one host and route between them, which is where the model field in the request finally does something.

Prerequisites#

Step 1: Start the server#

Detailed breakdown#

Step 2: The built-in web UI#

The UI hits the thinking problem first#

Detailed breakdown#

Step 3: The first request#

Detailed breakdown#

Step 4: Turn the thinking off#

Detailed breakdown#

Step 5: Context size and slots#

Detailed breakdown#

Step 6: Scaffold the client project#

Create the files#

Add the code: .gitignore#

Detailed breakdown#

Step 7: Initialize with uv#

Create the file#

Add the code: pyproject.toml (generated, shown for reference)#

Detailed breakdown#

Step 8: The client module#

Create the file#

Add the code: src/local_llm_client/client.py#

Detailed breakdown#

Step 9: The command-line entry point#

Create the file#

Add the code: src/local_llm_client/__main__.py#

Detailed breakdown#

Step 10: The tests#

Create the file#

Add the code: tests/test_client.py#

Detailed breakdown#

Step 11: The Makefile#

Create the file#

Add the code: Makefile#

Detailed breakdown#

Step 12: Run everything#

Detailed breakdown#

Step 13: Lock it down before it leaves your machine#

Detailed breakdown#

Troubleshooting#

Recap#

Next improvements#

Prerequisites

Step 1: Start the server

Detailed breakdown

Step 2: The built-in web UI

The UI hits the thinking problem first

Detailed breakdown

Step 3: The first request

Detailed breakdown

Step 4: Turn the thinking off

Detailed breakdown

Step 5: Context size and slots

Detailed breakdown

Step 6: Scaffold the client project

Create the files

Add the code: `.gitignore`

Detailed breakdown

Step 7: Initialize with uv

Create the file

Add the code: `pyproject.toml` (generated, shown for reference)

Detailed breakdown

Step 8: The client module

Create the file

Add the code: `src/local_llm_client/client.py`

Detailed breakdown

Step 9: The command-line entry point

Create the file

Add the code: `src/local_llm_client/main.py`

Detailed breakdown

Step 10: The tests

Create the file

Add the code: `tests/test_client.py`

Detailed breakdown

Step 11: The Makefile

Create the file

Add the code: `Makefile`

Detailed breakdown

Step 12: Run everything

Detailed breakdown

Step 13: Lock it down before it leaves your machine

Detailed breakdown

Troubleshooting

Recap

Next improvements