Llama-Cpp

If you already have Python code that talks to a local model through openai.OpenAI, there is a second way to run that model: import it directly. The llama-cpp-python package binds llama.cpp into your own process, so the weights load inside your program and generation is a function call rather than an HTTP request. By the end of this article you will have taken a working OpenAI-SDK client and rewritten it around the Llama constructor. No server to start, no base_url, no API key, and no port to keep free. The finished module is about forty lines and comes with a test suite that runs without loading a model at all. ...

An agent is a program that lets a language model decide which of your functions to run, runs them, and hands back the results until the model says it is done. There is no framework in this article and no API key. By the end you will have a coding agent on your own machine that lists, reads and writes files inside a directory you choose, asks permission before running a shell command, and stops instead of spinning when the model gets stuck. The core agent is a few hundred lines of Python, plus a model-free pytest suite, for a 795 MiB model. ...

Qwen3.8-27B is a 27-billion-parameter dense model from the Qwen team, released August 2026 under Apache 2.0. It reads images and video as well as text, ships a multi-token-prediction head for faster decoding, and claims 262,144 tokens of native context. Quantized to Q4_K_M it is 17.7 GiB on disk. The context number is the interesting one. A conventional 27B model with 64 attention layers would need roughly 64 GiB of KV cache to hold 262,144 tokens, which is more memory than most machines have for the cache alone. Qwen3.8 gives only every fourth layer a real attention cache and runs the other 48 layers on a fixed-size recurrent state, so the same context costs 16 GiB. Step 6 reads those numbers straight out of the loader. ...

Muse Glimmer is a 30-billion-parameter agentic model from Meta Superintelligence Lab, released in August 2026 with a perception encoder for image input and a speculative-decoding drafter. Quantized to roughly 4 bits it fits in about 17 GB, which puts a capable agentic model inside the memory budget of a single consumer machine. This article installs a llama.cpp new enough to load it, pulls the three GGUF files, and runs the model four ways: a one-shot CLI answer, an OpenAI-compatible server, an image description, and a speculative-decoding run. It ends with a small Python project that measures decode throughput, because the numbers published for Apple Silicon were measured with a different runtime and do not transfer to this one. ...

One command turns a downloaded GGUF file into an HTTP server that speaks the OpenAI API. Code written against openai.OpenAI runs against it with one line changed — the base_url — and nothing leaves your machine. This article starts that server, reads what its startup log is telling you, drives it with curl, and then builds a small Python client on the official OpenAI SDK. The interesting part is not the happy path, which takes about ninety seconds. It is the two places where “OpenAI-compatible” stops being the whole story: a reasoning model can hand your SDK an empty content field, and the -c you pass is not the context each request gets. ...

llama.cpp runs large language models directly on your machine, with no Python runtime, no server process you did not start, and no account. On Apple Silicon it uses Metal for GPU work and the unified memory architecture means a model does not have to be copied to a separate card before it can run. This article installs llama.cpp with Homebrew, pulls a model from Hugging Face, runs it two ways, and shows exactly where the weights land on disk. It closes with a small Python project that reads a GGUF file’s header with nothing but the standard library, so the model file stops being an opaque blob you downloaded and becomes something you can inspect and assert against. ...