Llama-Cpp

Qwen3.8-27B is a 27-billion-parameter dense model from the Qwen team, released August 2026 under Apache 2.0. It reads images and video as well as text, ships a multi-token-prediction head for faster decoding, and claims 262,144 tokens of native context. Quantized to Q4_K_M it is 17.7 GiB on disk. The context number is the interesting one. A conventional 27B model with 64 attention layers would need roughly 64 GiB of KV cache to hold 262,144 tokens, which is more memory than most machines have for the cache alone. Qwen3.8 gives only every fourth layer a real attention cache and runs the other 48 layers on a fixed-size recurrent state, so the same context costs 16 GiB. Step 6 reads those numbers straight out of the loader. ...

Muse Glimmer is a 30-billion-parameter agentic model from Meta Superintelligence Lab, released in August 2026 with a perception encoder for image input and a speculative-decoding drafter. Quantized to roughly 4 bits it fits in about 17 GB, which puts a capable agentic model inside the memory budget of a single consumer machine. This article installs a llama.cpp new enough to load it, pulls the three GGUF files, and runs the model four ways: a one-shot CLI answer, an OpenAI-compatible server, an image description, and a speculative-decoding run. It ends with a small Python project that measures decode throughput, because the numbers published for Apple Silicon were measured with a different runtime and do not transfer to this one. ...

One command turns a downloaded GGUF file into an HTTP server that speaks the OpenAI API. Code written against openai.OpenAI runs against it with one line changed — the base_url — and nothing leaves your machine. This article starts that server, reads what its startup log is telling you, drives it with curl, and then builds a small Python client on the official OpenAI SDK. The interesting part is not the happy path, which takes about ninety seconds. It is the two places where “OpenAI-compatible” stops being the whole story: a reasoning model can hand your SDK an empty content field, and the -c you pass is not the context each request gets. ...

llama.cpp runs large language models directly on your machine, with no Python runtime, no server process you did not start, and no account. On Apple Silicon it uses Metal for GPU work and the unified memory architecture means a model does not have to be copied to a separate card before it can run. This article installs llama.cpp with Homebrew, pulls a model from Hugging Face, runs it two ways, and shows exactly where the weights land on disk. It closes with a small Python project that reads a GGUF file’s header with nothing but the standard library, so the model file stops being an opaque blob you downloaded and becomes something you can inspect and assert against. ...