Inkling 2 Mini is the small open-weights model Thinking Machines released on 2026-11-20: 26.4 billion parameters, text and image input, Apache 2.0, and a stated minimum of an Apple Silicon Mac with 32 GB of memory. This article gets it running on your Mac from its original weights, with no conversion step. You will have it answering prompts in the terminal, reading a chart you generated yourself and getting it right, serving an OpenAI-compatible API on localhost that your own Python code calls, and calling a tool you wrote to answer a question it could not answer alone.

You will also measure the two settings that change how it runs. One is the reasoning-effort dial, which the model card says cannot fully switch reasoning off; you will see what each level costs in tokens and whether it changes the answers. The other is a small draft model, which guesses tokens ahead for the full model to check, bundled with the checkpoint (the published set of weight and configuration files). On the M5 Max this was written on, it raised generation from about 37 tokens per second to between 46 and 48.

The tool that makes this possible is MLX, Apple’s array framework for Apple Silicon, through the mlx-vlm package, which runs vision-language models on top of it. The common alternative for local models is llama.cpp, an inference engine that loads models converted to its single-file GGUF format. The reason to use MLX instead is specific to this model, and Step 2 shows it in the checkpoint’s own files.

Prerequisites

  • A Mac with Apple Silicon and at least 32 GB of unified memory. MLX has no Intel path. Generation peaks at 21.5 to 22.5 GB here, so 32 GB is a real floor, not a cautious one. Validated on an Apple M5 Max with 128 GB, macOS 27.2.1.
  • uv 0.11 or later (brew install uv), the Python package and project manager used for every command. Validated with 0.13.6. uv downloads Python 3.12 itself if you do not have it.
  • About 21 GB of free disk for the checkpoint, which lands in the shared Hugging Face cache.
  • jq (brew install jq), for reading the checkpoint’s JSON in Step 2. Validated with 1.8.2.
  • make, for Step 10. The one in Apple’s command-line developer tools is fine.
  • Familiarity with chat-completion APIs (a list of messages, a model name, max_tokens). OpenAI’s chat completions guide covers the shape if it is new to you.

No Hugging Face account or token is needed; the model is public.

Step 1: Set up the project

Inkling 2 Mini support in mlx-vlm was merged on 2026-11-20, four days after the newest release (0.8.0, 2026-11-16) was cut, so uv add mlx-vlm installs a package that has never heard of this model. The fix is to install from a pinned git commit instead. This step creates the project, puts that pin in pyproject.toml, and adds a release-age gate for everything else it resolves.

Create the file

mkdir -p ~/src/inkling-mlx
cd ~/src/inkling-mlx
touch .gitignore

Add the code: ~/src/inkling-mlx/.gitignore

.venv/
__pycache__/
.pytest_cache/
*.log
.DS_Store
tmp/
out/

Detailed breakdown

  • The .gitignore comes first so nothing the later steps generate (the virtual environment, test caches, the chart image Step 5 writes under out/, and stray logs or scratch files) can be committed by accident.
  • The model weights are not listed because they never land in the project. Step 2 downloads them into the shared Hugging Face cache under ~/.cache/huggingface/hub/, so there is nothing 20 GB in this folder to ignore.

Now create the project and pin the dependency. uv init --bare writes only a pyproject.toml, with no sample code to delete. --python 3.12 sets its requires-python; without it, uv writes the newest Python it knows (>=3.14 here) and the uv python pin 3.12 on the next line fails as incompatible. The pin writes a .python-version so every uv run uses the same interpreter:

cd ~/src/inkling-mlx
uv init --bare --name inkling-mlx --python 3.12
uv python pin 3.12
printf '\n[tool.uv]\nexclude-newer = "3 days"\n' >> pyproject.toml
REV=f1618d6d52be4ef25d3fe65270f7528a59c55df3
uv add "mlx-vlm @ git+https://github.com/Blaizzy/mlx-vlm@$REV"
uv add openai
uv add --dev pytest

The resulting pyproject.toml should look like this (the lower bounds on openai and pytest are whatever was current for you):

[project]
name = "inkling-mlx"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = [
    "mlx-vlm",
    "openai>=3.55.1",
]

[tool.uv]
exclude-newer = "3 days"

[tool.uv.sources.mlx-vlm]
git = "https://github.com/Blaizzy/mlx-vlm"
rev = "f1618d6d52be4ef25d3fe65270f7528a59c55df3"

[dependency-groups]
dev = [
    "pytest>=9.2.0",
]

Confirm what you installed. The version string will mislead you, so ask the loader itself whether it can resolve this model. The checkpoint’s config.json declares "model_type": "inkling2_loop", and mlx-vlm picks the model code from that string:

uv run python - <<'EOF'
import mlx.core as mx, mlx_vlm
from mlx_vlm.utils import get_model_and_args

print(mx.__version__, mlx_vlm.__version__)
arch, model_type = get_model_and_args({"model_type": "inkling2_loop"})
print("inkling2_loop ->", model_type)
EOF
0.32.4 0.8.0
inkling2_loop -> inkling2

Detailed breakdown

  • The commit hash in REV is what matters in the uv add line. A bare git+https://... URL tracks main, which moves several times a day; the hash freezes the exact code this article was validated against. uv records it in pyproject.toml under [tool.uv.sources] and in uv.lock. Your file will have that source as one line, mlx-vlm = { git = "...", rev = "..." }; the listing above shows the same setting as a [tool.uv.sources.mlx-vlm] table, which TOML treats as identical, so it fits the page. Once a release newer than 0.8.0 ships, run uv remove mlx-vlm and then uv add "mlx-vlm>=<that version>". The removal matters: a plain uv add updates the version range but keeps the [tool.uv.sources] git entry, so you would still be on this commit.
  • exclude-newer = "3 days" refuses any package version younger than three days. It applies to the registry packages mlx-vlm pulls in (mlx, transformers and the rest), not to the git commit you named explicitly. It is the same gate described in Make npm and uv Wait Before Installing New Releases on macOS.
  • openai and pytest are for Steps 7 and 10. mlx-vlm serves an OpenAI-compatible API, so the official client is the simplest way to call it.
  • mlx_vlm.__version__ says 0.8.0 even at this commit. The version number in the source was not bumped after the release, so a git install between releases reports the old number.
  • get_model_and_args is the function mlx-vlm calls when it loads a checkpoint, so asking it directly is the same test the real load would run. Here it maps inkling2_loop onto the inkling2 model code. On a plain uv add mlx-vlm install of 0.8.0 the same line fails with ValueError: Model type inkling2_loop not supported, the same error that loading the model prints.
  • Do not test with import mlx_vlm.models.inkling2. That module exists in 0.8.0 too: it was added for the much larger Inkling 2 Small, and imports cleanly on a version that cannot load Mini.
  • mlx 0.32.4 is the array framework itself, pulled in by mlx-vlm. It is the part that runs NVFP4 and MXFP8 weights on the GPU, which Step 2 explains.

Step 2: Download the original checkpoint

Every later step would download the model on first use anyway, but a 20 GB download hidden inside a generation command looks like a hang. Fetching it on its own makes the wait visible, and fetching it at a pinned revision gets you the exact files this article was validated with. The hf command comes with huggingface-hub, which mlx-vlm installed in Step 1:

cd ~/src/inkling-mlx
uv run hf download thinkingmachines/Inkling-2-Mini \
  --revision b85c3cfa615fc413cc9af3379f491b963f3b586a

On this connection it took about four minutes. When it finishes, list what arrived. MODEL_DIR holds this model’s folder in the Hugging Face cache, so later commands stay short; set it again in any new terminal:

MODEL_DIR=~/.cache/huggingface/hub/models--thinkingmachines--Inkling-2-Mini
ls -lL $MODEL_DIR/snapshots/*/ | awk 'NR>1 {printf "%14s  %s\n", $5, $9}'
          6868  chat_template.jinja
          2722  config.json
            98  generation_config.json
         67924  hf_quant_config.json
   19526019168  model-00001-of-00001.safetensors
        109300  model.safetensors.index.json
     635801416  mtp.safetensors
           461  processor_config.json
         54870  README.md
           529  special_tokens_map.json
            96  tiktoken
         13235  tokenizer_config.json
      26178678  tokenizer.json

Detailed breakdown

  • model-00001-of-00001.safetensors is the whole model in one 19.5 GB file: the language model and the vision encoder (the part that turns an image into tokens the language model reads) together, so the vision side needs no second download.
  • mtp.safetensors (636 MB) holds the draft model Step 9 measures, which mlx-vlm calls the drafter. It ships in the same repository, so the download above already includes it.
  • chat_template.jinja decides how a conversation becomes tokens, and it is where Step 4’s reasoning-effort setting is implemented. You will read part of it there.
  • ls -L follows symbolic links. The snapshot folder holds links into a content-addressed store, so a plain ls -l or du -sh on it reports a few hundred kilobytes and makes a finished download look empty.
  • The revision hash pins what this download fetches, and only that. Later commands name the model as thinkingmachines/Inkling-2-Mini, which resolves to the repository’s main branch. Today that is the same commit, but if Thinking Machines pushes an update, the next online load fetches it. To keep every later step on these exact files, prefix the commands with HF_HUB_OFFLINE=1, which loads from the cache and never contacts Hugging Face.

Why MLX instead of a GGUF build

The other file worth reading is hf_quant_config.json, because it explains the choice of runtime. It lists every tensor (a named array of weights) in the model by the number format it is stored in:

cd $MODEL_DIR/snapshots/*/
jq -r '.model.parameter_groups[]
  | "\(.compute_dtype)\t\(.tensors | length) tensors"' hf_quant_config.json
jq -r '.model.parameter_groups[] | select(.compute_dtype == "mxfp8")
  | .tensors[]' hf_quant_config.json | head -4
cd ~/src/inkling-mlx
float32	74 tensors
bfloat16	993 tensors
nvfp4	40 tensors
mxfp8	8 tensors
bfloat16	1 tensors
model.llm.layers.20.mlp.experts.w13_weight
model.llm.layers.20.mlp.experts.w2_weight
model.llm.layers.21.mlp.experts.w13_weight
model.llm.layers.21.mlp.experts.w2_weight

Detailed breakdown

  • Inkling 2 Mini is a mixture-of-experts model. Each of its 24 layers holds 128 small feed-forward networks called experts, and a router sends each token to 8 of them, so only 3.4 billion of the 26.4 billion parameters are active for any one token. The middle 12 layers then run three times per token, the loop that gives the architecture its name, inkling2_loop, which raises the effective active figure to 5.4 billion. The experts are most of the model’s size.
  • The 48 expert tensors use a 4-bit and an 8-bit floating-point format. NVFP4 is NVIDIA’s 4-bit format (each value is a 4-bit float, and every block of 16 values shares a scale); MXFP8 is an 8-bit microscaling format, which likewise shares one scale across each block of values. Layers 0 to 19 store their two expert tensors in NVFP4 (40 tensors), layers 20 to 23 in MXFP8 (8 tensors). Everything else, attention and the vision encoder included, is 16- or 32-bit.
  • These are the formats the model was trained in, not a compression applied afterwards. Thinking Machines trained the experts in NVFP4, so these weights are the model itself. Running them unchanged is the only way to run exactly what was evaluated.
  • MLX 0.32 implements nvfp4 and mxfp8 as native quantization modes. Quantization means storing weights in a low-bit format with shared scales, usually to shrink a model after training; here the low-bit format is the original. The Inkling loader in mlx-vlm maps each parameter_groups entry above onto those modes, so there is no conversion step in this article. The llama.cpp route (ggml-org/Inkling-2-Mini-GGUF) requantizes the NVFP4 experts into a different 4-bit format, IQ4_NL, which its own model card measures as slightly less faithful than an 8-bit copy.

Step 3: Generate text from the command line

The first run proves the whole chain works (the pinned mlx-vlm, the native weights, the Metal GPU) before any of your own code is involved. It also shows the model’s raw output format, which every later step depends on: Inkling 2 Mini writes its reasoning and its answer as two separate messages, and you can see both here before the server starts hiding the markers in Step 6.

--temperature 0 makes the model pick its most likely token every time, so your output should match the text below word for word. --verbose prints the token counts and memory figures at the end:

uv run mlx_vlm.generate --model thinkingmachines/Inkling-2-Mini \
  --prompt "In two sentences, what does a Makefile do?" \
  --reasoning-effort none --temperature 0 --max-tokens 400 --verbose
==========
Files: [] 

Prompt: <|message_system|><|content_text|>Thinking effort level: 
0<|end_message|><|message_user|><|content_text|>In two sentences, what does a 
Makefile do?<|end_message|>
<|message_model|><|content_thinking|>The user asks: "In two sentences, what 
does a Makefile do?" Need answer in two sentences. Need ensure exactly two 
sentences. Explain Makefile: build automation file specifying 
rules/targets/dependencies and commands to build/compile/link/clean. Mention 
make reads it to execute tasks in correct order. Need concise. Final only two 
sentences. Ensure no extra sentences. Count sentences: 1) A Makefile is a 
build-automation script that lists targets, their dependencies, and the 
commands to run. 2) When you run make, it reads the file and executes the 
needed commands in the correct order to produce or update the requested 
outputs. That's two sentences. 
Good.<|end_message|><|message_model|><|content_text|>A Makefile is a 
build-automation script that lists targets, their dependencies, and the 
commands to run. When you run make, it reads the file and executes the needed 
commands in the correct order to produce or update the requested 
outputs.<|end_message|>
==========
Prompt: 24 tokens, 107.700 tokens-per-sec
Generation: 203 tokens, 37.474 tokens-per-sec
Peak memory: 21.476 GB

fold -s -w 79 wraps the output at 79 columns, breaking at spaces where it can, so the long model lines fit the page; without it each message is one long line. A progress bar for the prompt goes to standard error and is left out above. Loading takes a few seconds on later runs; the first run after the download also reads 20 GB from disk.

Detailed breakdown

  • The Prompt: line is your question after the chat template ran. Your one sentence became a system message, Thinking effort level: 0, followed by your user message. --reasoning-effort none is what produced that 0; Step 4 changes it.
  • <|content_thinking|> opens the reasoning, <|content_text|> opens the answer. Each is a separate <|message_model|> message. The reasoning reads like notes to itself (“Need ensure exactly two sentences”), and it is there even though you asked for effort none. The model card says as much: the minimum setting “does not fully disable reasoning”.
  • Reasoning tokens count against --max-tokens. Of the 203 tokens generated, most are reasoning. With --max-tokens 200 on this same prompt the answer stops just before its final period, with no error or warning. Set the limit for reasoning plus answer, never for the answer alone.
  • 37.5 tokens per second is the speed to compare against later. It varies by a few tenths between runs (37.7 on an earlier run of the same command), so treat the first decimal place as noise. Step 9 measures it properly.
  • Peak memory of 21.5 GB is the weights plus working memory for a short prompt. That is why 32 GB is the stated floor: the rest of macOS needs room too, and long prompts add more.

Step 4: Turn the reasoning effort up and down

Step 3 showed --reasoning-effort none turning into a system message. Knowing exactly how that happens tells you which values are valid, and the answer is narrower than the Hugging Face release post suggests, which lists numbers from 0.0 to 0.99 alongside the names. The mechanism lives in the first 20 lines of the chat template you downloaded. fold -s -w 79 wraps its longest lines to fit the page, so a few lines below continue onto the next:

MODEL_DIR=~/.cache/huggingface/hub/models--thinkingmachines--Inkling-2-Mini
sed -n 1,20p $MODEL_DIR/snapshots/*/chat_template.jinja | fold -s -w 79
{%- set effort_map = {"none": 0.0, "minimal": 0.1, "low": 0.2, "medium": 0.7, 
"high": 0.9, "xhigh": 0.99, "max": 0.99} -%}
{%- set role_token = {"user": "<|message_user|>", "assistant": 
"<|message_model|>", "system": "<|message_system|>", "tool": 
"<|message_tool|>"} -%}

{%- macro emit_thinking_effort() -%}
    {%- set eff = reasoning_effort if reasoning_effort is defined and 
reasoning_effort is not none else 0.9 -%}
    {%- if eff is string -%}
        {%- set key = eff | trim -%}
        {%- if key not in effort_map -%}
            {{- raise_exception("Unknown reasoning_effort: " ~ eff) -}}
        {%- endif -%}
        {%- set num = effort_map[key] -%}
    {%- else -%}
        {%- set num = eff | float -%}
    {%- endif -%}
    {%- if num < 0.0 or num > 0.99 -%}
        {{- raise_exception("reasoning_effort must be in [0.0, 0.99]") -}}
    {%- endif -%}
    {{- "<|message_system|><|content_text|>Thinking effort level: " -}}
    {%- if num == 0.0 -%}0{%- else -%}{{ num }}{%- endif -%}
    {{- "<|end_message|>" -}}

Detailed breakdown

  • The effort is a number the model reads, not a switch in the runtime. Seven names map to values between 0.0 and 0.99, and the template writes the number into a system message, Thinking effort level: 0.9. Nothing in MLX changes; the model was trained to spend more reasoning tokens when it reads a higher number.
  • Leaving it unset means high (0.9). That is the model’s default, so a client that never mentions effort gets a lot of reasoning.
  • xhigh and max are the same value, 0.99.
  • A string must be one of the seven names, or the template raises. That catches typos instead of ignoring them.
  • A number is only a number if it arrives as a number, and through mlx-vlm it never does. The template accepts 0.5 as a float, but mlx_vlm.generate passes the effort through as text, so --reasoning-effort 0.5 reaches the template as the string "0.5", fails the name lookup, and stops with TemplateError: Unknown reasoning_effort: 0.5. The server types the field as a string: a JSON number 0.5 is rejected with HTTP 422 before it reaches the template, and the string "0.5" fails in the template with HTTP 500. Only the seven names work.

Now run the same question at both ends of the dial. 391 is 17 × 23, so the right answer is no:

PATTERN='Thinking effort level: [0-9.]+'
PATTERN+='|content_text\|>[A-Za-z]+<|^Generation: .*'
for effort in none max; do
  uv run mlx_vlm.generate --model thinkingmachines/Inkling-2-Mini \
    --prompt "Is 391 a prime number? Answer yes or no." \
    --reasoning-effort "$effort" --temperature 0 --max-tokens 2000 \
    --verbose 2>/dev/null | grep -oE "$PATTERN"
done
Thinking effort level: 0
content_text|>no<
Generation: 70 tokens, 37.845 tokens-per-sec
Thinking effort level: 0.99
content_text|>no<
Generation: 196 tokens, 37.406 tokens-per-sec

Detailed breakdown

  • grep -oE "$PATTERN" pulls three things out of each run: the effort the model was shown, the answer between content_text|> and the next <, and the token count. The pattern is built in two pieces only to keep the lines short; += appends to a shell variable in both zsh and bash.
  • Same answer, almost three times the tokens. At max the model spent 196 tokens to reach the no it reached in 70 at none. At 38 tokens per second that is 1.8 seconds against 5.2 for a one-word answer.
  • One question shows the direction, not the size of the effect. Step 9 runs eight questions at five levels and finds the relationship is weaker and less tidy than this pair suggests.

Step 5: Ask about an image you drew yourself

A photo from your camera roll makes a poor test of the vision side, because you cannot tell a plausible description from a correct one. This step generates a chart whose right answer is fixed by the code that draws it: five bars, with the tallest in third place rather than last, so a model that assumes bars rise left to right gets it wrong. The test in Step 10 reuses the same script.

Create the file

cd ~/src/inkling-mlx
touch chart.py

Add the code: ~/src/inkling-mlx/chart.py

"""Draw a bar chart PNG whose correct description is known in advance.

A vision check is only a check if a wrong answer is recognisable, so the chart
is generated from constants rather than downloaded. The tallest bar is the
third one, not the last, so a model that guesses "the bars go up" is caught.
Only the standard library is used: `zlib` and `struct` are enough for a PNG.
"""

from __future__ import annotations

import struct
import sys
import zlib
from pathlib import Path

WIDTH, HEIGHT = 480, 320
BARS = (70, 140, 250, 110, 180)  # bar heights in pixels, left to right
BAR_COLOR = (230, 120, 20)  # orange
BAR_WIDTH, GAP = 60, 30
BACKGROUND = (255, 255, 255)


def expected_answer() -> tuple[int, int]:
    """Return (number of bars, 1-based position of the tallest bar)."""
    return len(BARS), BARS.index(max(BARS)) + 1


def pixels() -> list[list[tuple[int, int, int]]]:
    """Return the image as rows of RGB tuples, top row first."""
    rows = [[BACKGROUND] * WIDTH for _ in range(HEIGHT)]
    for i, height in enumerate(BARS):
        left = GAP + i * (BAR_WIDTH + GAP)
        for y in range(HEIGHT - height, HEIGHT):
            for x in range(left, left + BAR_WIDTH):
                rows[y][x] = BAR_COLOR
    return rows


def encode_png(rows: list[list[tuple[int, int, int]]]) -> bytes:
    """Encode RGB rows as an 8-bit truecolour PNG."""

    def chunk(kind: bytes, data: bytes) -> bytes:
        crc = zlib.crc32(kind + data)
        return (
            struct.pack(">I", len(data)) + kind + data + struct.pack(">I", crc)
        )

    raw = b"".join(
        b"\x00" + bytes(c for px in row for c in px) for row in rows
    )
    header = struct.pack(">IIBBBBB", len(rows[0]), len(rows), 8, 2, 0, 0, 0)
    return (
        b"\x89PNG\r\n\x1a\n"
        + chunk(b"IHDR", header)
        + chunk(b"IDAT", zlib.compress(raw, 9))
        + chunk(b"IEND", b"")
    )


def main() -> None:
    out = Path(sys.argv[1] if len(sys.argv) > 1 else "out/chart.png")
    out.parent.mkdir(parents=True, exist_ok=True)
    out.write_bytes(encode_png(pixels()))
    bars, tallest = expected_answer()
    print(f"wrote {out}: {bars} bars, tallest is bar {tallest}")


if __name__ == "__main__":
    main()

Detailed breakdown

  • BARS is the ground truth. Five heights, tallest third. Everything else is derived from it, including expected_answer(), which the tests in Step 10 compare against what the model says.
  • pixels() paints each bar from the bottom edge up. Bars are 60 pixels wide with 30-pixel gaps, so bar i starts at x = 30 + 90*i and the last one ends at x = 450, inside the 480-pixel width.
  • encode_png() writes the smallest valid PNG: the 8-byte signature, an IHDR chunk (width, height, 8 bits per channel, colour type 2 for RGB), one IDAT chunk of zlib-compressed rows, each prefixed with filter byte 0, and an empty IEND. Every chunk carries a CRC-32 of its type and data, which is what zlib.crc32 computes. No imaging library is needed.
  • The default output path is out/chart.png, which .gitignore already excludes.

Draw the chart, then ask the model about it. The question asks for a fixed reply format so the answer can be checked by string comparison:

uv run chart.py
uv run mlx_vlm.generate --model thinkingmachines/Inkling-2-Mini \
  --image out/chart.png \
  --prompt "How many bars are in this chart, and which bar, counting from \
the left, is the tallest? Reply exactly as: bars=<n> tallest=<k>" \
  --reasoning-effort none --temperature 0 --max-tokens 400 --verbose \
  2>/dev/null | fold -s -w 79
wrote out/chart.png: 5 bars, tallest is bar 3
==========
Files: ['out/chart.png'] 

Prompt: <|message_system|><|content_text|>Thinking effort level: 
0<|end_message|><|message_user|><|content_image|><|unused_200054|><|end_message
|><|message_user|><|content_text|>How many bars are in this chart, and which 
bar, counting from the left, is the tallest? Reply exactly as: bars=<n> 
tallest=<k><|end_message|>
<|message_model|><|content_thinking|>We need answer image counting bars. Need 
inspect image. Visible orange bars: left partial at x~30-89 height from y=290 
to 400; second x~120-180 y=180-400; third x~210-270 y=70-400; fourth x~300-360 
y=210-400; fifth x~390-450 y=140-400. So 5 bars. Tallest is third from left 
(top y=70, height 330). Need exact format bars=5 tallest=3. Ensure no 
extra.<|end_message|><|message_model|><|content_text|>bars=5 
tallest=3<|end_message|>
==========
Prompt: 149 tokens, 546.605 tokens-per-sec
Generation: 132 tokens, 37.461 tokens-per-sec
Peak memory: 21.659 GB

Detailed breakdown

  • bars=5 tallest=3 is correct, and it matches expected_answer().
  • The image became one placeholder in the prompt, <|unused_200054|>, but 149 prompt tokens. The prompt text, placeholder included, is 54 tokens; the vision encoder turned the 480 by 320 chart into 96 image tokens, which replace the placeholder before the language model runs.
  • The reasoning is worth reading against the code. Its x ranges are the real ones (30 to 90, 120 to 180, and so on), and four of the five bar tops it gives (180, 70, 210, 140) are exact: those are 320 minus each bar’s height. But it puts the floor at y = 400 in a 320-pixel image and the first bar’s top at 290 instead of 250. The final answer came out right; the measurements behind it were partly invented. Check answers, not the reasoning that led to them.
  • Memory rose by about 0.2 GB over Step 3, for the vision encoder’s work on one small image.

Step 6: Serve it on localhost

Every mlx_vlm.generate call so far loaded 20 GB of weights, answered once, and exited. A server loads them once and keeps them in memory, and because mlx_vlm.server speaks the OpenAI chat-completions protocol, any client written for that API can use the model by changing its base URL. Steps 7 and 8, the effort half of Step 9, and the live tests in Step 10 talk to this server, so start it in a second terminal and leave it running.

The --host flag matters here. The server’s default is 0.0.0.0, which listens on every network interface, so anyone on your Wi-Fi could send it prompts; there is no API key unless you pass --api-key. 127.0.0.1 keeps it on this Mac:

cd ~/src/inkling-mlx
uv run mlx_vlm.server --host 127.0.0.1 --port 8080 \
  --model thinkingmachines/Inkling-2-Mini --log-level WARNING
INFO:     Started server process [8568]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://127.0.0.1:8080 (Press CTRL+C to quit)

Your process ID will differ, and you will also see a Fetching 12 files progress line, left out above. From the first terminal, ask the server whether it is ready:

curl -s http://127.0.0.1:8080/health | jq '{status, model_ready,
  loaded_model, loaded_tool_parser, effective_context_limit}'
{
  "status": "healthy",
  "model_ready": true,
  "loaded_model": "thinkingmachines/Inkling-2-Mini",
  "loaded_tool_parser": "inkling",
  "effective_context_limit": 1048576
}

Detailed breakdown

  • --model preloads the weights at startup, so the first request does not wait for a 20 GB load. Without it, the server loads whichever model the first request names.
  • --log-level WARNING keeps the server terminal readable. At the default INFO level it logs each load stage and several timestamped lines per request, most of them wider than a terminal. The four lines above come from the web server underneath, which also prints one short line per request ("GET /health HTTP/1.1" 200 OK) whatever the level.
  • The server checks Hugging Face for a newer revision of main at load, even though the files are cached. It works offline too: set HF_HUB_OFFLINE=1 before the command and it loads from the cache without the check.
  • model_ready: true is the field to wait for, because it is about the weights rather than the HTTP server. The full response also carries preload_in_progress and a model_load_progress object while a load is under way.
  • loaded_tool_parser: "inkling" means mlx-vlm recognized the model’s tool-call format and will turn it into structured OpenAI tool calls. Step 8 depends on that.
  • effective_context_limit is 1,048,576 tokens, the model’s full 1M context. That is the most a single request can hold, not memory set aside in advance; memory grows with the tokens actually in use.

Step 7: Call it from Python with the OpenAI SDK

This is the step that makes the model usable from your own code. The script below sends a prompt, optionally with an image, through the official openai package, and prints the reasoning and the answer separately. That separation is the main thing the server adds over Step 3: it parses the <|content_thinking|> and <|content_text|> messages and returns them in different fields, so your code gets a clean answer without stripping markers.

Create the file

cd ~/src/inkling-mlx
touch ask.py

Add the code: ~/src/inkling-mlx/ask.py

"""Ask the local Inkling 2 Mini server a question, optionally about an image.

Talks to `mlx_vlm.server` through the official OpenAI client. The server keeps
the model's reasoning apart from its answer, so both are printed separately.
"""

from __future__ import annotations

import argparse
import base64
import os
import textwrap
from dataclasses import dataclass
from pathlib import Path

from openai import OpenAI

BASE_URL = os.environ.get("INKLING_URL", "http://127.0.0.1:8080/v1")
MODEL = "thinkingmachines/Inkling-2-Mini"
EFFORTS = ("none", "minimal", "low", "medium", "high", "xhigh", "max")


@dataclass
class Reply:
    answer: str
    reasoning: str
    completion_tokens: int
    finish_reason: str


def client() -> OpenAI:
    # The server ignores the key, but the client will not start without one.
    return OpenAI(base_url=BASE_URL, api_key="local")


def image_part(path: Path) -> dict:
    """Wrap a PNG as a data URL, the chat API's form for inline images."""
    data = base64.b64encode(path.read_bytes()).decode("ascii")
    return {
        "type": "image_url",
        "image_url": {"url": f"data:image/png;base64,{data}"},
    }


def ask(
    prompt: str,
    effort: str = "none",
    image: Path | None = None,
    max_tokens: int = 1024,
) -> Reply:
    if effort not in EFFORTS:
        raise ValueError(f"effort must be one of {EFFORTS}, got {effort!r}")
    content: list[dict] = [{"type": "text", "text": prompt}]
    if image is not None:
        content.insert(0, image_part(image))
    resp = client().chat.completions.create(
        model=MODEL,
        messages=[{"role": "user", "content": content}],
        max_tokens=max_tokens,
        temperature=0,
        reasoning_effort=effort,
    )
    choice = resp.choices[0]
    msg = choice.message
    return Reply(
        answer=msg.content or "",
        # Not in the OpenAI schema; the SDK keeps unknown fields as attributes.
        reasoning=getattr(msg, "reasoning_content", None) or "",
        completion_tokens=resp.usage.completion_tokens if resp.usage else 0,
        finish_reason=choice.finish_reason or "",
    )


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__)
    parser.add_argument("prompt")
    parser.add_argument("--effort", default="none", choices=EFFORTS)
    parser.add_argument("--image", type=Path)
    parser.add_argument("--max-tokens", type=int, default=1024)
    args = parser.parse_args()
    reply = ask(args.prompt, args.effort, args.image, args.max_tokens)
    print(f"reasoning: {len(reply.reasoning)} chars")
    answer = textwrap.fill(
        reply.answer.strip(),
        width=79,
        initial_indent="answer:    ",
        subsequent_indent=" " * 11,
    )
    print(answer)
    print(f"tokens:    {reply.completion_tokens} ({reply.finish_reason})")


if __name__ == "__main__":
    main()

Detailed breakdown

  • BASE_URL is the only thing that makes this a local client. Everything else is the standard OpenAI SDK call. INKLING_URL lets you point it at another port or machine without editing the file.
  • api_key="local" is a placeholder. The SDK refuses to build a client without a key; the server, started without --api-key, ignores it.
  • reasoning_effort is a standard parameter of the SDK’s chat.completions.create, and the server feeds it to the chat template from Step 4. The script defaults to none because the model’s own default, used when the field is absent, is high.
  • ask() rejects anything but the seven names before sending. The server would answer 0.5 or a typo with an HTTP 500 carrying Unknown reasoning_effort: 0.5; checking locally gives a clear error without a round trip.
  • Images go in as a data: URL inside an image_url content part, the chat API’s form for inline images, placed before the text as in Step 5.
  • textwrap.fill wraps the answer at 79 columns, indenting continuation lines to line up under the first, so a long answer stays readable in a terminal.
  • reasoning_content is not in the OpenAI response schema, so the SDK does not declare it, but it keeps unknown fields from the JSON as attributes on the message. getattr with a default reads it without breaking against a server that does not send it.
  • finish_reason tells you whether the answer is complete. stop means the model ended its answer; length means max_tokens cut it off, which is the silent truncation from Step 3.

With the server from Step 6 still running, ask the Step 3 question and the Step 5 chart question again:

uv run ask.py "In two sentences, what does a Makefile do?"
uv run ask.py "How many bars are in this chart, and which bar, counting from \
the left, is the tallest? Reply exactly as: bars=<n> tallest=<k>" \
  --image out/chart.png
reasoning: 659 chars
answer:    A Makefile is a build-automation script that lists targets, their
           dependencies, and the commands to run. When you run make, it reads
           the file and executes the needed commands in the correct order to
           produce or update the requested outputs.
tokens:    203 (stop)
reasoning: 350 chars
answer:    bars=5 tallest=3
tokens:    132 (stop)

Detailed breakdown

  • Both answers and both token counts match the command-line runs in Steps 3 and 5 (203 and 132 tokens). Do not rely on that being exact: the server batches requests through a different code path, and the Step 4 prime question at max took 194 tokens here against 196 on the command line, with the same answer.
  • The 659 characters of reasoning arrived in reasoning_content, with no <|...|> markers in either field.
  • Both finished with stop, so neither was truncated by the 1,024-token default in ask().

Step 8: Let it call a tool

A tool call is how a model reaches past what it knows: you describe a function, the model replies with a request to call it instead of an answer, your code runs it, and the result goes back as a new message. The tool here computes a file’s SHA-256, which a language model cannot compute from its weights, so a correct final answer proves the call happened. It is also restricted to files inside the project, which is the habit to build before giving a model any tool that touches the disk.

Create the file

cd ~/src/inkling-mlx
touch agent.py

Add the code: ~/src/inkling-mlx/agent.py

"""Give the local model one tool and let it decide when to call it.

The tool computes a file's SHA-256, something no language model can do in its
head, so a correct final answer proves the tool really ran. The loop is the
standard one: send the tools, run any call the model makes, send the result
back, and repeat until the model answers in plain text.
"""

from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path

from ask import MODEL, client

ROOT = Path(__file__).resolve().parent
MAX_TURNS = 4

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "file_sha256",
            "description": "Return the SHA-256 hex digest of a project file.",
            "parameters": {
                "type": "object",
                "properties": {
                    "path": {
                        "type": "string",
                        "description": "Path relative to the project root.",
                    }
                },
                "required": ["path"],
            },
        },
    }
]


def file_sha256(path: str) -> str:
    """Hash a file, refusing anything outside the project folder."""
    target = (ROOT / path).resolve()
    if not target.is_relative_to(ROOT):
        return f"error: {path} is outside the project"
    if not target.is_file():
        return f"error: {path} does not exist"
    return hashlib.sha256(target.read_bytes()).hexdigest()


def run(prompt: str) -> str:
    messages: list[dict] = [{"role": "user", "content": prompt}]
    for _ in range(MAX_TURNS):
        resp = client().chat.completions.create(
            model=MODEL,
            messages=messages,
            tools=TOOLS,
            max_tokens=1024,
            temperature=0,
            reasoning_effort="low",
        )
        msg = resp.choices[0].message
        if not msg.tool_calls:
            return msg.content or ""
        messages.append(
            {
                "role": "assistant",
                "content": msg.content or "",
                "tool_calls": [
                    {
                        "id": call.id,
                        "type": "function",
                        "function": {
                            "name": call.function.name,
                            "arguments": call.function.arguments,
                        },
                    }
                    for call in msg.tool_calls
                ],
            }
        )
        for call in msg.tool_calls:
            args = json.loads(call.function.arguments or "{}")
            result = (
                file_sha256(**args)
                if call.function.name == "file_sha256"
                else "error: unknown tool"
            )
            print(f"tool call: {call.function.name}({args})")
            print(f"  result:  {result}")
            messages.append(
                {"role": "tool", "tool_call_id": call.id, "content": result}
            )
    return "error: no answer after the turn limit"


if __name__ == "__main__":
    question = (
        sys.argv[1]
        if len(sys.argv) > 1
        else "What is the SHA-256 of out/chart.png?"
    )
    print(f"answer: {run(question)}")

Detailed breakdown

  • TOOLS is a JSON Schema description of the function. The model never sees file_sha256 itself, only this name, description and parameter list, which the chat template writes into a system message. The description is what the model uses to decide the tool fits the question.
  • file_sha256() resolves the path before checking it. resolve() collapses .. and follows symlinks, so ../outside.txt and /etc/hosts both land outside ROOT and get an error string instead of a hash. Errors go back to the model as text rather than raising, so it can read them and respond.
  • The loop is the agent. A reply with tool_calls gets appended to the history as an assistant message, each call is run, and each result goes back as a tool message whose tool_call_id matches the call. The model then sees its own request and the answer, and either calls again or replies in plain text, which ends the loop.
  • MAX_TURNS = 4 bounds the loop to four requests. The model card warns that Inkling 2 Mini can get “stuck in long reasoning loops”; a model that keeps calling tools would otherwise run forever. The same warning is why every request sets max_tokens.
  • reasoning_effort="low" is enough to pick the right tool for a question this direct. Raise it for tasks where choosing between tools takes thought.
  • arguments arrives as a JSON string, as the OpenAI format specifies, and goes back into the history unchanged. The chat template itself demands a parsed object there, and the server converts it before rendering, so the standard client code works as written.

Run it, then check the model’s answer against shasum:

uv run agent.py
shasum -a 256 out/chart.png
tool call: file_sha256({'path': 'out/chart.png'})
  result:  0b593372d7b20e33fec9e938c3e7f9275295090ddede6eef755db779198528d8
answer: 0b593372d7b20e33fec9e938c3e7f9275295090ddede6eef755db779198528d8
0b593372d7b20e33fec9e938c3e7f9275295090ddede6eef755db779198528d8  out/chart.png

Detailed breakdown

  • The model chose to call the tool, with the right path, then repeated the 64-character digest exactly in its final answer.
  • The digests agree: the tool’s result, the model’s answer, and what shasum computes independently. agent.py prints the call and its result on separate lines because a 64-character digest after the call does not fit a terminal line.
  • Your digest may differ from this one if your Python’s zlib compresses the chart differently. What must hold is that your three digests match each other.
  • Two requests were made: one that came back with finish_reason: tool_calls and no text, and one, after the tool result was appended, that came back with the answer.

Step 9: Measure the effort dial and the drafter

Two settings change how this model runs, and neither should be taken on trust. The effort dial is supposed to trade tokens for quality; one question in Step 4 showed the token side, but not whether the answers improve. The draft model is supposed to make generation faster; how much depends on your hardware and your text. This script measures both, so the numbers you act on are your own.

Create the file

cd ~/src/inkling-mlx
touch probe.py

Add the code: ~/src/inkling-mlx/probe.py

"""Measure what the reasoning-effort dial and the bundled drafter actually buy.

`effort` sends the same questions at several effort levels through the local
server and reports, per level, how many questions came back right and how many
tokens the model spent. `draft` runs one prompt through `mlx_vlm.generate`
with and without the drafter and reports tokens per second.
"""

from __future__ import annotations

import argparse
import re
import statistics
import subprocess
import time

from ask import MODEL, ask

# Questions with one checkable answer each. Answers are compared after
# lower-casing and stripping punctuation, so "Saturday." matches "saturday".
QUESTIONS = [
    ("What is 47 * 83?", "3901"),
    ("Is 391 a prime number? Answer yes or no.", "no"),
    ("What is the remainder when 1000 is divided by 37?", "1"),
    ("How many prime numbers are less than 50?", "15"),
    ("What day of the week was 1 January 2000?", "saturday"),
    ("What is the sum of the integers from 1 to 250?", "31375"),
    (
        "How many times does the letter r appear in 'strawberry raspberry'?",
        "6",
    ),
    ("What is 2 to the power of 20?", "1048576"),
]
SUFFIX = " Reply with only the answer."
GENERATION = re.compile(
    r"^Generation: (\d+) tokens, ([\d.]+) tokens-per-sec", re.M
)
ACCEPTED = re.compile(r"([\d.]+)% of drafted")


def normalize(text: str) -> str:
    return re.sub(r"[^a-z0-9]", "", text.lower())


def effort_sweep(efforts: list[str], max_tokens: int) -> None:
    print(
        f"{'effort':<8} {'correct':>7} {'mean tokens':>11}"
        f" {'most tokens':>11} {'seconds':>8}"
    )
    for effort in efforts:
        right, tokens = 0, []
        start = time.monotonic()
        for question, expected in QUESTIONS:
            reply = ask(
                question + SUFFIX, effort=effort, max_tokens=max_tokens
            )
            right += normalize(reply.answer) == expected
            tokens.append(reply.completion_tokens)
        elapsed = time.monotonic() - start
        print(
            f"{effort:<8} {right:>4}/{len(QUESTIONS):<2}"
            f" {statistics.mean(tokens):>11.1f}"
            f" {max(tokens):>11} {elapsed:>8.1f}"
        )


def generate(
    prompt: str, draft: bool, max_tokens: int
) -> tuple[int, float, str]:
    cmd = [
        "mlx_vlm.generate",
        "--model",
        MODEL,
        "--prompt",
        prompt,
        "--reasoning-effort",
        "none",
        "--temperature",
        "0",
        "--max-tokens",
        str(max_tokens),
        "--verbose",
    ]
    if draft:
        cmd += ["--draft-model", MODEL]
    out = subprocess.run(cmd, capture_output=True, text=True, check=True)
    log = out.stdout + out.stderr
    match = GENERATION.search(log)
    if match is None:
        raise RuntimeError("no Generation line in mlx_vlm.generate output")
    accepted = ACCEPTED.search(log)
    return (
        int(match.group(1)),
        float(match.group(2)),
        f"{accepted.group(1)}%" if accepted else "-",
    )


def draft_compare(runs: int, max_tokens: int) -> None:
    prompt = (
        "Write a Python function that returns the n-th Fibonacci number "
        "iteratively, with a docstring and three doctest examples."
    )
    for draft in (False, True):
        label = "yes" if draft else "no"
        rates = []
        for i in range(runs):
            tokens, rate, accepted = generate(prompt, draft, max_tokens)
            rates.append(rate)
            print(
                f"draft={label:<3} run {i + 1}: {tokens} tokens, "
                f"{rate:.1f} tok/s, accepted {accepted}"
            )
        median = statistics.median(rates)
        print(f"draft={label:<3} median: {median:.1f} tok/s")


def main() -> None:
    parser = argparse.ArgumentParser(description=__doc__)
    sub = parser.add_subparsers(dest="command", required=True)
    e = sub.add_parser(
        "effort", help="sweep reasoning effort through the server"
    )
    e.add_argument("--efforts", default="none,low,medium,high,max")
    e.add_argument("--max-tokens", type=int, default=4096)
    d = sub.add_parser(
        "draft", help="compare speed with and without the drafter"
    )
    d.add_argument("--runs", type=int, default=5)
    d.add_argument("--max-tokens", type=int, default=800)
    args = parser.parse_args()
    if args.command == "effort":
        effort_sweep(args.efforts.split(","), args.max_tokens)
    else:
        draft_compare(args.runs, args.max_tokens)


if __name__ == "__main__":
    main()

Detailed breakdown

  • QUESTIONS pairs each prompt with its one right answer. 47 × 83 = 3,901; 391 = 17 × 23; 1,000 = 37 × 27 + 1; there are 15 primes below 50; 1 January 2000 was a Saturday; 1 + 2 + … + 250 = 250 × 251 / 2 = 31,375; “strawberry” and “raspberry” have three r’s each; 2²⁰ = 1,048,576.
  • normalize() makes the comparison forgiving about format but not about content. 1,048,576, Saturday. and No all match; 1048575 does not.
  • effort_sweep() goes through the server, reusing ask() from Step 7, so it measures exactly the path your own code uses. max_tokens defaults to 4,096 so that no level is cut off before it finishes reasoning.
  • generate() shells out to mlx_vlm.generate rather than the server, because the --verbose summary reports tokens per second for generation alone, without HTTP time, and the drafter’s acceptance rate, which the server does not return.
  • draft_compare() takes the median of five runs per setting. Single runs vary by up to a few tokens per second, and one sample would turn that noise into a result.
  • --draft-model names the same repository as --model. That is how mlx-vlm finds the bundled mtp.safetensors.

With the server still running, run the effort sweep. It sends 40 requests and takes about three minutes:

uv run probe.py effort
effort   correct mean tokens most tokens  seconds
none        8/8        134.8         338     30.9
low         8/8        110.9         232     26.0
medium      8/8        138.4         408     31.8
high        8/8        155.6         471     36.5
max         8/8        196.1         574     47.0

Detailed breakdown

  • Every level got every question right. On questions this size the dial does not buy correctness; it costs tokens, from 110.9 on average at low to 196.1 at max, 1.8 times as many, and 26.0 seconds against 47.0 for the whole set.
  • The token columns repeat exactly between runs, because temperature 0 makes each answer deterministic. Only the seconds column varies.
  • The dial is not monotonic, and none is not the cheapest level. none averaged more tokens than low. Per question, low spent fewer tokens than none on five of the eight, the same on two, and more on one; even max spent fewer than none on two. The higher levels tend to cost more, but you cannot predict any one answer’s cost from the level.
  • The expensive question is the one models find hard: counting letters. “How many times does the letter r appear” produced the largest count at every level, which is the most tokens column: 338, 232, 408, 471 and 574.
  • The practical setting is low for simple requests. It was the cheapest level here and lost nothing. Save the higher levels for problems where you can show they change the answer, and measure that on your own questions before paying for it.

The drafter comparison loads the model itself in each of its ten runs, so stop the server first (Ctrl+C in its terminal). Running both at once needs room for two 21 GB copies, which a 32 GB Mac does not have. It takes about five minutes:

uv run probe.py draft
draft=no  run 1: 510 tokens, 35.8 tok/s, accepted -
draft=no  run 2: 510 tokens, 36.3 tok/s, accepted -
draft=no  run 3: 510 tokens, 36.9 tok/s, accepted -
draft=no  run 4: 510 tokens, 36.9 tok/s, accepted -
draft=no  run 5: 510 tokens, 36.9 tok/s, accepted -
draft=no  median: 36.9 tok/s
draft=yes run 1: 385 tokens, 48.4 tok/s, accepted 38.4%
draft=yes run 2: 385 tokens, 45.5 tok/s, accepted 38.4%
draft=yes run 3: 385 tokens, 46.9 tok/s, accepted 38.4%
draft=yes run 4: 385 tokens, 46.6 tok/s, accepted 38.4%
draft=yes run 5: 385 tokens, 46.7 tok/s, accepted 38.4%
draft=yes median: 46.7 tok/s

Detailed breakdown

  • The drafter raised generation from 36.9 to 46.7 tokens per second, a factor of 1.27. Three other sessions on the same machine gave 1.32, 1.30 and 1.26 (drafted medians of 48.2, 48.0 and 46.6), so treat it as about 1.3.
  • How the drafter works. The small draft model guesses several tokens ahead, about seven per round here, and the full model checks all of the guesses in a single pass, keeping the ones it agrees with. 38.4% of the guesses were kept. Each round cost one pass of the full model and produced several tokens instead of one, which is where the speedup comes from. The technique is called speculative decoding.
  • The two settings did not produce the same text. Speculative decoding is described as lossless, meaning the full model’s choices are unchanged, yet the plain run wrote 510 tokens and the drafted run 385, and their reasoning diverges about 180 characters in. The likely cause is arithmetic: checking several tokens in one batched pass rounds slightly differently from checking one at a time, and at temperature 0 a near-tie can tip to a different token, after which the two texts go separate ways. Both outputs were working functions whose three doctests pass; they were not the same function.
  • Turn it on in the server by adding --draft-model thinkingmachines/Inkling-2-Mini to the Step 6 command. It costs about 1 GB more memory: peak memory on the command line went from 21.5 GB to 22.5 GB.

Step 10: Lock it down with tests and a Makefile

Everything so far was checked by eye. Tests turn the three facts this article depends on into checks that fail loudly: the chart has the answer chart.py claims, and the model, through the server, reads it and uses the tool. They are split in two files because only one of them needs a 20 GB model in memory: the offline tests run anywhere in under a second, and the live tests skip themselves when no server is listening.

First tell pytest where the modules are. The project has no package structure, so pythonpath = ["."] lets the tests import chart, ask and agent from the project root:

cd ~/src/inkling-mlx
mkdir -p tests
cat >> pyproject.toml <<'EOF'

[tool.pytest.ini_options]
pythonpath = ["."]
testpaths = ["tests"]
EOF

Create the file

cd ~/src/inkling-mlx
touch tests/test_offline.py

Add the code: ~/src/inkling-mlx/tests/test_offline.py

"""Tests that need no model: the chart, the tool, and argument checks."""

from __future__ import annotations

import struct
import zlib

import pytest

import agent
import ask
import chart


def decode_png(data: bytes) -> list[list[tuple[int, int, int]]]:
    """Decode the PNGs chart.py writes (8-bit RGB, filter type 0 only)."""
    assert data[:8] == b"\x89PNG\r\n\x1a\n"
    pos, idat, width, height = 8, b"", 0, 0
    while pos < len(data):
        (length,) = struct.unpack(">I", data[pos : pos + 4])
        kind, body = data[pos + 4 : pos + 8], data[pos + 8 : pos + 8 + length]
        if kind == b"IHDR":
            width, height = struct.unpack(">II", body[:8])
        elif kind == b"IDAT":
            idat += body
        pos += 12 + length
    raw, stride = zlib.decompress(idat), 1 + width * 3
    return [
        [
            tuple(raw[y * stride + 1 + x * 3 : y * stride + 4 + x * 3])
            for x in range(width)
        ]
        for y in range(height)
    ]


def test_chart_matches_its_answer(tmp_path):
    out = tmp_path / "chart.png"
    out.write_bytes(chart.encode_png(chart.pixels()))
    rows = decode_png(out.read_bytes())
    # Measure each bar's height from the decoded pixels, not the constants.
    heights = []
    for i in range(len(chart.BARS)):
        x = (
            chart.GAP
            + i * (chart.BAR_WIDTH + chart.GAP)
            + chart.BAR_WIDTH // 2
        )
        heights.append(sum(row[x] == chart.BAR_COLOR for row in rows))
    assert heights == list(chart.BARS)
    assert chart.expected_answer() == (5, 3)


def test_tool_hashes_a_project_file():
    digest = agent.file_sha256("chart.py")
    assert len(digest) == 64 and all(c in "0123456789abcdef" for c in digest)


@pytest.mark.parametrize(
    "path", ["../outside.txt", "/etc/hosts"], ids=["relative", "absolute"]
)
def test_tool_refuses_outside_paths(path):
    assert agent.file_sha256(path).startswith("error:")


def test_bad_effort_rejected_locally():
    with pytest.raises(ValueError):
        ask.ask("hello", effort="0.5")

Detailed breakdown

  • decode_png() reads the image back from bytes, so the chart test checks the file the model would see rather than the constants that produced it. It handles only what chart.py writes: 8-bit RGB with filter type 0 on every row.
  • test_chart_matches_its_answer measures each bar by counting orange pixels down the middle column of each bar. If a change to chart.py ever drew a bar wrong, the expected answer the live test relies on would be wrong too, and this test is what catches it.
  • The two path tests cover both escapes: a relative path that climbs out with .., and an absolute path. Both must come back as error strings, not hashes. The ids give them short names in pytest’s output, relative and absolute, instead of the paths themselves.
  • The effort test proves ask() refuses "0.5" locally, before any request, which is the guard Step 7 added after Step 4 showed the template rejecting it.

Create the file

cd ~/src/inkling-mlx
touch tests/test_live.py

Add the code: ~/src/inkling-mlx/tests/test_live.py

"""Tests against the running server. Skipped when nothing answers."""

from __future__ import annotations

import urllib.request

import pytest

import agent
import ask
import chart


def server_up() -> bool:
    try:
        with urllib.request.urlopen(
            ask.BASE_URL.removesuffix("/v1") + "/health", timeout=2
        ):
            return True
    except OSError:
        return False


pytestmark = pytest.mark.skipif(
    not server_up(), reason="mlx_vlm.server is not running"
)


def test_model_reads_the_chart(tmp_path):
    image = tmp_path / "chart.png"
    image.write_bytes(chart.encode_png(chart.pixels()))
    reply = ask.ask(
        "How many bars are in this chart, and which bar, counting from "
        "the left, is the tallest? Reply exactly as: bars=<n> tallest=<k>",
        image=image,
    )
    bars, tallest = chart.expected_answer()
    assert reply.answer.strip() == f"bars={bars} tallest={tallest}"


def test_model_uses_the_tool():
    expected = agent.file_sha256("chart.py")
    assert expected in agent.run("What is the SHA-256 of chart.py?")

Detailed breakdown

  • server_up() probes /health with a two-second timeout, and pytestmark applies the skip to every test in the file. Without the server, these tests report SKIPPED with the reason instead of failing on a connection error.
  • test_model_reads_the_chart compares the whole reply string against bars=5 tallest=3 built from expected_answer(), so the test and the chart cannot drift apart.
  • test_model_uses_the_tool hashes chart.py, not out/chart.png, so it does not depend on Step 5 having been run, and asserts the true digest appears in the model’s answer. A model that skipped the tool could not produce it.
  • These are the slow tests, about nine seconds together, nearly all of it generation.

Create the file

cd ~/src/inkling-mlx
touch Makefile

Add the code: ~/src/inkling-mlx/Makefile

MODEL    := thinkingmachines/Inkling-2-Mini
REVISION := b85c3cfa615fc413cc9af3379f491b963f3b586a
PORT     := 8080
QUESTION := How many bars are in this chart, and which bar, counting from \
            the left, is the tallest? Reply exactly as: bars=<n> tallest=<k>

.DEFAULT_GOAL := help
.PHONY: help download chart generate image serve ask agent effort draft test

help: ## Show this help
	@grep -E '^[a-z-]+:.*## ' $(MAKEFILE_LIST) \
	  | awk -F':.*## ' '{printf "  %-10s %s\n", $$1, $$2}'

download: ## Download the pinned checkpoint (about 20 GB)
	uv run hf download $(MODEL) --revision $(REVISION)

chart: ## Draw the test chart into out/chart.png
	uv run chart.py

generate: ## One text prompt through mlx_vlm.generate
	uv run mlx_vlm.generate --model $(MODEL) \
	  --prompt "In two sentences, what does a Makefile do?" \
	  --reasoning-effort none --temperature 0 --max-tokens 400 --verbose

image: chart ## Ask the CLI about the test chart
	uv run mlx_vlm.generate --model $(MODEL) \
	  --image out/chart.png --prompt "$(QUESTION)" \
	  --reasoning-effort none --temperature 0 --max-tokens 400 --verbose

serve: ## Run the OpenAI-compatible server on 127.0.0.1:8080
	uv run mlx_vlm.server --host 127.0.0.1 --port $(PORT) \
	  --model $(MODEL) --log-level WARNING

ask: chart ## Ask the server about the test chart (needs make serve)
	uv run ask.py "$(QUESTION)" --image out/chart.png

agent: chart ## Run the tool-calling loop (needs make serve)
	uv run agent.py

effort: ## Sweep reasoning effort through the server (needs make serve)
	uv run probe.py effort

draft: ## Compare drafter speed (stop make serve first)
	uv run probe.py draft

test: ## Run the tests (live ones skip unless make serve is running)
	uv run pytest -v

Detailed breakdown

  • .DEFAULT_GOAL := help makes plain make print the target list. The help recipe greps the Makefile for target: ## description lines, so a new target documents itself by carrying a ## comment.
  • Recipe lines must start with a tab, not spaces; make stops with missing separator otherwise. Copying from a web page sometimes converts them.
  • image, ask and agent depend on chart, so they draw out/chart.png first and work in a fresh checkout.
  • QUESTION holds the chart prompt once and is reused by the CLI and server targets, so both ask exactly the same thing. The backslash at the end of its first line continues the value onto the next, joined with one space.
  • Each target’s help text says what it needs: make serve running for the server targets, and stopped for draft.

Step 9 had you stop the server, so start it again in its own terminal (the Step 6 command, or make serve) and wait for model_ready: true. Then check the help screen and run the tests:

make
make test
  help       Show this help
  download   Download the pinned checkpoint (about 20 GB)
  chart      Draw the test chart into out/chart.png
  generate   One text prompt through mlx_vlm.generate
  image      Ask the CLI about the test chart
  serve      Run the OpenAI-compatible server on 127.0.0.1:8080
  ask        Ask the server about the test chart (needs make serve)
  agent      Run the tool-calling loop (needs make serve)
  effort     Sweep reasoning effort through the server (needs make serve)
  draft      Compare drafter speed (stop make serve first)
  test       Run the tests (live ones skip unless make serve is running)
uv run pytest -v
...
tests/test_live.py::test_model_reads_the_chart PASSED                    [ 14%]
tests/test_live.py::test_model_uses_the_tool PASSED                      [ 28%]
tests/test_offline.py::test_chart_matches_its_answer PASSED              [ 42%]
tests/test_offline.py::test_tool_hashes_a_project_file PASSED            [ 57%]
tests/test_offline.py::test_tool_refuses_outside_paths[relative] PASSED  [ 71%]
tests/test_offline.py::test_tool_refuses_outside_paths[absolute] PASSED  [ 85%]
tests/test_offline.py::test_bad_effort_rejected_locally PASSED           [100%]

============================== 7 passed in 9.21s ===============================

The ... stands for pytest’s header lines (platform, rootdir, plugins), which differ per machine. With the server stopped, the same command reports 5 passed, 2 skipped in under a second.

Troubleshooting

ValueError: Model type inkling2_loop not supported. Your mlx-vlm is a release from before Mini support (0.8.0 or earlier). Re-run the git uv add from Step 1 and check with the get_model_and_args line there. Avoid a plain uv add mlx-vlm, or deleting [tool.uv.sources] and running uv sync: both bring the old release back.

TemplateError: Unknown reasoning_effort: ..., or HTTP 500 with that message. The effort was not one of none, minimal, low, medium, high, xhigh or max. Numbers fail too, through both the command line and the server, because they arrive as strings (Step 4).

An answer stops mid-sentence with no error. Reasoning used up max_tokens. Through the server, finish_reason is length instead of stop; raise the limit.

The Mac slows to a crawl or the process is killed. Two copies of the model are loaded, usually the server plus an mlx_vlm.generate run or make draft. Each needs about 21.5 GB. Stop the server first, or close other large applications on a 32 GB machine.

address already in use when the server starts. The full line is ERROR: [Errno 48] error while attempting to bind on address ('127.0.0.1', 8080): [errno 48] address already in use. Something else, often an earlier server you forgot, is on port 8080. Find it with lsof -i :8080, or start on another port with --port 8081 and set INKLING_URL=http://127.0.0.1:8081/v1 for the scripts.

The download folder looks empty. The snapshot holds symbolic links; use ls -lL or du -shL (Step 2).

Starting the server without a network connection. Set HF_HUB_OFFLINE=1 so it loads from the cache without checking Hugging Face for a newer revision.

What you built

Inkling 2 Mini now runs on your Mac from the weights Thinking Machines trained, in their native NVFP4 and MXFP8 formats, with no conversion in between. You generated text and read an image from the command line, served the model on localhost behind an OpenAI-compatible API, called it from Python with the official SDK, and gave it a tool it used correctly. Seven tests and a Makefile let you rerun all of it.

Along the way you found the limits that the model card does not spell out:

  • mlx-vlm needs a pinned git commit until the next release, and its version string cannot tell you whether you have the right one.
  • Reasoning effort accepts seven names and no numbers through mlx-vlm, and even none still reasons.
  • Effort cost tokens and bought no correctness on eight checkable questions: every level scored 8/8, and low was the cheapest.
  • The bundled drafter is worth about 1.3x in generation speed on this machine, at the cost of output that is not identical to the plain run.

Next steps:

  • Compare it with the GGUF build. llama.cpp supports this model from release b12619 (2026-11-21), newer than the 1.0.0 that Homebrew ships at the time of writing, and Run Qwen3.8-27B Locally with llama.cpp on macOS shows the same kind of measurement on another large model.
  • Fine-tune a model with MLX. Fine-Tune a Local Model with LoRA on Apple Silicon using MLX trains an adapter on a small model in seconds, using the same framework.
  • Put the tool loop to real work. Swap file_sha256 for a tool that does something you need, keep the path check, and keep MAX_TURNS.