Inkling 2 Mini is the small open-weights model Thinking Machines released on
2026-11-20: 26.4 billion parameters, text and image input, Apache 2.0, and a
stated minimum of an Apple Silicon Mac with 32 GB of memory. This article gets
it running on your Mac from its original weights, with no conversion step. You
will have it answering prompts in the terminal, reading a chart you generated
yourself and getting it right, serving an
OpenAI-compatible API on localhost that your own Python code calls, and
calling a tool you wrote to answer a question it could not answer alone.
You will also measure the two settings that change how it runs. One is the reasoning-effort dial, which the model card says cannot fully switch reasoning off; you will see what each level costs in tokens and whether it changes the answers. The other is a small draft model, which guesses tokens ahead for the full model to check, bundled with the checkpoint (the published set of weight and configuration files). On the M5 Max this was written on, it raised generation from about 37 tokens per second to between 46 and 48.
The tool that makes this possible is MLX, Apple’s array framework for Apple
Silicon, through the mlx-vlm package, which runs vision-language models on top
of it. The common alternative for local models is llama.cpp, an inference
engine that loads models converted to its single-file GGUF format. The reason
to use MLX instead is specific to this model, and Step 2 shows it in the
checkpoint’s own files.
Prerequisites
- A Mac with Apple Silicon and at least 32 GB of unified memory. MLX has no Intel path. Generation peaks at 21.5 to 22.5 GB here, so 32 GB is a real floor, not a cautious one. Validated on an Apple M5 Max with 128 GB, macOS 27.2.1.
- uv 0.11 or
later (
brew install uv), the Python package and project manager used for every command. Validated with 0.13.6. uv downloads Python 3.12 itself if you do not have it. - About 21 GB of free disk for the checkpoint, which lands in the shared Hugging Face cache.
jq(brew install jq), for reading the checkpoint’s JSON in Step 2. Validated with 1.8.2.make, for Step 10. The one in Apple’s command-line developer tools is fine.- Familiarity with chat-completion APIs (a list of
messages, a model name,max_tokens). OpenAI’s chat completions guide covers the shape if it is new to you.
No Hugging Face account or token is needed; the model is public.
Step 1: Set up the project
Inkling 2 Mini support in mlx-vlm was merged on 2026-11-20, four days after
the newest release (0.8.0, 2026-11-16) was cut, so uv add mlx-vlm installs a
package that has never heard of this model. The fix is to install from a pinned
git commit instead. This step creates the project, puts that pin in
pyproject.toml, and adds a release-age gate for everything else it resolves.
Create the file
mkdir -p ~/src/inkling-mlx
cd ~/src/inkling-mlx
touch .gitignore
Add the code: ~/src/inkling-mlx/.gitignore
.venv/
__pycache__/
.pytest_cache/
*.log
.DS_Store
tmp/
out/
Detailed breakdown
- The
.gitignorecomes first so nothing the later steps generate (the virtual environment, test caches, the chart image Step 5 writes underout/, and stray logs or scratch files) can be committed by accident. - The model weights are not listed because they never land in the project.
Step 2 downloads them into the shared Hugging Face cache under
~/.cache/huggingface/hub/, so there is nothing 20 GB in this folder to ignore.
Now create the project and pin the dependency. uv init --bare writes only a
pyproject.toml, with no sample code to delete. --python 3.12 sets its
requires-python; without it, uv writes the newest Python it knows (>=3.14
here) and the uv python pin 3.12 on the next line fails as incompatible. The
pin writes a .python-version so every uv run uses the same interpreter:
cd ~/src/inkling-mlx
uv init --bare --name inkling-mlx --python 3.12
uv python pin 3.12
printf '\n[tool.uv]\nexclude-newer = "3 days"\n' >> pyproject.toml
REV=f1618d6d52be4ef25d3fe65270f7528a59c55df3
uv add "mlx-vlm @ git+https://github.com/Blaizzy/mlx-vlm@$REV"
uv add openai
uv add --dev pytest
The resulting pyproject.toml should look like this (the lower bounds on
openai and pytest are whatever was current for you):
[project]
name = "inkling-mlx"
version = "0.1.0"
requires-python = ">=3.12"
dependencies = [
"mlx-vlm",
"openai>=3.55.1",
]
[tool.uv]
exclude-newer = "3 days"
[tool.uv.sources.mlx-vlm]
git = "https://github.com/Blaizzy/mlx-vlm"
rev = "f1618d6d52be4ef25d3fe65270f7528a59c55df3"
[dependency-groups]
dev = [
"pytest>=9.2.0",
]
Confirm what you installed. The version string will mislead you, so ask the
loader itself whether it can resolve this model. The checkpoint’s
config.json declares "model_type": "inkling2_loop", and mlx-vlm picks the
model code from that string:
uv run python - <<'EOF'
import mlx.core as mx, mlx_vlm
from mlx_vlm.utils import get_model_and_args
print(mx.__version__, mlx_vlm.__version__)
arch, model_type = get_model_and_args({"model_type": "inkling2_loop"})
print("inkling2_loop ->", model_type)
EOF
0.32.4 0.8.0
inkling2_loop -> inkling2
Detailed breakdown
- The commit hash in
REVis what matters in theuv addline. A baregit+https://...URL tracksmain, which moves several times a day; the hash freezes the exact code this article was validated against.uvrecords it inpyproject.tomlunder[tool.uv.sources]and inuv.lock. Your file will have that source as one line,mlx-vlm = { git = "...", rev = "..." }; the listing above shows the same setting as a[tool.uv.sources.mlx-vlm]table, which TOML treats as identical, so it fits the page. Once a release newer than 0.8.0 ships, runuv remove mlx-vlmand thenuv add "mlx-vlm>=<that version>". The removal matters: a plainuv addupdates the version range but keeps the[tool.uv.sources]git entry, so you would still be on this commit. exclude-newer = "3 days"refuses any package version younger than three days. It applies to the registry packagesmlx-vlmpulls in (mlx,transformersand the rest), not to the git commit you named explicitly. It is the same gate described in Make npm and uv Wait Before Installing New Releases on macOS.openaiandpytestare for Steps 7 and 10.mlx-vlmserves an OpenAI-compatible API, so the official client is the simplest way to call it.mlx_vlm.__version__says0.8.0even at this commit. The version number in the source was not bumped after the release, so a git install between releases reports the old number.get_model_and_argsis the functionmlx-vlmcalls when it loads a checkpoint, so asking it directly is the same test the real load would run. Here it mapsinkling2_looponto theinkling2model code. On a plainuv add mlx-vlminstall of 0.8.0 the same line fails withValueError: Model type inkling2_loop not supported, the same error that loading the model prints.- Do not test with
import mlx_vlm.models.inkling2. That module exists in 0.8.0 too: it was added for the much larger Inkling 2 Small, and imports cleanly on a version that cannot load Mini. mlx0.32.4 is the array framework itself, pulled in bymlx-vlm. It is the part that runs NVFP4 and MXFP8 weights on the GPU, which Step 2 explains.
Step 2: Download the original checkpoint
Every later step would download the model on first use anyway, but a 20 GB
download hidden inside a generation command looks like a hang. Fetching it on
its own makes the wait visible, and fetching it at a pinned revision gets you
the exact files this article was validated with. The hf command
comes with huggingface-hub, which mlx-vlm installed in Step 1:
cd ~/src/inkling-mlx
uv run hf download thinkingmachines/Inkling-2-Mini \
--revision b85c3cfa615fc413cc9af3379f491b963f3b586a
On this connection it took about four minutes. When it finishes, list what
arrived. MODEL_DIR holds this model’s folder in the Hugging Face cache, so
later commands stay short; set it again in any new terminal:
MODEL_DIR=~/.cache/huggingface/hub/models--thinkingmachines--Inkling-2-Mini
ls -lL $MODEL_DIR/snapshots/*/ | awk 'NR>1 {printf "%14s %s\n", $5, $9}'
6868 chat_template.jinja
2722 config.json
98 generation_config.json
67924 hf_quant_config.json
19526019168 model-00001-of-00001.safetensors
109300 model.safetensors.index.json
635801416 mtp.safetensors
461 processor_config.json
54870 README.md
529 special_tokens_map.json
96 tiktoken
13235 tokenizer_config.json
26178678 tokenizer.json
Detailed breakdown
model-00001-of-00001.safetensorsis the whole model in one 19.5 GB file: the language model and the vision encoder (the part that turns an image into tokens the language model reads) together, so the vision side needs no second download.mtp.safetensors(636 MB) holds the draft model Step 9 measures, whichmlx-vlmcalls the drafter. It ships in the same repository, so the download above already includes it.chat_template.jinjadecides how a conversation becomes tokens, and it is where Step 4’s reasoning-effort setting is implemented. You will read part of it there.ls -Lfollows symbolic links. The snapshot folder holds links into a content-addressed store, so a plainls -lordu -shon it reports a few hundred kilobytes and makes a finished download look empty.- The revision hash pins what this download fetches, and only that. Later
commands name the model as
thinkingmachines/Inkling-2-Mini, which resolves to the repository’smainbranch. Today that is the same commit, but if Thinking Machines pushes an update, the next online load fetches it. To keep every later step on these exact files, prefix the commands withHF_HUB_OFFLINE=1, which loads from the cache and never contacts Hugging Face.
Why MLX instead of a GGUF build
The other file worth reading is hf_quant_config.json, because it explains the
choice of runtime. It lists every tensor (a named array of weights) in the
model by the number format it is stored in:
cd $MODEL_DIR/snapshots/*/
jq -r '.model.parameter_groups[]
| "\(.compute_dtype)\t\(.tensors | length) tensors"' hf_quant_config.json
jq -r '.model.parameter_groups[] | select(.compute_dtype == "mxfp8")
| .tensors[]' hf_quant_config.json | head -4
cd ~/src/inkling-mlx
float32 74 tensors
bfloat16 993 tensors
nvfp4 40 tensors
mxfp8 8 tensors
bfloat16 1 tensors
model.llm.layers.20.mlp.experts.w13_weight
model.llm.layers.20.mlp.experts.w2_weight
model.llm.layers.21.mlp.experts.w13_weight
model.llm.layers.21.mlp.experts.w2_weight
Detailed breakdown
- Inkling 2 Mini is a mixture-of-experts model. Each of its 24 layers holds
128 small feed-forward networks called experts, and a router sends each token
to 8 of them, so only 3.4 billion of the 26.4 billion parameters are active
for any one token. The middle 12 layers then run three times per token, the
loop that gives the architecture its name,
inkling2_loop, which raises the effective active figure to 5.4 billion. The experts are most of the model’s size. - The 48 expert tensors use a 4-bit and an 8-bit floating-point format. NVFP4 is NVIDIA’s 4-bit format (each value is a 4-bit float, and every block of 16 values shares a scale); MXFP8 is an 8-bit microscaling format, which likewise shares one scale across each block of values. Layers 0 to 19 store their two expert tensors in NVFP4 (40 tensors), layers 20 to 23 in MXFP8 (8 tensors). Everything else, attention and the vision encoder included, is 16- or 32-bit.
- These are the formats the model was trained in, not a compression applied afterwards. Thinking Machines trained the experts in NVFP4, so these weights are the model itself. Running them unchanged is the only way to run exactly what was evaluated.
- MLX 0.32 implements
nvfp4andmxfp8as native quantization modes. Quantization means storing weights in a low-bit format with shared scales, usually to shrink a model after training; here the low-bit format is the original. The Inkling loader inmlx-vlmmaps eachparameter_groupsentry above onto those modes, so there is no conversion step in this article. The llama.cpp route (ggml-org/Inkling-2-Mini-GGUF) requantizes the NVFP4 experts into a different 4-bit format, IQ4_NL, which its own model card measures as slightly less faithful than an 8-bit copy.
Step 3: Generate text from the command line
The first run proves the whole chain works (the pinned mlx-vlm, the native
weights, the Metal GPU) before any of your own code is involved. It also shows
the model’s raw output format, which every later step depends on: Inkling 2
Mini writes its reasoning and its answer as two separate messages, and you can
see both here before the server starts hiding the markers in Step 6.
--temperature 0 makes the model pick its most likely token every time, so
your output should match the text below word for word. --verbose prints the
token counts and memory figures at the end:
uv run mlx_vlm.generate --model thinkingmachines/Inkling-2-Mini \
--prompt "In two sentences, what does a Makefile do?" \
--reasoning-effort none --temperature 0 --max-tokens 400 --verbose
==========
Files: []
Prompt: <|message_system|><|content_text|>Thinking effort level:
0<|end_message|><|message_user|><|content_text|>In two sentences, what does a
Makefile do?<|end_message|>
<|message_model|><|content_thinking|>The user asks: "In two sentences, what
does a Makefile do?" Need answer in two sentences. Need ensure exactly two
sentences. Explain Makefile: build automation file specifying
rules/targets/dependencies and commands to build/compile/link/clean. Mention
make reads it to execute tasks in correct order. Need concise. Final only two
sentences. Ensure no extra sentences. Count sentences: 1) A Makefile is a
build-automation script that lists targets, their dependencies, and the
commands to run. 2) When you run make, it reads the file and executes the
needed commands in the correct order to produce or update the requested
outputs. That's two sentences.
Good.<|end_message|><|message_model|><|content_text|>A Makefile is a
build-automation script that lists targets, their dependencies, and the
commands to run. When you run make, it reads the file and executes the needed
commands in the correct order to produce or update the requested
outputs.<|end_message|>
==========
Prompt: 24 tokens, 107.700 tokens-per-sec
Generation: 203 tokens, 37.474 tokens-per-sec
Peak memory: 21.476 GB
fold -s -w 79 wraps the output at 79 columns, breaking at spaces where it
can, so the long model lines fit the page; without it each message is one
long line. A progress bar for the prompt goes to standard error and is left
out above. Loading takes a few seconds on later runs; the first run after the
download also reads 20 GB from disk.
Detailed breakdown
- The
Prompt:line is your question after the chat template ran. Your one sentence became a system message,Thinking effort level: 0, followed by your user message.--reasoning-effort noneis what produced that0; Step 4 changes it. <|content_thinking|>opens the reasoning,<|content_text|>opens the answer. Each is a separate<|message_model|>message. The reasoning reads like notes to itself (“Need ensure exactly two sentences”), and it is there even though you asked for effortnone. The model card says as much: the minimum setting “does not fully disable reasoning”.- Reasoning tokens count against
--max-tokens. Of the 203 tokens generated, most are reasoning. With--max-tokens 200on this same prompt the answer stops just before its final period, with no error or warning. Set the limit for reasoning plus answer, never for the answer alone. - 37.5 tokens per second is the speed to compare against later. It varies by a few tenths between runs (37.7 on an earlier run of the same command), so treat the first decimal place as noise. Step 9 measures it properly.
- Peak memory of 21.5 GB is the weights plus working memory for a short prompt. That is why 32 GB is the stated floor: the rest of macOS needs room too, and long prompts add more.
Step 4: Turn the reasoning effort up and down
Step 3 showed --reasoning-effort none turning into a system message. Knowing
exactly how that happens tells you which values are valid, and the answer is
narrower than the Hugging Face release post suggests, which lists numbers from
0.0 to 0.99 alongside the names. The mechanism lives in the first 20
lines of the chat template you downloaded. fold -s -w 79 wraps its longest
lines to fit the page, so a few lines below continue onto the next:
MODEL_DIR=~/.cache/huggingface/hub/models--thinkingmachines--Inkling-2-Mini
sed -n 1,20p $MODEL_DIR/snapshots/*/chat_template.jinja | fold -s -w 79
{%- set effort_map = {"none": 0.0, "minimal": 0.1, "low": 0.2, "medium": 0.7,
"high": 0.9, "xhigh": 0.99, "max": 0.99} -%}
{%- set role_token = {"user": "<|message_user|>", "assistant":
"<|message_model|>", "system": "<|message_system|>", "tool":
"<|message_tool|>"} -%}
{%- macro emit_thinking_effort() -%}
{%- set eff = reasoning_effort if reasoning_effort is defined and
reasoning_effort is not none else 0.9 -%}
{%- if eff is string -%}
{%- set key = eff | trim -%}
{%- if key not in effort_map -%}
{{- raise_exception("Unknown reasoning_effort: " ~ eff) -}}
{%- endif -%}
{%- set num = effort_map[key] -%}
{%- else -%}
{%- set num = eff | float -%}
{%- endif -%}
{%- if num < 0.0 or num > 0.99 -%}
{{- raise_exception("reasoning_effort must be in [0.0, 0.99]") -}}
{%- endif -%}
{{- "<|message_system|><|content_text|>Thinking effort level: " -}}
{%- if num == 0.0 -%}0{%- else -%}{{ num }}{%- endif -%}
{{- "<|end_message|>" -}}
Detailed breakdown
- The effort is a number the model reads, not a switch in the runtime.
Seven names map to values between 0.0 and 0.99, and the template writes the
number into a system message,
Thinking effort level: 0.9. Nothing in MLX changes; the model was trained to spend more reasoning tokens when it reads a higher number. - Leaving it unset means
high(0.9). That is the model’s default, so a client that never mentions effort gets a lot of reasoning. xhighandmaxare the same value, 0.99.- A string must be one of the seven names, or the template raises. That catches typos instead of ignoring them.
- A number is only a number if it arrives as a number, and through
mlx-vlmit never does. The template accepts0.5as a float, butmlx_vlm.generatepasses the effort through as text, so--reasoning-effort 0.5reaches the template as the string"0.5", fails the name lookup, and stops withTemplateError: Unknown reasoning_effort: 0.5. The server types the field as a string: a JSON number0.5is rejected with HTTP 422 before it reaches the template, and the string"0.5"fails in the template with HTTP 500. Only the seven names work.
Now run the same question at both ends of the dial. 391 is 17 × 23, so the
right answer is no:
PATTERN='Thinking effort level: [0-9.]+'
PATTERN+='|content_text\|>[A-Za-z]+<|^Generation: .*'
for effort in none max; do
uv run mlx_vlm.generate --model thinkingmachines/Inkling-2-Mini \
--prompt "Is 391 a prime number? Answer yes or no." \
--reasoning-effort "$effort" --temperature 0 --max-tokens 2000 \
--verbose 2>/dev/null | grep -oE "$PATTERN"
done
Thinking effort level: 0
content_text|>no<
Generation: 70 tokens, 37.845 tokens-per-sec
Thinking effort level: 0.99
content_text|>no<
Generation: 196 tokens, 37.406 tokens-per-sec
Detailed breakdown
grep -oE "$PATTERN"pulls three things out of each run: the effort the model was shown, the answer betweencontent_text|>and the next<, and the token count. The pattern is built in two pieces only to keep the lines short;+=appends to a shell variable in both zsh and bash.- Same answer, almost three times the tokens. At
maxthe model spent 196 tokens to reach thenoit reached in 70 atnone. At 38 tokens per second that is 1.8 seconds against 5.2 for a one-word answer. - One question shows the direction, not the size of the effect. Step 9 runs eight questions at five levels and finds the relationship is weaker and less tidy than this pair suggests.
Step 5: Ask about an image you drew yourself
A photo from your camera roll makes a poor test of the vision side, because you cannot tell a plausible description from a correct one. This step generates a chart whose right answer is fixed by the code that draws it: five bars, with the tallest in third place rather than last, so a model that assumes bars rise left to right gets it wrong. The test in Step 10 reuses the same script.
Create the file
cd ~/src/inkling-mlx
touch chart.py
Add the code: ~/src/inkling-mlx/chart.py
"""Draw a bar chart PNG whose correct description is known in advance.
A vision check is only a check if a wrong answer is recognisable, so the chart
is generated from constants rather than downloaded. The tallest bar is the
third one, not the last, so a model that guesses "the bars go up" is caught.
Only the standard library is used: `zlib` and `struct` are enough for a PNG.
"""
from __future__ import annotations
import struct
import sys
import zlib
from pathlib import Path
WIDTH, HEIGHT = 480, 320
BARS = (70, 140, 250, 110, 180) # bar heights in pixels, left to right
BAR_COLOR = (230, 120, 20) # orange
BAR_WIDTH, GAP = 60, 30
BACKGROUND = (255, 255, 255)
def expected_answer() -> tuple[int, int]:
"""Return (number of bars, 1-based position of the tallest bar)."""
return len(BARS), BARS.index(max(BARS)) + 1
def pixels() -> list[list[tuple[int, int, int]]]:
"""Return the image as rows of RGB tuples, top row first."""
rows = [[BACKGROUND] * WIDTH for _ in range(HEIGHT)]
for i, height in enumerate(BARS):
left = GAP + i * (BAR_WIDTH + GAP)
for y in range(HEIGHT - height, HEIGHT):
for x in range(left, left + BAR_WIDTH):
rows[y][x] = BAR_COLOR
return rows
def encode_png(rows: list[list[tuple[int, int, int]]]) -> bytes:
"""Encode RGB rows as an 8-bit truecolour PNG."""
def chunk(kind: bytes, data: bytes) -> bytes:
crc = zlib.crc32(kind + data)
return (
struct.pack(">I", len(data)) + kind + data + struct.pack(">I", crc)
)
raw = b"".join(
b"\x00" + bytes(c for px in row for c in px) for row in rows
)
header = struct.pack(">IIBBBBB", len(rows[0]), len(rows), 8, 2, 0, 0, 0)
return (
b"\x89PNG\r\n\x1a\n"
+ chunk(b"IHDR", header)
+ chunk(b"IDAT", zlib.compress(raw, 9))
+ chunk(b"IEND", b"")
)
def main() -> None:
out = Path(sys.argv[1] if len(sys.argv) > 1 else "out/chart.png")
out.parent.mkdir(parents=True, exist_ok=True)
out.write_bytes(encode_png(pixels()))
bars, tallest = expected_answer()
print(f"wrote {out}: {bars} bars, tallest is bar {tallest}")
if __name__ == "__main__":
main()
Detailed breakdown
BARSis the ground truth. Five heights, tallest third. Everything else is derived from it, includingexpected_answer(), which the tests in Step 10 compare against what the model says.pixels()paints each bar from the bottom edge up. Bars are 60 pixels wide with 30-pixel gaps, so bar i starts at x = 30 + 90*i and the last one ends at x = 450, inside the 480-pixel width.encode_png()writes the smallest valid PNG: the 8-byte signature, anIHDRchunk (width, height, 8 bits per channel, colour type 2 for RGB), oneIDATchunk of zlib-compressed rows, each prefixed with filter byte 0, and an emptyIEND. Every chunk carries a CRC-32 of its type and data, which is whatzlib.crc32computes. No imaging library is needed.- The default output path is
out/chart.png, which.gitignorealready excludes.
Draw the chart, then ask the model about it. The question asks for a fixed reply format so the answer can be checked by string comparison:
uv run chart.py
uv run mlx_vlm.generate --model thinkingmachines/Inkling-2-Mini \
--image out/chart.png \
--prompt "How many bars are in this chart, and which bar, counting from \
the left, is the tallest? Reply exactly as: bars=<n> tallest=<k>" \
--reasoning-effort none --temperature 0 --max-tokens 400 --verbose \
2>/dev/null | fold -s -w 79
wrote out/chart.png: 5 bars, tallest is bar 3
==========
Files: ['out/chart.png']
Prompt: <|message_system|><|content_text|>Thinking effort level:
0<|end_message|><|message_user|><|content_image|><|unused_200054|><|end_message
|><|message_user|><|content_text|>How many bars are in this chart, and which
bar, counting from the left, is the tallest? Reply exactly as: bars=<n>
tallest=<k><|end_message|>
<|message_model|><|content_thinking|>We need answer image counting bars. Need
inspect image. Visible orange bars: left partial at x~30-89 height from y=290
to 400; second x~120-180 y=180-400; third x~210-270 y=70-400; fourth x~300-360
y=210-400; fifth x~390-450 y=140-400. So 5 bars. Tallest is third from left
(top y=70, height 330). Need exact format bars=5 tallest=3. Ensure no
extra.<|end_message|><|message_model|><|content_text|>bars=5
tallest=3<|end_message|>
==========
Prompt: 149 tokens, 546.605 tokens-per-sec
Generation: 132 tokens, 37.461 tokens-per-sec
Peak memory: 21.659 GB
Detailed breakdown
bars=5 tallest=3is correct, and it matchesexpected_answer().- The image became one placeholder in the prompt,
<|unused_200054|>, but 149 prompt tokens. The prompt text, placeholder included, is 54 tokens; the vision encoder turned the 480 by 320 chart into 96 image tokens, which replace the placeholder before the language model runs. - The reasoning is worth reading against the code. Its x ranges are the real ones (30 to 90, 120 to 180, and so on), and four of the five bar tops it gives (180, 70, 210, 140) are exact: those are 320 minus each bar’s height. But it puts the floor at y = 400 in a 320-pixel image and the first bar’s top at 290 instead of 250. The final answer came out right; the measurements behind it were partly invented. Check answers, not the reasoning that led to them.
- Memory rose by about 0.2 GB over Step 3, for the vision encoder’s work on one small image.
Step 6: Serve it on localhost
Every mlx_vlm.generate call so far loaded 20 GB of weights, answered once,
and exited. A server loads them once and keeps them in memory, and because
mlx_vlm.server speaks the OpenAI chat-completions protocol, any client
written for that API can use the model by changing its base URL. Steps 7 and 8,
the effort half of Step 9, and the live tests in Step 10 talk to this server, so
start it in a second terminal and leave it running.
The --host flag matters here. The server’s default is 0.0.0.0, which
listens on every network interface, so anyone on your Wi-Fi could send it
prompts; there is no API key unless you pass --api-key. 127.0.0.1 keeps it
on this Mac:
cd ~/src/inkling-mlx
uv run mlx_vlm.server --host 127.0.0.1 --port 8080 \
--model thinkingmachines/Inkling-2-Mini --log-level WARNING
INFO: Started server process [8568]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://127.0.0.1:8080 (Press CTRL+C to quit)
Your process ID will differ, and you will also see a Fetching 12 files
progress line, left out above. From the first terminal, ask the server whether
it is ready:
curl -s http://127.0.0.1:8080/health | jq '{status, model_ready,
loaded_model, loaded_tool_parser, effective_context_limit}'
{
"status": "healthy",
"model_ready": true,
"loaded_model": "thinkingmachines/Inkling-2-Mini",
"loaded_tool_parser": "inkling",
"effective_context_limit": 1048576
}
Detailed breakdown
--modelpreloads the weights at startup, so the first request does not wait for a 20 GB load. Without it, the server loads whichever model the first request names.--log-level WARNINGkeeps the server terminal readable. At the defaultINFOlevel it logs each load stage and several timestamped lines per request, most of them wider than a terminal. The four lines above come from the web server underneath, which also prints one short line per request ("GET /health HTTP/1.1" 200 OK) whatever the level.- The server checks Hugging Face for a newer revision of
mainat load, even though the files are cached. It works offline too: setHF_HUB_OFFLINE=1before the command and it loads from the cache without the check. model_ready: trueis the field to wait for, because it is about the weights rather than the HTTP server. The full response also carriespreload_in_progressand amodel_load_progressobject while a load is under way.loaded_tool_parser: "inkling"meansmlx-vlmrecognized the model’s tool-call format and will turn it into structured OpenAI tool calls. Step 8 depends on that.effective_context_limitis 1,048,576 tokens, the model’s full 1M context. That is the most a single request can hold, not memory set aside in advance; memory grows with the tokens actually in use.
Step 7: Call it from Python with the OpenAI SDK
This is the step that makes the model usable from your own code. The script
below sends a prompt, optionally with an image, through the official openai
package, and prints the reasoning and the answer separately. That separation
is the main thing the server adds over Step 3: it parses the
<|content_thinking|> and <|content_text|> messages and returns them in
different fields, so your code gets a clean answer without stripping markers.
Create the file
cd ~/src/inkling-mlx
touch ask.py
Add the code: ~/src/inkling-mlx/ask.py
"""Ask the local Inkling 2 Mini server a question, optionally about an image.
Talks to `mlx_vlm.server` through the official OpenAI client. The server keeps
the model's reasoning apart from its answer, so both are printed separately.
"""
from __future__ import annotations
import argparse
import base64
import os
import textwrap
from dataclasses import dataclass
from pathlib import Path
from openai import OpenAI
BASE_URL = os.environ.get("INKLING_URL", "http://127.0.0.1:8080/v1")
MODEL = "thinkingmachines/Inkling-2-Mini"
EFFORTS = ("none", "minimal", "low", "medium", "high", "xhigh", "max")
@dataclass
class Reply:
answer: str
reasoning: str
completion_tokens: int
finish_reason: str
def client() -> OpenAI:
# The server ignores the key, but the client will not start without one.
return OpenAI(base_url=BASE_URL, api_key="local")
def image_part(path: Path) -> dict:
"""Wrap a PNG as a data URL, the chat API's form for inline images."""
data = base64.b64encode(path.read_bytes()).decode("ascii")
return {
"type": "image_url",
"image_url": {"url": f"data:image/png;base64,{data}"},
}
def ask(
prompt: str,
effort: str = "none",
image: Path | None = None,
max_tokens: int = 1024,
) -> Reply:
if effort not in EFFORTS:
raise ValueError(f"effort must be one of {EFFORTS}, got {effort!r}")
content: list[dict] = [{"type": "text", "text": prompt}]
if image is not None:
content.insert(0, image_part(image))
resp = client().chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": content}],
max_tokens=max_tokens,
temperature=0,
reasoning_effort=effort,
)
choice = resp.choices[0]
msg = choice.message
return Reply(
answer=msg.content or "",
# Not in the OpenAI schema; the SDK keeps unknown fields as attributes.
reasoning=getattr(msg, "reasoning_content", None) or "",
completion_tokens=resp.usage.completion_tokens if resp.usage else 0,
finish_reason=choice.finish_reason or "",
)
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("prompt")
parser.add_argument("--effort", default="none", choices=EFFORTS)
parser.add_argument("--image", type=Path)
parser.add_argument("--max-tokens", type=int, default=1024)
args = parser.parse_args()
reply = ask(args.prompt, args.effort, args.image, args.max_tokens)
print(f"reasoning: {len(reply.reasoning)} chars")
answer = textwrap.fill(
reply.answer.strip(),
width=79,
initial_indent="answer: ",
subsequent_indent=" " * 11,
)
print(answer)
print(f"tokens: {reply.completion_tokens} ({reply.finish_reason})")
if __name__ == "__main__":
main()
Detailed breakdown
BASE_URLis the only thing that makes this a local client. Everything else is the standard OpenAI SDK call.INKLING_URLlets you point it at another port or machine without editing the file.api_key="local"is a placeholder. The SDK refuses to build a client without a key; the server, started without--api-key, ignores it.reasoning_effortis a standard parameter of the SDK’schat.completions.create, and the server feeds it to the chat template from Step 4. The script defaults tononebecause the model’s own default, used when the field is absent, ishigh.ask()rejects anything but the seven names before sending. The server would answer0.5or a typo with an HTTP 500 carryingUnknown reasoning_effort: 0.5; checking locally gives a clear error without a round trip.- Images go in as a
data:URL inside animage_urlcontent part, the chat API’s form for inline images, placed before the text as in Step 5. textwrap.fillwraps the answer at 79 columns, indenting continuation lines to line up under the first, so a long answer stays readable in a terminal.reasoning_contentis not in the OpenAI response schema, so the SDK does not declare it, but it keeps unknown fields from the JSON as attributes on the message.getattrwith a default reads it without breaking against a server that does not send it.finish_reasontells you whether the answer is complete.stopmeans the model ended its answer;lengthmeansmax_tokenscut it off, which is the silent truncation from Step 3.
With the server from Step 6 still running, ask the Step 3 question and the Step 5 chart question again:
uv run ask.py "In two sentences, what does a Makefile do?"
uv run ask.py "How many bars are in this chart, and which bar, counting from \
the left, is the tallest? Reply exactly as: bars=<n> tallest=<k>" \
--image out/chart.png
reasoning: 659 chars
answer: A Makefile is a build-automation script that lists targets, their
dependencies, and the commands to run. When you run make, it reads
the file and executes the needed commands in the correct order to
produce or update the requested outputs.
tokens: 203 (stop)
reasoning: 350 chars
answer: bars=5 tallest=3
tokens: 132 (stop)
Detailed breakdown
- Both answers and both token counts match the command-line runs in Steps
3 and 5 (203 and 132 tokens). Do not rely on that being exact: the server
batches requests through a different code path, and the Step 4 prime
question at
maxtook 194 tokens here against 196 on the command line, with the same answer. - The 659 characters of reasoning arrived in
reasoning_content, with no<|...|>markers in either field. - Both finished with
stop, so neither was truncated by the 1,024-token default inask().
Step 8: Let it call a tool
A tool call is how a model reaches past what it knows: you describe a function, the model replies with a request to call it instead of an answer, your code runs it, and the result goes back as a new message. The tool here computes a file’s SHA-256, which a language model cannot compute from its weights, so a correct final answer proves the call happened. It is also restricted to files inside the project, which is the habit to build before giving a model any tool that touches the disk.
Create the file
cd ~/src/inkling-mlx
touch agent.py
Add the code: ~/src/inkling-mlx/agent.py
"""Give the local model one tool and let it decide when to call it.
The tool computes a file's SHA-256, something no language model can do in its
head, so a correct final answer proves the tool really ran. The loop is the
standard one: send the tools, run any call the model makes, send the result
back, and repeat until the model answers in plain text.
"""
from __future__ import annotations
import hashlib
import json
import sys
from pathlib import Path
from ask import MODEL, client
ROOT = Path(__file__).resolve().parent
MAX_TURNS = 4
TOOLS = [
{
"type": "function",
"function": {
"name": "file_sha256",
"description": "Return the SHA-256 hex digest of a project file.",
"parameters": {
"type": "object",
"properties": {
"path": {
"type": "string",
"description": "Path relative to the project root.",
}
},
"required": ["path"],
},
},
}
]
def file_sha256(path: str) -> str:
"""Hash a file, refusing anything outside the project folder."""
target = (ROOT / path).resolve()
if not target.is_relative_to(ROOT):
return f"error: {path} is outside the project"
if not target.is_file():
return f"error: {path} does not exist"
return hashlib.sha256(target.read_bytes()).hexdigest()
def run(prompt: str) -> str:
messages: list[dict] = [{"role": "user", "content": prompt}]
for _ in range(MAX_TURNS):
resp = client().chat.completions.create(
model=MODEL,
messages=messages,
tools=TOOLS,
max_tokens=1024,
temperature=0,
reasoning_effort="low",
)
msg = resp.choices[0].message
if not msg.tool_calls:
return msg.content or ""
messages.append(
{
"role": "assistant",
"content": msg.content or "",
"tool_calls": [
{
"id": call.id,
"type": "function",
"function": {
"name": call.function.name,
"arguments": call.function.arguments,
},
}
for call in msg.tool_calls
],
}
)
for call in msg.tool_calls:
args = json.loads(call.function.arguments or "{}")
result = (
file_sha256(**args)
if call.function.name == "file_sha256"
else "error: unknown tool"
)
print(f"tool call: {call.function.name}({args})")
print(f" result: {result}")
messages.append(
{"role": "tool", "tool_call_id": call.id, "content": result}
)
return "error: no answer after the turn limit"
if __name__ == "__main__":
question = (
sys.argv[1]
if len(sys.argv) > 1
else "What is the SHA-256 of out/chart.png?"
)
print(f"answer: {run(question)}")
Detailed breakdown
TOOLSis a JSON Schema description of the function. The model never seesfile_sha256itself, only this name, description and parameter list, which the chat template writes into a system message. The description is what the model uses to decide the tool fits the question.file_sha256()resolves the path before checking it.resolve()collapses..and follows symlinks, so../outside.txtand/etc/hostsboth land outsideROOTand get an error string instead of a hash. Errors go back to the model as text rather than raising, so it can read them and respond.- The loop is the agent. A reply with
tool_callsgets appended to the history as an assistant message, each call is run, and each result goes back as atoolmessage whosetool_call_idmatches the call. The model then sees its own request and the answer, and either calls again or replies in plain text, which ends the loop. MAX_TURNS = 4bounds the loop to four requests. The model card warns that Inkling 2 Mini can get “stuck in long reasoning loops”; a model that keeps calling tools would otherwise run forever. The same warning is why every request setsmax_tokens.reasoning_effort="low"is enough to pick the right tool for a question this direct. Raise it for tasks where choosing between tools takes thought.argumentsarrives as a JSON string, as the OpenAI format specifies, and goes back into the history unchanged. The chat template itself demands a parsed object there, and the server converts it before rendering, so the standard client code works as written.
Run it, then check the model’s answer against shasum:
uv run agent.py
shasum -a 256 out/chart.png
tool call: file_sha256({'path': 'out/chart.png'})
result: 0b593372d7b20e33fec9e938c3e7f9275295090ddede6eef755db779198528d8
answer: 0b593372d7b20e33fec9e938c3e7f9275295090ddede6eef755db779198528d8
0b593372d7b20e33fec9e938c3e7f9275295090ddede6eef755db779198528d8 out/chart.png
Detailed breakdown
- The model chose to call the tool, with the right path, then repeated the 64-character digest exactly in its final answer.
- The digests agree: the tool’s
result, the model’sanswer, and whatshasumcomputes independently.agent.pyprints the call and its result on separate lines because a 64-character digest after the call does not fit a terminal line. - Your digest may differ from this one if your Python’s zlib compresses the chart differently. What must hold is that your three digests match each other.
- Two requests were made: one that came back with
finish_reason: tool_callsand no text, and one, after the tool result was appended, that came back with the answer.
Step 9: Measure the effort dial and the drafter
Two settings change how this model runs, and neither should be taken on trust. The effort dial is supposed to trade tokens for quality; one question in Step 4 showed the token side, but not whether the answers improve. The draft model is supposed to make generation faster; how much depends on your hardware and your text. This script measures both, so the numbers you act on are your own.
Create the file
cd ~/src/inkling-mlx
touch probe.py
Add the code: ~/src/inkling-mlx/probe.py
"""Measure what the reasoning-effort dial and the bundled drafter actually buy.
`effort` sends the same questions at several effort levels through the local
server and reports, per level, how many questions came back right and how many
tokens the model spent. `draft` runs one prompt through `mlx_vlm.generate`
with and without the drafter and reports tokens per second.
"""
from __future__ import annotations
import argparse
import re
import statistics
import subprocess
import time
from ask import MODEL, ask
# Questions with one checkable answer each. Answers are compared after
# lower-casing and stripping punctuation, so "Saturday." matches "saturday".
QUESTIONS = [
("What is 47 * 83?", "3901"),
("Is 391 a prime number? Answer yes or no.", "no"),
("What is the remainder when 1000 is divided by 37?", "1"),
("How many prime numbers are less than 50?", "15"),
("What day of the week was 1 January 2000?", "saturday"),
("What is the sum of the integers from 1 to 250?", "31375"),
(
"How many times does the letter r appear in 'strawberry raspberry'?",
"6",
),
("What is 2 to the power of 20?", "1048576"),
]
SUFFIX = " Reply with only the answer."
GENERATION = re.compile(
r"^Generation: (\d+) tokens, ([\d.]+) tokens-per-sec", re.M
)
ACCEPTED = re.compile(r"([\d.]+)% of drafted")
def normalize(text: str) -> str:
return re.sub(r"[^a-z0-9]", "", text.lower())
def effort_sweep(efforts: list[str], max_tokens: int) -> None:
print(
f"{'effort':<8} {'correct':>7} {'mean tokens':>11}"
f" {'most tokens':>11} {'seconds':>8}"
)
for effort in efforts:
right, tokens = 0, []
start = time.monotonic()
for question, expected in QUESTIONS:
reply = ask(
question + SUFFIX, effort=effort, max_tokens=max_tokens
)
right += normalize(reply.answer) == expected
tokens.append(reply.completion_tokens)
elapsed = time.monotonic() - start
print(
f"{effort:<8} {right:>4}/{len(QUESTIONS):<2}"
f" {statistics.mean(tokens):>11.1f}"
f" {max(tokens):>11} {elapsed:>8.1f}"
)
def generate(
prompt: str, draft: bool, max_tokens: int
) -> tuple[int, float, str]:
cmd = [
"mlx_vlm.generate",
"--model",
MODEL,
"--prompt",
prompt,
"--reasoning-effort",
"none",
"--temperature",
"0",
"--max-tokens",
str(max_tokens),
"--verbose",
]
if draft:
cmd += ["--draft-model", MODEL]
out = subprocess.run(cmd, capture_output=True, text=True, check=True)
log = out.stdout + out.stderr
match = GENERATION.search(log)
if match is None:
raise RuntimeError("no Generation line in mlx_vlm.generate output")
accepted = ACCEPTED.search(log)
return (
int(match.group(1)),
float(match.group(2)),
f"{accepted.group(1)}%" if accepted else "-",
)
def draft_compare(runs: int, max_tokens: int) -> None:
prompt = (
"Write a Python function that returns the n-th Fibonacci number "
"iteratively, with a docstring and three doctest examples."
)
for draft in (False, True):
label = "yes" if draft else "no"
rates = []
for i in range(runs):
tokens, rate, accepted = generate(prompt, draft, max_tokens)
rates.append(rate)
print(
f"draft={label:<3} run {i + 1}: {tokens} tokens, "
f"{rate:.1f} tok/s, accepted {accepted}"
)
median = statistics.median(rates)
print(f"draft={label:<3} median: {median:.1f} tok/s")
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__)
sub = parser.add_subparsers(dest="command", required=True)
e = sub.add_parser(
"effort", help="sweep reasoning effort through the server"
)
e.add_argument("--efforts", default="none,low,medium,high,max")
e.add_argument("--max-tokens", type=int, default=4096)
d = sub.add_parser(
"draft", help="compare speed with and without the drafter"
)
d.add_argument("--runs", type=int, default=5)
d.add_argument("--max-tokens", type=int, default=800)
args = parser.parse_args()
if args.command == "effort":
effort_sweep(args.efforts.split(","), args.max_tokens)
else:
draft_compare(args.runs, args.max_tokens)
if __name__ == "__main__":
main()
Detailed breakdown
QUESTIONSpairs each prompt with its one right answer. 47 × 83 = 3,901; 391 = 17 × 23; 1,000 = 37 × 27 + 1; there are 15 primes below 50; 1 January 2000 was a Saturday; 1 + 2 + … + 250 = 250 × 251 / 2 = 31,375; “strawberry” and “raspberry” have three r’s each; 2²⁰ = 1,048,576.normalize()makes the comparison forgiving about format but not about content.1,048,576,Saturday.andNoall match;1048575does not.effort_sweep()goes through the server, reusingask()from Step 7, so it measures exactly the path your own code uses.max_tokensdefaults to 4,096 so that no level is cut off before it finishes reasoning.generate()shells out tomlx_vlm.generaterather than the server, because the--verbosesummary reports tokens per second for generation alone, without HTTP time, and the drafter’s acceptance rate, which the server does not return.draft_compare()takes the median of five runs per setting. Single runs vary by up to a few tokens per second, and one sample would turn that noise into a result.--draft-modelnames the same repository as--model. That is howmlx-vlmfinds the bundledmtp.safetensors.
With the server still running, run the effort sweep. It sends 40 requests and takes about three minutes:
uv run probe.py effort
effort correct mean tokens most tokens seconds
none 8/8 134.8 338 30.9
low 8/8 110.9 232 26.0
medium 8/8 138.4 408 31.8
high 8/8 155.6 471 36.5
max 8/8 196.1 574 47.0
Detailed breakdown
- Every level got every question right. On questions this size the dial
does not buy correctness; it costs tokens, from 110.9 on average at
lowto 196.1 atmax, 1.8 times as many, and 26.0 seconds against 47.0 for the whole set. - The token columns repeat exactly between runs, because temperature 0 makes each answer deterministic. Only the seconds column varies.
- The dial is not monotonic, and
noneis not the cheapest level.noneaveraged more tokens thanlow. Per question,lowspent fewer tokens thannoneon five of the eight, the same on two, and more on one; evenmaxspent fewer thannoneon two. The higher levels tend to cost more, but you cannot predict any one answer’s cost from the level. - The expensive question is the one models find hard: counting letters.
“How many times does the letter r appear” produced the largest count at
every level, which is the
most tokenscolumn: 338, 232, 408, 471 and 574. - The practical setting is
lowfor simple requests. It was the cheapest level here and lost nothing. Save the higher levels for problems where you can show they change the answer, and measure that on your own questions before paying for it.
The drafter comparison loads the model itself in each of its ten runs, so stop the server first (Ctrl+C in its terminal). Running both at once needs room for two 21 GB copies, which a 32 GB Mac does not have. It takes about five minutes:
uv run probe.py draft
draft=no run 1: 510 tokens, 35.8 tok/s, accepted -
draft=no run 2: 510 tokens, 36.3 tok/s, accepted -
draft=no run 3: 510 tokens, 36.9 tok/s, accepted -
draft=no run 4: 510 tokens, 36.9 tok/s, accepted -
draft=no run 5: 510 tokens, 36.9 tok/s, accepted -
draft=no median: 36.9 tok/s
draft=yes run 1: 385 tokens, 48.4 tok/s, accepted 38.4%
draft=yes run 2: 385 tokens, 45.5 tok/s, accepted 38.4%
draft=yes run 3: 385 tokens, 46.9 tok/s, accepted 38.4%
draft=yes run 4: 385 tokens, 46.6 tok/s, accepted 38.4%
draft=yes run 5: 385 tokens, 46.7 tok/s, accepted 38.4%
draft=yes median: 46.7 tok/s
Detailed breakdown
- The drafter raised generation from 36.9 to 46.7 tokens per second, a factor of 1.27. Three other sessions on the same machine gave 1.32, 1.30 and 1.26 (drafted medians of 48.2, 48.0 and 46.6), so treat it as about 1.3.
- How the drafter works. The small draft model guesses several tokens ahead, about seven per round here, and the full model checks all of the guesses in a single pass, keeping the ones it agrees with. 38.4% of the guesses were kept. Each round cost one pass of the full model and produced several tokens instead of one, which is where the speedup comes from. The technique is called speculative decoding.
- The two settings did not produce the same text. Speculative decoding is described as lossless, meaning the full model’s choices are unchanged, yet the plain run wrote 510 tokens and the drafted run 385, and their reasoning diverges about 180 characters in. The likely cause is arithmetic: checking several tokens in one batched pass rounds slightly differently from checking one at a time, and at temperature 0 a near-tie can tip to a different token, after which the two texts go separate ways. Both outputs were working functions whose three doctests pass; they were not the same function.
- Turn it on in the server by adding
--draft-model thinkingmachines/Inkling-2-Minito the Step 6 command. It costs about 1 GB more memory: peak memory on the command line went from 21.5 GB to 22.5 GB.
Step 10: Lock it down with tests and a Makefile
Everything so far was checked by eye. Tests turn the three facts this article
depends on into checks that fail loudly: the chart has the answer chart.py
claims, and the model, through the server, reads it and uses the tool. They are split in two files because only one of them
needs a 20 GB model in memory: the offline tests run anywhere in under a
second, and the live tests skip themselves when no server is listening.
First tell pytest where the modules are. The project has no package
structure, so pythonpath = ["."] lets the tests import chart, ask and
agent from the project root:
cd ~/src/inkling-mlx
mkdir -p tests
cat >> pyproject.toml <<'EOF'
[tool.pytest.ini_options]
pythonpath = ["."]
testpaths = ["tests"]
EOF
Create the file
cd ~/src/inkling-mlx
touch tests/test_offline.py
Add the code: ~/src/inkling-mlx/tests/test_offline.py
"""Tests that need no model: the chart, the tool, and argument checks."""
from __future__ import annotations
import struct
import zlib
import pytest
import agent
import ask
import chart
def decode_png(data: bytes) -> list[list[tuple[int, int, int]]]:
"""Decode the PNGs chart.py writes (8-bit RGB, filter type 0 only)."""
assert data[:8] == b"\x89PNG\r\n\x1a\n"
pos, idat, width, height = 8, b"", 0, 0
while pos < len(data):
(length,) = struct.unpack(">I", data[pos : pos + 4])
kind, body = data[pos + 4 : pos + 8], data[pos + 8 : pos + 8 + length]
if kind == b"IHDR":
width, height = struct.unpack(">II", body[:8])
elif kind == b"IDAT":
idat += body
pos += 12 + length
raw, stride = zlib.decompress(idat), 1 + width * 3
return [
[
tuple(raw[y * stride + 1 + x * 3 : y * stride + 4 + x * 3])
for x in range(width)
]
for y in range(height)
]
def test_chart_matches_its_answer(tmp_path):
out = tmp_path / "chart.png"
out.write_bytes(chart.encode_png(chart.pixels()))
rows = decode_png(out.read_bytes())
# Measure each bar's height from the decoded pixels, not the constants.
heights = []
for i in range(len(chart.BARS)):
x = (
chart.GAP
+ i * (chart.BAR_WIDTH + chart.GAP)
+ chart.BAR_WIDTH // 2
)
heights.append(sum(row[x] == chart.BAR_COLOR for row in rows))
assert heights == list(chart.BARS)
assert chart.expected_answer() == (5, 3)
def test_tool_hashes_a_project_file():
digest = agent.file_sha256("chart.py")
assert len(digest) == 64 and all(c in "0123456789abcdef" for c in digest)
@pytest.mark.parametrize(
"path", ["../outside.txt", "/etc/hosts"], ids=["relative", "absolute"]
)
def test_tool_refuses_outside_paths(path):
assert agent.file_sha256(path).startswith("error:")
def test_bad_effort_rejected_locally():
with pytest.raises(ValueError):
ask.ask("hello", effort="0.5")
Detailed breakdown
decode_png()reads the image back from bytes, so the chart test checks the file the model would see rather than the constants that produced it. It handles only whatchart.pywrites: 8-bit RGB with filter type 0 on every row.test_chart_matches_its_answermeasures each bar by counting orange pixels down the middle column of each bar. If a change tochart.pyever drew a bar wrong, the expected answer the live test relies on would be wrong too, and this test is what catches it.- The two path tests cover both escapes: a relative path that climbs out
with
.., and an absolute path. Both must come back as error strings, not hashes. Theidsgive them short names in pytest’s output,relativeandabsolute, instead of the paths themselves. - The effort test proves
ask()refuses"0.5"locally, before any request, which is the guard Step 7 added after Step 4 showed the template rejecting it.
Create the file
cd ~/src/inkling-mlx
touch tests/test_live.py
Add the code: ~/src/inkling-mlx/tests/test_live.py
"""Tests against the running server. Skipped when nothing answers."""
from __future__ import annotations
import urllib.request
import pytest
import agent
import ask
import chart
def server_up() -> bool:
try:
with urllib.request.urlopen(
ask.BASE_URL.removesuffix("/v1") + "/health", timeout=2
):
return True
except OSError:
return False
pytestmark = pytest.mark.skipif(
not server_up(), reason="mlx_vlm.server is not running"
)
def test_model_reads_the_chart(tmp_path):
image = tmp_path / "chart.png"
image.write_bytes(chart.encode_png(chart.pixels()))
reply = ask.ask(
"How many bars are in this chart, and which bar, counting from "
"the left, is the tallest? Reply exactly as: bars=<n> tallest=<k>",
image=image,
)
bars, tallest = chart.expected_answer()
assert reply.answer.strip() == f"bars={bars} tallest={tallest}"
def test_model_uses_the_tool():
expected = agent.file_sha256("chart.py")
assert expected in agent.run("What is the SHA-256 of chart.py?")
Detailed breakdown
server_up()probes/healthwith a two-second timeout, andpytestmarkapplies the skip to every test in the file. Without the server, these tests reportSKIPPEDwith the reason instead of failing on a connection error.test_model_reads_the_chartcompares the whole reply string againstbars=5 tallest=3built fromexpected_answer(), so the test and the chart cannot drift apart.test_model_uses_the_toolhasheschart.py, notout/chart.png, so it does not depend on Step 5 having been run, and asserts the true digest appears in the model’s answer. A model that skipped the tool could not produce it.- These are the slow tests, about nine seconds together, nearly all of it generation.
Create the file
cd ~/src/inkling-mlx
touch Makefile
Add the code: ~/src/inkling-mlx/Makefile
MODEL := thinkingmachines/Inkling-2-Mini
REVISION := b85c3cfa615fc413cc9af3379f491b963f3b586a
PORT := 8080
QUESTION := How many bars are in this chart, and which bar, counting from \
the left, is the tallest? Reply exactly as: bars=<n> tallest=<k>
.DEFAULT_GOAL := help
.PHONY: help download chart generate image serve ask agent effort draft test
help: ## Show this help
@grep -E '^[a-z-]+:.*## ' $(MAKEFILE_LIST) \
| awk -F':.*## ' '{printf " %-10s %s\n", $$1, $$2}'
download: ## Download the pinned checkpoint (about 20 GB)
uv run hf download $(MODEL) --revision $(REVISION)
chart: ## Draw the test chart into out/chart.png
uv run chart.py
generate: ## One text prompt through mlx_vlm.generate
uv run mlx_vlm.generate --model $(MODEL) \
--prompt "In two sentences, what does a Makefile do?" \
--reasoning-effort none --temperature 0 --max-tokens 400 --verbose
image: chart ## Ask the CLI about the test chart
uv run mlx_vlm.generate --model $(MODEL) \
--image out/chart.png --prompt "$(QUESTION)" \
--reasoning-effort none --temperature 0 --max-tokens 400 --verbose
serve: ## Run the OpenAI-compatible server on 127.0.0.1:8080
uv run mlx_vlm.server --host 127.0.0.1 --port $(PORT) \
--model $(MODEL) --log-level WARNING
ask: chart ## Ask the server about the test chart (needs make serve)
uv run ask.py "$(QUESTION)" --image out/chart.png
agent: chart ## Run the tool-calling loop (needs make serve)
uv run agent.py
effort: ## Sweep reasoning effort through the server (needs make serve)
uv run probe.py effort
draft: ## Compare drafter speed (stop make serve first)
uv run probe.py draft
test: ## Run the tests (live ones skip unless make serve is running)
uv run pytest -v
Detailed breakdown
.DEFAULT_GOAL := helpmakes plainmakeprint the target list. Thehelprecipe greps the Makefile fortarget: ## descriptionlines, so a new target documents itself by carrying a##comment.- Recipe lines must start with a tab, not spaces;
makestops withmissing separatorotherwise. Copying from a web page sometimes converts them. image,askandagentdepend onchart, so they drawout/chart.pngfirst and work in a fresh checkout.QUESTIONholds the chart prompt once and is reused by the CLI and server targets, so both ask exactly the same thing. The backslash at the end of its first line continues the value onto the next, joined with one space.- Each target’s help text says what it needs:
make serverunning for the server targets, and stopped fordraft.
Step 9 had you stop the server, so start it again in its own terminal (the
Step 6 command, or make serve) and wait for model_ready: true. Then check
the help screen and run the tests:
make
make test
help Show this help
download Download the pinned checkpoint (about 20 GB)
chart Draw the test chart into out/chart.png
generate One text prompt through mlx_vlm.generate
image Ask the CLI about the test chart
serve Run the OpenAI-compatible server on 127.0.0.1:8080
ask Ask the server about the test chart (needs make serve)
agent Run the tool-calling loop (needs make serve)
effort Sweep reasoning effort through the server (needs make serve)
draft Compare drafter speed (stop make serve first)
test Run the tests (live ones skip unless make serve is running)
uv run pytest -v
...
tests/test_live.py::test_model_reads_the_chart PASSED [ 14%]
tests/test_live.py::test_model_uses_the_tool PASSED [ 28%]
tests/test_offline.py::test_chart_matches_its_answer PASSED [ 42%]
tests/test_offline.py::test_tool_hashes_a_project_file PASSED [ 57%]
tests/test_offline.py::test_tool_refuses_outside_paths[relative] PASSED [ 71%]
tests/test_offline.py::test_tool_refuses_outside_paths[absolute] PASSED [ 85%]
tests/test_offline.py::test_bad_effort_rejected_locally PASSED [100%]
============================== 7 passed in 9.21s ===============================
The ... stands for pytest’s header lines (platform, rootdir, plugins), which
differ per machine. With the server stopped, the same command reports
5 passed, 2 skipped in under a second.
Troubleshooting
ValueError: Model type inkling2_loop not supported. Your mlx-vlm is a
release from before Mini support (0.8.0 or earlier). Re-run the git uv add
from Step 1 and check with the get_model_and_args line there. Avoid a plain
uv add mlx-vlm, or deleting [tool.uv.sources] and running uv sync: both
bring the old release back.
TemplateError: Unknown reasoning_effort: ..., or HTTP 500 with that
message. The effort was not one of none, minimal, low, medium,
high, xhigh or max. Numbers fail too, through both the command line and
the server, because they arrive as strings (Step 4).
An answer stops mid-sentence with no error. Reasoning used up
max_tokens. Through the server, finish_reason is length instead of
stop; raise the limit.
The Mac slows to a crawl or the process is killed. Two copies of the model
are loaded, usually the server plus an mlx_vlm.generate run or make draft.
Each needs about 21.5 GB. Stop the server first, or close other large
applications on a 32 GB machine.
address already in use when the server starts. The full line is
ERROR: [Errno 48] error while attempting to bind on address ('127.0.0.1', 8080): [errno 48] address already in use.
Something else, often an earlier server you forgot, is on port 8080. Find it
with lsof -i :8080, or start on another port with --port 8081 and set
INKLING_URL=http://127.0.0.1:8081/v1 for the scripts.
The download folder looks empty. The snapshot holds symbolic links; use
ls -lL or du -shL (Step 2).
Starting the server without a network connection. Set
HF_HUB_OFFLINE=1 so it loads from the cache without checking Hugging Face
for a newer revision.
What you built
Inkling 2 Mini now runs on your Mac from the weights Thinking Machines trained,
in their native NVFP4 and MXFP8 formats, with no conversion in between. You
generated text and read an image from the command line, served the model on
localhost behind an OpenAI-compatible API, called it from Python with the
official SDK, and gave it a tool it used correctly. Seven tests and a
Makefile let you rerun all of it.
Along the way you found the limits that the model card does not spell out:
mlx-vlmneeds a pinned git commit until the next release, and its version string cannot tell you whether you have the right one.- Reasoning effort accepts seven names and no numbers through
mlx-vlm, and evennonestill reasons. - Effort cost tokens and bought no correctness on eight checkable questions:
every level scored 8/8, and
lowwas the cheapest. - The bundled drafter is worth about 1.3x in generation speed on this machine, at the cost of output that is not identical to the plain run.
Next steps:
- Compare it with the GGUF build. llama.cpp supports this model from release b12619 (2026-11-21), newer than the 1.0.0 that Homebrew ships at the time of writing, and Run Qwen3.8-27B Locally with llama.cpp on macOS shows the same kind of measurement on another large model.
- Fine-tune a model with MLX. Fine-Tune a Local Model with LoRA on Apple Silicon using MLX trains an adapter on a small model in seconds, using the same framework.
- Put the tool loop to real work. Swap
file_sha256for a tool that does something you need, keep the path check, and keepMAX_TURNS.