Fine-Tune a Local Model with LoRA on Apple Silicon using MLX

By the end of this article you will have trained a language model to do something it currently cannot do, on your own Mac, in about twenty seconds of compute. You will hand it a bug report in plain English and it will answer with exactly one line:

severity=high component=auth summary=login fails after the token refresh runs

The base model does not do this. Asked the same question it starts explaining token refresh and never reaches a parseable answer. You will measure that difference rather than eyeball it: twenty test reports held out of training so the model has never seen them, which the base model scores 0/20 on and your fine-tuned model scores 20/20 on, run by a script you write in Step 8.

Along the way you will learn which knobs actually move that number. You will measure how many training steps the model needs before it is not just well-formatted but correct, and you will find the setting that silently throws your training away if you ship the model the obvious way.

Everything runs locally against a 0.6-billion-parameter model that fits in about 1.2 GB of memory while training. No API key, no cloud GPU, no account.

What LoRA is, and why it fits on a laptop

A language model’s weights are large. Retraining all of them means holding the model, its gradients, and an optimizer’s state in memory at once, which is why full fine-tuning is a datacenter activity. LoRA (Low-Rank Adaptation) sidesteps this. Instead of changing the original weights, it freezes them and learns a small pair of extra matrices alongside each one. Only those extras are trained, and only they are saved.

The saved file is called an adapter. In the run below it holds 2.9 million trainable parameters against the model’s 596 million, which is 0.484% of the total, and lands on disk at 11 MB. You load it on top of the base model at inference time, and the base model is untouched and reusable. Training one adapter per task and swapping between them is the normal way to work.

MLX is Apple’s array framework for Apple Silicon, built to use the unified memory that the CPU and GPU share on an M-series chip. mlx-lm is the language model library on top of it, and it ships the three commands this article uses: mlx_lm.lora to train, mlx_lm.generate to run, and mlx_lm.fuse to merge an adapter back into a model.

Prerequisites

• A Mac with Apple Silicon (M1 or later). MLX has no Intel path. This article was validated on an M5 Max with 128 GB of unified memory running macOS 26.6.2, but the training run peaks at about 1.16 GB and will fit comfortably on a base configuration.
• uv 0.11.26 or later, the Python package and project manager used throughout. Install it with brew install uv.
• About 1.6 GB of disk. The model is 335 MB; each adapter is 6–23 MB depending on its rank; and the fused model in Step 11 is a further 1.1 GB.
• make, for Step 13. The version Apple ships with the developer tools is fine.
• A network connection twice: once to download the model, and once more in Step 11, which needs files the first download deliberately skips.

No prior fine-tuning experience is assumed. Familiarity with a terminal and with reading Python is.

Step 1: Set up the project

Start with an empty project directory and a virtual environment. uv init --bare creates a pyproject.toml without the sample package layout that the default uv init scaffolds, which is what you want here because the scripts sit at the project root rather than inside a package. The tests/ directory is for Step 12.

Create the files

mkdir -p lora-fine-tune-mlx-macos/tests
cd lora-fine-tune-mlx-macos
uv init --bare
touch .gitignore

Add the code: .gitignore

# Python / uv
.venv/
__pycache__/
*.pyc
.pytest_cache/

# Generated by `make data` — rebuilt from make_data.py with a fixed seed.
data/

# Training output: adapters and fused models are large and reproducible.
adapters/
adapters-*/
fused/
fused-dequantized/

Detailed breakdown

The .gitignore comes first, before any other file, because everything this project generates is both large and reproducible. The dataset is rebuilt from a seeded generator, the adapters are rebuilt by a training command, and a fused model is over a gigabyte. Committing any of them would be committing build output.

adapters-* has a wildcard because Steps 9 and 10 train several adapters side by side to compare settings, and they should all stay out of git.

Now add the dependency. mlx-lm pulls in mlx and mlx-metal, which are the framework and its Metal backend.

Add the dependencies

uv add mlx-lm
uv add --dev pytest

Detailed breakdown

This resolves and installs the framework. The versions this article was validated against are mlx-lm 0.31.3, mlx 0.32.2, mlx-metal 0.32.2, transformers 5.17.0 and huggingface-hub 1.32.0, on Python 3.12.9. mlx-lm moves quickly and its command-line flags change between releases, so treat the flag names below as things to confirm against --help rather than memorize.

pytest goes in the development dependency group. The tests in Step 12 run without loading a model at all, which is what makes them fast enough to run on every change.

Step 2: Watch the base model fail

Before training anything, establish what you are starting from. Without a recorded baseline you have no way to tell a fine-tune that worked from one that merely produced confident-looking output.

The model is mlx-community/Qwen3-0.6B-4bit. The mlx-community organization on Hugging Face publishes models already converted to MLX’s format, so no conversion step is needed. The 4bit suffix means the weights are quantized — stored at 4 bits per parameter instead of 16, which cuts the download to roughly 330 MB and the memory footprint with it. Training a LoRA adapter on top of a quantized base is common enough to have its own name, QLoRA, and mlx_lm.lora does it without any extra flags.

Run the base model

uv run mlx_lm.generate \
  --model mlx-community/Qwen3-0.6B-4bit \
  --prompt "Login fails after the token refresh runs." \
  --max-tokens 60

The first run downloads the model. Later runs read it from ~/.cache/huggingface/hub/.

==========
<think>
Okay, the user is encountering a problem where they try to log in after a token refresh, but it doesn't work. Let me break this down step by step.

First, I need to recall how token refresh works. When a user logs in, their session is typically stored in a
==========
Prompt: 16 tokens, 22.780 tokens-per-sec
Generation: 60 tokens, 589.662 tokens-per-sec
Peak memory: 0.384 GB

Detailed breakdown

Two things in that output shape the rest of the article.

The first is that the model is answering a question nobody asked. It was given a bug report and it started explaining token refresh. That is reasonable default behavior for a chat model and useless if what you wanted was a triage record. The fine-tune closes that gap, which is one of behavior, not knowledge. The model already knows what a token refresh is.

The second is the <think> tag. Qwen3 is a reasoning model: it works in a separate channel before answering, and the content of that channel is not the answer. A token is the unit the model reads and writes, roughly a word or a word-piece, and --max-tokens 60 capped this run at sixty of them — not enough to finish thinking, so there is no answer here at all, just the opening of one. (The report itself mentions a token refresh; that is an authentication token, unrelated to the model’s tokens.) Remember this tag, because it does not go away when you fine-tune, and Step 7 explains why.

Step 3: Design a task you can actually score

Pick something vague, like “make it sound more friendly”, and you cannot tell whether training worked. Pick something with exactly one right answer per input and you can count.

The task here is triage. Given a one-sentence bug report, emit one line with three fields: a severity, a component, and a summary. The trick that makes it scoreable is that the mapping is deterministic by construction. The verb in the report decides the severity, and the subject decides the component. “Login fails” is severity=high component=auth because fails is a high-severity verb and login belongs to auth, and nothing else is a defensible answer.

That gives two separate numbers to track, and the distance between them is the point:

• parse — the answer matched the required line shape at all
• exact — the answer was also the correct triage for that report

An answer that fails the first test is off-format, and that is the word the code, its output and the rest of this article use for it.

A model that has learned the format but not the task scores full marks on the first and poorly on the second. Step 9 shows exactly that happening.

Step 4: Write the dataset generator

Rather than hand-write two hundred training examples, generate them from a seeded template. The seed makes the dataset reproducible, so the records and the losses printed later in this article are the ones you will get.

Create the file

touch make_data.py

Add the code: make_data.py

"""Build the triage dataset that teaches the model one fixed output line.

Every record maps a plain-English bug report to a single line of the form

    severity=<low|medium|high> component=<name> summary=<text>

The mapping is deterministic: the verb decides the severity and the subject
decides the component. That matters for evaluation — because there is exactly
one correct answer per report, `evaluate.py` can score exact matches instead of
asking a human whether the output "looks about right".
"""

import json
import random
from pathlib import Path

# The verb carries the severity. Keep these disjoint: a verb that appears under
# two severities makes some reports genuinely ambiguous, and the exact-match
# score would then punish the model for the dataset's mistake.
SEVERITIES = {
    "high": ["cannot start", "fails", "crashes", "is down", "loses data"],
    "medium": ["is slow", "times out", "returns the wrong total", "is stale"],
    "low": ["is misaligned", "has a typo", "is truncated", "wraps oddly"],
}

# The subject carries the component, under the same disjointness rule.
COMPONENTS = {
    "auth": ["login", "the session", "token refresh", "SSO"],
    "billing": ["the invoice", "checkout", "the receipt", "a refund"],
    "search": ["the search index", "autocomplete", "a saved filter"],
    "upload": ["file upload", "the image resizer", "a large attachment"],
    "api": ["the REST endpoint", "a webhook", "rate limiting"],
    "email": ["the welcome email", "a password reset", "digest mail"],
    "export": ["CSV export", "the PDF report", "a data dump"],
    "ui": ["the sidebar", "a modal", "the date picker"],
}

# Trailing context the model must copy into the summary but ignore when picking
# severity and component. The empty strings make some reports context-free.
CONTEXT = [
    "after the last deploy", "on mobile Safari", "for new accounts",
    "under load", "in the EU region", "when the cache is cold",
    "for admin users", "on first run", "", "",
]

TRAIN, VALID, TEST = 200, 20, 20


def make_record(rng: random.Random) -> dict[str, str]:
    """Draw one report and the single line that correctly triages it."""
    severity = rng.choice(list(SEVERITIES))
    component = rng.choice(list(COMPONENTS))
    subject = rng.choice(COMPONENTS[component])
    verb = rng.choice(SEVERITIES[severity])
    context = rng.choice(CONTEXT)

    body = " ".join(part for part in (subject, verb, context) if part)
    report = body[0].upper() + body[1:] + "."
    return {
        "prompt": report,
        "completion": f"severity={severity} component={component} summary={body}",
    }


def main() -> None:
    # A fixed seed makes the split reproducible, so the numbers in the article
    # are the numbers you get.
    rng = random.Random(11)
    seen: set[str] = set()
    rows: list[dict[str, str]] = []
    while len(rows) < TRAIN + VALID + TEST:
        record = make_record(rng)
        # Duplicates across the train/test boundary would leak answers and
        # inflate the test score, so keep prompts unique.
        if record["prompt"] in seen:
            continue
        seen.add(record["prompt"])
        rows.append(record)

    out = Path("data")
    out.mkdir(exist_ok=True)
    splits = (
        ("train", rows[:TRAIN]),
        ("valid", rows[TRAIN:TRAIN + VALID]),
        ("test", rows[TRAIN + VALID:]),
    )
    for name, chunk in splits:
        path = out / f"{name}.jsonl"
        with open(path, "w") as handle:
            for record in chunk:
                handle.write(json.dumps(record) + "\n")
        print(f"wrote {path} ({len(chunk)} records)")


if __name__ == "__main__":
    main()

Detailed breakdown

The two dictionaries at the top are the task definition. Their keys are the labels the model must learn to emit and their values are the surface forms that imply each label. The comment about disjointness is load-bearing: if is slow appeared under both medium and high, some generated reports would have two defensible answers, and the exact-match score would dock the model for the dataset’s ambiguity rather than for a mistake. Step 12 enforces this with a test rather than trusting the comment.

CONTEXT contains two empty strings among ten entries, so roughly a fifth of reports carry no trailing context. This stops the model from learning that every report ends with a prepositional phrase. The context must be copied into the summary but must not affect the severity or component, which is the part of the task that requires the model to actually attend to structure.

The uniqueness check is not cosmetic. The splits are taken as three slices of one shuffled list, so without it a report could appear in both train and test, and the test score would be measuring memorization.

The output is JSONL — one JSON object per line, not a JSON array. This is the format mlx_lm.lora expects, and it expects exactly three files named train.jsonl, valid.jsonl and test.jsonl inside the directory you point --data at.

Generate the data

uv run python make_data.py

wrote data/train.jsonl (200 records)
wrote data/valid.jsonl (20 records)
wrote data/test.jsonl (20 records)

Look at a couple of records before moving on:

head -2 data/train.jsonl

{"prompt": "A modal times out for new accounts.", "completion": "severity=medium component=ui summary=a modal times out for new accounts"}
{"prompt": "The date picker has a typo on mobile Safari.", "completion": "severity=low component=ui summary=the date picker has a typo on mobile Safari"}

Step 5: Know which format you are handing the trainer

mlx_lm.lora accepts three different JSONL shapes and picks between them by inspecting your data, which is convenient right up until it guesses differently than you expected. Learn the rule before you train: the failure mode is a successful-looking run that taught the wrong thing.

It reads the first record of the file and checks for keys in a fixed order:

1. prompt and completion present — treated as a prompt/completion pair
2. messages present — treated as a chat transcript
3. text present — treated as raw text with no structure

This project uses the first shape. In that shape mlx-lm wraps your two fields into a user turn and an assistant turn and applies the model’s own chat template, which means what the model trains on matches what it will see at inference time. That correspondence is the reason to prefer prompt/completion over text for instruction-style tasks.

Because only record zero is inspected, a file that mixes shapes is classified by its first line. A later record missing the chosen shape’s keys fails with a bare KeyError: 'prompt' once training reaches it, rather than with a message about data formats — so the error arrives late and names the wrong problem. Keep one shape per file.

Only train.jsonl is strictly required. A missing valid.jsonl prints a warning and trains without validation, which costs you the one number that tells you whether anything generalized, and test.jsonl is read only when you pass --test. This project writes all three.

Step 6: Train the adapter

The whole article builds to this one command. The flags below are the ones that matter for a first run; everything else in mlx_lm.lora --help has a default that is reasonable for this size of task.

Train

uv run mlx_lm.lora \
  --model mlx-community/Qwen3-0.6B-4bit \
  --train \
  --data data \
  --mask-prompt \
  --iters 300 \
  --batch-size 4 \
  --steps-per-report 25 \
  --steps-per-eval 100 \
  --adapter-path adapters

Loading pretrained model
Loading datasets
Training
Trainable parameters: 0.484% (2.884M/596.050M)
Starting training..., iters: 300
Iter 1: Val loss 5.801, Val took 0.145s
Iter 25: Train loss 0.998, Learning Rate 1.000e-05, It/sec 18.080, Tokens/sec 1571.508, Trained Tokens 2173, Peak mem 1.144 GB
Iter 50: Train loss 0.099, Learning Rate 1.000e-05, It/sec 18.528, Tokens/sec 1577.097, Trained Tokens 4301, Peak mem 1.157 GB
Iter 75: Train loss 0.036, Learning Rate 1.000e-05, It/sec 18.615, Tokens/sec 1620.988, Trained Tokens 6478, Peak mem 1.157 GB
Iter 100: Val loss 0.033, Val took 0.123s
Iter 100: Train loss 0.033, Learning Rate 1.000e-05, It/sec 18.641, Tokens/sec 1583.741, Trained Tokens 8602, Peak mem 1.157 GB
Iter 100: Saved adapter weights to adapters/adapters.safetensors and adapters/0000100_adapters.safetensors.
Iter 125: Train loss 0.014, Learning Rate 1.000e-05, It/sec 18.564, Tokens/sec 1612.821, Trained Tokens 10774, Peak mem 1.157 GB
Iter 150: Train loss 0.013, Learning Rate 1.000e-05, It/sec 18.557, Tokens/sec 1580.321, Trained Tokens 12903, Peak mem 1.157 GB
Iter 175: Train loss 0.004, Learning Rate 1.000e-05, It/sec 18.577, Tokens/sec 1593.161, Trained Tokens 15047, Peak mem 1.157 GB
Iter 200: Val loss 0.005, Val took 0.124s
Iter 200: Train loss 0.004, Learning Rate 1.000e-05, It/sec 18.555, Tokens/sec 1600.917, Trained Tokens 17204, Peak mem 1.157 GB
Iter 200: Saved adapter weights to adapters/adapters.safetensors and adapters/0000200_adapters.safetensors.
Iter 225: Train loss 0.002, Learning Rate 1.000e-05, It/sec 18.578, Tokens/sec 1599.212, Trained Tokens 19356, Peak mem 1.157 GB
Iter 250: Train loss 0.002, Learning Rate 1.000e-05, It/sec 18.539, Tokens/sec 1593.635, Trained Tokens 21505, Peak mem 1.157 GB
Iter 275: Train loss 0.001, Learning Rate 1.000e-05, It/sec 18.533, Tokens/sec 1587.892, Trained Tokens 23647, Peak mem 1.157 GB
Iter 300: Val loss 0.001, Val took 0.124s
Iter 300: Train loss 0.001, Learning Rate 1.000e-05, It/sec 18.508, Tokens/sec 1598.328, Trained Tokens 25806, Peak mem 1.157 GB
Iter 300: Saved adapter weights to adapters/adapters.safetensors and adapters/0000300_adapters.safetensors.
Saved final weights to adapters/adapters.safetensors.

That is the whole run, printed in full, and make train reported 17.7 seconds for it — including regenerating the dataset, which train depends on.

Learning Rate 1.000e-05 appears on every line and this article never changes it. It is the third knob, it is the one most likely to matter once the first two stop helping, and tuning it is out of scope here; the default is what produced every number below.

Two kinds of number appear in that output and they behave differently on a re-run. The losses and the trained-token counts are deterministic: mlx_lm.lora seeds its generator at 0 by default and the dataset came from a seeded generator, so a second run reproduces Val loss 5.801 and Trained Tokens 25806 exactly, and the saved adapter is bit-identical. The throughput columns, the wall-clock, and the third decimal of Peak mem will not match, since they depend on what else the machine is doing — runs on this machine reported 1.144 GB early and settled between 1.151 and 1.157 GB. If your losses diverge from these, something about the data or the flags differs; if only It/sec does, nothing is wrong.

Detailed breakdown

--mask-prompt excludes the bug report from the loss — the number measuring how wrong the model’s predictions are, which training works to make smaller — so the model is scored only on the line it is supposed to produce and not on its ability to echo back the question. For instruction-style tasks this is almost always what you want. It is off by default, and leaving it off spends part of the model’s limited capacity learning to reproduce inputs.

--iters 300 is the number of optimizer steps, not passes over the data. With --batch-size 4 against 200 training records, one pass is 50 steps, so 300 iterations is six passes. Step 9 measures what happens at 25, 50 and 100.

--steps-per-report 25 and --steps-per-eval 100 set the two printing cadences: a training-loss line every 25 iterations, a validation pass every 100. They are why the log has the lines it has, and changing them changes only what you see, not what is learned.

--adapter-path adapters is both the output directory and, later, the flag you pass at inference. Alongside the final adapters.safetensors you get numbered checkpoints every 100 steps and an adapter_config.json recording every setting used, which makes a training run self-documenting.

One default does more than it looks: --num-layers is 16, and this model has 28 blocks. LoRA is attached to the last 16 of them rather than to all, which is where the 0.484% figure comes from. Passing --num-layers -1 adapts every block and raises both the parameter count and the memory figure.

Read the two loss columns, not one. Train loss is measured on data the model is being fitted to and always falls. Val loss is measured on the 20 held-out records in valid.jsonl — the validation split, which the trainer abbreviates to Val — and is the one that tells you whether anything transferred. Here it goes from 5.801 to 0.001, which is the direction you want. Val loss flattening or climbing while train loss keeps falling is overfitting, and on a dataset this small it is a realistic outcome if you push the iteration count much higher.

The peak memory figure, a shade over 1.15 GB, is the whole training footprint. That number is why this runs on a laptop.

Step 7: Ask the tuned model, and handle what comes back

Now use it. The adapter is applied at load time with --adapter-path, and the base model files are read from the same cache as before.

Run the tuned model

uv run mlx_lm.generate \
  --model mlx-community/Qwen3-0.6B-4bit \
  --adapter-path adapters \
  --prompt "Login fails after the token refresh runs." \
  --max-tokens 40

==========
<think>

</think>

severity=high component=auth summary=login fails after the token refresh runs
==========

That is the transformation, against the identical prompt that produced open-ended reasoning and no answer in Step 2.

Detailed breakdown

The <think> tag is still there, and it is now empty. Step 2 promised this detail, and it is the likeliest thing to break a parser written against this model.

The reason is not what it looks like. The tags are not being forced on you by the template at inference time — the prompt the model actually receives contains no <think> at all:

<|im_start|>user\nLogin fails.<|im_end|>\n<|im_start|>assistant\n

They come from training. When the chat template renders a finished assistant turn that carries no reasoning content, it writes the answer preceded by an empty pair of tags:

<|im_start|>assistant\n<think>\n\n</think>\n\nseverity=high component=auth summary=login fails<|im_end|>\n

That rendering is what mlx_lm.lora trained on in Step 6, and --mask-prompt keeps everything after the user turn inside the loss. So all 200 training targets began with <think>\n\n</think>\n\n, and the model learned to emit that pair before every answer. Fine-tuning did not fail to remove the reasoning channel; fine-tuning is the reason the empty channel is there.

The practical consequence is the same either way: anything reading this output has to strip the tags before parsing, and a parser built against a non-reasoning model will not expect them. Qwen3’s template does accept enable_thinking=False, which pre-closes the channel at inference time, but that is a separate mechanism from the one training installed here.

Write that handling down now, as a function, rather than inlining a regular expression at each call site.

Create the file

touch triage.py

Add the code: triage.py

"""Answer one bug report with the fine-tuned adapter.

This is the inference side of the project. The parsing helpers live here rather
than in `evaluate.py` because the tests exercise them without loading a model —
`strip_reasoning` and `parse_line` are plain string functions, and every claim
the article makes about the output shape is checked against them.
"""

import argparse
import re

MODEL = "mlx-community/Qwen3-0.6B-4bit"
ADAPTER = "adapters"
MAX_TOKENS = 60

# Qwen3's chat template renders an assistant turn that carries no reasoning
# content as an EMPTY `<think></think>` pair followed by the answer. Training
# sees that rendering, and `--mask-prompt` keeps it inside the loss, so the
# model is taught to emit the empty pair before every answer. It is not the
# template speaking at inference time: the generation prompt contains no
# `<think>` at all. Either way the tags reach this code, so strip them.
#
# Only a CLOSED channel is removed. The base model, capped at MAX_TOKENS, often
# never closes its `<think>`, and leaving that text in place is what makes the
# off-format case visible instead of silently blank.
REASONING = re.compile(r"<think>.*?</think>", re.DOTALL)

TRIAGE_LINE = re.compile(r"^severity=(\S+) component=(\S+) summary=(.+)$")


def strip_reasoning(text: str) -> str:
    """Remove a closed reasoning channel and surrounding whitespace."""
    return REASONING.sub("", text).strip()


def parse_line(line: str) -> dict[str, str] | None:
    """Return the three triage fields, or None if the line is off-format."""
    match = TRIAGE_LINE.match(line)
    if match is None:
        return None
    severity, component, summary = match.groups()
    return {"severity": severity, "component": component, "summary": summary}


def triage(report: str, adapter: str | None = ADAPTER,
           max_tokens: int = MAX_TOKENS) -> str:
    """Load the model and return its raw answer for one report."""
    # Imported here so that `--help` and the tests do not pay for MLX.
    from mlx_lm import generate, load

    model, tokenizer = load(MODEL, adapter_path=adapter)
    prompt = tokenizer.apply_chat_template(
        [{"role": "user", "content": report}],
        add_generation_prompt=True,
        tokenize=False,
    )
    answer = generate(model, tokenizer, prompt=prompt,
                      max_tokens=max_tokens, verbose=False)
    return strip_reasoning(answer)


def main() -> None:
    parser = argparse.ArgumentParser(description="Triage one bug report.")
    parser.add_argument("report", help="The bug report, in plain English.")
    parser.add_argument("--base", action="store_true",
                        help="Use the base model with no adapter, for comparison.")
    parser.add_argument("--adapter", default=ADAPTER,
                        help=f"Adapter directory to load (default: {ADAPTER}).")
    parser.add_argument("--max-tokens", type=int, default=MAX_TOKENS,
                        help=f"Generation budget (default: {MAX_TOKENS}).")
    args = parser.parse_args()

    adapter = None if args.base else args.adapter
    line = triage(args.report, adapter=adapter, max_tokens=args.max_tokens)
    print(line)

    fields = parse_line(line)
    if fields is None:
        print("\n-- off-format: nothing to parse --")
    else:
        print()
        for key, value in fields.items():
            print(f"{key:>10}: {value}")


if __name__ == "__main__":
    main()

Detailed breakdown

strip_reasoning uses a non-greedy match with re.DOTALL so it spans the newlines inside the channel and stops at the first closing tag rather than running to the last one in the string.

parse_line returns None rather than raising, because off-format output is an expected result here and not an error. Step 8 counts those Nones, and the --base flag exists so you can produce one on demand.

The from mlx_lm import ... sits inside triage() rather than at module scope. Importing MLX costs noticeable time, and the tests in Step 12 import this module purely for the two string functions. A top-level import would make a test suite that touches no model pay the framework’s startup cost on every run.

apply_chat_template with add_generation_prompt=True is what makes the inference path match the training path. mlx_lm.lora applied the same template when it built the training batches in Step 6, so skipping it here would present the model with text in a shape it was never trained on.

Confirm both paths

uv run python triage.py "Login fails after the token refresh runs."

severity=high component=auth summary=login fails after the token refresh runs

  severity: high
 component: auth
   summary: login fails after the token refresh runs

uv run python triage.py --base "Login fails after the token refresh runs."

<think>
Okay, the user is encountering a problem where they try to log in after a token refresh, but it doesn't work. Let me break this down step by step.

First, I need to recall how token refresh works. When a user logs in, their session is typically stored in a

-- off-format: nothing to parse --

Step 8: Measure it instead of trusting it

One good answer is an anecdote. The claim that matters is about the twenty reports the model has never seen, in data/test.jsonl, which make_data.py set aside and training never touched.

Create the file

touch evaluate.py

Add the code: evaluate.py

"""Score a model on the held-out test set.

Two numbers, and the gap between them is the point:

  parse  — the answer matched the required line shape at all
  exact  — the answer was also the correct triage for that report

Format is learned long before content is. Measured on this project, 25 training
iterations score 20/20 parse and 3/20 exact. Reporting only the first number
would call that a success.
"""

import argparse
import json
from pathlib import Path

from triage import MODEL, parse_line, strip_reasoning

TEST_SET = Path("data/test.jsonl")
MAX_TOKENS = 60


def load_records() -> list[dict[str, str]]:
    if not TEST_SET.exists():
        raise SystemExit(f"{TEST_SET} not found — run `make data` first.")
    with open(TEST_SET) as handle:
        return [json.loads(line) for line in handle]


def score(adapter: str | None, max_tokens: int = MAX_TOKENS,
          model_path: str = MODEL) -> tuple[int, int, int]:
    # Read the test set BEFORE loading the model. Loading first would mean a
    # missing dataset surfaces as an MLX traceback from deep inside the
    # library, and the message above would never be reached.
    records = load_records()
    if adapter is not None and not Path(adapter).exists():
        raise SystemExit(f"{adapter}/ not found — run `make train` first.")

    from mlx_lm import generate, load

    model, tokenizer = load(model_path, adapter_path=adapter)
    parsed = exact = 0
    for record in records:
        prompt = tokenizer.apply_chat_template(
            [{"role": "user", "content": record["prompt"]}],
            add_generation_prompt=True,
            tokenize=False,
        )
        answer = strip_reasoning(
            generate(model, tokenizer, prompt=prompt,
                     max_tokens=max_tokens, verbose=False)
        )
        if parse_line(answer) is not None:
            parsed += 1
        if answer == record["completion"]:
            exact += 1
    return parsed, exact, len(records)


def main() -> None:
    parser = argparse.ArgumentParser(description="Score a model on the test set.")
    parser.add_argument("--adapter", default="adapters",
                        help="Adapter directory, or 'none' for the base model.")
    parser.add_argument("--max-tokens", type=int, default=MAX_TOKENS,
                        help="Generation budget. The base model does not stop, "
                             "so give it more before calling it a failure.")
    # A fused model is a model, not an adapter, so scoring one means replacing
    # the base rather than layering on top of it.
    parser.add_argument("--model", default=MODEL,
                        help="Model to score (default: the 4-bit base).")
    args = parser.parse_args()

    adapter = None if args.adapter == "none" else args.adapter
    label = args.model if args.model != MODEL else (
        "base model" if adapter is None else adapter)
    parsed, exact, total = score(adapter, args.max_tokens, args.model)
    print(f"{label}: {parsed}/{total} parse, {exact}/{total} exact")


if __name__ == "__main__":
    main()

Detailed breakdown

The module imports MODEL, parse_line and strip_reasoning from triage.py rather than redefining them. If the evaluator stripped reasoning differently from the tool you actually ship, its score would be measuring a program nobody runs.

--max-tokens is exposed as a flag specifically so the base model gets a fair hearing. The tuned model answers in well under 60 tokens, but the base model is still mid-explanation at that point, and scoring it against a budget it cannot finish in would prove nothing. Giving it 120 tokens changes its score not at all, which is a stronger result than giving it 60.

Counting parse and exact separately is the design decision that makes the next step possible. They are collected in the same loop because they are two questions about one answer.

Score both models

uv run python evaluate.py --adapter none --max-tokens 120

base model: 0/20 parse, 0/20 exact

uv run python evaluate.py --adapter adapters

adapters: 20/20 parse, 20/20 exact

Zero to twenty, from under twenty seconds of training and an 11 MB file. The article set out to produce that result, measured rather than asserted.

Step 9: Find out how much training you actually needed

Three hundred iterations worked, but the number was picked before any evidence existed. Three more training runs, scored against the adapter you already have, answer the question properly and take about a minute in total.

Train and score a sweep

for N in 25 50 100; do
  uv run mlx_lm.lora --model mlx-community/Qwen3-0.6B-4bit --train --data data \
    --mask-prompt --iters $N --batch-size 4 --steps-per-report 25 \
    --steps-per-eval 100 --adapter-path adapters-$N > /dev/null 2>&1
  uv run python evaluate.py --adapter adapters-$N
done

adapters-25: 20/20 parse, 3/20 exact
adapters-50: 20/20 parse, 10/20 exact
adapters-100: 20/20 parse, 14/20 exact

Together with the 300-iteration result from Step 8:

Detailed breakdown

The format column is already full at 25 iterations, before the model has even finished one pass over the training data. After half a pass it produces perfectly-shaped triage lines, and 17 out of 20 of them say the wrong thing.

This is the most useful thing to take away from the article, and the reason Step 3 insisted on a task with one right answer. A run stopped at 25 iterations looks finished. Every output is well-formed, every field is present, and a spot check of two or three examples would very likely pass. Only the second column reveals that the model has learned the shape of the answer and almost nothing about choosing it.

Format is cheap to learn because it is the same in every training example. Content is expensive because it is different in every one. Any evaluation that only checks “did it produce valid output” is measuring the cheap half.

The practical rule that follows: score the thing you actually care about, and treat a well-formatted answer as the start of the evaluation rather than the end of it.

Step 10: Change a setting that has no command-line flag

The other knob that matters here is rank — the size of the small matrices LoRA learns. Rank sets how much capacity the adapter has: a higher rank can represent more, costs more disk, and risks fitting noise on a small dataset.

Rank has no flag on mlx_lm.lora. It lives in a nested lora_parameters block reachable only through a YAML config file passed with -c, and --help lists neither name. Guessing at a flag is not the trap, because mlx_lm.lora --rank 16 is rejected outright with unrecognized arguments. The trap is the YAML: put rank: 16 at the top level of the config and it is accepted, ignored, and never mentioned again, because rank is only read from inside lora_parameters.

Create the file

touch rank16.yaml

Add the code: rank16.yaml

model: mlx-community/Qwen3-0.6B-4bit
train: true
data: data
mask_prompt: true
iters: 100
batch_size: 4
adapter_path: adapters-rank16
steps_per_report: 25
steps_per_eval: 100
lora_parameters:
  rank: 16
  dropout: 0.0
  scale: 20.0

Detailed breakdown

Most keys in the config file are the underscored form of a command-line flag, so --mask-prompt becomes mask_prompt and --adapter-path becomes adapter_path. A few exist only here, lora_parameters among them. A config file can replace the flags entirely, which pays off once a run has more than a couple of non-default settings, because the file is then a record of the experiment.

The defaults, visible in adapters/adapter_config.json after any run, are rank: 8, dropout: 0.0 and scale: 20.0. Supply all three. The lora_parameters block is not merged key-by-key with the defaults; it replaces them wholesale, so a config giving only rank: 16 fails with KeyError: 'dropout' the moment the adapters are built.

Train and score at three ranks

uv run mlx_lm.lora -c rank16.yaml > /dev/null 2>&1
uv run python evaluate.py --adapter adapters-rank16

Do not edit rank16.yaml to get the rank-4 number. Its adapter_path points at adapters-rank16, so editing in place overwrites the adapter you just scored and leaves a file called rank16.yaml containing rank 4. Copy it instead, changing both values:

sed -e 's/rank: 16/rank: 4/' -e 's/adapters-rank16/adapters-rank4/' \
  rank16.yaml > rank4.yaml
uv run mlx_lm.lora --config rank4.yaml > /dev/null 2>&1
uv run python evaluate.py --adapter adapters-rank4

The rank-8 row needs no new run: it is adapters-100 from Step 9, the same 100 iterations at the default rank. The three together give:

The adapter doubles in size with each step, as expected from doubling the rank. The score does not follow the same shape. Going from rank 4 to rank 8 cost twice the disk and bought nothing at all on this task, while going from 8 to 16 bought five more correct answers.

That is the honest result, more instructive than a clean line would have been. Rank is not a quality dial to be turned up. Rank 4 already had as much capacity as 100 iterations could fill, so paying for rank 8 bought nothing; rank 16 helping at that same iteration count is then evidence that rank 8 was a binding constraint after all. Which knob is limiting you is a question to measure, not to assume.

One caveat on reading that table as a capacity result. mlx-lm applies the LoRA scale directly rather than dividing it by the rank, as some implementations do, and this config holds scale at 20.0 across all three runs. Raising the rank therefore raises the size of the update as well as the capacity, and this experiment does not separate the two effects.

Step 11: Ship it, and avoid the trap

An adapter needs its base model present at load time. To hand someone a single self-contained model instead, merge the adapter into the weights with mlx_lm.fuse. This step holds the most expensive mistake in the article.

Complete the model snapshot first

uv run python -c "from huggingface_hub import snapshot_download; \
  print(snapshot_download('mlx-community/Qwen3-0.6B-4bit'))"

Detailed breakdown

Without this, mlx_lm.fuse fails on a model you have already trained against, with an error that points in the wrong direction:

huggingface_hub.errors.IncompleteSnapshotError: The cached snapshot for
'mlx-community/Qwen3-0.6B-4bit' (revision 'main', ...) is incomplete:
2 file(s) are missing (.gitattributes, README.md). Outgoing traffic is disabled
('local_files_only=True'). Re-run the download with network access to complete
the snapshot.

The message says outgoing traffic is disabled, which reads like a network problem and is not one. The cause is that mlx-lm downloads models with an allowlist of patterns covering weights, tokenizer and configuration, so the cached snapshot is deliberately partial and has never contained README.md. Saving the fused model then asks the cache for a complete snapshot with local_files_only=True, which forbids fetching the two files it just decided were missing. The check happens on the way out, after the model has been loaded, fused and dequantized, so you lose that work as well as the command. Downloading the full snapshot once satisfies it permanently.

Fuse the adapter

uv run mlx_lm.fuse \
  --model mlx-community/Qwen3-0.6B-4bit \
  --adapter-path adapters \
  --save-path fused-dequantized \
  --dequantize

Loading pretrained model
Dequantizing model

Detailed breakdown

--dequantize is not optional here, and leaving it off produces a model that silently contains none of your training. That is the trap, and nothing in the output warns you.

Fusing without it writes a 331 MB model that loads, runs, and answers every prompt exactly like the untrained base:

The third row is the one to produce yourself, since it is the claim that costs you if it is wrong:

uv run mlx_lm.fuse --model mlx-community/Qwen3-0.6B-4bit \
  --adapter-path adapters --save-path /tmp/fused-quantized
uv run python evaluate.py --model /tmp/fused-quantized --adapter none

/tmp/fused-quantized: 0/20 parse, 0/20 exact

Note the --model flag rather than --adapter: a fused model replaces the base rather than layering on top of it, so scoring one means pointing evaluate.py at a different model with no adapter at all.

The base model here is quantized to 4 bits. Merging LoRA’s learned matrices into a weight means adding a small delta to it, and writing that sum back into a 4-bit representation rounds it to the nearest representable value. On this adapter the delta is a few per cent of the distance between neighbouring representable values and never reaches one full step, so it rounds away and the fused weight is the weight you started with. The command reports success because nothing failed. --dequantize writes the merged weights back at the model’s original 16-bit precision instead, which is why the output is three and a half times larger and why it is the only fused variant that works.

Given that, the adapter is usually the better artifact to ship. It is 11 MB against 1.1 GB, it leaves the base model shared between tasks, and nothing rounds it away.

Verify the fused model

uv run mlx_lm.generate --model ./fused-dequantized \
  --prompt "The PDF report is truncated for admin users." --max-tokens 40

==========
<think>

</think>

severity=low component=export summary=the PDF report is truncated for admin users

Step 12: Lock the behavior down with tests

The claims this article makes about output shape are exactly the claims a test can hold. These tests load no model and need no network, so they run in hundredths of a second and can run on every edit.

Create the files

touch tests/test_triage.py tests/test_make_data.py

Add the code: tests/test_triage.py

"""Tests for the output parsing, which run without loading a model."""

from triage import parse_line, strip_reasoning


def test_strip_reasoning_removes_an_empty_channel():
    # What the fine-tuned model actually emits: the tags survive, empty.
    raw = "<think>\n\n</think>\n\nseverity=high component=auth summary=login fails"
    assert strip_reasoning(raw) == "severity=high component=auth summary=login fails"


def test_strip_reasoning_removes_a_filled_channel():
    raw = "<think>Let me consider this.</think>\nseverity=low component=ui summary=a modal is stale"
    assert strip_reasoning(raw).startswith("severity=low")


def test_strip_reasoning_leaves_a_clean_line_alone():
    line = "severity=medium component=api summary=a webhook is slow"
    assert strip_reasoning(line) == line


def test_parse_line_splits_the_three_fields():
    fields = parse_line("severity=high component=billing summary=checkout crashes under load")
    assert fields == {
        "severity": "high",
        "component": "billing",
        "summary": "checkout crashes under load",
    }


def test_parse_line_keeps_spaces_in_the_summary():
    fields = parse_line("severity=low component=export summary=the PDF report is truncated")
    assert fields["summary"] == "the PDF report is truncated"


def test_parse_line_rejects_prose():
    # The base model's answer, which is what the off-format branch exists for.
    assert parse_line("Okay, the user is reporting that login fails.") is None


def test_parse_line_rejects_a_reordered_line():
    assert parse_line("component=auth severity=high summary=login fails") is None


def test_parse_line_rejects_a_missing_field():
    assert parse_line("severity=high component=auth") is None


def test_strip_reasoning_leaves_an_unclosed_channel_alone():
    # The base model, capped at 60 tokens, often never closes `<think>`. The
    # regex needs a closing tag, so the text stays — which is what makes the
    # off-format case visible rather than silently blank.
    raw = "<think>\nOkay, the user is encountering a problem where they"
    assert strip_reasoning(raw) == raw


def test_strip_reasoning_removes_only_the_first_closed_channel():
    raw = "<think>a</think>severity=low component=ui summary=x <think>b</think>"
    assert "<think>" not in strip_reasoning(raw)

Add the code: tests/test_make_data.py

"""Tests for the dataset generator, which needs no model and no network."""

import json
import random

from make_data import COMPONENTS, SEVERITIES, make_record
from triage import parse_line


def test_verbs_are_unique_to_one_severity():
    # Overlap would make some reports genuinely ambiguous and would punish the
    # model for the dataset's mistake at exact-match time.
    verbs = [v for group in SEVERITIES.values() for v in group]
    assert len(verbs) == len(set(verbs))


def test_subjects_are_unique_to_one_component():
    subjects = [s for group in COMPONENTS.values() for s in group]
    assert len(subjects) == len(set(subjects))


def test_every_completion_is_on_format():
    rng = random.Random(11)
    for _ in range(200):
        assert parse_line(make_record(rng)["completion"]) is not None


def test_severity_follows_the_verb_and_component_the_subject():
    rng = random.Random(3)
    for _ in range(200):
        record = make_record(rng)
        fields = parse_line(record["completion"])
        verbs = SEVERITIES[fields["severity"]]
        subjects = COMPONENTS[fields["component"]]
        assert any(verb in record["prompt"].lower() for verb in verbs)
        assert any(subject in fields["summary"] for subject in subjects)


def test_the_report_is_a_capitalised_sentence():
    rng = random.Random(7)
    for _ in range(50):
        prompt = make_record(rng)["prompt"]
        assert prompt[0].isupper()
        assert prompt.endswith(".")


def test_records_are_json_serialisable():
    rng = random.Random(11)
    record = make_record(rng)
    assert set(json.loads(json.dumps(record))) == {"prompt", "completion"}

Detailed breakdown

The first two tests in test_make_data.py enforce the disjointness rule that Step 4 stated as a comment. A comment describing an invariant is a wish; a test enforces it. If a later edit adds is slow to the high list, the suite fails instead of the exact-match score quietly dropping for reasons that look like a training problem.

test_severity_follows_the_verb_and_component_the_subject checks the property the whole evaluation rests on: that each generated record really is answerable from its text. It runs 200 draws rather than one because the generator is random, and a single draw would exercise one path through it.

In test_triage.py, test_parse_line_rejects_prose uses the base model’s real Step 2 output as its input. The off-format branch is not hypothetical, and the test says where its example came from.

The tests import triage and make_data from the project root, and pytest will not find them there without being told. That configuration goes in the file uv init already created.

Create the file

# pyproject.toml already exists; this appends to it.

Add the code: pyproject.toml

[tool.pytest.ini_options]
# The modules under test live at the project root, not in a package, so pytest
# needs the root on sys.path to import them from tests/.
pythonpath = ["."]
testpaths = ["tests"]

Detailed breakdown

pythonpath = ["."] puts the project root on sys.path for the test run, which is what lets from triage import parse_line resolve from inside tests/. Without it, collection fails before a single test runs, with ModuleNotFoundError: No module named 'triage'. This is the usual way a flat project layout trips pytest.

testpaths = ["tests"] means a bare pytest collects only that directory rather than scanning the whole project, so a stray test_ function elsewhere does not join the suite by accident.

Run the tests

uv run pytest -q

................                                                         [100%]
16 passed in 0.01s

Step 13: Wrap it in a Makefile

Every command so far has been long enough to get wrong. A Makefile turns them into names and documents the workflow in the process.

Create the file

touch Makefile

Add the code: Makefile

MODEL   := mlx-community/Qwen3-0.6B-4bit
ADAPTER ?= adapters
ITERS   ?= 300
REPORT  ?= Login fails after the token refresh runs.

.DEFAULT_GOAL := help

.PHONY: help
help:  ## Show this help screen
	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) \
		| awk 'BEGIN {FS = ":.*?## "}; {printf "  \033[36m%-16s\033[0m %s\n", $$1, $$2}'

.PHONY: setup
setup:  ## Install dependencies
	uv sync

.PHONY: data
data:  ## Generate data/{train,valid,test}.jsonl
	uv run python make_data.py

.PHONY: snapshot
snapshot:  ## Download the complete model snapshot (needed by `make fuse`)
	uv run python -c "from huggingface_hub import snapshot_download; \
		print(snapshot_download('$(MODEL)'))"

.PHONY: train
train: data  ## Train the LoRA adapter (ITERS=300 ADAPTER=adapters)
	uv run mlx_lm.lora --model $(MODEL) --train --data data --mask-prompt \
		--iters $(ITERS) --batch-size 4 --steps-per-report 25 \
		--steps-per-eval 100 --adapter-path $(ADAPTER)

.PHONY: train-rank
train-rank: data  ## Train at a non-default rank (RANK=4 or 16, via rank<N>.yaml)
	uv run mlx_lm.lora --config rank$(RANK).yaml

.PHONY: ask
ask:  ## Triage one report with the adapter (REPORT='...' ADAPTER=...)
	uv run python triage.py --adapter $(ADAPTER) "$(REPORT)"

.PHONY: ask-base
ask-base:  ## Triage the same report with no adapter, for comparison
	uv run python triage.py --base "$(REPORT)"

.PHONY: evaluate
evaluate:  ## Score the adapter on the held-out test set
	uv run python evaluate.py --adapter $(ADAPTER)

.PHONY: evaluate-base
evaluate-base:  ## Score the base model on the same test set
	uv run python evaluate.py --adapter none --max-tokens 120

.PHONY: fuse
fuse: snapshot  ## Fuse the adapter into a standalone dequantized model
	uv run mlx_lm.fuse --model $(MODEL) --adapter-path $(ADAPTER) \
		--save-path fused-dequantized --dequantize

.PHONY: test
test:  ## Run the unit tests (no model required)
	uv run pytest -q

.PHONY: clean
clean:  ## Remove generated data, adapters, fused models and caches
	rm -rf data adapters adapters-* fused-dequantized .pytest_cache \
		__pycache__ tests/__pycache__

Detailed breakdown

.DEFAULT_GOAL := help means a bare make prints the target list rather than running anything, so the project explains itself.

train depends on data, so a fresh clone cannot train against a dataset that does not exist. fuse depends on snapshot for the same reason, which turns the confusing IncompleteSnapshotError from Step 11 into a step that simply happens.

ADAPTER, ITERS and REPORT use ?= so that an environment variable can supply a default. Overriding on the command line works either way — a command-line assignment beats both ?= and := — so this is about where a default may come from, not about whether you can override it. Either of these works:

make train ITERS=50 ADAPTER=adapters-50
make ask REPORT="The invoice is stale in the EU region."

Confirm the default target

make

  help             Show this help screen
  setup            Install dependencies
  data             Generate data/{train,valid,test}.jsonl
  snapshot         Download the complete model snapshot (needed by `make fuse`)
  train            Train the LoRA adapter (ITERS=300 ADAPTER=adapters)
  train-rank       Train at a non-default rank (RANK=4 or 16, via rank<N>.yaml)
  ask              Triage one report with the adapter (REPORT='...' ADAPTER=...)
  ask-base         Triage the same report with no adapter, for comparison
  evaluate         Score the adapter on the held-out test set
  evaluate-base    Score the base model on the same test set
  fuse             Fuse the adapter into a standalone dequantized model
  test             Run the unit tests (no model required)
  clean            Remove generated data, adapters, fused models and caches

What you built

Starting from a stock 0.6B model that answered a bug report by explaining it at length, you now have an 11 MB adapter that answers the same report with one parseable line, and a measurement proving it: 0/20 to 20/20 on twenty reports the model never saw during training.

The whole loop is four commands and under a minute of compute:

make data      # 240 records from a seeded generator
make train     # about 18 seconds, ~1.16 GB peak memory
make evaluate  # 20/20 parse, 20/20 exact
make test      # 16 tests, no model required

Three findings to carry to your next fine-tune:

• Format is learned long before correctness. At 25 iterations the model produced perfectly-shaped output and got 3 of 20 answers right. An evaluation that only checks whether output parses would have called that done.
• Rank is a ceiling, not a dial. Doubling rank from 4 to 8 changed the score not at all on this task, while doubling again to 16 helped. Which knob is limiting you is a question to answer by measuring, not by assuming.
• Fusing into a quantized model discards the adapter without saying so. --dequantize is what makes the merged weights survive, and shipping the adapter instead avoids the question.

Where to go next, roughly in order of payoff:

• Swap in your own data. Nothing above is specific to triage. Replace make_data.py with a loader for whatever prompt-and-answer pairs you have, keep the prompt/completion shape from Step 5, and the rest of the pipeline works unchanged.
• Try --fine-tune-type dora. mlx_lm.lora accepts lora, dora and full. DoRA separates each weight’s magnitude from its direction and often gets closer to full fine-tuning at similar cost. The evaluation harness you built is what makes that comparison meaningful.
• Serve it. mlx_lm.server exposes an OpenAI-compatible endpoint and takes the same --adapter-path, which puts your adapter behind the same API shape the rest of your tooling already speaks.
• Watch for overfitting on smaller data. With 200 records and 300 iterations, validation loss fell the whole way. With 30 records it may turn upward partway, and that turn is the signal to stop.