Run Laya Locally on Apple Silicon and Measure Its Confidence Yourself

Laya is an open-weights model from Convai Innovations that answers typed questions about a piece of text instead of writing prose. You hand it a state (a support thread, an email, a JSON record) and a set of questions, each of which declares the shape of its answer in advance: pick one option from a list, place the text on an ordered scale, or give the probability that a statement is true. Every answer comes back as a probability distribution over the options you declared. Laya is an encoder, a network that reads text and scores it rather than writing any. Each call is one forward pass (a single run of the network over the input), with no generated text to parse.

What you build here is a support-ticket triage script that runs entirely on your Mac. It asks Laya five typed questions about each ticket in roughly 35 to 60 ms on the Apple GPU, acts on the answers it is confident about, and escalates the rest to a human. “Confident” will not be Laya’s word for it: it will be a threshold you fitted yourself, after measuring on 100 labeled tickets how accurate each checkpoint (a trained set of weights: you will compare Laya’s base model with a version fine-tuned for this kind of task) is, how fast it runs on your GPU and on your CPU, and how far its stated probabilities are from the accuracy you observed.

That last measurement is the reason to do this rather than read the model card. Convai’s model card is candid about Laya’s weaknesses, and you will reproduce the most important of them: the base checkpoint scores below a baseline that ignores the input entirely. You will also find a result that runs the other way from what the card says about its confidence, which you would not have known without measuring.

Everything was run on 2026-09-27 against laya 0.3.20 (released 2026-09-24), the newest release on PyPI at the time of writing. The project was first published on 2026-09-18 and shipped 28 releases between then and 2026-09-24, so pin the version this article uses. Versions used throughout: macOS 26.6.2 on an Apple M5 Max, Python 3.12.9, uv 0.11.26, torch 2.14.0, transformers 5.17.0, and pytest 9.1.1. Timings depend heavily on the chip; accuracy and calibration numbers should not.

What you end up with

• laya_triage/data.py: the 100 labeled customer-service test cases (and 300 training cases) from a public benchmark, loaded as Python objects.
• laya_triage/model.py and laya_triage/first_call.py: one checkpoint loaded onto the GPU, and the raw answers for one ticket.
• laya_triage/evaluate.py and laya_triage/report.py: accuracy per question for two checkpoints on two devices, next to a baseline that ignores the input, with p50/p90 latency and a count of answers the GPU and CPU disagree on.
• laya_triage/metrics.py and laya_triage/calibrate.py: temperatures and an act-or-escalate threshold fitted on half the cases and checked on the other half, both ways round.
• laya_triage/triage.py: the triage script, run on two new tickets.
• A pytest suite for the calibration arithmetic that needs no model, and a Makefile whose default target prints help.

Prerequisites

• macOS on Apple Silicon. The GPU path uses MPS (Metal Performance Shaders), PyTorch’s backend for the Apple GPU. Any M-series Mac works; expect different timings from the ones printed here.
• About 3 GB of free disk. The virtual environment is 921 MB, almost all of it PyTorch, and each Laya checkpoint is an 843 MB weights file. You will download two.
• uv 0.11 or newer. uv installs Python 3.12 for the project if you do not have it.
• Network access to Hugging Face on first run, for the checkpoints and the benchmark data. No account or token is needed. Many runs print a warning about unauthenticated requests to the Hugging Face Hub, which you can ignore; the article’s output listings leave it out, along with download progress bars. After the first run, prefixing a command with HF_HUB_OFFLINE=1 silences it.
• Familiarity with Python and the command line. You do not need to know how the model is built; the parts that matter for using it are explained where they come up.

Step 1: Create the project

The project is a small Python package, laya_triage, with each script run as a module (uv run python -m laya_triage.<name>) so they can import each other without an install step. The .gitignore comes first so that nothing the later steps download or write lands in version control: results/ will hold the evaluation output, which you regenerate rather than commit.

Create the file

mkdir -p laya-triage && cd laya-triage
touch .gitignore

Add the code: .gitignore

# Python
__pycache__/
*.py[cod]
.venv/
.pytest_cache/

# Downloaded evaluation data and results
data/
results/

# OS / editor
.DS_Store
*.log
tmp/

Detailed breakdown

• .venv/ holds PyTorch and the rest of the environment, close to a gigabyte.
• results/ holds the JSON the evaluation writes. It is derived data, so it is ignored along with data/, which is reserved for any labeled data of your own.
• The checkpoints do not appear here because they never enter the project: huggingface_hub caches them under ~/.cache/huggingface/, and make clean in Step 8 leaves that cache alone so you do not download 1.7 GB twice.

Now create the project and add its dependencies. --vcs none stops uv init from creating a Git repository and a second .gitignore, and the rm removes the placeholder main.py it writes:

uv init --name laya-triage --python 3.12 --no-readme --vcs none \
  --description "Support-ticket triage with Laya, with calibration you measure yourself" .
rm main.py
uv add "laya==0.3.20" pandas pyarrow huggingface_hub
uv add --dev pytest
mkdir -p laya_triage tests tickets
touch laya_triage/__init__.py

The version pin matters more than usual. Laya went from 0.3.3 to 0.3.20 in a week, and the GitHub repository already has merged changes that are not in any release. pandas and pyarrow read the benchmark’s Parquet files, and huggingface_hub downloads them.

Confirm the install without loading a model:

uv run python -c "import laya, torch; print(laya.__version__, torch.__version__, torch.backends.mps.is_available())"

0.3.20 2.14.0 True

True means PyTorch can see the Apple GPU. If it prints False, the rest of the article still runs, only on the CPU and much more slowly.

Step 2: Load labeled cases from the benchmark

Measuring accuracy needs questions whose right answers are already known, asked about text Laya has not seen. The typed-decisions benchmark (LocalLLaMA/typed-decisions, Apache 2.0) provides exactly that for four workflows, separate task domains that each ask their own set of questions. This article uses the customer-service one: 300 training cases and 100 test cases, each a support thread plus account details, with the same five questions asked of every case.

The right answers need one caveat before you trust any score built on them. The benchmark calls each one the gold label. It is not human ground truth. It is the answer a teacher model (a separate model, which the benchmark describes as of “roughly 4B-class capability”) gave most weight across three samples, so accuracy here means agreement with that teacher. The benchmark’s documentation puts a fresh teacher sample’s agreement with those labels at 0.735 across all four workflows, and warns that a model scoring much higher has learned the teacher’s quirks rather than the task.

The five questions

Every case asks the same five questions. Each declares a type, which Step 3 explains in full: choice picks one option, score places the ticket on an ordered scale whose levels are numbered from 0, and noul is the probability that a statement is true.

Each option and level carries a sentence of description in the data, which Laya reads along with the label. The action question has nothing to do with the action field that appears inside every answer in Step 3.

Create the file

touch laya_triage/data.py

Add the code: laya_triage/data.py

"""Load labeled customer-service cases from the typed-decisions benchmark."""

import json
from dataclasses import dataclass

import pandas as pd
from huggingface_hub import hf_hub_download

DATASET = "LocalLLaMA/typed-decisions"
WORKFLOW = "customer_service"


@dataclass
class Case:
    id: str
    state: dict
    questions: dict
    gold: dict[str, str]  # question name -> the label the benchmark counts as correct


def load_cases(split: str = "test") -> list[Case]:
    path = hf_hub_download(
        DATASET,
        f"{WORKFLOW}/{split}-00000-of-00001.parquet",
        repo_type="dataset",
    )
    frame = pd.read_parquet(path).sort_values("id")
    cases = []
    for _, row in frame.iterrows():
        questions = json.loads(row["questions"])
        gold = {name: str(row[f"{name}__label"]) for name in questions}
        cases.append(Case(row["id"], json.loads(row["state"]), questions, gold))
    return cases


def question_schema() -> dict:
    """The five customer-service questions, exactly as the benchmark asks them."""
    return load_cases("test")[0].questions

Detailed breakdown

• hf_hub_download fetches one Parquet file per split and caches it, so only the first call downloads anything. Later calls still check the Hub, which is where the warning in Prerequisites comes from.
• state and questions are stored in the dataset as JSON strings. Together they are exactly what Laya’s predict() takes, so a case can be replayed without reshaping.
• The gold label for each question lives in a column named <question>__label. The score questions store their gold level as a string such as "2", which matches the string keys Laya returns its levels under; str() guards against a copy of the data that stores them as numbers.
• Sorting by id fixes the case order. Step 6 splits the cases into alternating halves, and a stable order makes that split the same on every machine.
• question_schema() returns the five questions from the first test case. Every one of the 400 cases asks the identical five, so the triage script in Step 7 can ask new tickets the same questions the evaluation measured.

Step 3: Ask the five questions about one ticket

Before measuring anything, look at what one answer contains, because two of the fields that look most useful are not. You need a ticket and a way to load a checkpoint.

Convai ships three checkpoints in one Hugging Face repository, convaiinnovations/laya: the base English model at the root (421M parameters, built on the ModernBERT-large encoder), a multilingual one, and laya-typed-decisions, the English model fine-tuned on the training split of the benchmark from Step 2. This article compares the base model and the fine-tuned one, and ignores the multilingual checkpoint because the tickets are English.

Create the file

touch laya_triage/model.py

Add the code: laya_triage/model.py

"""Load a Laya checkpoint, and read its answers as probability distributions."""

import warnings

import laya

REPO = "convaiinnovations/laya"
# The base English checkpoint sits at the repo root; the fine-tuned one is a subfolder.
SUBFOLDERS = {"english": None, "typed-decisions": "typed-decisions"}


def load_checkpoint(name: str = "typed-decisions", device: str | None = None) -> laya.Agent:
    # Both checkpoints warn on load that one shipped temperature is out of range
    # and has been clamped. We fit our own, so the warning adds nothing here.
    warnings.filterwarnings("ignore", message="laya: this checkpoint ships")
    return laya.load(REPO, subfolder=SUBFOLDERS[name], device=device)


def distribution(answer: dict) -> dict[str, float]:
    """Any Laya answer as {label: probability}; a noul answer becomes true/false."""
    if answer["type"] == "noul":
        return {"true": answer["noul"], "false": 1.0 - answer["noul"]}
    return dict(answer["probabilities"])


def top(probs: dict[str, float]) -> tuple[str, float]:
    """The most probable label and its probability."""
    label = max(probs, key=probs.get)
    return label, probs[label]

Detailed breakdown

• laya.load() takes the repository and a subfolder; only the requested checkpoint’s files download. The base checkpoint is at the root, so its subfolder is None.
• device=None lets Laya choose. Its loader tries CUDA, then MPS, then Intel XPU, then the CPU, so on a Mac it picks the GPU without being told. The card’s own examples pass device="cuda", which on a Mac prints a warning and falls back to the CPU, not to MPS; leave it unset or pass "mps".
• Both checkpoints print a RuntimeWarning on load: one of the temperatures they ship (settings that sharpen or flatten the probabilities, explained in Step 6) is out of range and has been clamped. It applies only to choice questions with 11 or more options, which this article never asks, and Step 6 fits its own temperatures anyway, so the filter keeps the warning out of every output.
• distribution() gives every answer the same shape, {label: probability}. A noul answer (defined below) carries a single probability, so it becomes a two-label distribution, true and false.
• top() returns the most probable label. That is the answer the evaluation scores.

Next, a ticket. The state is a JSON object in the benchmark’s shape: account details and the conversation so far.

Create the file

touch tickets/double-charge.json tickets/vague.json

Add the code: tickets/double-charge.json

{
  "account": {"tier": "pro", "seats": 12, "tenure_months": 31, "lifetime_value_usd": 8640, "prior_tickets_90d": 2},
  "thread": [
    {"role": "customer", "text": "Our card was charged twice for the September invoice, $720 each time. I can see both charges on the statement."},
    {"role": "agent", "text": "Thanks for flagging this. Could you confirm the last four digits of the card?"},
    {"role": "customer", "text": "It ends in 4417. This is the second time this year. Refund the duplicate today or we will move the whole team to another vendor."}
  ]
}

Add the code: tickets/vague.json

{
  "account": {"tier": "free", "seats": 1, "tenure_months": 2, "lifetime_value_usd": 0, "prior_tickets_90d": 0},
  "thread": [
    {"role": "customer", "text": "hey, something seems off with my account since last week. not sure what changed."}
  ]
}

Detailed breakdown

• The two tickets are chosen to sit at opposite ends. The first names the problem, the amount, the history and a threat to leave. The second says almost nothing, so the questions about it have no clear answer.
• Neither ticket appears in the benchmark. They are the “new tickets” Step 7 triages with numbers fitted on the benchmark’s cases.

Now the script that asks the questions.

Create the file

touch laya_triage/first_call.py

Add the code: laya_triage/first_call.py

"""Ask Laya the five customer-service questions about one ticket and print what comes back."""

import json
import sys
import time
from pathlib import Path

from laya_triage.data import question_schema
from laya_triage.model import load_checkpoint


def main() -> None:
    ticket = Path(sys.argv[1] if len(sys.argv) > 1 else "tickets/double-charge.json")
    questions = question_schema()

    start = time.perf_counter()
    agent = load_checkpoint("typed-decisions")  # no device given: Laya picks one
    print(f"loaded on {agent.device} in {time.perf_counter() - start:.1f} s")

    result = agent.predict(json.loads(ticket.read_text()), questions)
    for name in ("category", "urgency", "needs_human"):
        print(f"\n{name} ({questions[name]['type']}):")
        print(json.dumps(result["answers"][name], indent=2))


if __name__ == "__main__":
    main()

Detailed breakdown

• The five questions come from the benchmark, so this call is shaped exactly like every call the evaluation makes.
• All five are answered in a single predict() call. Laya encodes each question together with the state as one row of a batch and runs every row in one forward pass of the encoder.
• The script prints three answers, one of each question type, which is enough to see every field Laya returns.

Run it. The first run downloads the fine-tuned checkpoint and the benchmark file:

uv run python -m laya_triage.first_call

loaded on mps in 1.2 s

category (choice):
{
  "type": "choice",
  "choice": "billing",
  "probabilities": {
    "account": 0.0893,
    "billing": 0.534,
    "delivery": 0.0291,
    "refund": 0.2826,
    "technical": 0.0651
  },
  "confidence": 0.2615,
  "answer_confidence": 0.534,
  "action": {
    "act_probability": 1.0
  }
}

urgency (score):
{
  "type": "score",
  "score": 2.672,
  "legend": {
    "0": "No time pressure; can wait indefinitely.",
    "1": "Routine; handle within the normal queue.",
    "2": "Elevated; should be handled within the same week.",
    "3": "Critical; requires action within the same day."
  },
  "probabilities": {
    "0": 0.0091,
    "1": 0.0574,
    "2": 0.1859,
    "3": 0.7476
  },
  "confidence": 0.4683,
  "answer_confidence": 0.7476,
  "action": {
    "act_probability": 1.0
  }
}

needs_human (noul):
{
  "type": "noul",
  "noul": 0.6333,
  "confidence": 0.6333,
  "answer_confidence": 0.6333,
  "action": {
    "act_probability": 1.0
  }
}

The load time is from a warm cache; the first run adds the download. Your probabilities may differ in the last digit or two, for a reason Step 5 measures.

Reading the three answer types

Laya’s three question types, which the rest of the article uses by name:

• choice picks one option from a list you define. choice holds the most probable option and probabilities the full distribution.
• score places the state on an ordered scale. The levels are numbered from 0 in the order you listed them, and legend echoes their descriptions. score is the expected level, the probability-weighted average (2.672 here), so it can fall between levels. The most probable level is 3.
• noul is the probability that a statement is true. noul: 0.6333 means Laya puts a 63% chance on “this conversation requires a human agent”.

Two fields that mislead

Both look like the number to gate a decision on:

• confidence is not the probability of the answer. For choice and score it is 1 minus the normalised entropy of the distribution, a measure of how concentrated the distribution is. billing has probability 0.534 and confidence 0.2615. The probability of the answer is in answer_confidence, which is the number the rest of this article uses, because it is a probability you can calibrate and compare across question types. The README on Laya’s GitHub main branch advises the same; the model card’s note on the next field suggests confidence, which can rank answers but is not a probability.
• action.act_probability is 1.0 on all three, and in practice on almost every input. Convai’s card lists this as a known issue (#185): the field carries no usable signal yet. Do not build an act-or-escalate rule on it; Step 6 builds one on answer_confidence instead.

Step 4: Score both checkpoints on the GPU and the CPU

One ticket shows what an answer looks like, not whether it is right. This step runs all 100 test cases through a checkpoint on a chosen device and saves every distribution Laya returned, along with the gold label and the time the call took. It runs four times: two checkpoints, each on MPS and on the CPU.

Create the file

touch laya_triage/evaluate.py

Add the code: laya_triage/evaluate.py

"""Run one Laya checkpoint over every labeled test case and save what it answered."""

import argparse
import json
import time
from pathlib import Path

from laya_triage.data import load_cases
from laya_triage.model import distribution, load_checkpoint

RESULTS = Path("results")


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--model", default="typed-decisions",
                        choices=["english", "typed-decisions"])
    parser.add_argument("--device", default="mps", choices=["mps", "cpu"])
    args = parser.parse_args()

    cases = load_cases("test")
    agent = load_checkpoint(args.model, args.device)

    # The first calls on a GPU pay one-off setup costs; keep them out of the timings.
    for case in cases[:3]:
        agent.predict(case.state, case.questions)

    records = []
    for case in cases:
        start = time.perf_counter()
        result = agent.predict(case.state, case.questions)
        elapsed_ms = (time.perf_counter() - start) * 1000
        records.append({
            "id": case.id,
            "ms": round(elapsed_ms, 2),
            "input_tokens": result["usage"]["input_tokens"],
            "answers": {
                name: {
                    "type": answer["type"],
                    "probabilities": distribution(answer),
                    "gold": case.gold[name],
                }
                for name, answer in result["answers"].items()
            },
        })

    RESULTS.mkdir(exist_ok=True)
    out = RESULTS / f"{args.model}-{args.device}.json"
    out.write_text(json.dumps(records, indent=1))
    print(f"{len(records)} cases, {len(records) * len(cases[0].questions)} answers -> {out}")


if __name__ == "__main__":
    main()

Detailed breakdown

• The three untimed warm-up calls matter on MPS. The first GPU calls pay setup costs that a long-running service pays only once; timing them would inflate the results.
• Each case is timed as one predict() call with all five questions, which is how a triage service would use it: one ticket arrives, five answers go out.
• input_tokens is Laya’s own count of the tokens (the sub-word pieces a model reads text as) it read, summed over every row. Each row carries the whole state plus one question and its options, so the state is counted five times: about 1,700 tokens for a ticket whose rows average roughly 350.
• The file stores each distribution, not only the winning label. Step 6 needs the full distributions to refit the probabilities without re-running the model.

Run the four evaluations. The CPU runs take about two minutes each:

uv run python -m laya_triage.evaluate --model english --device mps
uv run python -m laya_triage.evaluate --model english --device cpu
uv run python -m laya_triage.evaluate --model typed-decisions --device mps
uv run python -m laya_triage.evaluate --model typed-decisions --device cpu

100 cases, 500 answers -> results/english-mps.json
100 cases, 500 answers -> results/english-cpu.json
100 cases, 500 answers -> results/typed-decisions-mps.json
100 cases, 500 answers -> results/typed-decisions-cpu.json

The first english run also downloads the base checkpoint.

Step 5: Read accuracy, latency, and device agreement

Four result files still need reading. This step turns them into one table, and puts a baseline next to it that every checkpoint has to beat to be worth running: the majority class, which ignores the ticket and always answers each question’s most common label from the training split. A model that loses to it does worse, overall, than not reading the text at all.

Create the file

touch laya_triage/report.py

Add the code: laya_triage/report.py

"""Compare saved evaluation runs: accuracy per question, latency, and device agreement."""

import json
import statistics
from collections import Counter
from pathlib import Path

from laya_triage.data import load_cases
from laya_triage.model import top

RESULTS = Path("results")


def majority_baseline(names: list[str]) -> dict[str, float]:
    """Accuracy on test of always answering each question's commonest train label."""
    train, test = load_cases("train"), load_cases("test")
    scores = {}
    for name in names:
        commonest = Counter(c.gold[name] for c in train).most_common(1)[0][0]
        scores[name] = sum(c.gold[name] == commonest for c in test) / len(test)
    return scores


def per_question_accuracy(records: list[dict]) -> dict[str, float]:
    names = list(records[0]["answers"])
    return {
        n: sum(top(r["answers"][n]["probabilities"])[0] == r["answers"][n]["gold"]
               for r in records) / len(records)
        for n in names
    }


def main() -> None:
    # Evaluation files are named <model>-<device>.json; calibration.json has no hyphen.
    runs = {p.stem: json.loads(p.read_text()) for p in sorted(RESULTS.glob("*-*.json"))}
    if not runs:
        raise SystemExit("no results yet: run laya_triage.evaluate first")
    names = list(next(iter(runs.values()))[0]["answers"])

    rows = {"majority class": majority_baseline(names)}
    rows.update({run: per_question_accuracy(recs) for run, recs in runs.items()})

    print(f"{'run':<22}" + "".join(f"{n:>13}" for n in names) + f"{'overall':>10}")
    for run, acc in rows.items():
        overall = sum(acc.values()) / len(acc)
        print(f"{run:<22}" + "".join(f"{acc[n]:>13.2f}" for n in names) + f"{overall:>10.3f}")

    print()
    for run, recs in runs.items():
        ms = sorted(r["ms"] for r in recs)
        tokens = statistics.mean(r["input_tokens"] for r in recs)
        print(f"{run:<22} p50 {statistics.median(ms):7.1f} ms   p90 {ms[int(0.9 * len(ms)) - 1]:7.1f} ms"
              f"   mean input {tokens:.0f} tokens")

    for model in ("english", "typed-decisions"):
        mps, cpu = runs.get(f"{model}-mps"), runs.get(f"{model}-cpu")
        if mps and cpu:
            cpu_by_id = {r["id"]: r for r in cpu}
            flips = sum(top(a["answers"][n]["probabilities"])[0]
                        != top(cpu_by_id[a["id"]]["answers"][n]["probabilities"])[0]
                        for a in mps for n in names)
            print(f"{model}: MPS and CPU pick different answers on {flips} of {len(mps) * len(names)}")


if __name__ == "__main__":
    main()

Detailed breakdown

• majority_baseline() takes its labels from the training split and scores them on the test split. Taking the commonest label from the test split itself would let the baseline peek at the answers.
• per_question_accuracy() compares the most probable label with the gold label. For a score question that means the most probable level, not the rounded expected score.
• p50 and p90 are the median and 90th-percentile call times: half the calls finished within the p50 figure, nine in ten within the p90.
• overall is the plain mean of the five per-question accuracies.
• The last block pairs each MPS answer with the CPU answer for the same case id and counts the pairs that chose a different label.
• The glob matches <model>-<device>.json only, so it skips the calibration.json that Step 6 writes to the same folder.

uv run python -m laya_triage.report

run                          action     category   churn_risk  needs_human      urgency   overall
majority class                 0.52         0.18         0.47         0.67         0.42     0.452
english-cpu                    0.05         0.48         0.46         0.67         0.25     0.382
english-mps                    0.05         0.49         0.46         0.67         0.25     0.384
typed-decisions-cpu            0.58         0.94         0.80         0.76         0.74     0.764
typed-decisions-mps            0.58         0.94         0.81         0.76         0.74     0.766

english-cpu            p50   942.8 ms   p90  1314.7 ms   mean input 1723 tokens
english-mps            p50    59.0 ms   p90    73.7 ms   mean input 1723 tokens
typed-decisions-cpu    p50   949.0 ms   p90  1312.0 ms   mean input 1738 tokens
typed-decisions-mps    p50    59.2 ms   p90    75.6 ms   mean input 1738 tokens
english: MPS and CPU pick different answers on 1 of 500
typed-decisions: MPS and CPU pick different answers on 1 of 500

The accuracy columns should match yours to the digit. The millisecond columns are from an M5 Max and will not; rerun the report on your own Mac and use your numbers.

The base checkpoint loses to a fixed answer

The base english checkpoint scores 0.382 overall, below the 0.452 of a rule that never reads the ticket. The action column shows why: it answers close_no_action on 87 of the 100 cases, a label the teacher never chose for any of them, and scores 0.05. This is the weakness Convai’s card states first: “Base checkpoints are near chance on typed-decisions zero-shot”. Zero-shot means answering questions it was never trained on. The base model is a starting point for fine-tuning, not a decision engine.

The fine-tuned checkpoint scores 0.764 on the CPU, the same as the 0.764 Convai publishes for this workflow, and wins every column against the baseline. Treat that as a check that your setup matches theirs rather than as independent evidence: laya-typed-decisions was trained on this benchmark’s training split, and these test cases come from the same generator. Its score is also above the teacher’s 0.735 from Step 2, which the benchmark reads as partly learning the teacher’s quirks. It measures how well the checkpoint learned this benchmark, which is what its card says it is for.

The GPU is about 16 times faster, and slightly different

On this Mac, MPS answers a five-question ticket in a median of 59 ms against 949 ms on the CPU. Convai’s headline 33 ms is for a single question on an NVIDIA Tesla T4; these are five questions and about 1,700 input tokens per call, so the two figures measure different work. The card’s one Apple datapoint, “about 1.7 s” for a 4,000-token input on the multilingual checkpoint, is a different workload again.

The disagreements come from precision, not randomness. On MPS, Laya switches to 16-bit floating point (fp16 autocast) for any call with at least five question rows, because that is faster there, and the CPU runs in 32-bit. Five questions per ticket is exactly that minimum. To see the trade-off, raise the minimum with the environment variable Laya reads for it, rerun the fine-tuned checkpoint’s MPS evaluation, and report again (the excerpt below shows the lines that change):

LAYA_MPS_AMP_MIN_ROWS=6 uv run python -m laya_triage.evaluate --model typed-decisions --device mps
uv run python -m laya_triage.report

typed-decisions-mps    p50   129.1 ms   p90   173.5 ms   mean input 1738 tokens
english: MPS and CPU pick different answers on 1 of 500
typed-decisions: MPS and CPU pick different answers on 0 of 500

In 32-bit, the GPU and CPU agree on all 500 of the fine-tuned checkpoint’s answers, and the median call slows from 59 ms to 129 ms. Many services would accept one changed answer in 500 for twice the speed, but it means a threshold should be fitted on output from the device and precision you will serve with. Rerun the normal evaluation before moving on, so the next step fits on the fp16 numbers the triage script will see:

uv run python -m laya_triage.evaluate --model typed-decisions --device mps

Step 6: Fit the probabilities and an act-or-escalate threshold

A triage script needs a rule of the form “act on this answer if its probability is at least t”. For t to mean anything, the probabilities have to. A model is calibrated when its stated probabilities match how often it is right: of all the answers it gives at 0.8, about 80% should be correct.

Convai trains Laya with what it calls RLCD, reinforcement learning against strictly proper scoring rules: reward functions under which reporting your true belief is the best strategy. That is the basis for the card calling the probabilities calibrated. The card also says the checkpoints “ship over-confident” and recommends refitting on your own data. This step does that, and measures the result.

Measuring and fixing calibration

ECE (expected calibration error) turns calibration into one number. Group the answers into ten bins by stated probability (0.0 to 0.1, 0.1 to 0.2, and so on), take the gap between each bin’s average probability and its accuracy, and average the gaps weighted by how many answers each bin holds. 0 is perfect.

Temperature scaling is the standard fix. Raise every probability to the power 1/T and renormalise. A temperature above 1 flattens the distribution, which corrects over-confidence. A temperature below 1 sharpens it, which corrects under-confidence. The top answer never changes, only how sure the model claims to be. One temperature is fitted per question type, choosing the T that gives the gold labels the highest probability on average (the lowest log loss).

All of this is arithmetic on saved distributions, so it lives in its own module that never imports Laya.

Create the file

touch laya_triage/metrics.py

Add the code: laya_triage/metrics.py

"""Calibration and threshold arithmetic. Pure functions: no model, no Laya import."""

import math

# Candidate temperatures: 0.05 to 5.00 in steps of 0.05.
TEMPERATURES = [round(0.05 * i, 2) for i in range(1, 101)]


def rescale(probs: dict[str, float], temperature: float) -> dict[str, float]:
    """Temperature scaling: p ** (1/T), renormalised. T > 1 softens, T < 1 sharpens."""
    floor = 1e-9  # a probability of exactly 0 would make the power undefined
    powered = {k: max(v, floor) ** (1.0 / temperature) for k, v in probs.items()}
    total = sum(powered.values())
    return {k: v / total for k, v in powered.items()}


def log_loss(answers: list[dict], temperature: float) -> float:
    """Mean negative log probability given to the gold label."""
    total = 0.0
    for d in answers:
        p = rescale(d["probabilities"], temperature).get(d["gold"], 0.0)
        total -= math.log(max(p, 1e-9))
    return total / len(answers)


def fit_temperature(answers: list[dict]) -> float:
    """The temperature that minimises log loss on these answers."""
    return min(TEMPERATURES, key=lambda t: log_loss(answers, t))


def confidence_and_correct(answers: list[dict],
                           temperature: float = 1.0) -> list[tuple[float, bool]]:
    pairs = []
    for d in answers:
        scaled = rescale(d["probabilities"], temperature)
        label = max(scaled, key=scaled.get)
        pairs.append((scaled[label], label == d["gold"]))
    return pairs


def accuracy(pairs: list[tuple[float, bool]]) -> float:
    return sum(ok for _, ok in pairs) / len(pairs)


def expected_calibration_error(pairs: list[tuple[float, bool]], bins: int = 10) -> float:
    """Average gap between stated confidence and observed accuracy, over 10 equal-width bins."""
    gap = 0.0
    for b in range(bins):
        lo, hi = b / bins, (b + 1) / bins
        in_bin = [(c, ok) for c, ok in pairs if lo < c <= hi or (b == 0 and c == 0)]
        if in_bin:
            mean_conf = sum(c for c, _ in in_bin) / len(in_bin)
            gap += len(in_bin) / len(pairs) * abs(accuracy(in_bin) - mean_conf)
    return gap


def pick_threshold(pairs: list[tuple[float, bool]], target: float) -> float | None:
    """Lowest confidence cut-off whose accepted answers are at least `target` accurate."""
    for cut in sorted({c for c, _ in pairs}):
        accepted = [(c, ok) for c, ok in pairs if c >= cut]
        if accuracy(accepted) >= target:
            return cut
    return None

Detailed breakdown

• rescale() works on probabilities rather than on the model’s internal scores. Raising probabilities to 1/T and renormalising gives the same result as dividing those scores by T, so no model access is needed. The floor stops a probability of exactly 0 from breaking the power.
• fit_temperature() is a grid search over 100 values from 0.05 to 5.00. With a single parameter and a few hundred answers, a grid search is fast and simple.
• Each item in answers is one saved answer from Step 4: its type, its distribution, and the gold label.
• confidence_and_correct() produces the pairs everything else consumes: for each answer, the probability of the top label after scaling, and whether that label was right.
• expected_calibration_error() bins on (lo, hi], with an exact 0 counted in the first bin so no pair is dropped.
• pick_threshold() walks the cut-offs from lowest to highest and returns the first whose accepted answers reach the target accuracy. The lowest such cut-off acts on the most answers. It returns None when even the most confident answers miss the target.

Testing the arithmetic without a model

The functions above decide which answers a human sees, so they get tests. None of the tests loads a checkpoint, so the suite runs in milliseconds.

Create the file

touch tests/test_metrics.py

Add the code: tests/test_metrics.py

import pytest

from laya_triage.metrics import (
    expected_calibration_error,
    fit_temperature,
    pick_threshold,
    rescale,
)


def test_temperature_one_changes_nothing():
    probs = {"a": 0.6, "b": 0.3, "c": 0.1}
    assert rescale(probs, 1.0) == pytest.approx(probs)


def test_temperature_sharpens_or_softens_but_keeps_the_answer():
    probs = {"a": 0.6, "b": 0.3, "c": 0.1}
    sharper, softer = rescale(probs, 0.5), rescale(probs, 2.0)
    assert max(sharper, key=sharper.get) == max(softer, key=softer.get) == "a"
    assert sharper["a"] > 0.6 > softer["a"]
    assert sum(sharper.values()) == pytest.approx(1.0)


def test_underconfident_model_gets_a_temperature_below_one():
    # Always right, but only 60% sure: the fit should sharpen it.
    answers = [{"probabilities": {"a": 0.6, "b": 0.4}, "gold": "a"}] * 10
    assert fit_temperature(answers) < 1.0


def test_overconfident_model_gets_a_temperature_above_one():
    # 90% sure, right half the time: the fit should soften it.
    answers = ([{"probabilities": {"a": 0.9, "b": 0.1}, "gold": "a"}] * 5
                 + [{"probabilities": {"a": 0.9, "b": 0.1}, "gold": "b"}] * 5)
    assert fit_temperature(answers) > 1.0


def test_fit_finds_a_temperature_inside_the_grid():
    # 90% sure, right 80% of the time: the best T sets p(a) to 0.8, which is
    # 0.9 ** (1/T) / (0.9 ** (1/T) + 0.1 ** (1/T)) = 0.8, so T = ln 9 / ln 4, about 1.58.
    answers = ([{"probabilities": {"a": 0.9, "b": 0.1}, "gold": "a"}] * 8
               + [{"probabilities": {"a": 0.9, "b": 0.1}, "gold": "b"}] * 2)
    assert fit_temperature(answers) == pytest.approx(1.6)


def test_ece_is_zero_when_confidence_matches_accuracy():
    pairs = [(0.75, True), (0.75, True), (0.75, True), (0.75, False)]
    assert expected_calibration_error(pairs) == pytest.approx(0.0)


def test_ece_is_the_gap_when_confidently_wrong():
    assert expected_calibration_error([(0.9, False)] * 4) == pytest.approx(0.9)


def test_threshold_is_the_lowest_cut_that_meets_the_target():
    pairs = [(0.9, True), (0.8, True), (0.7, False), (0.6, True)]
    assert pick_threshold(pairs, 1.0) == 0.8
    assert pick_threshold(pairs, 0.75) == 0.6


def test_no_threshold_when_the_target_is_out_of_reach():
    assert pick_threshold([(0.5, False), (0.9, False)], 0.9) is None

Detailed breakdown

• The two fit_temperature tests pin the direction of the fit: an answer that is always right at 60% should be sharpened (T < 1), and one that is right half the time at 90% should be softened (T > 1). The calibration output below is read in exactly those terms.
• The ECE tests cover both ends: a bin whose confidence equals its accuracy contributes nothing, and a confidently wrong bin contributes its whole confidence.
• The interior test guards the search itself. The two direction tests are satisfied by the grid’s end values (0.05 and 5.00), so a broken search that always returned an end would pass them; this one has a known optimum in the middle, worked out in its comment.
• The threshold tests check that the lowest qualifying cut-off wins, and that an unreachable target returns None rather than a threshold that does not work.

Tell pytest where the package and tests are by appending to pyproject.toml:

cat >> pyproject.toml <<'EOF'

[tool.pytest.ini_options]
pythonpath = ["."]
testpaths = ["tests"]
EOF
uv run pytest -q

.........                                                                [100%]
9 passed in 0.01s

Fitting on one half and checking on the other

Fitting temperatures and a threshold on the same cases you then score them on would overstate how well they hold up. So the script splits the 100 test cases into even- and odd-numbered halves, fits on one, checks on the other (the held-out half), and then swaps them. Each fit-and-check is called a fold. If the two folds agree, the fitted numbers are probably stable. If they disagree, you have too little labeled data to trust them.

The training split is not used for this. laya-typed-decisions was trained on it, and Convai’s card notes that the per-type temperatures shipped with this checkpoint were fitted on training data (#186). The same mistake here would make the probabilities look better calibrated than they are.

Create the file

touch laya_triage/calibrate.py

Add the code: laya_triage/calibrate.py

"""Fit one temperature per question type and an act/escalate threshold, and measure how well they hold up."""

import argparse
import json
from pathlib import Path

from laya_triage.metrics import (
    accuracy,
    confidence_and_correct,
    expected_calibration_error,
    fit_temperature,
    pick_threshold,
)

RESULTS = Path("results")
KINDS = ("choice", "noul", "score")


def flatten(records: list[dict]) -> list[dict]:
    return [answer for r in records for answer in r["answers"].values()]


def fit(answers: list[dict], target: float) -> tuple[dict[str, float], float | None]:
    temperatures = {k: fit_temperature([d for d in answers if d["type"] == k]) for k in KINDS}
    return temperatures, pick_threshold(scaled_pairs(answers, temperatures), target)


def scaled_pairs(answers: list[dict], temperatures: dict[str, float]):
    pairs = []
    for kind, t in temperatures.items():
        pairs += confidence_and_correct([d for d in answers if d["type"] == kind], t)
    return pairs


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--run", default="typed-decisions-mps")
    parser.add_argument("--target", type=float, default=0.90,
                        help="accuracy the automatically handled answers must reach")
    args = parser.parse_args()

    records = json.loads((RESULTS / f"{args.run}.json").read_text())
    even, odd = flatten(records[0::2]), flatten(records[1::2])

    # Fit on one half, judge on the other, then swap. Two folds that disagree
    # are the sign that there is not yet enough labeled data to trust a threshold.
    for name, fit_half, check_half in (("fold A", even, odd), ("fold B", odd, even)):
        temperatures, threshold = fit(fit_half, args.target)
        raw = scaled_pairs(check_half, dict.fromkeys(KINDS, 1.0))
        fitted = scaled_pairs(check_half, temperatures)
        acted = [p for p in fitted if threshold is not None and p[0] >= threshold]
        print(f"{name}: temperatures {temperatures}")
        print(f"        ECE on held-out half {expected_calibration_error(raw):.3f} raw"
              f" -> {expected_calibration_error(fitted):.3f} fitted")
        if acted:
            print(f"        threshold {threshold:.3f}: acts on {len(acted)}/{len(fitted)}"
                  f" ({len(acted) / len(fitted):.0%}) at {accuracy(acted):.3f} accuracy")
        else:
            print(f"        no threshold reached {args.target:.0%} on the fitting half")

    # The numbers the triage script uses come from all the labeled cases.
    temperatures, threshold = fit(even + odd, args.target)
    if threshold is None:
        raise SystemExit(f"no threshold reaches {args.target:.0%} on this data")
    out = RESULTS / "calibration.json"
    out.write_text(json.dumps({"run": args.run, "target": args.target,
                               "temperatures": temperatures, "threshold": threshold}, indent=1))
    print(f"\nall {len(records)} cases: temperatures {temperatures}, threshold {threshold:.3f} -> {out}")


if __name__ == "__main__":
    main()

Detailed breakdown

• flatten() pools every answer in a half. With five questions per case, each half holds 250 answers.
• fit() fits one temperature per question type (choice, noul, score), then picks a threshold on the scaled probabilities of the same half.
• Each fold prints the held-out half’s ECE before and after its temperatures, and what its threshold does there: how many answers it acts on, and how accurate those answers turn out to be.
• The final fit uses all 100 cases. The folds estimate how well that procedure transfers; the final numbers use every labeled case available, and are saved to results/calibration.json for the triage script.
• --target 0.90 asks for a threshold whose accepted answers are at least 90% accurate. It is a policy choice: raise it and more answers reach a human.

uv run python -m laya_triage.calibrate

fold A: temperatures {'choice': 0.3, 'noul': 0.2, 'score': 0.35}
        ECE on held-out half 0.165 raw -> 0.100 fitted
        threshold 0.714: acts on 179/250 (72%) at 0.793 accuracy
fold B: temperatures {'choice': 0.45, 'noul': 0.35, 'score': 0.4}
        ECE on held-out half 0.258 raw -> 0.085 fitted
        threshold 0.739: acts on 115/250 (46%) at 0.939 accuracy

all 100 cases: temperatures {'choice': 0.4, 'noul': 0.3, 'score': 0.4}, threshold 0.739 -> results/calibration.json

What the fit found

Every temperature is below 1. On this workflow the fine-tuned checkpoint is under-confident: it is right more often than it says. The card for this checkpoint describes it as “still over-confident”, with an ECE of 0.213. ECE measures the size of the gap between stated probability and accuracy, not its direction. Running the same checkpoint over each of the four workflows’ full test sets while preparing this article (a check outside the companion project) gave an ECE between 0.17 and 0.25 in each, in line with the card, and in every one the average stated probability (0.50 to 0.64) sat below the accuracy (0.73 to 0.81). The size of the miscalibration reproduces; its direction, on this measure, is the opposite of the card’s label.

The card itself points at a likely cause. The probabilities Laya returns have already been divided by temperatures shipped in the checkpoint’s config, and for these five questions those are 1.25 to 1.98: values the card says were “inherited from the base checkpoint” and override the ones fitted for this model. Temperatures that large flatten a distribution, which suits the over-confident base model; on the fine-tuned model they appear to flatten too much, and a fitted T below 1 takes some of that back. Here the fit cut the held-out ECE by 39% in fold A (0.165 to 0.100) and 67% in fold B (0.258 to 0.085).

The threshold did not transfer as well. Both folds asked for 90% accuracy. Fold B delivered 93.9% on its held-out half; fold A delivered 79.3%. pick_threshold() takes the lowest cut-off that reaches the target, so on its own half each fold lands at barely 90% with no margin, and 50 cases are too few for that to carry over. Fold A’s sharper temperatures also push more held-out answers over its line. Treat the 0.739 threshold as a starting point, and plan on labeling a few hundred of your own tickets before an automated action depends on it.

To see the same procedure on the base checkpoint, run uv run python -m laya_triage.calibrate --run english-mps. Its temperatures for choice and score come out above 1 (over-confident, as the card says), and its thresholds act on almost nothing. Rerun the default command afterwards so results/calibration.json holds the fine-tuned checkpoint’s numbers.

Step 7: Triage new tickets

The triage script loads the fine-tuned checkpoint on the GPU, asks the five questions about each ticket, rescales each answer with the fitted temperature for its type, and acts on it only if the rescaled probability clears the fitted threshold.

Create the file

touch laya_triage/triage.py

Add the code: laya_triage/triage.py

"""Triage support tickets locally: answer five typed questions, act on the confident answers, escalate the rest."""

import argparse
import json
import time
from pathlib import Path

from laya_triage.data import question_schema
from laya_triage.metrics import rescale
from laya_triage.model import distribution, load_checkpoint, top

CALIBRATION = Path("results/calibration.json")


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("tickets", nargs="+", type=Path)
    parser.add_argument("--device", default="mps", choices=["mps", "cpu"])
    args = parser.parse_args()

    if not CALIBRATION.exists():
        raise SystemExit("no calibration yet: run laya_triage.calibrate first")
    calibration = json.loads(CALIBRATION.read_text())
    expected = f"typed-decisions-{args.device}"
    if calibration["run"] != expected:
        raise SystemExit(f"calibration.json was fitted on {calibration['run']}, not {expected}: "
                         f"run laya_triage.calibrate --run {expected} first")
    temperatures, threshold = calibration["temperatures"], calibration["threshold"]

    questions = question_schema()
    agent = load_checkpoint("typed-decisions", args.device)
    agent.predict(json.loads(args.tickets[0].read_text()), questions)  # warm-up, untimed

    print(f"threshold {threshold:.3f} (fitted on {expected} for {calibration['target']:.0%} accuracy)")
    for path in args.tickets:
        state = json.loads(path.read_text())
        start = time.perf_counter()
        result = agent.predict(state, questions)
        elapsed_ms = (time.perf_counter() - start) * 1000

        print(f"\n{path.name}  ({elapsed_ms:.0f} ms)")
        for name, answer in result["answers"].items():
            raw = distribution(answer)
            label, p = top(rescale(raw, temperatures[answer["type"]]))
            if answer["type"] == "score":
                label = f"level {label}"  # an index into the question's legend, 0 = lowest
            verdict = "act" if p >= threshold else "ESCALATE"
            print(f"  {name:<12} {label:<20} raw {top(raw)[1]:.2f}"
                  f"  calibrated {p:.2f}  {verdict}")


if __name__ == "__main__":
    main()

Detailed breakdown

• The script exits if results/calibration.json is missing, rather than silently using Laya’s raw probabilities. It also exits if the file was fitted on a different run, for example after calibrate --run english-mps, or when you pass --device cpu to numbers fitted on MPS output.
• The questions come from question_schema(), the exact schema the temperatures and threshold were fitted on. A new question, or a reworded one, needs its own labeled cases and its own fit.
• The warm-up call is untimed, for the same reason as in Step 4.
• Each line prints the raw probability Laya returned and the calibrated one, so you can see what the fit changed.
• score labels print as level N, an index into that question’s legend, 0 being the lowest. For urgency, level 3 is “Critical; requires action within the same day”; for churn_risk, it is “Imminent: threatening to cancel, dispute or leave”.
• The decision is made per answer, not per ticket. A ticket can have a confident category and an uncertain next action; the routing is automated and the reply goes to a person.

uv run python -m laya_triage.triage tickets/double-charge.json tickets/vague.json

threshold 0.739 (fitted on typed-decisions-mps for 90% accuracy)

double-charge.json  (44 ms)
  action       escalate_to_human    raw 0.40  calibrated 0.64  ESCALATE
  category     billing              raw 0.53  calibrated 0.82  act
  churn_risk   level 3              raw 0.70  calibrated 0.93  act
  needs_human  true                 raw 0.63  calibrated 0.86  act
  urgency      level 3              raw 0.75  calibrated 0.97  act

vague.json  (36 ms)
  action       request_information  raw 0.47  calibrated 0.76  act
  category     account              raw 0.42  calibrated 0.59  ESCALATE
  churn_risk   level 1              raw 0.51  calibrated 0.79  act
  needs_human  false                raw 0.59  calibrated 0.78  act
  urgency      level 1              raw 0.51  calibrated 0.66  ESCALATE

Read the category rows side by side. The same question gets a confident answer on the double-charge ticket (billing, 0.82, act) and an uncertain one on the vague ticket (account, 0.59, escalate), which is the behaviour you want: the uncertainty is part of the output.

What the temperatures buy is a threshold that means what it says, not better decisions. Scaling never changes which label is on top and rarely changes how answers of one type rank by confidence, so a threshold fitted on raw probabilities makes nearly the same calls: rerunning the two folds on raw probabilities gives cut-offs of 0.470 and 0.549, and 79.4% and 94.2% accuracy on the held-out halves, against 79.3% and 93.9% calibrated. The difference is that 0.739 reads as “about 74% sure” and holds roughly true, where a raw cut-off of 0.52 is a number that only works because it was fitted.

Some acted-on answers are plausible but unverified. request_information is a sensible next step for “something seems off”, and the script acts on it at 0.76. Whether that is acceptable is the threshold question from Step 6, answered by your labeled data, not by this output.

Step 8: Wrap it in a Makefile

The steps above are now commands you will rerun whenever the model, the data, or the target changes. A Makefile records them in order, and running make with no target prints what each one does.

Create the file

touch Makefile

Add the code: Makefile

.DEFAULT_GOAL := help
.PHONY: help install first-call evaluate report calibrate triage test clean

help: ## Show this help
	@grep -E '^[a-z-]+:.*## ' $(MAKEFILE_LIST) | awk 'BEGIN {FS = ":.*## "}; {printf "  %-12s %s\n", $$1, $$2}'

install: ## Install dependencies into .venv
	uv sync

first-call: ## Ask the five questions about one ticket and print three raw answers
	uv run python -m laya_triage.first_call

evaluate: ## Score both checkpoints on MPS and CPU against the 100 labeled test cases
	uv run python -m laya_triage.evaluate --model english --device mps
	uv run python -m laya_triage.evaluate --model english --device cpu
	uv run python -m laya_triage.evaluate --model typed-decisions --device mps
	uv run python -m laya_triage.evaluate --model typed-decisions --device cpu

report: ## Print accuracy, latency and MPS/CPU agreement from saved results
	uv run python -m laya_triage.report

calibrate: ## Fit temperatures and the act/escalate threshold
	uv run python -m laya_triage.calibrate

triage: ## Triage the sample tickets with the fitted numbers
	uv run python -m laya_triage.triage tickets/double-charge.json tickets/vague.json

test: ## Run the unit tests
	uv run pytest -v

clean: ## Remove saved results and the test cache (keeps downloaded models)
	rm -rf results .pytest_cache

Detailed breakdown

• .DEFAULT_GOAL := help makes a bare make print the help screen. The help target reads the ## comments from the file, so each target’s help text sits on the same line as the target.
• evaluate runs all four evaluations in the order Step 4 did, so calibrate sees the fp16 MPS results.
• clean deletes results/ but not the Hugging Face cache, so a clean rebuild does not download 1.7 GB of checkpoints again.
• Makefile recipes must be indented with a tab, not spaces.

make

  help         Show this help
  install      Install dependencies into .venv
  first-call   Ask the five questions about one ticket and print three raw answers
  evaluate     Score both checkpoints on MPS and CPU against the 100 labeled test cases
  report       Print accuracy, latency and MPS/CPU agreement from saved results
  calibrate    Fit temperatures and the act/escalate threshold
  triage       Triage the sample tickets with the fitted numbers
  test         Run the unit tests
  clean        Remove saved results and the test cache (keeps downloaded models)

The full sequence from nothing is make clean evaluate report calibrate triage.

Troubleshooting

• laya.load() hangs. Convai’s card documents this: if TensorFlow is installed in the same environment, transformers imports it, and its runtime can deadlock model construction. Run with USE_TF=0 in front of the command.
• Everything runs on the CPU. Check that torch.backends.mps.is_available() prints True (Step 1). Passing device="cuda", as the card’s examples do, falls back to the CPU on a Mac, with a printed warning.
• The first call is much slower than later ones. That is the download on the first run and GPU setup on every run. Steps 4 and 7 exclude warm-up calls from their timings for this reason.
• no calibration yet: run laya_triage.calibrate first. triage.py needs results/calibration.json. Run Steps 4 and 6 (or make evaluate calibrate) first.
• A noul answer looks stuck on “no”. The card lists this as a known issue on the base English checkpoint (#156): the false: / true: option labels can outweigh the text. Its suggested workaround is to ask a two-option choice with neutral keys and the yes/no wording in the descriptions.
• A choice question with many options answers badly. Options share a fixed budget of tokens; the card says accuracy falls off above roughly 20 options. Keep option lists short or split them into two questions.

Recap

You installed Laya 0.3.20 into a pinned uv project and ran it on the Apple GPU, which Laya selects without being told. On 100 labeled customer-service tickets you measured that the base checkpoint scores below a baseline that ignores the input, while the fine-tuned laya-typed-decisions checkpoint scores 0.764 on the CPU (0.766 on MPS), matching Convai’s published 0.764 for this workflow. On an M5 Max a five-question ticket took a median of 59 ms on MPS against 949 ms on the CPU, and the GPU’s fp16 path changed one answer in 500.

You then tested the calibration claim rather than accepting it. On this workflow the fine-tuned checkpoint was under-confident, one temperature per question type cut the held-out calibration error by 39% and 67% in the two folds, and a threshold fitted for 90% accuracy delivered between 79% and 94% depending on which half it was fitted on. The triage script uses those fitted numbers to act on confident answers and escalate the rest, in about 40 ms per ticket.

For any model that claims calibrated confidence, the card’s numbers are a starting point: here the size of the miscalibration matched the card while its direction did not, and it differed between two checkpoints of the same model. Fit it on labeled examples of your own inputs, and check the fit on examples it did not see.

Where to go next:

• Your own labels. Replace load_cases() with a few hundred of your own tickets and answers, and rerun make evaluate calibrate. If your questions differ from the benchmark’s, the base checkpoint’s numbers in Step 5 are the honest estimate of zero-shot quality, and Convai’s fine-tuning notebook is the route to a checkpoint of your own.
• A service. uv add "laya[serve]" installs laya-serve, an HTTP server that answers POST /v1/systemone, the request shape of TypeSafe’s hosted Jev model. It binds 0.0.0.0 with no authentication unless LAYA_API_KEY is set.
• An MCP server. The laya[mcp] extra exposes the same predictions as tools for an agent.
• Other Apple runtimes. Laya’s README links a third-party port, laya-apple, that runs on MLX and the Apple Neural Engine. The report and calibration steps work unchanged on distributions saved from any runtime, which makes them a direct way to check a port’s accuracy as well as its speed.