Tool-calling support matrix

Calibration notes

Method and limitations

Each cell measures one full stack. The stack combines a model, quant, server, and server version. A cell does not describe the model alone.

A failed observation applies only to the tested combination. It does not mean the weights are bad. The same weights can pass on one server and fail on another. When the evidence proves that difference, the cell includes a cause annotation.

When the result includes a transcript path, a failing observation links to the full request and response transcript. Legacy schema v1 results do not record transcript paths. Read the case studies in docs/case-studies/ for controlled comparisons.

The servers use different decode methods. llama.cpp compiles the supplied tool definitions into a GBNF grammar. It uses that grammar to constrain decoding. Ollama and MLX LM generate unconstrained text. They parse the tool call after decoding. Cross-band differences therefore reflect the full stack. Compare adjacent models only when they use the same server. Each row states whether the run recorded its decode mode or the site read the mode from the cited preset mapping. An unmapped preset has an unknown mode.

The site includes 50 distinct scenarios. Each published cell represents one run. Hatching marks that the cell has no verdict.

The case studies draw a verdict only after at least five runs per arm. The current case studies cover 90 runs across 18 quantization arms, and 40 runs across 8 arms for the peg-native anomaly.

Excluded rows

The quantization conclusion excludes Meta-Llama-3.1-8B-Instruct on llama.cpp (Q8_0, Q4_K_M, Q3_K_M). For this model, llama.cpp returns HTTP 500 on 7-9 of 50 scenarios per run ("does not match the expected peg-native format"). These results are server errors. They are not model failures and cannot be compared across arms. Read the peg-native case study.

All measurements used Apple M4 Max, 64GB; macOS 26.5.2.