Evidence layer

Observed outcomes

These rasters show the scenario-level evidence behind the matrix. They keep the same row order as the matrix. They do not rank observations from a single run.

Each mark shows one scenario outcome for one published stack. The data files include scenario descriptions and rationales. They also contain status text and evidence paths. They provide citation details. The appendix includes recorded stack metadata and transcript links.

Scenario evidence

The complete raster shows all 50 scenarios. The focused raster shows the seven multi-turn scenarios. The signature inventory groups identical 50-outcome vectors. Figure captions link to the complete text alternative in JSON and CSV.

Figure 1. Scenario-status raster Rows show published stack runs. Columns show scenario ids grouped by capability. We omit the n=1 hatch because every raster row is a single run. The hatch would mark every cell and add no distinction. What this does not show: This figure does not rank stacks or assign an overall score. It does not predict how a stack will behave across repeated runs. Method and limitations - Data: JSON, CSV

Single call

Stacksingle-array-tagssingle-boolean-flagsingle-date-rangesingle-decimal-pricesingle-empty-argumentssingle-enum-formatsingle-integer-limitsingle-nested-optionssingle-optional-omittedsingle-optional-presentsingle-string-listsingle-weathersingle-zero-and-negative gemma3:12b | Q4_K_M | ollama gemma3:4b | Q4_K_M | ollama granite3.1-dense:8b | Q4_K_M | llama.cpp phi4-mini:latest | Q4_K_M | llama.cpp llama3-groq-tool-use:8b | Q4_0 | ollama hermes3:8b | Q4_0 | ollama Qwen/Qwen2.5-1.5B-Instruct | Q3_K_M | llama.cpp Qwen/Qwen2.5-1.5B-Instruct | Q4_K_M | llama.cpp Qwen/Qwen2.5-1.5B-Instruct | Q8_0 | llama.cpp Qwen/Qwen2.5-1.5B-Instruct | 4bit | MLX LM Qwen/Qwen2.5-7B-Instruct | Q3_K_M | llama.cpp Qwen/Qwen2.5-7B-Instruct | Q4_K_M | llama.cpp Qwen/Qwen2.5-7B-Instruct | Q8_0 | llama.cpp Qwen/Qwen2.5-7B-Instruct | 4bit | MLX LM Qwen/Qwen2.5-7B-Instruct | 8bit | MLX LM qwen2.5:7b-instruct | Q4_K_M | ollama qwen3:0.6b | Q4_K_M | ollama qwen3:1.7b | Q4_K_M | ollama qwen3:14b | Q4_K_M | ollama qwen3:4b | Q4_K_M | ollama qwen3:8b | Q4_K_M | ollama granite3.1-dense:8b | Q4_K_M | ollama llama3.1:8b | Q4_K_M | ollama meta-llama/Meta-Llama-3.1-8B-Instruct | Q3_K_M | llama.cpp meta-llama/Meta-Llama-3.1-8B-Instruct | Q4_K_M | llama.cpp meta-llama/Meta-Llama-3.1-8B-Instruct | Q8_0 | llama.cpp meta-llama/Meta-Llama-3.1-8B-Instruct | 4bit | MLX LM microsoft/Phi-4-mini-instruct | 4bit | MLX LM phi4-mini:latest | Q4_K_M | ollama mistral:7b | Q4_K_M | ollama mistralai/Mistral-7B-Instruct-v0.3 | 4bit | MLX LM watt-ai/watt-tool-8B | Q4_K_M | llama.cpp

Parallel calls

Stackparallel-city-timeparallel-different-toolsparallel-three-citiesparallel-three-different-toolsparallel-two-calculationsparallel-two-multiple-argsparallel-two-searchesparallel-weather-and-time gemma3:12b | Q4_K_M | ollama gemma3:4b | Q4_K_M | ollama granite3.1-dense:8b | Q4_K_M | llama.cpp phi4-mini:latest | Q4_K_M | llama.cpp llama3-groq-tool-use:8b | Q4_0 | ollama hermes3:8b | Q4_0 | ollama Qwen/Qwen2.5-1.5B-Instruct | Q3_K_M | llama.cpp Qwen/Qwen2.5-1.5B-Instruct | Q4_K_M | llama.cpp Qwen/Qwen2.5-1.5B-Instruct | Q8_0 | llama.cpp Qwen/Qwen2.5-1.5B-Instruct | 4bit | MLX LM Qwen/Qwen2.5-7B-Instruct | Q3_K_M | llama.cpp Qwen/Qwen2.5-7B-Instruct | Q4_K_M | llama.cpp Qwen/Qwen2.5-7B-Instruct | Q8_0 | llama.cpp Qwen/Qwen2.5-7B-Instruct | 4bit | MLX LM Qwen/Qwen2.5-7B-Instruct | 8bit | MLX LM qwen2.5:7b-instruct | Q4_K_M | ollama qwen3:0.6b | Q4_K_M | ollama qwen3:1.7b | Q4_K_M | ollama qwen3:14b | Q4_K_M | ollama qwen3:4b | Q4_K_M | ollama qwen3:8b | Q4_K_M | ollama granite3.1-dense:8b | Q4_K_M | ollama llama3.1:8b | Q4_K_M | ollama meta-llama/Meta-Llama-3.1-8B-Instruct | Q3_K_M | llama.cpp meta-llama/Meta-Llama-3.1-8B-Instruct | Q4_K_M | llama.cpp meta-llama/Meta-Llama-3.1-8B-Instruct | Q8_0 | llama.cpp meta-llama/Meta-Llama-3.1-8B-Instruct | 4bit | MLX LM microsoft/Phi-4-mini-instruct | 4bit | MLX LM phi4-mini:latest | Q4_K_M | ollama mistral:7b | Q4_K_M | ollama mistralai/Mistral-7B-Instruct-v0.3 | 4bit | MLX LM watt-ai/watt-tool-8B | Q4_K_M | llama.cpp

Streaming

Stackstreaming-array-argumentsstreaming-boolean-argumentstreaming-empty-argumentsstreaming-enum-argumentsstreaming-nested-argumentsstreaming-parallel-threestreaming-parallel-twostreaming-weather gemma3:12b | Q4_K_M | ollama gemma3:4b | Q4_K_M | ollama granite3.1-dense:8b | Q4_K_M | llama.cpp phi4-mini:latest | Q4_K_M | llama.cpp llama3-groq-tool-use:8b | Q4_0 | ollama hermes3:8b | Q4_0 | ollama Qwen/Qwen2.5-1.5B-Instruct | Q3_K_M | llama.cpp Qwen/Qwen2.5-1.5B-Instruct | Q4_K_M | llama.cpp Qwen/Qwen2.5-1.5B-Instruct | Q8_0 | llama.cpp Qwen/Qwen2.5-1.5B-Instruct | 4bit | MLX LM Qwen/Qwen2.5-7B-Instruct | Q3_K_M | llama.cpp Qwen/Qwen2.5-7B-Instruct | Q4_K_M | llama.cpp Qwen/Qwen2.5-7B-Instruct | Q8_0 | llama.cpp Qwen/Qwen2.5-7B-Instruct | 4bit | MLX LM Qwen/Qwen2.5-7B-Instruct | 8bit | MLX LM qwen2.5:7b-instruct | Q4_K_M | ollama qwen3:0.6b | Q4_K_M | ollama qwen3:1.7b | Q4_K_M | ollama qwen3:14b | Q4_K_M | ollama qwen3:4b | Q4_K_M | ollama qwen3:8b | Q4_K_M | ollama granite3.1-dense:8b | Q4_K_M | ollama llama3.1:8b | Q4_K_M | ollama meta-llama/Meta-Llama-3.1-8B-Instruct | Q3_K_M | llama.cpp meta-llama/Meta-Llama-3.1-8B-Instruct | Q4_K_M | llama.cpp meta-llama/Meta-Llama-3.1-8B-Instruct | Q8_0 | llama.cpp meta-llama/Meta-Llama-3.1-8B-Instruct | 4bit | MLX LM microsoft/Phi-4-mini-instruct | 4bit | MLX LM phi4-mini:latest | Q4_K_M | ollama mistral:7b | Q4_K_M | ollama mistralai/Mistral-7B-Instruct-v0.3 | 4bit | MLX LM watt-ai/watt-tool-8B | Q4_K_M | llama.cpp

Tool choice

Stacktool-choice-auto-calltool-choice-auto-no-calltool-choice-namedtool-choice-named-emptytool-choice-nonetool-choice-requiredtool-choice-required-empty gemma3:12b | Q4_K_M | ollama gemma3:4b | Q4_K_M | ollama granite3.1-dense:8b | Q4_K_M | llama.cpp phi4-mini:latest | Q4_K_M | llama.cpp llama3-groq-tool-use:8b | Q4_0 | ollama hermes3:8b | Q4_0 | ollama Qwen/Qwen2.5-1.5B-Instruct | Q3_K_M | llama.cpp Qwen/Qwen2.5-1.5B-Instruct | Q4_K_M | llama.cpp Qwen/Qwen2.5-1.5B-Instruct | Q8_0 | llama.cpp Qwen/Qwen2.5-1.5B-Instruct | 4bit | MLX LM Qwen/Qwen2.5-7B-Instruct | Q3_K_M | llama.cpp Qwen/Qwen2.5-7B-Instruct | Q4_K_M | llama.cpp Qwen/Qwen2.5-7B-Instruct | Q8_0 | llama.cpp Qwen/Qwen2.5-7B-Instruct | 4bit | MLX LM Qwen/Qwen2.5-7B-Instruct | 8bit | MLX LM qwen2.5:7b-instruct | Q4_K_M | ollama qwen3:0.6b | Q4_K_M | ollama qwen3:1.7b | Q4_K_M | ollama qwen3:14b | Q4_K_M | ollama qwen3:4b | Q4_K_M | ollama qwen3:8b | Q4_K_M | ollama granite3.1-dense:8b | Q4_K_M | ollama llama3.1:8b | Q4_K_M | ollama meta-llama/Meta-Llama-3.1-8B-Instruct | Q3_K_M | llama.cpp meta-llama/Meta-Llama-3.1-8B-Instruct | Q4_K_M | llama.cpp meta-llama/Meta-Llama-3.1-8B-Instruct | Q8_0 | llama.cpp meta-llama/Meta-Llama-3.1-8B-Instruct | 4bit | MLX LM microsoft/Phi-4-mini-instruct | 4bit | MLX LM phi4-mini:latest | Q4_K_M | ollama mistral:7b | Q4_K_M | ollama mistralai/Mistral-7B-Instruct-v0.3 | 4bit | MLX LM watt-ai/watt-tool-8B | Q4_K_M | llama.cpp

Multi-turn

Stackmulti-turn-calendar-followupmulti-turn-flight-bookingmulti-turn-inventory-ordermulti-turn-parallel-followupmulti-turn-routemulti-turn-ticket-updatemulti-turn-two-hop-chain gemma3:12b | Q4_K_M | ollama gemma3:4b | Q4_K_M | ollama granite3.1-dense:8b | Q4_K_M | llama.cpp phi4-mini:latest | Q4_K_M | llama.cpp llama3-groq-tool-use:8b | Q4_0 | ollama hermes3:8b | Q4_0 | ollama Qwen/Qwen2.5-1.5B-Instruct | Q3_K_M | llama.cpp Qwen/Qwen2.5-1.5B-Instruct | Q4_K_M | llama.cpp Qwen/Qwen2.5-1.5B-Instruct | Q8_0 | llama.cpp Qwen/Qwen2.5-1.5B-Instruct | 4bit | MLX LM Qwen/Qwen2.5-7B-Instruct | Q3_K_M | llama.cpp Qwen/Qwen2.5-7B-Instruct | Q4_K_M | llama.cpp Qwen/Qwen2.5-7B-Instruct | Q8_0 | llama.cpp Qwen/Qwen2.5-7B-Instruct | 4bit | MLX LM Qwen/Qwen2.5-7B-Instruct | 8bit | MLX LM qwen2.5:7b-instruct | Q4_K_M | ollama qwen3:0.6b | Q4_K_M | ollama qwen3:1.7b | Q4_K_M | ollama qwen3:14b | Q4_K_M | ollama qwen3:4b | Q4_K_M | ollama qwen3:8b | Q4_K_M | ollama granite3.1-dense:8b | Q4_K_M | ollama llama3.1:8b | Q4_K_M | ollama meta-llama/Meta-Llama-3.1-8B-Instruct | Q3_K_M | llama.cpp meta-llama/Meta-Llama-3.1-8B-Instruct | Q4_K_M | llama.cpp meta-llama/Meta-Llama-3.1-8B-Instruct | Q8_0 | llama.cpp meta-llama/Meta-Llama-3.1-8B-Instruct | 4bit | MLX LM microsoft/Phi-4-mini-instruct | 4bit | MLX LM phi4-mini:latest | Q4_K_M | ollama mistral:7b | Q4_K_M | ollama mistralai/Mistral-7B-Instruct-v0.3 | 4bit | MLX LM watt-ai/watt-tool-8B | Q4_K_M | llama.cpp

Correctly declines

Stacknegative-auto-arithmeticnegative-auto-knowledgenegative-greetingnegative-invalid-schemanegative-long-argumentnegative-plain-textnegative-unicode-argument gemma3:12b | Q4_K_M | ollama gemma3:4b | Q4_K_M | ollama granite3.1-dense:8b | Q4_K_M | llama.cpp phi4-mini:latest | Q4_K_M | llama.cpp llama3-groq-tool-use:8b | Q4_0 | ollama hermes3:8b | Q4_0 | ollama Qwen/Qwen2.5-1.5B-Instruct | Q3_K_M | llama.cpp Qwen/Qwen2.5-1.5B-Instruct | Q4_K_M | llama.cpp Qwen/Qwen2.5-1.5B-Instruct | Q8_0 | llama.cpp Qwen/Qwen2.5-1.5B-Instruct | 4bit | MLX LM Qwen/Qwen2.5-7B-Instruct | Q3_K_M | llama.cpp Qwen/Qwen2.5-7B-Instruct | Q4_K_M | llama.cpp Qwen/Qwen2.5-7B-Instruct | Q8_0 | llama.cpp Qwen/Qwen2.5-7B-Instruct | 4bit | MLX LM Qwen/Qwen2.5-7B-Instruct | 8bit | MLX LM qwen2.5:7b-instruct | Q4_K_M | ollama qwen3:0.6b | Q4_K_M | ollama qwen3:1.7b | Q4_K_M | ollama qwen3:14b | Q4_K_M | ollama qwen3:4b | Q4_K_M | ollama qwen3:8b | Q4_K_M | ollama granite3.1-dense:8b | Q4_K_M | ollama llama3.1:8b | Q4_K_M | ollama meta-llama/Meta-Llama-3.1-8B-Instruct | Q3_K_M | llama.cpp meta-llama/Meta-Llama-3.1-8B-Instruct | Q4_K_M | llama.cpp meta-llama/Meta-Llama-3.1-8B-Instruct | Q8_0 | llama.cpp meta-llama/Meta-Llama-3.1-8B-Instruct | 4bit | MLX LM microsoft/Phi-4-mini-instruct | 4bit | MLX LM phi4-mini:latest | Q4_K_M | ollama mistral:7b | Q4_K_M | ollama mistralai/Mistral-7B-Instruct-v0.3 | 4bit | MLX LM watt-ai/watt-tool-8B | Q4_K_M | llama.cpp

Figure 2. Multi-turn raster In these published observations, multi_turn passes 37/224 (17%). Of 32 rows, 22 pass none of the multi_turn scenarios. What this does not show: This figure does not explain why a turn failed. It does not isolate model and server effects or measure reliability across repeated runs. Method and limitations - Data: JSON, CSV

Figure 3. Outcome-signature inventory Each row shows one exact status vector. 9 stacks share the repeated 7-pass signature. The stacks include the granite observations. Every row has the same 5 negative_trap passes and 2 tool_choice_modes passes. What this does not show: This figure does not establish shared model identity or a common cause. It does not predict future behavior. Method and limitations - Data: JSON, CSV