Model Leaderboard - Toolathlon

Model	Type	Agent	Date	Pass@1	Pass@3	Pass^3	# Turns	# Tool Calls
Kimi K3 (max)	Open-Weights	Default	2026-07-16	76.5_{± 1.9}	83.3	68.5	22.8	39.1
Claude Opus 4.8 (max)	Proprietary	Default	2026-06-30	76.2_{± 3.4}	84.3	66.7	19.9	36.3
Muse Spark 1.1 (xhigh)	Proprietary	Default	2026-07-09	75.6_{± 0.4}	86.1	63.9	28.9	42.7
GPT-5.5 (xhigh)	Proprietary	Default	2026-06-30	73.5_{± 1.2}	82.4	62.0	24.4	37.8
Claude Sonnet 5 (max)	Proprietary	Default	2026-07-01	71.6_{± 1.2}	83.3	53.7	27.1	51.5
Gemini 3.5 Flash (high)	Proprietary	Default	2026-06-30	67.3_{± 1.2}	79.6	53.7	45.3	45.6
Gemini 3.1 Pro (high)	Proprietary	Default	2026-07-01	61.1_{± 1.3}	72.2	48.1	26.2	32.2
GLM 5.2 (max)	Open-Weights	Default	2026-06-30	59.9_{± 1.9}	78.7	41.7	23.3	41.9
Kimi K2.6	Open-Weights	Default	2026-07-15	58.0_{± 4.9}	72.2	41.7	30.8	42.3
kimi-k2.7-code	Open-Weights	Default	2026-06-30	58.0_{± 4.3}	74.1	41.7	23.1	37.6
Deepseek-v4-pro (max)	Open-Weights	Default	2026-06-30	55.9_{± 1.2}	68.5	41.7	23.6	39.1
Inkling (xhigh)	Open-Weights	Default	2026-07-15	45.5_{± 3.7}	63.8	25.5	47.3	50.2
Nemotron-3-ultra	Open-Weights	Default	2026-07-15	34.3_{± 1.5}	50.0	19.4	28.6	31.9
Kimi K2.5	Open-Weights	Default	2026-07-15	33.0_{± 1.2}	50.9	15.7	20.3	31.3

This archived snapshot shows the leaderboard immediately before Toolathlon-Verified. It is retained for historical context; these scores use earlier task definitions and evaluation infrastructure and are not directly comparable with the Verified results above.

Model	Type	Agent	Date	Pass@1	Pass@3	Pass^3	# Turns
Gemini-3.5-Flash	Proprietary	Default	2026-05-19	56.5_{± 2.7}	68.5	43.5	44.7
GPT-5.5-xhigh	Proprietary	Default	2026-04-24	55.6	—	—	—
GPT-5.4-xhigh	Proprietary	Default	2026-03-06	54.6	—	—	—
DeepSeek-V4-Pro Max	Open-Source	Default	2026-04-25	52.8_{± 1.9}	63.9	38.9	24.1
Claude-Opus-4.7	Proprietary	Default	2026-04-25	52.8^†	—	—	16.2
GPT-5.3-Codex-xhigh	Proprietary	Default	2026-03-06	51.9	—	—	—
Kimi-K2.6	Open-Source	Default	2026-04-21	50.0	—	—	—
Gemini-3-Flash	Proprietary	Default	2025-12-18	49.4_{± 0.4}	59.3	36.1	28.6
Gemini-3.1-Pro	Proprietary	Default	2026-03-13	48.8_{± 2.3}	62.0	34.3	27.9
DeepSeek-V4-Flash Max	Open-Source	Default	2026-04-25	48.2_{± 0.9}	57.4	37.0	26.1
Claude-Opus-4.6	Proprietary	Claude Agent SDK	2026-03-06	47.2^†	—	—	—
MiniMax-M2.7	Open-Source	Default	2026-03-18	46.3	—	—	—
Claude-Sonnet-4.6	Proprietary	Default	2026-02-23	44.8_{± 2.9}	59.3	30.6	23.4
GPT-5.2-xhigh^‡	Proprietary	Default	2025-12-18	43.8_{± 1.2}	50.9	33.3	28.2
Claude-Opus-4.5	Proprietary	Default	2025-11-27	43.5_{± 0.8}	57.4	30.6	18.7
GPT-5.4-mini-xhigh	Proprietary	Default	2026-03-17	42.9	—	—	—
GPT-5.2-high^‡	Proprietary	Default	2025-12-17	41.7_{± 1.3}	54.6	28.7	23.9
MiniMax-M2.1	Open-Source	Default	2025-12-25	40.7_{± 0.8}	51.9	27.8	17.8
GLM-5.1	Open-Source	Default	2026-04-07	40.7	—	—	—
Qwen3.6-Plus	Proprietary	Default	2026-04-02	39.8	—	—	—
GLM-5	Open-Source	Default	2026-02-13	39.2_{± 1.2}	51.9	25.9	16.5
Claude-Sonnet-4.5	Proprietary	Default	2025-10-28	38.9_{± 3.0}	52.8	20.4	20.2
MiniMax-M2.5	Open-Source	Default	2026-03-18	38.3	—	—	—
Qwen3.5-397B-A17B	Open-Source	Default	2026-04-02	38.3	—	—	—
GPT-5-high^‡	Proprietary	Default	2025-12-17	37.7_{± 1.2}	50.9	19.4	25.7
Qwen3.5-Plus	Open-Source	Default	2026-02-21	37.7_{± 1.2}	49.1	25.9	17.4
GPT-5.1-high^‡	Proprietary	Default	2025-12-17	37.0_{± 2.7}	50.0	20.4	19.0
Gemini-3-Pro	Proprietary	Default	2025-11-22	36.4_{± 0.4}	48.1	23.1	19.0
GPT-5.4-nano-xhigh	Proprietary	Default	2026-03-17	35.5	—	—	—
DeepSeek-V3.2-Thinking	Open-Source	Default	2025-12-01	35.2_{± 0.8}	54.6	16.7	43.7
Qwen3.5-27B	Open-Source	Default	2026-04-17	31.5	—	—	—
Claude-Sonnet-4	Proprietary	Default	2025-10-28	29.9_{± 1.6}	41.7	17.6	27.3
Qwen3.5-35BA3B	Open-Source	Default	2026-04-17	28.7	—	—	—
Kimi-K2.5	Open-Source	Default	2026-02-04	27.8_{± 0.8}	38.9	14.8	17.2
Grok-4	Proprietary	Default	2025-10-28	27.5_{± 1.7}	38.9	16.7	20.3
Qwen3.6-35BA3B	Open-Source	Default	2026-04-17	26.9	—	—	—
Claude-Haiku-4.5	Proprietary	Default	2025-10-28	26.2_{± 1.9}	39.8	13.0	21.9
GLM-4.7	Open-Source	Default	2025-12-25	23.8_{± 1.2}	36.1	10.2	27.8
DeepSeek-V3.2-Exp	Open-Source	Default	2025-10-28	20.1_{± 1.2}	27.8	12.0	26.0
GLM-4.6	Open-Source	Default	2025-10-28	18.8_{± 2.2}	29.6	9.3	27.9
Grok-Code-Fast-1	Proprietary	Default	2025-10-28	18.5_{± 2.0}	30.6	9.3	20.2
Grok-4-Fast	Proprietary	Default	2025-10-28	18.5_{± 2.0}	32.4	5.6	15.9
Kimi-K2-thinking	Open-Source	Default	2025-11-22	17.6_{± 2.0}	29.6	4.6	24.4
o3	Proprietary	Default	2025-10-28	17.0_{± 0.9}	25.0	9.3	19.4
o4-mini	Proprietary	Default	2025-10-28	14.8_{± 0.8}	26.9	3.7	16.6
GPT-5-mini	Proprietary	Default	2025-10-28	14.5_{± 1.2}	23.1	5.6	19.7
Qwen-3-Coder	Open-Source	Default	2025-10-28	14.5_{± 1.9}	21.3	6.5	28.5
Gemini-3.1-Flash-Lite	Proprietary	Default	2026-03-13	14.2_{± 1.2}	20.4	7.4	31.2
Kimi-K2-0905	Open-Source	Default	2025-10-28	13.0_{± 2.0}	22.2	5.6	26.6
Gemini-2.5-Pro	Proprietary	Default	2025-10-28	10.5_{± 1.9}	21.3	2.8	26.5
Gemini-2.5-Flash	Proprietary	Default	2025-10-28	3.7_{± 1.5}	8.3	0.0	8.3

Results bearing this badge were independently evaluated by us; sources for all other results are linked in the corresponding model names.

^† Claude-Opus was evaluated once due to budget constraints.

^‡ OpenAI models require the Responses API to achieve better native performance. Therefore, we modified the codebase to support the Responses API for these models, updated the results accordingly, and removed the previous results that were based on the Chat Completions API.