The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution
Real-world language agents must handle complex, multi-step workflows across diverse applications. The Tool Decathlon (dubbed as Toolathlon) is a benchmark for language agents offering diverse applications and tools, realistic environment setup, and reliable execution-based evaluation. Toolathlon spans 32 software applications and 604 tools, ranging from everyday platforms such as Google Calendar and Notion to professional applications like WooCommerce, Kubernetes, and BigQuery. And it includes 108 manually sourced or crafted tasks in total, requiring interacting with multiple applications over approximately 20 interaction turns on average to complete. Each task is strictly verifiable through dedicated evaluation scripts.
| Model | Type | Agent | Date | Pass@1 | Pass@3 | Pass^3 | # Turns |
|---|---|---|---|---|---|---|---|
| GPT-5.4-xHigh* | Proprietary | Default | 2026-03-06 | 54.6 | — | — | — |
| Gemini-3-Flash | Proprietary | Default | 2025-12-18 | 49.4± 0.4 | 59.3 | 36.1 | 28.6 |
| Claude-4.6-Opus | Proprietary | Claude Agent SDK | 2026-03-06 | 47.2 | — | — | — |
| MiniMax-M2.1 | Open-Source | Default | 2025-12-25 | 40.7± 0.8 | 51.9 | 27.8 | 17.8 |
| GLM-5 | Open-Source | Default | 2026-02-13 | 39.2± 1.2 | 51.9 | 25.9 | 16.5 |
| Qwen3.5-Plus | Open-Source | Default | 2026-02-21 | 37.7± 1.2 | 49.1 | 25.9 | 17.4 |
| DeepSeek-V3.2-Thinking | Open-Source | Default | 2025-12-01 | 35.2± 0.8 | 54.6 | 16.7 | 43.7 |
| Kimi-K2.5 | Open-Source | Default | 2026-02-04 | 27.8± 0.8 | 38.9 | 14.8 | 17.2 |
| Grok-4 | Proprietary | Default | 2025-10-28 | 27.5± 1.7 | 38.9 | 16.7 | 20.3 |
* This result is sourced from the provider’s published report and has not yet been independently reproduced by ToolAthlon. The source document is linked in the model name.
- 💰NV Market
- 🏢Travel Reimbursement
- 💻Exp Recordings
- 🛒Product Recall
- 🎓Homework Grader
- 💻K8S PR Preview Testing
- 🎤Final Performance Analysis