The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution

Real-world language agents must handle complex, multi-step workflows across diverse applications. The Tool Decathlon (dubbed as Toolathlon) is a benchmark for language agents offering diverse applications and tools, realistic environment setup, and reliable execution-based evaluation. Toolathlon spans 32 software applications and 604 tools, ranging from everyday platforms such as Google Calendar and Notion to professional applications like WooCommerce, Kubernetes, and BigQuery. And it includes 108 manually sourced or crafted tasks in total, requiring interacting with multiple applications over approximately 20 interaction turns on average to complete. Each task is strictly verifiable through dedicated evaluation scripts.

Github arXiv Trajectories Quick Start

view full leaderboard ↗

Model	Type	Agent	Date	Pass\@1	Pass\@3	Pass^3	# Turns
Gemini-3.5-Flash	Proprietary	Default	2026-05-19	56.5_{± 2.7}	68.5	43.5	44.7
GPT-5.5-xhigh	Proprietary	Default	2026-04-24	55.6	--	--	--
DeepSeek-V4-Pro Max	Open-Source	Default	2026-04-25	52.8_{± 1.9}	63.9	38.9	24.1
Claude-Opus-4.7	Proprietary	Default	2026-04-25	52.8^†	--	--	16.2
Kimi-K2.6	Open-Source	Default	2026-04-21	50.0	--	--	--
MiniMax-M2.7	Open-Source	Default	2026-03-18	46.3	--	--	--
GLM-5.1	Open-Source	Default	2026-04-07	40.7	--	--	--
Qwen3.6-Plus	Proprietary	Default	2026-04-02	39.8	--	--	--
Grok-4	Proprietary	Default	2025-10-28	27.5_{± 1.7}	38.9	16.7	20.3

view all 108 tasks ↗

NVIDIA Market

💰 Finance & Market

Analyze NVIDIA's institutional ownership trends across 8 quarters, adjust for stock split, populate results\_template.xlsx with common holdings only.

Travel Expense Reimbursement

🏢 Office & Business

Validate expense claims against invoices.

Exp Recordings

💻 Tech & Dev

Update the Notion table with best scores and steps per benchmark from W\&B runs, combining same-named runs and averaging available metrics.

Woocommerce Product Recall

🛒 Shopping & E-commerce

Sync latest unupdated product inventories from each city's SQLite warehouse database to WooCommerce online store.

Canvas Homework Grader Python

🎓 Campus & Study

Grade Homework2 by downloading latest Python submissions from email, running them to check for errors, and assigning 10 (pass) or 0 (fail) in Canvas based on correctness.

K8S PR Preview Testing

💻 Tech & Dev

Deploy the feature/pr-123 branch of SimpleShopping to Kubernetes.

Inter Final Performance Analysis

🎤 Daily & Entertainment

Populate Inter Milan's 2023 and 2025 UCL final stats into three Google Sheets tabs, compute differences in "StatsDifference", and mark missing data.

Built by HKUST NLP