Closing the Agentic Tool-Use Gap for a Frontier Foundation Model

Background

The client's foundation model showed significant gaps in agentic scenarios that require autonomously executing MCP (Model Context Protocol) tools. Its capabilities needed to improve so the agent could reliably complete complex workflows through the full closed loop of understand the goal → decompose the task → select the tool → execute the call → validate the result → iterate and correct.

Problem

Through our testing and evaluation, we identified the following issues with the client's model on long-horizon agentic tasks:

1. Insufficient tool-call effectiveness: the model could not correctly and effectively invoke MCP tools to resolve user tasks, and struggled to translate user requirements into precise tool-execution instructions.
2. Lack of execution stability: on a subset of tasks, repeated runs showed pronounced variance. Outcomes swung widely and reliability was poor, keeping task-failure rates persistently high.
3. Long-horizon tasks stalling out: the model frequently exhibited self-doubt behavior, easily falling into unproductive loops until it exceeded the context window, blocking task progress and ultimately causing failure.

Method

1. Model diagnosis: ran benchmark evaluations across MCP-related datasets against the client's model, analyzed the results, and pinpointed the model's specific capability gaps.
2. Production framework design: built a data-production execution framework and task-constraint mechanism around the diagnosed capability gaps, and stood up the tooling needed for data production and verification.
3. Dataset composition design: defined task standards and target dataset ratios based on the diagnosed problems.

   Standard	Criterion	Target Ratio	Rationale
   1	SOTA model pass@3 = 1 or 2	10%	SOTA models succeed at a high rate
   2	SOTA model sum@5 ∈ (0, 16]	25%–50%	SOTA model execution is unstable
   3	Client model pass@3 = 0 or 1	25%–50%	Tasks the client's model struggles to execute
   4	High variance in client model pass@3 across runs	40%	Client model execution is unstable

4. Task design: each task package includes:
   • Task prompt: the task description, covering the goal, constraints, and output requirements
   • Task environment files and initialization scripts: the baseline file environment or database initialization scripts required to execute the task
   • Result-verification script: validates whether the task executed successfully and surfaces the points of divergence in the result, usable for RL (reinforcement learning)
   • Task results: aggregate statistics across models, execution trajectories (usable for SFT, meaning supervised fine-tuning), and execution logs
5. Data production: partnered directly with domain experts to source real-world business scenarios, then built and validated the task data against them.

Results

• After the first batch of 100 tasks: model capability improved 9%.
• After full delivery: model capability improved 22%, and leaderboard rank moved from #10 to #2, surpassing SOTA models including Gemini 3 Pro, GPT-5, and Claude Opus 4.5.