ToolGrad: Google’s Answer-First Method for Training LLMs to Use Tools

Google XR researchers Zhongyi Zhou and Ruofei Du published ToolGrad at ACL 2026, a framework that flips the standard approach to building tool-use training data for large language models. Instead of starting with a user query and searching for a solution, ToolGrad generates the tool-use answer chain first and then writes the query to match. Fine-tuning Gemma-3 on just 500 samples produced a 12B model that scored 83.1 on the Berkeley Function Calling Leaderboard, beating several state-of-the-art proprietary models.

What happened

Detail	Fact
Published at	ACL 2026
Authors	Zhongyi Zhou and Ruofei Du, Google XR
API database used	ToolBench (16,000+ real-world APIs)
Training dataset size	ToolGrad-500 (500 samples)
Models fine-tuned	Gemma-3 1B, 4B, and 12B
BFCL score (12B)	83.1
Benchmark	Berkeley Function Calling Leaderboard (BFCL)

Teaching an LLM to call external tools reliably requires large, high-quality datasets of tool-use sequences paired with the user queries that prompted them. The usual method, used by prior systems like ToolBench and ToolACE, works in two steps: invent a plausible user instruction, then run a depth-first search (DFS) agent to find a working sequence of API calls that satisfies it. The problem is that many searches fail, making the whole process expensive and slow.

ToolGrad reverses the order. It builds a verified API call chain first, then generates a user query to describe what that chain accomplishes. Because a concrete, working solution is far less ambiguous than an open-ended prompt, the query-writing step requires only one LLM call instead of a lengthy search. The result is a higher pass rate and lower generation cost per sample.

How ToolGrad actually works

The framework borrows the concept of “textual gradients” from TextGrad, a prompt-engineering technique that uses an LLM critic to write rich, plain-text feedback. In standard machine learning, numerical gradients tell the optimizer which direction to adjust model weights. Textual gradients do the same thing for text: they describe, in prose, how a draft should change to get closer to the target.

ToolGrad applies this idea to iterative API chain construction through four modules that run in sequence on each step:

1. API Proposer: Narrows a sampled set of APIs down to a few promising candidates for extending the current workflow.
2. API Executors: Test the candidate APIs in parallel and produce detailed execution reports.
3. API Selector: Reviews those reports and picks the single best-performing call. This selection acts as the textual gradient, providing directional feedback on what works.
4. LLM Updater: Revises the synthetic user query and AI response to stay consistent with the updated API sequence.

The loop repeats until the chain is complete. The final output is a verified API workflow, a matching user query, and the model’s expected response. All three go into the training dataset.

What the benchmark results show

Google evaluated the fine-tuned Gemma-3 models on the Berkeley Function Calling Leaderboard (BFCL), a benchmark that uses a different tool set from ToolBench, making it a genuine out-of-distribution test. The comparison groups were base Gemma-3 models without any fine-tuning, proprietary models (Gemini, GPT, and Claude), and specialist tool-use models including ToolACE and Hammer-2.1-7B.

Key findings from the paper:

• Fine-tuning on ToolGrad-500 improved tool-use performance for all three Gemma-3 sizes (1B, 4B, 12B).
• ToolGrad-12B scored 83.1 on BFCL, beating the proprietary models tested.
• The model matched or exceeded state-of-the-art results even on tools it had never seen during training.

That last point matters. OOD generalisation (performing well on tools outside the training set) is the real test of whether a model has learned to reason about tool use rather than memorise specific APIs.

Why it matters

AI agents that can call APIs reliably are the foundation of most serious automation work, from web search to file management to code execution. The quality of the training data determines how well those agents perform in production. If you can generate better data cheaper, you can fine-tune smaller models to do the same work that previously needed a large proprietary model.

ToolGrad-12B outperforming GPT and Claude on BFCL with only 500 training samples is a striking demonstration of that principle. It also suggests that answer-first data generation could apply beyond tool use, anywhere you want to build complex, verifiable reasoning chains for training data.

For teams building AI integrations on top of LLMs, this research signals that fine-tuning on small, high-quality datasets may soon be a practical alternative to prompt engineering with large closed models. The cost of custom fine-tuning is dropping.

Our take

The core insight here is simple and worth internalising: if you know the answer, writing the question is easy. The DFS approach was always fighting the problem from the hard end. ToolGrad flips the constraint and the efficiency gain follows naturally.

The 500-sample number deserves some scrutiny. The paper compares against ToolBench’s DFS baseline, but the benchmark (BFCL) uses different tools than the training data, which does make the comparison meaningful. Still, 83.1 on a leaderboard with a specific test format is not the same as reliable tool use in a messy real-world deployment. The gap between benchmark scores and production reliability is something we see regularly on client projects.

That said, the direction is correct. More coverage of the AI research driving these improvements is in our AI news section, including related work on agents and safety. If you are considering a fine-tuning project, this paper is a good argument for investing in data quality over data volume.

What to do about it

1. Read the ToolGrad paper if you are building or evaluating LLM agents that call external APIs. The four-module architecture is directly applicable to custom tool-use pipelines.
2. Run your current agent against the Berkeley Function Calling Leaderboard to get a baseline score before comparing it to fine-tuned alternatives.
3. If you are planning a fine-tuning project, prioritise answer-first data generation: build verified examples first, then write the prompts that fit them.
4. Watch Gemma-3 closely. A 12B open model that matches closed proprietary models on tool use changes the cost equation for production deployments significantly.

The practical takeaway: better training data beats bigger models. 500 well-constructed examples outperformed models trained on far more but noisier data.

Source: Google Research Blog