How AI Tool Calling Works Under the Hood

The Request-Response Cycle

Every tool calling interaction begins with the calling application sending a request to the model API. This request contains three critical components: the conversation messages (system prompt, user messages, and any previous assistant responses), the tool definitions (a JSON array describing each available function), and model parameters (temperature, max tokens, and other configuration). The tool definitions are the key differentiator from a normal chat request. They tell the model what functions exist, what each function does, and what arguments each function accepts.

The model processes the full request and makes a decision. If the user question can be answered directly from the model training data and no tool would improve the response, the model generates a normal text response. If a tool call would produce a better, more accurate, or more complete response, the model generates a structured tool call object instead of or alongside text. This decision is not rule-based. The model uses its trained understanding of when tools are helpful, informed by the tool descriptions and the user request context.

When the model decides to make a tool call, it generates a response that includes a tool call block containing the function name and a JSON object of arguments. The arguments must conform to the JSON Schema defined in the tool definition. The model does not execute anything, it produces data that describes what it wants to execute. This distinction between intent and execution is fundamental to the security model of tool calling.

The calling application receives the model response, extracts the tool call, validates the arguments against the expected schema, and executes the function in its own runtime environment. The function result is then formatted as a tool result message and appended to the conversation. The entire conversation, now including the tool call and its result, is sent back to the model in a follow-up request. The model processes the result and either generates a final text response or makes additional tool calls.

The Conversation Loop in Detail

A complete tool calling interaction typically involves multiple API round trips. Consider a user asking "What is the current temperature in Tokyo and New York?" with a weather API tool available. The first request sends the user message and tool definitions to the model. The model recognizes that its training data does not contain current weather data and generates two parallel tool calls: one for Tokyo weather and one for New York weather.

The application receives both tool calls, executes them concurrently against the weather API, and receives results like {"city": "Tokyo", "temp_c": 22, "condition": "Partly cloudy"} and {"city": "New York", "temp_c": 18, "condition": "Clear"}. These results are formatted as tool result messages and sent back to the model along with the full conversation history.

The model receives the results and now has the real data it needs to answer the user question. It generates a natural language response like "Tokyo is currently 22 degrees Celsius and partly cloudy, while New York is 18 degrees Celsius with clear skies." The conversation is complete, having required two API round trips: one to generate the tool calls and one to generate the final response from the results.

More complex tasks involve many more round trips. A research task might require the model to search for information, read several documents, extract specific data points, cross-reference findings, and compile a summary. Each step involves a tool call and a result, potentially spanning 10 to 20 round trips. The model maintains context across all these turns, building up its understanding of the task as each tool result provides new information.

Structured Output Generation

The mechanism by which models generate structured tool calls is worth understanding. Models do not have separate "text mode" and "tool mode" capabilities. They are always generating tokens, which are the fundamental units of model output. When tool definitions are present, the model training includes examples of generating structured tool call tokens alongside regular text tokens. The model learns to produce well-formed JSON arguments that match the tool parameter schemas.

Provider implementations add constraints on top of raw token generation to improve reliability. Some providers use constrained decoding, which forces the model output to conform to valid JSON syntax during generation. Others use post-processing to validate and repair the generated JSON. The result is that modern models produce valid, schema-conformant tool calls with very high reliability, typically above 95% accuracy on well-defined tools with clear descriptions.

The model decision about when to use tools versus generating text directly is probabilistic, not deterministic. The same user query with the same tools might sometimes result in a tool call and sometimes result in a direct text response, depending on sampling parameters like temperature. Most production systems use low temperature values (0.0 to 0.3) for tool calling to maximize consistency and reduce variability in tool call generation.

Multi-Turn State Management

Tool calling conversations maintain state through the conversation history. Each message in the conversation, including system prompts, user messages, assistant responses, tool calls, and tool results, is included in every subsequent API request. This means the model has access to the full history of what tools were called, what arguments were used, and what results were returned. It uses this history to make informed decisions about what to do next.

This stateless design (from the API perspective) means that every request must include the full conversation context. For long tool calling sessions with many round trips, the conversation history can grow large. Applications must manage context window limits by truncating old messages, summarizing previous interactions, or implementing sliding window strategies that keep the most recent and most relevant messages while dropping older ones.

Some applications implement their own state management layer on top of the conversation history. Rather than relying solely on the model ability to recall information from earlier in the conversation, they maintain structured state objects that track task progress, accumulated results, and decision history. This state is injected into the system prompt or as a special context message, giving the model reliable access to critical information without depending on its ability to extract it from a long conversation history.

Provider Implementation Differences

While the core mechanism is consistent across providers, implementation details vary in ways that affect application design. OpenAI, Anthropic, and Google each have different approaches to tool definition format, parallel tool calling behavior, streaming support, and error reporting.

OpenAI uses a "functions" array (now called "tools" in newer API versions) with strict JSON Schema support. They support parallel tool calls where the model can generate multiple tool calls in a single response. Streaming is supported, with tool call arguments streamed token by token as they are generated.

Anthropic uses a "tools" array with similar JSON Schema support but includes additional features like tool choice controls that let developers force the model to use a specific tool or prevent tool use entirely. Claude supports parallel tool calls and provides detailed stop reason information that distinguishes between "end_turn" (model is done) and "tool_use" (model wants to call a tool).

Google Gemini uses a function declaration format that is similar to but not identical to OpenAI and Anthropic formats. Gemini supports parallel function calls and provides function calling modes that control how aggressively the model uses available functions.

The Model Context Protocol (MCP) aims to abstract these differences by providing a standard interface for tool definitions and invocations. MCP clients handle the translation between the standard format and each provider specific format, allowing developers to write tool definitions once and use them across multiple providers.