AI Engineer Summit 2025

Function Calling is All You Need

Ilan Bigio· Developer Experience Engineer, OpenAI1:42:54

Loading slides…

Read the talk

Function Calling Is All You Need

Build from a weather-function round trip to persistent memory, delegated model calls, background tasks, and agents that can change their own tools.

From a talk by Ilan Bigio

Before you start: Basic Python, JSON, and chat API familiarity will help; the article introduces the function-calling loop before extending it to asynchronous tasks.

From completing a question to taking an action

Ask a base language model how to get to the park, and it might continue the story by explaining that Sally asked that question yesterday. The text is plausible, but the question remains unanswered. With GPT, GPT-2, and GPT-3, building a chatbot meant arranging examples of questions and answers so that answering became the likely continuation.

InstructGPT made instruction following an explicitly trained behavior. User and assistant roles then gave conversations a more useful structure. Adding functions extended that progression again: an assistant could request an interaction with external state. The old completion-only Playground makes the difference tangible—one text window had become a conversation with instructions and available actions.

Four linked boxes show text completion, instruction following, user/assistant chat, and instructions plus functions, with examples alongside.Open full source frame
From text completion to instructions with functions.

Coherent continuation was itself a milestone. Bigio recalls the GPT-2 unicorns-in-the-Andes example: the model produced multiple paragraphs whose later details referred back to earlier context. But maintaining a story and obtaining information from outside that story require different capabilities.

WebGPT supplied an early action interface. In 2021, a GPT-3 model was trained to use a fixed set of browser commands. The application parsed its requested actions, performed them, and returned the observations to context. Human demonstrations on Reddit Explain Like I’m Five questions provided examples of searching and answering; preference training helped select desirable responses. The model could now learn a cycle of action and observation, although its tools were specialized.

Meta’s Toolformer explored learning API use from a small number of demonstrations, with tools such as question answering, calculation, and translation. Bigio describes the central intuition through a calculator call: insert the call where its answer makes the remaining sentence easier to predict. Comparing continuation probabilities provides a signal for whether the inserted tool use helped, reducing the need to label every useful call by hand.

OpenAI’s June 2023 function-calling launch turned that capability into a general interface developers could supply at request time. Bigio corrects himself from pre-training to post-training: the model had been taught to select functions and generate arguments in the expected format. Additional systems and training can still matter, but this interface is enough to begin constructing agent loops, retrieval, memory, workflows, and delegation from ordinary code.

1:451:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:55 · section reference included

The model requests; the application executes

Functions serve two broad purposes: fetching information and changing something. Reading APIs, retrieving documents, and loading memories belong to the first group. Writing through an API, updating frontend or backend state, and advancing a workflow belong to the second. A workflow action can also change the assistant itself by replacing its prompt, loading another tool set, or handing the conversation to a different agent.

A function call expresses intent; it does not execute your code. The exchange has five steps:

  1. Supply function definitions and the user’s input.
  2. Receive the model’s requested function and arguments.
  3. Parse the arguments and execute the corresponding application code.
  4. Return the result to the model.
  5. Let the model use that result in its next response.

The application owns the boundary between a proposed operation and an actual state change.

A five-step list runs from specifying functions and input through executing code and returning results to the model, beside a developer/model sequence diagram.Open full source frame
The five steps of a function-calling exchange.

The function-calling guide treats tool design as interface design. Explain each parameter, describe when the function should be used, and provide instructions or examples where necessary. Apply the principle of least surprise: if a person cannot infer how to use an interface, the model may struggle too.

Types should rule out contradictory requests. A toggle represented by two Boolean parameters can allow both on and off to be selected together. An enum such as state: "on" | "off" expresses the actual choice directly. Object structure and enums reduce the invalid states that the model can represent before execution even begins.

In OpenAI’s terminology, tools are the broader category. A raw function is an interface whose execution you manage; hosted tools such as Code Interpreter and file search also include an execution service. Bigio presents this as an adopted API convention rather than a universal definition. Audience questions immediately push beyond the single-function case: how should a model choose among a large library, could permissions help narrow it, and what happens when one tool supplies another tool’s inputs? Those questions motivate the later routing and delegation examples; the workshop does not build a separate dependency scheduler.

10:1710:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:17 · section reference included

A weather call becomes an agent loop

The first weather example stops at a requested call. Nothing handles it yet. Bigio then fills in the application side using the documentation’s weather implementation: parse the generated arguments, invoke get_weather, append its result, and request another completion. The final answer reports the temperature in Paris, although Zoom obscures the numerical value during the demonstration. This completes one request–execution–response cycle.

An agent extends that cycle until the model has no more tool calls to make. Bigio names the operation run_full_turn: one user turn can contain several model requests and several rounds of tool execution. Each assistant message enters the history before its tool results; each result is associated with the call that requested it. When the assistant produces a message without tool calls, control returns to the user.

The essential Python control flow can be written independently of the functions it will execute:

python

import json
from openai import OpenAI

client = OpenAI()


def run_full_turn(messages, tools, functions, model):
    while True:
        response = client.chat.completions.create(
            model=model,
            messages=messages,
            tools=tools,
        )
        message = response.choices[0].message
        messages.append(message.model_dump(exclude_none=True))

        if not message.tool_calls:
            return message.content

        for call in message.tool_calls:
            arguments = json.loads(call.function.arguments)
            result = functions[call.function.name](**arguments)
            messages.append({
                "role": "tool",
                "tool_call_id": call.id,
                "content": json.dumps(result),
            })

Here tools contains the schemas visible to the model, while functions maps allowed names to Python callables. The loop is the same whether a function reads weather, searches documents, or updates application state.

Writing schemas by hand is not necessary for every example. Bigio adds a functions-to-schema utility that converts a Python function object into the API’s schema representation, then supplies a mock get_weather that returns 20 degrees Celsius. After correcting a helper invocation that was mistakenly treated as part of the agent class, the call produces a completion through agents.run_full_turn.

Python editor showing get_weather, an Emergency Reference Agent with tools=[get_weather], a run_full_turn call, and print(messages).Open full source frame
A weather function is attached to an agent and passed to run_full_turn with a question about Paris.

The workshop then switches to Swarm for its convenient demonstration loop. Its basic execution pattern remains the one just built: call the model, handle functions, append observations, and continue. Keeping the underlying loop visible makes the framework a convenience rather than a prerequisite for understanding the examples.

16:0216:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:02 · section reference included

Memory starts with a list and a file

The smallest useful memory bank is a list with two functions: append a memory and retrieve memories. The interesting part is the interface description. Bigio tells the model to store factual information about the user’s life or preferences and to keep the stored text concise. A function docstring becomes part of the tool description, so it influences both when the model writes and what it chooses to preserve. Expiration is floated as an additional field, but no expiry behavior is demonstrated.

Persistence requires moving the list outside the conversation. The example reads a local JSON file at startup and writes it after changes. To fit the existing demo loop, Bigio instructs the assistant to call get_memory on its first turn. That instruction is a workaround for initialization, not a property of memory itself: an application could load the relevant memories before making the first model request.

The storage layer is small enough to inspect directly:

python

import json
from pathlib import Path

memory_path = Path("memory.json")
memory_bank = (
    json.loads(memory_path.read_text())
    if memory_path.exists()
    else []
)


def add_to_memory(memory_text: str):
    """Store a concise fact about the user's life or preferences."""
    memory_bank.append(memory_text)
    memory_path.write_text(json.dumps(memory_bank, indent=2))
    return "Memory stored."


def get_memory():
    """Retrieve the user's stored memories."""
    return memory_bank

Registering these functions gives the model an interface to the file-backed list; the file operations remain ordinary application code.

The first test exposes a wiring error: the functions were not registered under Swarm’s expected functions field, so apparent memory behavior was not backed by the intended calls. Once corrected, get_memory returns an empty bank. Bigio supplies a fact about his height, checks that it was written to the file, ends the session, and starts another conversation asking how tall he is. The new conversation retrieves the stored fact.

Loading every memory is sufficient for this example. A larger bank could use semantic similarity or search to retrieve only relevant entries. That is also the connection to retrieval-augmented generation: expose retrieval as a function, execute the search in application code, and put its results into the next model context. Selective memory retrieval is proposed here, not implemented in the live example.

26:1026:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

26:10 · section reference included

Delegation is another function call

Before adding delegation, Bigio opens Swarm’s implementation. The non-streaming client.run path repeats the same completion-and-tool-result loop, with additional handling for context and handoffs. Delegation can therefore be introduced at the function boundary without replacing the whole agent.

PatternWhat changesWhere the result goes
HandoffActive prompt and toolsThe new agent continues the conversation
Nested callA function invokes another modelBack to the calling agent
Managed taskWork runs under a task handleRetrieved or delivered later

A handoff transfers control. A nested call preserves the foreground agent and gives it another source of answers. Managed tasks add an explicit lifecycle for work that should not hold up the conversation.

The first implementation is a nested call to o1. A function accepts a task_description, calls client.chat.completions.create, and returns the result. Its description tells the foreground model to use it for difficult work instead of trying everything itself. The request supplies the task as a user message without a system message:

python

from openai import OpenAI

client = OpenAI()


def delegate_to_smarter_model(task_description: str):
    """Use for difficult tasks that need additional reasoning."""
    response = client.chat.completions.create(
        model="o1",
        messages=[{"role": "user", "content": task_description}],
    )
    return response.choices[0].message.content

The stronger model is reached through the same function interface as weather or memory. Even the parameter name helps communicate what information the caller should supply.

The test begins with a poem about dogs and Earth whose alternating words follow the alphabet. Bigio then adds a one-sentence response limit and changes the request into a haiku with both starting-letter and ending-letter constraints, explicitly describing it as hard to encourage delegation. The terminal appears idle because the function call is printed only after the nested request returns.

Eventually a result arrives. Bigio judges it only tentatively correct and suggests GPT-4o might have struggled with the task; the workshop does not establish a controlled model comparison. The clear result is an interaction problem: while the nested call runs, the user waits. Delegation has moved the computation elsewhere without making the foreground conversation available.

32:4333:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

32:43 · section reference included

Choose the background-work interaction first

Before choosing concurrency primitives, decide what should happen when delegated work finishes. Should the result enter the conversation automatically? Should it trigger a new response? Can the user inspect running tasks, launch more work, or batch requests? Audience suggestions range from asyncio and JavaScript’s event loop to batching and smaller models. These can affect implementation or latency, but they do not by themselves define the interaction.

Bigio sketches a smaller contract: return a pending acknowledgment immediately, then expose check_tasks for later inspection. At this point the code is only an interface sketch. Returning a string that says work is pending does not make a blocking model request run in the background. The executor still needs to start the work and retain a handle to it.

Python code returns “delegated, response pending,” defines a check_tasks stub, and includes it in the agent’s functions list.Open full source frame
Sketching a pending response and a task-checking function.

The terminal also needs a different arrangement. A loop blocked on user input is awkward for displaying unsolicited results, especially if output appears in the middle of the user’s typing. Bigio separates input from processing with a socket server and a terminal client. The server listens on a host and port, accepts user input, and starts a message processor; the client connects and handles input. He corrects his initial description from WebSockets to sockets.

The agent uses AsyncOpenAI and awaits completions. Its tool handler creates tasks for multiple calls and awaits their results together. This allows network waits to overlap because those operations yield control to the event loop. CPU-heavy code that does not yield still occupies the execution thread; adding async does not make it parallel CPU work.

Concurrent work needs serialized conversation updates. If two results each start a generation against the same old history, they can produce conflicting continuations. A message queue provides a single place to accept user input and, when supported, completion events. The processor removes one message, runs a full turn, updates the history, and then proceeds to the next. A simple name-recall exchange checks that ordinary conversation still works before background delegation is added.

40:5841:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

40:58 · section reference included

Parallel calls can still hold up the conversation

The async example first reproduces a normal weather round trip with a mock response of 67 and sunny. Bigio then adds a location parameter and randomizes temperatures between 50 and 80. The same function is copied into the synchronous version so that both implementations can answer requests for several cities. These are mock weather values; the comparison concerns scheduling.

A model can emit several tool calls in one response while the application still executes them one after another. With immediate-return functions, that distinction is easy to miss. Bigio adds time.sleep(1) to make it visible. He observes that five mock weather calls, each using time.sleep(1) in a sequential executor, take over five seconds.

Replacing the blocking sleep with await asyncio.sleep(1) allows the waits to overlap in the async executor. The live example returns the group together, without reporting an exact asynchronous runtime. Real network requests could occupy the same place as the artificial waits.

Execution strategyDuring tool workForeground conversation
Sequential callsOne call runs at a timeWaits
Concurrent calls, awaited togetherI/O waits overlapStill waits for the group
Background tasks with handlesWork continues independentlyCan accept another turn

Several o1 calls could run concurrently too. But if the foreground turn awaits all of them, the user still cannot continue that conversation until they finish. Shortening the aggregate wait and releasing the foreground turn are separate changes.

49:0949:25
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

49:09 · section reference included

Return a task ID before the result

create_task gives background work an identity. It generates a short random ID, schedules the operation, stores the handle, and returns the ID to the model. A separate check_task function looks up that handle and reports whether the operation is still running or has a result. The initial tool call can now finish while the underlying work continues.

The following version keeps the workshop’s SF weather example and makes the pending state explicit:

python

import asyncio
import random
import secrets

_tasks: dict[str, asyncio.Task] = {}


async def get_weather(location: str):
    await asyncio.sleep(10)
    return {
        "location": location,
        "weather": f"{random.randint(50, 80)} and sunny",
    }


async def create_task(description: str):
    """Start a weather task; use a location as its description."""
    task_id = secrets.token_hex(4)
    _tasks[task_id] = asyncio.create_task(get_weather(description))
    return {"task_id": task_id, "status": "pending"}


async def check_task(task_id: str):
    """Check a previously created task without waiting for it."""
    task = _tasks.get(task_id)
    if task is None:
        return {"task_id": task_id, "status": "unknown"}
    if not task.done():
        return {"task_id": task_id, "status": "pending"}
    try:
        result = task.result()
    except Exception as error:
        return {
            "task_id": task_id,
            "status": "failed",
            "error": str(error),
        }
    return {"task_id": task_id, "status": "completed", "result": result}

The pending response acknowledges scheduled work, not a weather answer. The task handles must remain in a running event loop for the work to continue.

Tool registration is missed once more and then corrected. To make the behavior unambiguous, Bigio increases the artificial weather delay from five to ten seconds. He creates an SF task, exchanges greetings, checks its status, asks for a joke, and checks again. The completed mock result is 78 and sunny. The useful demonstration is that those intervening turns happen while the task is outstanding.

The next change replaces get_weather with a model call. Bigio requests a haiku as a task, then asks for a second one with a theme chosen by the model. Both can be inspected through the same task interface while the foreground chat remains available. He questions whether the unexpectedly quick result actually used o1, so the example establishes multiple background model tasks without establishing which model produced those particular poems.

Polling is only one delivery mechanism. The separate input and output architecture was intended to support pushing a completed result into the conversation without a new user message. Bigio identifies that as the next extension, but the demonstrated version still requires checking tasks.

54:2854:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

54:28 · section reference included

Expose progress without hiding the loop

An audience question suggests representing agents as Python generators, composing nested agents with yield from, and yielding tool calls as they occur. Bigio agrees: instead of exposing only a final answer, an agent can emit steps, calls, and results throughout its execution. That would make the long nested call visible rather than leaving the terminal apparently idle.

The additional work is event handling. With several agents emitting events, a consumer must listen to them and associate each event with its originating agent. A flat collection of event-producing agents is possible; the workshop instead keeps one foreground manager responsible for starting and inspecting subordinate work. Generator-based progress reporting is recommended as an extension, but not implemented live.

Framework choice follows the same preference for visible mechanics. Bigio likes Swarm for prototyping because he knows it well, but often copies a small custom loop between projects to retain granular control and avoid unnecessary dependencies. Handoffs, for example, require special handling of particular tool results. He also likes the interface of Pydantic AI, while explicitly noting that he has not used it extensively.

1:01:501:03:32
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:01:50 · section reference included

Selecting from a larger tool library

A large function library presents a selection problem before it presents an execution problem. Bigio offers three approaches, each changing what the model needs to choose from:

  • Group by responsibility. Give related functions to separate agents and invoke the relevant group for the task.
  • Fine-tune flat selection. Keep functions available together when routing overhead is undesirable. Bigio reports fine-tuning GPT-3.5 for approximately 120 functions in an OpenAI project.
  • Load functions dynamically. Use the input or conversation to retrieve likely relevant definitions, through embeddings or an initial function call that loads another set.

The last approach starts to resemble a handoff: a call changes the functions available for the next model request. The reported GPT-3.5 project illustrates a possible approach, but no accuracy metric or evaluation set is supplied.

The ensuing question asks whether reasoning models call tools inside their hidden reasoning. For the o1 API at the time of the workshop, Bigio says the exposed function calls occur after reasoning rather than within it. That is a historical description of the demonstrated API, not a general restriction on what post-training could enable.

1:07:411:07:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:07:41 · section reference included

Route to the right functions with a handoff

The routing example divides functions into email and calendar groups. The email agent gets operations such as sending and checking email; the calendar agent gets event creation. A triage agent chooses the appropriate destination. Before adding this structure to a real application, Bigio recommends trying the complete function set on one agent and using evaluations to determine whether the extra routing is needed. His hypothetical larger library motivates the split; it is not evidence that a particular count requires multiple agents.

Swarm represents a transfer with a function that returns an Agent object. Register those transfer functions on triage, and the framework can replace the active agent when one is called:

python

from swarm import Agent

email_agent = Agent(
    name="Email agent",
    instructions="Handle email requests with the available email tools.",
)
calendar_agent = Agent(
    name="Calendar agent",
    instructions="Handle scheduling requests with the available calendar tools.",
)


def transfer_to_email():
    """Transfer email requests to the email agent."""
    return email_agent


def transfer_to_calendar():
    """Transfer scheduling requests to the calendar agent."""
    return calendar_agent


triage_agent = Agent(
    name="Triage agent",
    instructions="Transfer the user to the appropriate specialist.",
    functions=[transfer_to_email, transfer_to_calendar],
)

The transfer functions are the routing interface; each specialist’s application functions can be attached to its own functions list.

A request to send an email first transfers the conversation to the email agent. Bigio then makes the request complete: send to Bob, use a taxes subject, and put a short reminder to do taxes in the body. With instructions to act when the necessary information is already present, the agent transfers and immediately invokes the demonstration’s send-email function. It feels like one operation to the user, although routing adds a model hop. The demonstration shows function dispatch, not independently confirmed delivery to a mailbox.

A request for a Streamlit- or Gradio-style one-command application wrapper receives no concrete implementation plan. On function counts, Bigio is tentative about hard API limits, then gives practical guidance: Bigio suggests roughly 10–20 available functions as a rule of thumb for reliability without extensive prompting. He explains that the earlier 120-function fine-tune served an unusually latency-sensitive application that needed flat selection. The decision is therefore about evaluated selection quality and acceptable routing latency, not a universal cutoff.

1:10:191:10:31
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:10:19 · section reference included

An agent that adds its own functions

An audience question about code-generating agents leads to an unplanned experiment: let the agent write a Python function, register it, and use it. Bigio starts a new Swarm agent and considers a self-handoff—returning the same agent after modifying its functions—to refresh the available interface. The proposed add_tool accepts a string containing a Python implementation.

The key operation is converting source text into a callable object. Bigio initially reaches for eval, then corrects it to exec, which can execute a function definition. The desired output is the newly defined function object so that it can be appended to the agent’s tool list. Merely running a script in a subprocess would not provide that in-process object. Cursor assistance and an audience implementation help work through the parser and registration code.

The first behavioral test is deliberately observable: create a tool that prints hello. When the generated tool merely returns text, Bigio asks for printing instead, then calls the revised tool and sees the output. He next asks the agent to make a calculator and tries a multiplication expression. The useful result is the cycle of generating, registering, and invoking a new callable; the spoken arithmetic is not clear enough to recover an exact calculation.

Executing model-generated Python inside the application process is an unsafe capability boundary. It grants generated code the process’s access rather than confining it to the apparent tool description. Bigio repeatedly labels the example dangerous and treats it as an experiment. Dynamic tool registration is a powerful mechanism, but this implementation does not provide an isolation or permission system for the code it accepts.

1:17:051:17:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:17:05 · section reference included

A function can also mean staying silent

Voice interfaces introduce another kind of action: deciding not to respond yet. A silence-based voice activity detector can signal that the user might be finished even when they have only paused mid-thought. Bigio proposes treating that signal as an eager candidate boundary, then letting the model choose a stay_silent function if the utterance still sounds unfinished.

For example, a user might pause after saying they have been thinking about something, before identifying what it is. The detector triggers, but the model selects silence so the user can continue. Bigio reports that this works well with an appropriate tool description; he does not demonstrate it live. The mechanism adds semantic judgment after the acoustic trigger rather than requiring every detected pause to produce speech.

A separate DevDay experiment uses XML-tagged scripts to direct how the Realtime model speaks. Bigio describes asking the model to read a script according to the tags, then tries to play an example. The room cannot hear the playback, so no shared-audio result is established. He also distinguishes this from function calling: it is an observed instruction-following behavior, not an explicitly trained XML speech-control interface.

1:26:511:27:02
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:26:51 · section reference included

When a remembered fact changes

Returning to memory, an audience question asks how to enforce consistency and represent more structured information. The append-only list can retain facts, but it cannot by itself distinguish a contradiction from a legitimate update. Bigio proposes retrieving semantically similar memories before storing a new one, then asking a model whether the new information updates or contradicts an existing entry.

Consider the project example: an earlier memory says the project is not ready; a later message says it is done. Both statements can accurately describe different moments. Bigio proposes timestamping the entries and adding an explicit link from the older memory to its successor. The older record stays available, while the relationship identifies which state is current.

With that link, retrieval can return either the latest state or the whole update chain. A question based on an old assumption—such as how long the project remains delayed—can be answered using the completion update, while a historical question can still use the earlier record. This is an off-the-cuff design proposal, not a feature added to the live memory implementation.

Preserve the old fact and link its update

Constructed example: Record IDs, JSON field names, and dates are teaching values. The project-state change and proposed successor-link mechanism come from the workshop.

New user message — unchanged
The project is done now.

Operation: Proposed storage operation: retrieve the related project memory, classify the message as an update, preserve the earlier record, and link it to the new record.

Earlier project memory

Before: Before the proposed update
{"id":"m1","text":"The project is not ready yet.","recorded_at":"2025-02-01","superseded_by":null}
After: After the proposed update · Changed
{"id":"m1","text":"The project is not ready yet.","recorded_at":"2025-02-01","superseded_by":"m2"}

Completion update

Before: Before the proposed update
Not present
After: After the proposed update · Added
{"id":"m2","text":"The project is done now.","recorded_at":"2025-02-10","superseded_by":null}
The successor link identifies current state without deleting the project's history.
1:31:251:31:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:31:25 · section reference included

Carrying the same contract into a phone conversation

The final demonstration opens the OpenAI Realtime API with Twilio Quickstart. Its setup interface tracks the account, phone number, and local web server with a live checklist. Bigio explains that repeated Twilio and ngrok setup had been tedious; the interface is intended to make progress visible and reduce trips through the Twilio console. His local application is already running, with sensitive settings masked.

He creates a whimsical universe-answers function schema in the Realtime Playground, pastes it into the tool configuration, saves the changes, and calls the phone number. The first attempts do not produce a working conversation. The interface nevertheless illustrates two distinct ways to supply tools:

  • Schema-only tools: the interface displays a requested call, and a person can enter a mock response.
  • Backend handlers: application code executes the function and returns its actual output.

Bigio points to a backend weather handler as the implemented example and suggests that delegation to o1 could be added through the same handler mechanism.

The consequential difference from the earlier chat loop is the pending-call interaction. Bigio says the Realtime API permits a function call to remain outstanding while the user continues talking to the model. He attributes that behavior to specific training: a voice conversation cannot simply stop whenever an external operation is slow. It is the background-task requirement from the Python example expressed in a realtime conversation interface.

A final retry also fails, so the workshop ends without a successful end-to-end phone call. Bigio then shares the experimental self-writing tool-agent code in Slack, repeats the unsafe-code warning, and offers to share the repository and slides. The shared code returns to the most expansive use of the execution contract explored here: application code can interpret a requested operation as a change to the functions the model will see on its next turn.

A Slack message contains Python code defining an agent, parsing function code, and adding a tool to the agent’s functions list.Open full source frame
The experimental tool-writing agent code shared in Slack.
1:35:211:35:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:35:21 · section reference included

Resources

From the talk

  • The educational orchestration framework used in the workshop. Its README now directs production users to the Agents SDK.

  • A phone-assistant example connecting Twilio and OpenAI through a WebSocket backend, with setup instructions and mock tool responses. The README cautions against deploying without security review.

  • The original announcement of function calling for GPT-4 and GPT-3.5 Turbo, including the historical API parameters.

  • ToolformerPaper8:20

    Research on teaching language models to select and use external APIs with only a few demonstrations per tool.

  • WebGPTArticle3:22

    OpenAI's 2021 account of training GPT-3 to browse, collect sources, and answer questions using human feedback.

  • Official documentation for the Python agent framework discussed during the framework Q&A.

Updates since the talk

Transcript download
Full transcript

Loading transcript…