AI Engineer World's Fair 2025
Real AI Agents Need Planning, Not Just Prompting
Loading slides…
Read the talk
Real AI Agents Need Planning, Not Just Prompting
As instructions accumulate, reliable output becomes a search and validation problem. Planning coordinates that work; an execution engine decides how to spend the time and compute.
From a talk by Yuval Belfer
Before you start: Familiarity with LLM prompts and tool calling is helpful; no prior planning-framework experience is required.
Why is instruction following still a problem?
In 2022, InstructGPT made following natural-language instructions substantially more useful. By the release of GPT-4.1 in 2025, instruction following was still an engineering problem. Why had better models not made it disappear? Yuval Belfer opens with that question, extending it beyond OpenAI to language models generally.
Consider the early request: “Explain the moon landing to a six-year-old.” A short, accessible explanation was enough to make the capability feel remarkable. As developers adopted language models, however, requests grew into long specifications. The task remained in the prompt, but so did the context, constraints, formatting rules, and business requirements. Putting everything into one input made the model responsible for satisfying all of it in one attempt.
A task can be easy to describe and difficult to satisfy. Even instruction following can demand more than a single generation. This is the opening for an agent architecture: not merely a longer prompt, but a way to plan the work required to meet the requirements.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From routing to reasoning and acting
Belfer works at AI21 Labs, which develops language models and an agentic framework. His approach to defining an agent is pragmatic: whether a working system is called an agent or a workflow matters less than whether it solves the problem. That leaves several useful architectures on the table.
| Pattern | What the model does |
|---|---|
| LLM router | Sends a query to a specialized language model. |
| Function calling | Selects from provided tools to interact with APIs or services such as Google Search. |
Belfer introduces MCP as standardizing these external interactions, then leaves the protocol details aside. Tool access is useful regardless of whether the surrounding system earns the agent label.
ReAct, short for reasoning and acting, adds an iterative loop: reason about the situation, take an action, observe the result, and repeat. Belfer characterizes this as choosing the next step from the current state rather than looking ahead across the entire plan. That is a distinction about the planning horizon of the pattern he describes, not an inherent prohibition on planning: the original ReAct work also discusses updating action plans.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A plan needs an execution engine
Planning means figuring out the steps needed to reach a goal. It becomes useful when the problem has no straightforward solution, particularly when work needs to run in parallel or the system needs to explain its choices. Belfer's concern with a ReAct trace is that seeing the sequence of thoughts and actions does not necessarily explain why that route was chosen.
Plans can take different forms:
- Text: Belfer points to Microsoft's Magentic-One as an example of text-based planning.
- Code: He points to Hugging Face's smolagents, where agents can express actions as executable code. This is a useful contrast in representation, rather than an exclusive product taxonomy; smolagents also supports conventional tool calling.
A dynamic plan can change during execution. Instead of committing to the initial sequence until completion, the system reassesses whether the plan is still useful and replans when necessary. That makes planning an ongoing responsibility rather than a one-time preamble to the task.
The execution engine makes the plan operational. It analyzes dependencies between steps, identifies work that can run in parallel, and manages trade-offs between speed and cost. Belfer also names branch prediction as a possible latency optimization, without specifying its implementation. AI21 Maestro brings these two responsibilities together: a planner chooses how to proceed, while a smart execution engine manages the work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Turn requirements into a search and repair process
Maestro's simplified instruction-following example starts by separating the context and task from the requirements. The requirements include no more than ten paragraphs, a formal tone, and positive mentions of the brand. These are the same conditions a developer might otherwise bury in the prompt; making them explicit makes them easier to validate.
The paragraph limit illustrates the benefit of a separate check. For text whose paragraphs are separated by blank lines, a small Python validator can turn that requirement into a concrete result:
python
import re
def check_paragraph_limit(text: str, maximum: int = 10) -> dict:
paragraphs = [
part for part in re.split(r"\n\s*\n", text.strip())
if part.strip()
]
return {
"requirement": "paragraph_limit",
"maximum": maximum,
"observed": len(paragraphs),
"passed": len(paragraphs) <= maximum,
}
Tone and the sentiment of brand mentions require different checks: counting paragraphs does not establish that either semantic requirement was met. Separating requirements gives each condition a place in validation.
The execution graph then turns generation into a search over possible answers. At each stage, the planner and execution engine choose candidates worth continuing. Best-of-N supplies diversity: generate several high-temperature samples from one model, or obtain candidates from different models, instead of relying on a single answer.
The remaining loop controls where additional effort goes:
- Discard candidates that do not look promising.
- Continue with the best candidate or candidates within a predefined budget.
- Validate those candidates, fix failures, and repeat the improvement process.
The diagram's branches represent successive candidates and repairs, not a requirement to exhaust every possible path. Candidate selection concentrates the available budget on answers that appear worth improving.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose a path, then reduce the results
For a more complex query, the same architecture chooses among alternative execution tracks. The execution engine estimates expected cost, latency, and success probability for those tracks, and the planner uses the estimates to select a path. The displayed graph expands the choices across retrieval parameters, model choices, and model parameters.
At the end, a reduce step turns the accumulated results into an answer. It can select the best result or combine several results into a complete response to the original query. The highlighted routes in the final diagram converge toward validation and the complete answer: branching creates alternatives, while reduction brings the work back to one deliverable.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Higher quality costs additional inference
Belfer reports improved IFEval results using Maestro's planner and execution engine with GPT-4o, Claude Sonnet 3.5, and o3-mini. IFEval evaluates instruction following through verifiable requirements. The spoken comparison gives no numerical scores, model snapshots, scoring variant, or inference budgets.
On an internal requirement-satisfaction dataset built from customer data, Belfer also reports improvement over a single LLM call, at the cost of more runtime and money. The internal evaluation is distinct from IFEval, and the talk provides no numerical cost or latency measurements. The engineering trade-off is explicit: candidate generation, validation, and repair buy quality by doing additional work.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use planning when the task requires it
The practical ending is an adoption ladder, not a mandate to put a planner around every model. Start with the simplest system that works. If a small language model is sufficient, use it. If a model with tools handles the task, stop there. If ReAct works, use that.
Move to a planner and execution engine when the task's complexity requires that additional coordination. Belfer closes by inviting developers to the Maestro waitlist—the access path offered in the recording—to see whether the system fits their needs. The decision is whether explicit planning, validation, and controlled execution solve a problem that the simpler architecture cannot.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
OpenAI’s original announcement explains instruction tuning and includes the child-friendly moon-landing example.
The original paper on interleaving reasoning and actions during interactive tasks.
Explains an orchestrator that maintains task progress, delegates work, and revises plans.
Hugging Face’s introduction to agents that express actions as executable code, with examples.
Introduces IFEval and its use of automatically verifiable instructions to evaluate model outputs.
Further reading
Yuval Belfer explains Maestro’s candidate search, validation, repair, execution budgets, and evaluation methods.
Google Research’s implementation and dataset for instruction-following evaluation.
Updates since the talk
Documentation for defining requirements, setting budgets, and inspecting validation reports.
AI21’s July 2025 announcement making Maestro available to enterprise customers.
Full transcript
Loading transcript…