# Real AI Agents Need Planning, Not Just Prompting

Yuval Belfer, AI21 Labs

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/Th5e4h-oVmc-real-ai-agents-need-planning-not-just) · [Video](https://www.youtube.com/watch?v=Th5e4h-oVmc)

## Real AI Agents Need Planning, Not Just Prompting

As instructions accumulate, reliable output becomes a search and validation problem. Planning coordinates that work; an execution engine decides how to spend the time and compute.

Familiarity with LLM prompts and tool calling is helpful; no prior planning-framework experience is required.

### Why is instruction following still a problem?

[0:00](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=0s)

In 2022, [**InstructGPT**](https://openai.com/index/instruction-following/) made following natural-language instructions substantially more useful. By the release of [**GPT-4.1**](https://openai.com/index/gpt-4-1/) in 2025, instruction following was still an engineering problem. Why had better models not made it disappear? Yuval Belfer opens with that question, extending it beyond OpenAI to language models generally.

Consider the early request: “Explain the moon landing to a six-year-old.” A short, accessible explanation was enough to make the capability feel remarkable. As developers adopted language models, however, requests grew into long specifications. The task remained in the prompt, but so did the context, constraints, formatting rules, and business requirements. Putting everything into one input made the model responsible for satisfying all of it in one attempt.

![Slide titled “How is it still a problem?” with a short moon-landing prompt, its completion, and a surprised cat image.](https://ai.engineer/api/talk-essay-media/aa69d037fd198f6cd43ca40232831b7c68d0e9ec1370af329080164ad969293a)

The early instruction-following example: explaining the moon landing to a six-year-old.

**A task can be easy to describe and difficult to satisfy.** Even instruction following can demand more than a single generation. This is the opening for an agent architecture: not merely a longer prompt, but a way to plan the work required to meet the requirements.

### From routing to reasoning and acting

[1:27](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=87s)

Belfer works at AI21 Labs, which develops language models and an agentic framework. His approach to defining an agent is pragmatic: whether a working system is called an agent or a workflow matters less than whether it solves the problem. That leaves several useful architectures on the table.

| Pattern | What the model does |
| --- | --- |
| LLM router | Sends a query to a specialized language model. |
| Function calling | Selects from provided tools to interact with APIs or services such as Google Search. |

Belfer introduces MCP as standardizing these external interactions, then leaves the protocol details aside. Tool access is useful regardless of whether the surrounding system earns the agent label.

[**ReAct**](https://arxiv.org/abs/2210.03629), short for reasoning and acting, adds an iterative loop: reason about the situation, take an action, observe the result, and repeat. Belfer characterizes this as choosing the next step from the current state rather than looking ahead across the entire plan. That is a distinction about the planning horizon of the pattern he describes, not an inherent prohibition on planning: the original ReAct work also discusses updating action plans.

### A plan needs an execution engine

[3:12](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=192s)

**Planning means figuring out the steps needed to reach a goal.** It becomes useful when the problem has no straightforward solution, particularly when work needs to run in parallel or the system needs to explain its choices. Belfer's concern with a ReAct trace is that seeing the sequence of thoughts and actions does not necessarily explain why that route was chosen.

Plans can take different forms:

- **Text:** Belfer points to Microsoft's [Magentic-One](https://www.microsoft.com/en-us/research/articles/magentic-one-a-generalist-multi-agent-system-for-solving-complex-tasks/) as an example of text-based planning.
- **Code:** He points to Hugging Face's [smolagents](https://huggingface.co/blog/smolagents), where agents can express actions as executable code. This is a useful contrast in representation, rather than an exclusive product taxonomy; smolagents also supports conventional tool calling.

A **dynamic plan** can change during execution. Instead of committing to the initial sequence until completion, the system reassesses whether the plan is still useful and replans when necessary. That makes planning an ongoing responsibility rather than a one-time preamble to the task.

The execution engine makes the plan operational. It analyzes dependencies between steps, identifies work that can run in parallel, and manages trade-offs between speed and cost. Belfer also names branch prediction as a possible latency optimization, without specifying its implementation. [AI21 Maestro](https://www.ai21.com/blog/maestro-ai-planning-orchestration/) brings these two responsibilities together: a planner chooses how to proceed, while a smart execution engine manages the work.

![Slide titled “Dynamic Planning + Smart Execution” listing execution-engine responsibilities: analyzing dependencies, enabling parallel execution, and branch prediction for better latency.](https://ai.engineer/api/talk-essay-media/85e4df895ca60d2ab0308eaf61d46d1da61b64ad9e84da21beabdcc0b1d28e15)

Dynamic planning plus smart execution: dependencies, parallel execution, and branch prediction.

### Turn requirements into a search and repair process

[4:49](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=289s)

Maestro's simplified instruction-following example starts by separating the **context and task** from the **requirements**. The requirements include no more than ten paragraphs, a formal tone, and positive mentions of the brand. These are the same conditions a developer might otherwise bury in the prompt; making them explicit makes them easier to validate.

The paragraph limit illustrates the benefit of a separate check. For text whose paragraphs are separated by blank lines, a small Python validator can turn that requirement into a concrete result:

```python
import re


def check_paragraph_limit(text: str, maximum: int = 10) -> dict:
    paragraphs = [
        part for part in re.split(r"\n\s*\n", text.strip())
        if part.strip()
    ]
    return {
        "requirement": "paragraph_limit",
        "maximum": maximum,
        "observed": len(paragraphs),
        "passed": len(paragraphs) <= maximum,
    }
```

Tone and the sentiment of brand mentions require different checks: counting paragraphs does not establish that either semantic requirement was met. Separating requirements gives each condition a place in validation.

The execution graph then turns generation into a search over possible answers. At each stage, the planner and execution engine choose candidates worth continuing. **Best-of-N** supplies diversity: generate several high-temperature samples from one model, or obtain candidates from different models, instead of relying on a single answer.

The remaining loop controls where additional effort goes:

1. Discard candidates that do not look promising.
2. Continue with the best candidate or candidates within a predefined budget.
3. Validate those candidates, fix failures, and repeat the improvement process.

The diagram's branches represent successive candidates and repairs, not a requirement to exhaust every possible path. Candidate selection concentrates the available budget on answers that appear worth improving.

![AI21 Maestro diagram with prompt+requirements branching to C1, C2, and C3; validate+fix branches lead to C4, C5, C6, and orange C7. A purple “Best candidate” annotation appears beside C1.](https://ai.engineer/api/talk-essay-media/749eebed81306bb30afea65886ae4a40a2817394989da7d910fc29c64d84aef6)

Maestro’s candidate graph highlights the best candidate and repeated validation and repair.

### Choose a path, then reduce the results

[6:05](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=365s)

For a more complex query, the same architecture chooses among alternative execution tracks. The execution engine estimates **expected cost, latency, and success probability** for those tracks, and the planner uses the estimates to select a path. The displayed graph expands the choices across retrieval parameters, model choices, and model parameters.

At the end, a **reduce** step turns the accumulated results into an answer. It can select the best result or combine several results into a complete response to the original query. The highlighted routes in the final diagram converge toward validation and the complete answer: branching creates alternatives, while reduction brings the work back to one deliverable.

![Branching diagram of retrieval parameters, model choices, and model parameters, with red highlighted paths converging before “Validate” and “Complete Answer.”](https://ai.engineer/api/talk-essay-media/9e6824338680bbd8b6ef633b97c82fac8b4620f867047df42d4533b98cce9f11)

Highlighted execution paths converge on validation and a complete answer.

### Higher quality costs additional inference

[6:41](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=401s)

Belfer reports improved [IFEval](https://arxiv.org/abs/2311.07911) results using Maestro's planner and execution engine with **GPT-4o**, **Claude Sonnet 3.5**, and **o3-mini**. IFEval evaluates instruction following through verifiable requirements. The spoken comparison gives no numerical scores, model snapshots, scoring variant, or inference budgets.

On an internal requirement-satisfaction dataset built from customer data, Belfer also reports improvement over a single LLM call, at the cost of more runtime and money. The internal evaluation is distinct from IFEval, and the talk provides no numerical cost or latency measurements. The engineering trade-off is explicit: candidate generation, validation, and repair buy quality by doing additional work.

### Use planning when the task requires it

[7:20](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=440s)

The practical ending is an adoption ladder, not a mandate to put a planner around every model. Start with the simplest system that works. If a small language model is sufficient, use it. If a model with tools handles the task, stop there. If ReAct works, use that.

Move to a planner and execution engine when the task's complexity requires that additional coordination. Belfer closes by inviting developers to the Maestro waitlist—the access path offered in the recording—to see whether the system fits their needs. The decision is whether explicit planning, validation, and controlled execution solve a problem that the simpler architecture cannot.

## Resources

- [InstructGPT and the moon-landing example](https://openai.com/index/instruction-following/) — OpenAI’s original announcement explains instruction tuning and includes the child-friendly moon-landing example.
- [ReAct](https://arxiv.org/abs/2210.03629) — The original paper on interleaving reasoning and actions during interactive tasks.
- [Microsoft’s Magentic-One](https://www.microsoft.com/en-us/research/articles/magentic-one-a-generalist-multi-agent-system-for-solving-complex-tasks/) — Explains an orchestrator that maintains task progress, delegates work, and revises plans.
- [Introducing smolagents](https://huggingface.co/blog/smolagents) — Hugging Face’s introduction to agents that express actions as executable code, with examples.
- [Maestro: Making Complex Tasks Simple](https://www.ai21.com/blog/maestro-technical-overview/) — Yuval Belfer explains Maestro’s candidate search, validation, repair, execution budgets, and evaluation methods.
- [Instruction-Following Evaluation](https://arxiv.org/abs/2311.07911) — Introduces IFEval and its use of automatically verifiable instructions to evaluate model outputs.
- [IFEval code and data](https://github.com/google-research/google-research/tree/master/instruction_following_eval) — Google Research’s implementation and dataset for instruction-following evaluation.
- [Maestro Validated Output walkthrough](https://docs.ai21.com/docs/walkthrough-guide) — Documentation for defining requirements, setting budgets, and inspecting validation reports.
- [Maestro enterprise availability](https://www.ai21.com/blog/maestro-ai-enterprise-knowledge-work/) — AI21’s July 2025 announcement making Maestro available to enterprise customers.

## Slides

### Aligning language models to follow instructions

In 2022, OpenAI introduced InstructGPT models trained with humans in the loop and deployed them as the API’s default language models.

[0:00](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=0s) [OpenAI’s 2022 InstructGPT announcement and a statement about human-in-the-loop training.](https://assets.ai.engineer/sites/aiecode2025/sha256/b419aec0a16b620316ac60bf44a4a2a77557c8fe8b86fbe45996c458c8d7d2de.webp)

Aligning language models to follow instructions

OpenAI
January 27, 2022
Publication

These InstructGPT models, which are trained with humans in the loop, are now deployed as the default language models on our API.

Interface text:
AI21labs
ON AIR

### GPT-4.1 instruction-following evaluations

GPT-4.1 scores 49% on OpenAI’s internal instruction-following evaluation and 87% on IFEval. Other models score higher on both, illustrating the speaker’s point that instruction following remains difficult.

[0:16](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=16s) [GPT-4.1 announcement above two instruction-following accuracy charts.](https://assets.ai.engineer/sites/aiecode2025/sha256/a499a2bbc9490582a4a0a37013cc7e11ca797065a8737982699d3d547100801e.webp)

AI21labs

OpenAI
April 14, 2025
Product  Research  Publication

Introducing GPT-4.1 in the API

A new series of GPT models featuring major improvements on coding, instruction following, and long context—plus our first-ever nano model.

Internal OpenAI Instructions following eval accura[illegible]
GPT-4.1: 49%
GPT-4o (2024-11-20): 29%
OpenAI o1 (high): 51%
OpenAI o3-mini (high): 50%
GPT-4.5: 54%
GPT-4.1 mini: 45%
GPT-4.1 nano: 32%
GPT-4o mini: 27%

IFEval accuracy
GPT-4.1: 87%
GPT-4o (2024-11-20): 81%
OpenAI o1 (high): 92%
OpenAI o3-mini (high): 94%
GPT-4.5: 88%
GPT-4.1 mini: 84%
GPT-4.1 nano: 75%
GPT-4o mini: 78%

Interface text:
ON AIR

### An early InstructGPT example

An early InstructGPT example answers a simple moon-landing prompt with a brief account of people taking pictures and sending them back to Earth. The speaker recalls how impressive this seemed at the time.

[0:33](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=33s) [A short moon-landing prompt alongside its completion.](https://assets.ai.engineer/sites/aiecode2025/sha256/0f963b0f9791b278781a40eee43c94e4490c3569f59220c6f65d00ce120504ae.webp)

How is it still a problem?

In the good old days of InstructGPT...

Prompt
Explain the moon landing to a 6 year old in a few sentences.

Completion
People went to the moon, and they took pictures of what they saw, and sent them back to the earth so we could all see them.

ON AIR

### A more constrained prompt

The moon-landing prompt adds child-friendly vocabulary and a 10-word sentence limit, illustrating how developers pack constraints into a single instruction.

[0:53](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=53s) [A moon-landing prompt with vocabulary and sentence-length constraints.](https://assets.ai.engineer/sites/aiecode2025/sha256/e765aa862eb5f9e583a89d34409638d56a7942cee90046494b9f3e625024b07b.webp)

How is it still a problem?

Nowadays

Prompt

Explain the moon landing to a 6 year old in a few sentences. Use only words a 6-year-old would understand (avoid technical jargon like "spacecraft," "lunar module," or "trajectory"). Keep sentences short (maximum 10 words per sentence).

ON AIR

### Real AI Agents Need Planning, Not Just Prompting

Yuval Belfer argues that putting all the context and constraints into one prompt is insufficient; AI agents also need planning.

[1:11](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=71s) [Title slide for Yuval Belfer's talk on AI agent planning.](https://assets.ai.engineer/sites/aiecode2025/sha256/4ed405d50ade4685d6e068eb672e803da7778ce5ef8f9cac57f035e3fe2a319f.webp)

Real AI Agents Need Planning,
Not Just Prompting

AI21labs

→ Yuval Belfer, Sr. Developer Advocate, AI21 Labs

ON AIR

### What Is an Agent?

Belfer sidesteps the philosophical debate over what counts as an agent: as an engineer, he cares about making it work, whether it is called an agent or a workflow.

[1:42](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=102s) [A Batman and Robin meme contrasts defining an agent with making it work.](https://assets.ai.engineer/sites/aiecode2025/sha256/57441dce1f1bc2f2074d206b0f019ac2a60b993ff7cb684a40df8917d8fb80ac.webp)

What Is an Agent?

WEBSTER'S DICTIONARY DEFINES AN AGENT AS...
JUST MAKE IT WORK

imgflip.com

Interface text:
ON AIR

### LLM as a Router

A routing model directs each query to a specialized language model. The diagram shows three possible domains; some people call this arrangement an agent.

[1:52](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=112s) [A router directs a query to one of three specialized language models.](https://assets.ai.engineer/sites/aiecode2025/sha256/cba1e2edb00eb8b0f9fbfd5fe5c4ff1b6b0d497b542075b04063a2ba07b38b7c.webp)

Agents

LLM as a Router

• Using a smart model to direct the query to a specialized model (usually SLM)

Router
LLM Domain A
LLM Domain B
LLM Domain C

Interface text:
ON AIR

### Agents

Function calling gives an LLM a list of external tools and lets it request one. The developer still has to build the system that uses the resulting function call. The speaker treats the “agent” label as less important than making it work.

[2:04](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=124s) [A meme asks whether the depicted example is an AI agent.](https://assets.ai.engineer/sites/aiecode2025/sha256/48dbc71cf89130d2048bfde7fd56631a281bfcc3b2829e140ffb9dfa4c265c99.webp)

Agents

LLM as a Router
• [illegible]cialized

Is this AN AI AGENT?
imgflip.com

Interface text:
ON AIR

[2:07](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=127s) [Slide explains providing external tools to an LLM and handling its function calls.](https://assets.ai.engineer/sites/aiecode2025/sha256/135cc7737b623334492b97acfabd86fe16032bfe1addae1fcc47b80f0702e1f7.webp)

Agents

LLM as a Router

Function calling
• Providing an LLM with a list of external tools it can use
• If it chooses to use one, the output will be a function call
• It’s the developer’s responsibility to build the system utilizing this feature

Interface text:
ON AIR

### Reasoning and Actions

ReAct cycles through thought, action, and observation. The agent acts on its current situation and uses the environment’s response to choose a next step, without looking ahead to an entire plan.

[2:35](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=155s) [Slide shows a thought, action, and observation cycle between an agent and its environment.](https://assets.ai.engineer/sites/aiecode2025/sha256/27b4e8a8c93ada8b9e3ca2b459b9367fa24e8d66707b8d595ec3d4cb299aecd4.webp)

Agents

LLM as a Router

Function calling

Reasoning & Actions (ReAct)
• Thought → Act → Observation

Action
Eat cake.

Thoughts
I’m hungry.
Agent (LLM)

Environment

The cake is a lie.
Observation

https://medium.com/@bryan.mckenney/teaching-llms-to-think-and-act-react-prompt-engineering-eef278555a2e

### Planning

Planning maps steps toward a goal for complex tasks, including those involving parallelization or explainability. Examples include the text-based Magentic-One and code-based smolagents.

[3:11](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=191s) [Slide defines planning and lists use cases and text-based and code-based planner examples.](https://assets.ai.engineer/sites/aiecode2025/sha256/6320f4b94b6f03849dd622c5983627146fbe78193d5647cd9cd2b25f72d3122e.webp)

Planning

Definition
• Figuring out what steps to take to reach your goal

When to use it?
• Non-straightforward problems (complex tasks)
• Tasks requiring parallelization
• When explainability matters

Format Types
• Text-based planners
  ◦ Example: Magentic-One (Microsoft)
• Code-based planners
  ◦ Example: smolagents (🤗)

### Dynamic planning

Dynamic planning leaves room to reassess a plan midway and replan if needed, rather than following the original plan all the way through.

[3:54](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=234s) [Slide defines dynamic planning as the option to replan.](https://assets.ai.engineer/sites/aiecode2025/sha256/0cbf7f1e751fb169205e9f1b546b3b9839bc36183e3386ad317887386067db65.webp)

Dynamic Planning

Definition

When to use it?

Format Types

What’s Dynamic?
• The option to replan

Other visible text: ON AIR

### Dynamic planning and smart execution

An execution engine analyzes dependencies between steps to enable parallel execution. It can also use branch prediction to improve latency.

[4:12](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=252s) [Slide lists dependency analysis, parallel execution, and branch prediction as execution-engine capabilities.](https://assets.ai.engineer/sites/aiecode2025/sha256/d456638ba9f7fd132d0d8f2a59d8b079d02bb9a7db72ffbb3875e4a40d526b23.webp)

Dynamic Planning + Smart Execution

Definition

When to use it?

Format Types

What’s Dynamic?

Execution Engine
• Analyze dependencies between steps
• Enables parallel execution
• Branch prediction for better latency

ON AIR

### AI21 Maestro example

In this simplified Maestro example, a prompt generates three candidates. The best candidate branches into validation and repair, and one branch goes through another iteration.

[4:38](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=278s) [A prompt generates three candidates; the best candidate branches into validation and repair steps.](https://assets.ai.engineer/sites/aiecode2025/sha256/fcfb021748b761d75a6de023358acdafb234997e807aca8566912e3d27a5b224.webp)

Example: AI21 Maestro

prompt+requirements

generate → C1
generate → C2
generate → C3

Best candidate: C1
validate+fix → C4
validate+fix → C5
validate+fix → C6
C4 → validate+fix → C7

Interface text:
ON AIR

### Maestro execution paths

The planner chooses among paths through retrieval settings, model choices and parameters using expected cost, latency and success probability. Results can then be selected or combined into an answer.

[6:04](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=364s) [A branching diagram connects paraphrased queries to retrieval choices, model choices, and model parameters.](https://assets.ai.engineer/sites/aiecode2025/sha256/3dd13eabf517491fd0a3148a6bfde398506570c887b7cd6e02aa0d37e8f958e8.webp)

Paraphrase queries

Retrieval parameters choice
Retrieval parameters choice
Retrieval parameters choice

Model choice
Model choice
Model choice
Model choice
Model choice
Model choice
Model choice
Model choice

Model parameters
Model parameters
Model parameters
Model parameters
Model parameters
Model parameters
Model parameters
Model parameters
Model parameters
Model parameters
Model parameters
Model parameters
Model parameters
Model parameters
Model parameters
Model parameters

ON AIR

### IFEval results

Maestro’s code-validation approach scores higher than a single LLM call across all three models: 91.9% for GPT-4o, 96.1% for Claude Sonnet 3.5, and 95.7% for o3-mini.

[6:40](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=400s) [Bar chart compares LLM calls and two Maestro validation approaches across GPT-4o, Claude Sonnet 3.5, and o3-mini.](https://assets.ai.engineer/sites/aiecode2025/sha256/96da771ff363e7e2e8cae5450c1d95b36cb11f5e8a8e10f0a299c85a50487d9e.webp)

IFEval

Legend: LLM Call; Maestro (LLM-as-a-Judge); Maestro (Code Validation)
Y-axis: % Accuracy
Y-axis ticks: 80%, 85%, 90%, 95%, 100%
Models: gpt-4o; Claude Sonnet 3.5; o3-mini
Visible values: 91.9%; 96.1%; 95.7%

Interface text: ON AIR

## Full transcript

[0:00](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=0s) The year is 2022. OpenAI just released InstructGPT, which is the first model who can take instructions and actually follow them, really do what you want it to do. They have even called their model InstructGPT.

[0:16](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=16s) So how come three and a half years later, in 2025, when they release GPT-4.1, they still have trouble to follow instruction? And it's not just them, right? Every language model has some problems with following instructions.

[0:34](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=34s) And why is that? So if you remember the good old days of InstructGPT, this was one of the examples that blew everybody's mind, right? Explain the moon landing to a [REDACTED:age].

[0:45](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=45s) We got something nice, and everybody went bananas. And then with time, more and more developers started to use language models, and now the instruction looks more something like this,

[0:58](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=58s) or this, or this. Right? And that's what everybody's doing, just taking all the information, all the context, all the constraints, all the requirements, and just put everything in one prompt and hope for the better.

[1:13](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=73s) And really, even for simple tasks such as instruction following, LLMs alone are no longer enough, and this is where AI agents come to play. And agents don't just use prompting, they need planning.

[1:27](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=87s) I'm Yuval. I work at AI21 Labs, where we developed our own language models and also an agentic framework, and I want to talk to you about planning. So let's maybe start first with what is an agent.

[1:40](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=100s) There's a philosophical debate, everybody's wondering, and the truth is, as an engineer myself, I don't care. Whatever works, just make it work, call it an agent, call it a workflow, call it whatever.

[1:54](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=114s) Some people call LLM as a router an agent, where we have some sort of a routing model that just takes any query and directs it into a specialized LLM.

[2:05](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=125s) Is, is this an AI agent? Who cares? We also have function calling, right, where essentially we're providing the LLM a list of external tools that it can use and let it interact with other APIs, with the world, Google Search, uh, where this is something that now a lot of people talk about MCP, where it standardize this and

[2:27](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=147s) everything MCP related is an agent. Again, doesn't matter, as long as it works, and there's a lot of information about MCP right now, so we will not get into it.

[2:36](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=156s) And maybe the most popular agent, uh, of it all is ReAct, Reasoning and Action, where this is a framework that can be done with any language model in the framework of thought, then act upon that thought, and observe, and so on, and so on.

[2:52](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=172s) Every time the model wants to do some sort of a step towards, uh, the solution, but this is something that happens each step at a time, and there is no look ahead to the entire plan.

[3:04](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=184s) Right? This is kind of like part of a plan. I know the situation right now, I know what the next step I want to go to, and so on, and so on.

[3:12](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=192s) And what is planning that, that we're talking about, that every AI agent essentially needs to do? So it's figuring out the steps that you have to take to reach your goal.

[3:24](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=204s) And when do you want to use it is when there is no straightforward problems, essentially complex tasks, and that it requires parallelization and explainability, right? Rather than ReAct, where you can't really understand what happened, you can just see all the thoughts that happened throughout, but you don't really know why.

[3:43](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=223s) You can talk about forms-based, uh, planners, such as text bl- based, as Magentic-One by Microsoft, or code-based planners by smolagents, uh, from Hugging Face.

[3:56](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=236s) What do you mean when I say dynamic planning, is that there is an option to do replan. And essentially, it's not just have one plan and go all the way in this plan.

[4:06](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=246s) In the middle, you can ask yourself, "Mm, is this plan good, or should I do replanning and go somewhere else?" And when we talk about efficiency, we want to put smart execution into that.

[4:18](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=258s) Essentially, every planner need an execution engine because an execution engine can analyze dependencies between steps, which enables paral- parallel execution, and can also have some sort of a trade-off between speed and cost.

[4:34](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=274s) For instance, you can use branch prediction for s- uh, faster systems. Let's take a look of a simplified version of AI21 Maestro, which is a system that uses both a planner and smart execution engine.

[4:49](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=289s) So you can see here the very simple case of instruction following. Really, we have a prompt, which is essentially some context and the task, and the requirements are all those, uh, shouldn't be more than 10 paragraphs, the tone should be formal, all mentions of our brand should be positive, and so on, and so on.

[5:08](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=308s) The thing that you usually just shove in the prompt, now they are separated, so it will be easier to validate. And you can see that in every step here, we have some sort of a, an execution tree or execution graph, if you will, where in every step the, uh, planner and the execution engine chooses several candidates and

[5:27](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=327s) only continue to fix and improve those who seem promising. And really, there are several techniques here that you can use for any system, such as using Best-of-N, where essentially instead of one generation, we sample from any LLM with high temperature, several generations, or maybe use different LLMs.

[5:49](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=349s) We also ditch the candidates that do not seem like they will be good and only pursue the best candidate or candidates based on predefined budget. And also, we have some sort of validation, so iterations, and keep fixing, and so on, and so on.

[6:05](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=365s) Here is how it looks in something where it may be a bit more complicated, where again we have the input, the original, uh, input of the system, and then we have those, uh, tracks where the execution engine can say what is the expected cost, the expected latency, latency, and success probability, and the planner will choose accordingly the

[6:27](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=387s) right path to go. Where at the end, we will have some sort of a reduce, where we will have a lot of, uh, results, and we want to take the best one or combine all of them for a complete answer to the query that we had in, in the beginning.

[6:41](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=401s) And you can see that it actually works. Right? You can actually see that for IFEval, for instance, uh, we get really high results for GPT-4o, or Claude Sonnet 3.5, or o3-mini.

[6:54](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=414s) We're getting very, very high results by just improving them and using our planner and smart execution engine. Also, for requirement satisfaction, which is a data set that we build internally based on customer data, again, you can see that we improve upon just a single call for LLMs, and we are paying a little bit here, uh, with, right,

[7:14](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=434s) more runtime and more money, but we get higher quality.

[7:20](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=440s) So this is really the takeaways here, that LLMs alone are not always enough, even for what we call simple tasks, such as just instruction following. I will always say start simple and then use whatever work.

[7:32](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=452s) So if you can use SLMs, use SLMs. If you can just take a model and put, uh, some tools into it, have at it, that's great. Use ReAct if you can.

[7:42](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=462s) And if you really need, if your task is so complex you have to use planning and execution e- engine, this is where you should go and explore, and I do invite you to join the Maestro waitlist to check it out and see if it works for you.

[7:57](https://www.youtube.com/watch?v=Th5e4h-oVmc&t=477s) Thank you.
