AI Engineer World's Fair 2025
How agents will unlock the $500B promise of AI
Loading slides…
Read the talk
How agents will unlock the $500B promise of AI
A small tool-calling loop can power an agent, but enterprise value depends on permissions, evaluation, observability, and choosing which systems deserve custom engineering.
From a talk by Donald Hruska
Before you start: Basic familiarity with JavaScript functions, API calls, and language models will help with the agent-loop example.
Where does the infrastructure investment become useful work?
Why are enterprises still experimenting with chatbots when AI infrastructure investment is already enormous? Donald Hruska opens with that mismatch, citing half a trillion dollars spent on infrastructure while many large companies remain at toy chatbots and code generation. At Retool, where he leads new product teams, the starting point is internal applications and integrations with AI providers. The newly released Retool Agents extends that work toward agents that connect to production systems with guardrails. The missing step is turning model capability into dependable business operations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Coding shows what changes when models can act
The revenue figures establish the launch-era context. Citing Reuters, Hruska reports that Anthropic's annualized revenue rose from $1 billion in December to $2 billion at the end of March and $3 billion at the end of May, tripling in five months. He contrasts that run rate with OpenAI's projected total 2025 revenue of $12 billion, more than three times the previous year's revenue. These are different measures—an annualized run rate and a full-year forecast—and Hruska attributes much of the growth to enterprise spending.
Coding is where he already sees workflows changing. His engineers use Cursor or Windsurf, spending more of their effort on prompting and code review while models handle everyday implementation. He describes the productivity improvement qualitatively. The same concentration appears in his snapshot of OpenRouter: a unified API exposing hundreds of models, with a top-app list dominated by code generation.
Providers are investing in that demand. On SWE-bench Verified, Hruska reports an approximately 21-percentage-point gain for GPT-4.1 over GPT-4o, followed by roughly nine more points for Gemini 2.5 Pro over GPT-4.1. SWE-bench Verified measures repository issue resolution, and these historical results depend on the surrounding prompts and tools: OpenAI excluded tasks that could not run on its infrastructure, while Google's result used a custom agent setup. The figures show the direction of investment, not a controlled comparison under a shared harness.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From an idea to an agent
Hruska invokes Rick Rubin's description on the Andreessen Horowitz podcast: “Vibe coding is the punk rock of software.” The analogy is about access. Just as punk's simplicity let someone with something to say make a song, coding agents let someone with an idea begin making software. A rough request to Cursor or Windsurf starts a sequence of deliberation, actions, and code changes. That differs from completing text or copying a ChatGPT answer into an editor: the system participates in carrying out the task.
The next question is whether that pattern can extend to other business problems. Code has an advantage: it has defined semantics and can be tested, giving the agent and its operator ways to check a result. General-purpose business agents need their own ways to validate actions and outcomes. The execution pattern transfers more readily than the assurance that the work is correct.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The agent is an execution loop
Hruska estimates that a basic agent can fit in about 100 lines of JavaScript or Python. His example uses ReAct: interleave reasoning and action until the model reaches a final answer. Tools are functions, whether they call an external service or execute application code. The resulting agent is an LLM inside a loop that can read, decide, call tools, and check the feedback.
The demonstration starts with a calculator tool, its calculation function, and a system prompt establishing the ReAct behavior. The tool definition tells the model what operation is available; the function is what the application actually executes. Keeping those roles separate matters: requesting a calculation does not itself perform one.
The control flow is a bounded for loop:
- Ask the model for its next action or final answer.
- If it requests a tool, execute that function and append the result to the conversation.
- Let the model use that feedback on its next turn.
- Return a final answer when one arrives, or stop when the iteration budget is exhausted.
This JavaScript version makes that application-side contract explicit. nextStep is the model adapter: it receives the conversation and tool definitions, then returns either a tool request or a final answer. For an input such as What is 17 × 23?, the calculator can return 391 as feedback for the model's next turn.
javascript
const tools = {
calculator: {
description: "Multiply two finite numbers",
parameters: {
type: "object",
properties: {
a: { type: "number" },
b: { type: "number" }
},
required: ["a", "b"],
additionalProperties: false
}
}
};
function calculator({ a, b }) {
if (!Number.isFinite(a) || !Number.isFinite(b)) {
throw new Error("Expected two finite numbers");
}
const result = a * b;
if (!Number.isFinite(result)) {
throw new Error("Calculation overflow");
}
return result;
}
export async function runAgent(input, nextStep, maxSteps = 8) {
const messages = [
{
role: "system",
content: "Use tools when needed. Check their results, then answer."
},
{ role: "user", content: input }
];
for (let step = 0; step < maxSteps; step++) {
const decision = await nextStep({ messages, tools });
if (decision.type === "final") {
return decision.answer;
}
if (decision.type !== "tool" || decision.name !== "calculator") {
throw new Error("Unsupported action");
}
messages.push({ role: "assistant", content: decision });
const result = calculator(decision.arguments);
messages.push({
role: "tool",
name: decision.name,
content: String(result)
});
}
throw new Error("Agent reached its iteration limit");
}
The iteration limit prevents the agent from repeatedly thinking and acting without an end, accumulating model costs. Tool results re-enter the conversation; the application then detects and returns the model's final answer.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What the working prototype leaves out
A working loop is only the beginning of deployment, much as a quickly generated web application still needs production engineering. Enterprise requirements include single sign-on, role-based access control, secure service integrations, audit logs, SOC 2 compliance, secrets management through systems such as AWS Secrets Manager, and internationalization. Hruska cites reporting from The Information about vulnerabilities shipped through insufficiently reviewed AI-generated code. These surrounding systems cannot be treated as incidental details just because generating the application became easier.
Agents also introduce operational risks within the loop itself. Outputs can be fabricated, inaccurate, or unpredictable. Access must be constrained to the resources the agent should use, and token consumption needs limits. Evals are a safeguard for more predictable behavior, not a way to make a probabilistic model mathematically deterministic. They help establish whether the agent behaves acceptably across the cases the business cares about.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Four ways to deliver an agent
The implementation choice determines how much of that surrounding system your team must own. Hruska groups the options into four categories:
| Approach | Control and flexibility | Engineering burden |
|---|---|---|
| From scratch | Maximum ownership; purpose-built behavior | High, including ancillary systems |
| Framework, such as LangGraph | Substantial control within a framework | Medium |
| Agent platform, such as Retool Agents | Opinionated defaults; platform dependence | Lower effort to production |
| Vertical agent | Tuned for one use case; little flexibility beyond it | Turnkey within its specialty |
Building from scratch can include fine-tuning and dedicated AI or ML engineers. A framework preserves substantial configurability, including choices about memory, while tying the implementation to its abstractions. A platform takes on hosting, service connectors, and fleet observability, which can suit the long tail of business workflows. A vertical offering goes further by concentrating on one task that it can perform particularly well.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Spend engineering effort where ownership matters
The decision starts with the value of controlling the implementation:
- Core product or competitive advantage: Hruska favors building, because the agent's behavior is part of what differentiates the business.
- Sensitive data, regulation, or hard SLAs: Examine both approaches against the actual requirements; neither route automatically satisfies them.
- Commodity workflow with a short deadline: Buying can make more sense when the business needs a result in days rather than quarters.
Then assess the operational risk. Is the best use of the team's time debugging business logic, or diagnosing a broken OAuth integration at two in the morning? Owning an implementation also means owning the failure modes around it.
For a managed platform, inspect what is included rather than assuming the platform removes every integration task. Check whether Salesforce, Databricks, and Snowflake connections are available out of the box. Then examine permissioning, compliance, audit trails, observability, and evals. A missing capability can mean custom engineering or another vendor to purchase and operate. Compare token costs, infrastructure costs, and engineering costs together.
Once agents are deployed, visibility needs two levels. At the fleet level, track token usage, estimated cost, and runtime information. From there, drill into a specific agent and an individual run to inspect whether it did what you expected. Aggregate spending tells you that something changed; a run-level view lets you investigate the behavior behind that change.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A few custom agents, a long tail of managed workflows
This can produce a mixed portfolio, much like ordinary business software. Hruska uses Stripe as an analogy: its core billing logic and critical customer applications warrant custom development, while external platforms serve internal needs. He describes Stripe using React for much of its customer-facing software and Retool for much of its internal tooling. His expectation is that businesses will similarly maintain a few purpose-built agents alongside many platform-hosted business agents.
Cursor makes the distinction especially clear. Its coding agent is its product, so control, ownership, iteration speed, and attention to implementation details are central to the business. But as the company grows, chargeback disputes and customer-support requests could become substantial operational workloads. Hruska offers those as hypothetical candidates for an agent platform, not as examples of Cursor deployments already in use.
Hruska describes working with AWS on automating mundane business processes, then points to existing Retool customers. Hruska reports that ClickUp saved over $200,000 in vendor costs and hundreds of thousands of dollars in additional headcount through its Retool AI tooling. These are customer savings claims, rather than a benchmark of the newly launched agent product.
Hruska reports Descript's estimate that its 50 apps save hundreds of work hours weekly. He also cites Retool's announcement that customers had automated over 100 million cumulative hours of work. That broader Retool milestone predates the impact of the newly launched Agents product; it is not an Agents-only result.
The intended benefit is more capacity for creative and strategic work. Hruska compares the concern about AI to fears that the printing press would undermine traditional knowledge: wider access to information instead expanded what people could do. He expects agents to enhance teams' capabilities and ultimately increase global GDP. That is the economic ambition behind the automation examples, rather than a measured outcome of them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Cheaper inference sharpens the engineering question
Hruska cites Mary Meeker's AI Trends Report for a 99.7% decline in cost per token from 2022 to 2024. The talk does not specify the model, workload, or quality baseline behind that figure. His economic observation is that provider revenue can rise rapidly while the marginal cost of inference falls. At the time of the talk, Hruska says Retool's cheapest agent cost $3 per hour. He expects that price to keep declining.
He also cites the Meeker report for an elevenfold increase in Google searches for AI agents over 16 months. But growing interest does not answer the deployment question. The useful question is where engineers can create the most value, and which tool fits that work—not how to find one system that puts an entire business on autopilot.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What Retool had decided—and what was still ahead
The first audience question tests the build-versus-buy framework against Retool itself: which core systems does it build by hand, and where does it use its own agents? Hruska says Retool already builds much of its internal software on its platform and prioritizes using its own product. But Agents had launched only the previous week, so the split between custom agents and platform implementations remained undecided. He offers no concrete core-agent example. Retool's company-specific policy is to use the platform wherever possible, then investigate and build missing capabilities when something cannot be done.
A second question, from someone building government and NGO applications, asks about on-premises deployment. Despite an initial affirmative answer, Hruska clarifies that Retool Agents launched cloud-only. At this June 2025 talk, he expected on-premises support within one or two weeks, possibly three, and eventual support for air-gapped customers without a firm date. Those were roadmap commitments, not capabilities already available at launch—a consequential distinction for teams whose deployment environment determines whether they can adopt the platform at all.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The original research on agents that interleave reasoning with actions and environmental feedback.
Retool's May 2025 announcement connects its Agents launch with its reported automation milestone.
Further reading
OpenAI's April 2025 benchmark results, including SWE-bench Verified scores and evaluation limitations.
Google's March 2025 model announcement includes its SWE-bench Verified result using a custom agent setup.
A customer account of AI applications for go-to-market operations, including order-form approval and avoided software costs.
A customer account of more than 50 internal applications spanning operational tools and AI evaluation.
Full transcript
Loading transcript…