AI Engineer Summit 2025
OpenAI for VPs of AI
Loading slides…
Read the talk
OpenAI for VPs of AI
Start with business priorities, build measurable use cases, and let observed failures guide the move from a single agent to specialized networks with explicit safety gates.
From a talk by Prashant Mital and Toki Sherbakov
Before you start: Basic familiarity with language-model APIs, prompts, and retrieval will help with the agent architecture examples.
Getting models into everyday work
How does an enterprise turn access to powerful models into useful work for employees, internal operations, and customers? The problem extends beyond choosing a model: capabilities must become products, those products must fit real workflows, and deployment must generate feedback that improves the next iteration.
OpenAI’s division of responsibilities follows that path. Research builds foundational models. Applied engineering turns them into products such as ChatGPT and the API. Go-to-market teams help enterprises put those products into employees’ hands, automate operations, and integrate AI into customer-facing products. Feedback from those deployments then returns to both product teams and model research, closing the loop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Three surfaces for enterprise adoption
Enterprise adoption commonly begins by giving employees AI they can use every day. That builds AI literacy through practice. Internal automation follows: copilots and workflows begin doing more of the organization’s work. Customer-facing integration brings those capabilities into the product itself. These are common phases, not a mandatory sequence.
| Adoption surface | Purpose | Typical product surface |
|---|---|---|
| Workforce | Daily assistance and AI literacy | ChatGPT |
| Internal operations | Automation and customized copilots | ChatGPT; API for greater customization |
| Customer products | AI inside the end-user experience | Primarily the API |
The transition to the API is driven by the integration and customization a use case requires. Internal operations can begin within ChatGPT; more complex workflows need closer control over how AI fits into the surrounding application.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Business strategy before organizational scale
Start with the business strategy, then identify where AI can advance it. A top-down mandate is useful when it establishes priorities and constraints, rather than treating AI adoption as an independent objective. From those priorities, select one or two substantial, high-impact use cases and scope them tightly enough to deliver.
Once those initial use cases are underway, build the organization’s capacity to repeat the work. That can mean enabling teams, establishing a center of excellence, or creating a shared technical platform that other divisions can build on. The progression is from strategic direction to bounded delivery, then to broader capability—not an organization-wide platform built before the first use case is understood.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Define success before development
Toki Sherbakov describes an illustrative three-month use-case journey, not a delivery guarantee. Its first steps establish what the system must do and how the team will know whether it works:
- Ideate and scope. Choose the workflow and bound the problem.
- Review the architecture. Determine how AI fits into the existing stack.
- Define success. Establish metrics and KPIs before implementation becomes the main activity.
These decisions give development a target beyond producing a convincing demonstration.
Most of the work then goes into iteration: changing prompts, incorporating retrieval-augmented generation, and improving whichever parts of the system constrain the use case. OpenAI’s collaboration with engineering teams can include workshops, office hours, paired programming, and webinars to accelerate that development.
Testing returns to the evaluations defined upfront. A/B tests and beta rollouts establish how the application behaves in practice before a broader launch. Production rollout adds scale optimization testing, followed by continuing maintenance. Deployment is therefore another stage of the use-case lifecycle, with its own checks, rather than the point at which development simply stops.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Dedicated teams, bounded roadmap visibility
This partnership requires dedicated teams on both sides. Early access to models and features can help an enterprise prepare for capabilities that are approaching release, but that access does not imply a long, dependable forecast. Sherbakov describes his visibility at the time as roughly six months, or two quarters; he explicitly cannot supply an eighteen-month roadmap.
Research engineering and product experts can support the work, while joint roadmap sessions connect the customer’s plans to the capabilities OpenAI can see coming. The useful planning horizon is the one the participants can actually discuss, with ongoing alignment as those plans develop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Morgan Stanley: improving the whole retrieval system
Morgan Stanley’s internal knowledge assistant gives wealth managers a way to ask questions across a large collection of information, including research reports and stock-related data. The operational requirement is accurate information that advisers can use when responding to clients. A fluent answer alone does not satisfy that requirement.
Sherbakov reports an initial accuracy of 45%. The team introduced HyDE retrieval, fine-tuned embeddings, and different chunking strategies. These changes address the retrieval side of the application: how a question connects to relevant material and how that material is represented and divided for use.
| Reported milestone | Accuracy | Development context |
|---|---|---|
| After further retrieval work | 85% | Added reranking and classification |
| Customer target | 90% | Desired accuracy |
| Final reported result | 98% | Further prompt engineering and query expansion |
These are Sherbakov’s deployment-specific accuracy figures. The talk does not define the evaluation set or scoring protocol, and the sequence does not isolate the contribution of any individual technique. It illustrates cumulative improvement against a customer metric, rather than a general benchmark for retrieval systems.
The next step in this progression is moving from systems that answer questions toward systems that carry out work. The speakers frame 2025 as an expected year of agents, then turn from enterprise delivery to the architecture lessons emerging from those deployments.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
The model controls an execution loop
Prashant Mital draws on customer work and OpenAI’s own agentic products, including deep research and the original Operator research preview. His framing is a transition from an assistant to a coworker: an application that can take successive steps toward an objective.
An agent combines a model, instructions, access to tools, and an execution loop. Instructions usually arrive through a prompt. Tools let the application retrieve information or interact with external systems. The defining feature of this account is that the model controls when the loop terminates.
Within that loop, the application proceeds through a recurring cycle:
- Receive and interpret natural-language instructions.
- Determine whether to call tools.
- Execute the requested tools.
- Incorporate their return values into a response.
- Decide whether the objective has been met and execution should end.
The tool results become information for the next model decision. That makes the trajectory of the interaction—not just one generated answer—part of the system’s behavior.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Understand the primitives before abstracting
Suppose an agent must orchestrate models, retrieve data, reason over it, and generate an output. Building directly with primitives means making API calls and recording outputs and failures yourself. A framework can handle many of those details and make a proof of concept much faster to assemble. The risk is accepting its design choices before understanding the system’s behavior or constraints.
Build enough of the system directly to understand what needs improvement. Observe how the task decomposes, where failures arise, and which constraints prevent better results. Abstractions become useful when they remove repetition you have actually encountered—for example, repeatedly implementing an embedding strategy or model graders.
The decision is therefore not primarily which framework to choose. It is whether an abstraction improves a system whose data, failure points, and constraints the team understands. Start simply, optimize the parts that need it, and introduce abstraction when it makes the application better.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learn from one focused agent in production
The same reasoning applies to agent count. Agents calling other agents, coordinating tasks, and reasoning across long trajectories create many interacting unknowns when introduced too early. Mital recommends beginning with one agent, purpose-built for one task, and deploying it to a limited set of production users.
That release exposes bottlenecks that an architectural diagram cannot resolve:
- Conversation reliability: hallucinations may emerge over an extended interaction.
- Latency and adoption: a system may be capable but too slow for users to adopt.
- Retrieval accuracy: poor evidence retrieval may constrain the quality of answers.
Once the team understands how the application underperforms and what users value, it can improve the relevant parts incrementally. Complexity grows as new intents, failure cases, and constraints become visible.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Handoffs preserve context while changing specialists
When the task calls for multiple specializations, a network of agents can divide a larger workflow into related subflows. Each agent handles a particular part of the request. A handoff transfers control of the active conversation from one agent to another—like a phone transfer, but with the ability to carry the conversation history forward so the user does not have to start again.
The sample customer-service architecture assigns models according to the work they perform:
| Role | Model | Responsibility |
|---|---|---|
| Triage | GPT-4o mini | Route the incoming request |
| Dispute agent | GPT-4o | Manage the conversation with the user |
| Eligibility specialist | o3-mini | Check refund eligibility |
This is a sample architecture, not a reported production result. It illustrates how a small model can handle triage, a conversational model can manage a dispute, and a reasoning model can handle an accuracy-sensitive decision within the same service flow.
The mechanism is straightforward: retain the conversation history and context while replacing the active model, prompt, and tool definitions. The new agent receives the prior interaction but operates with the instructions and capabilities appropriate to its specialization. Full-history preservation is the pattern illustrated here, rather than a requirement that every handoff implementation must follow.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Run guardrails separately, gate consequential actions
Guardrails enforce safety, security, and reliability around the application. Keeping the main instructions simple and focused on the target task makes the system easier to compose and its accuracy and performance easier to improve predictably. Safety checks need not all become extra instructions inside that main prompt; they can run separately, in parallel.
Faster, cheaper models such as GPT-4o mini make separate checks more accessible. But parallel checking must be paired with an explicit action gate: a high-stakes tool call, such as issuing a refund, or a response disclosing personal-account information should wait until the relevant checks have completed. A proposed action is not yet an authorized action.
For a refund workflow, the separation can be expressed with a small Python orchestration function. Here, propose_refund only prepares a proposal; the side effect belongs exclusively to issue_refund. Every check returns a Boolean approval.
python
import asyncio
async def guarded_refund(
request,
*,
propose_refund,
input_check,
output_checks,
issue_refund,
):
proposal, input_allowed = await asyncio.gather(
propose_refund(request),
input_check(request),
)
if not input_allowed:
return {"status": "blocked"}
approvals = await asyncio.gather(
*(check(proposal) for check in output_checks)
)
if not approvals or not all(approvals):
return {"status": "blocked"}
return await issue_refund(proposal)
The input check overlaps with proposal generation. Output checks require the proposal and then run concurrently with each other. Rejection or a raised exception prevents execution from reaching issue_refund; speculative work before that point must remain free of consequential side effects.
The illustrated guardrail arrangement uses an input check for prompt injection and multiple checks on the agent’s response. The slide shows a user and an appointment agent, with hallucination and misinformation detection and a content-policy filter around the output. The architecture separates the task from its checks while retaining the requirement to defer consequential tool calls and responses until those checks return.
These design choices fit together: understand the primitives before adding abstractions, learn from one focused agent, introduce specialists as the set of intents expands, and keep task prompts focused while guardrails handle edge cases. The system becomes more capable through observed needs, with explicit boundaries around what it may actually do.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The original HyDE paper explains retrieval using embeddings of generated hypothetical documents.
OpenAI’s launch account of multistep research in ChatGPT, with later product updates marked separately.
The original browser-agent research preview, including human oversight and safety controls, with a later integration notice.
Further reading
A customer case study covering expert evaluations, retrieval improvements, regression testing and advisor adoption.
Original research code with a demonstration notebook and TREC DL19 experiment. Uses historical GPT-3 and Contriever tooling.
Updates since the talk
Current guidance for transferring conversation control and configuring the history passed to another agent.
Current SDK guidance on parallel and blocking input checks, output validation and function-tool guardrails.
Full transcript
Loading transcript…