AI Engineer World's Fair 2024
Level Up Your Automation Game: Agentic Workflows on Vertex AI
Loading slides…
Read the talk
Level Up Your Automation Game: Agentic Workflows on Vertex AI
Agents turn model capabilities into user-facing actions, but useful automation depends on how reasoning, tools, state and quality checks work together.
From a talk by Rukma Sen
Where do everyday users meet AI?
Where will developers, employees, parents and students actually interact with generative AI? Powerful models can still be intimidating or inaccessible to the people who could benefit from them. Rukma Sen’s starting proposition is that, for many everyday tasks, an agent will become the interface between the model and the user. The user describes what they need in natural language; the agent makes model capabilities accessible through that interaction. This is a prediction about interface design, not a claim that every AI application must become an agent.
Developers occupy a useful position at that boundary: they understand both model capabilities and the people they are building for. Their work includes designing the interaction and setting limits that make it helpful, humane and safe. Sen compares this opportunity with earlier changes in how people used technology: household machines and appliances, followed by internet-era screens and the familiar gestures of swiping, zooming and scrolling. Agent interfaces offer another opportunity to shape those habits.
That opportunity brings responsibility. Sen invokes Spider-Man’s familiar maxim before turning to the practical obligations of designing systems that mediate people’s interactions with AI. The limits around an agent are part of its interface, just as much as the way a user asks it for help.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Responsibility extends from outputs to stored data
The first obligation is protecting users. Guardrails should limit misinformation and make clear the difference between generated material and information presented as factual. An interface that makes an answer easy to obtain also needs to help people understand what kind of answer they have received.
The next questions concern cybersecurity and privacy: where is the data used in model reasoning stored, and how is it safeguarded? Sen highlights wearables and unobtrusive computing because these systems place computation throughout everyday life. As the interface becomes less conspicuous, the handling of personal data becomes more consequential.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A model, tools and orchestration
For the architecture discussion, Sen uses a deliberately broad working definition: an AI agent is a system designed to achieve specific goals by interacting with its environment. The definition establishes a practical scope without trying to settle every disagreement about what counts as an agent.
Three components make that system work:
- Model: Reasons about the goal, determines the next plan of action and guides behavior—the agent’s brain.
- Tools: Let the agent act beyond reasoning or generating text—the agent’s hands. Examples include retrieving information from the internet and calling external APIs to book flights or process payments.
- Orchestration: Maintains memory and state, keeps track of goals and coordinates the pieces—the nervous system connecting brain and hands.
The distinction between thinking and acting matters here. Tools give model decisions a way to affect the environment, while orchestration maintains the context in which those decisions occur.
Together, these components support autonomous task completion. But the category is broader than today’s generative systems: some kinds of agents existed long before the recent growth of generative AI.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Deterministic, generative and hybrid behavior
Sen separates agent behavior into three broad categories. A deterministic system follows fixed rules: a calculator given 2 + 2 returns 4. A generative system can produce varied responses and combine rules in new ways. Her example is a customer-service chatbot that uses company product information to compose a helpful answer. A hybrid system assigns different parts of the task to each approach.
| Type | Behavior | Example |
|---|---|---|
| Deterministic | Fixed rules; consistent output | Calculator |
| Generative | Varied, composed responses | Product-support chatbot |
| Hybrid | Rules plus generated communication | Financial-advisor workflow |
In Sen’s illustrative financial-advisor workflow, deterministic agents perform market analysis and identify investments, while a generative agent communicates the strategy to customers. The useful architectural distinction is the separation of analysis from explanation; the example does not establish the accuracy of investment predictions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From one model to specialized roles
A single-agent architecture puts reasoning, planning and action under one model. Give it instructions, a goal and a set of tools, and it handles the task. This is a straightforward architecture to implement, but obtaining an answer is not the same as obtaining a dependable result.
Sen illustrates the reliability problem by asking a model to count occurrences of the letter A in banana. In her anecdote, the model answers four; when asked to check, it changes the answer to two; asked again, it says one. Repeated prompting changes the output without establishing correctness. This is an illustrative failure, not a measured failure rate, but it exposes a production concern: self-checking does not necessarily produce a trustworthy check.
A multi-agent architecture distributes work among specialized roles, much as a company assigns different responsibilities to different people. Sen’s customer-service example follows three levels:
- Dispatcher: Assess the incoming customer issue and route the case.
- Subject-matter expert: Respond using expertise in a particular product line or region.
- Supervisor: Check the work against a predefined dataset and resolve some issues.
The architecture separates routing, answering and reviewing rather than asking one model to perform every responsibility within a single role.
Adding a supervisor creates another component whose behavior must be designed carefully. In one system Sen built, the supervisor was supposed to return “This is not good enough. Please try again” when dissatisfied. It kept rejecting the first agent’s work until she recreated the setup. No specific repair is demonstrated, but the anecdote supplies a consequential qualification: specialized review can itself become a source of failure rather than an automatic guarantee of quality.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Applications by audience, expertise and modality
The application categories Sen describes use several different organizing principles. Customer and employee agents are defined by whom they serve; knowledge agents by their expertise; voice agents by how people interact with them.
- Customer agents: Support, e-commerce, B2B interactions and travel services.
- Employee agents: HR enrollment, benefits questions, sales, accounts payable and supply-chain work.
- Knowledge agents: Specialized subject expertise, such as answering legal questions.
- Voice agents: Spoken interactions, including fast-food drive-through ordering.
Sen reports seeing increased interest in voice agents as multimodal applications expand, without attaching an adoption figure to that observation.
The drive-through example makes the interface concrete. A customer speaks an order; the agent transcribes it and sends it to the ordering system; a person at the delivery window serves the customer. The agent connects a natural human input to an existing operational system, while the physical fulfillment remains with staff.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Vertex AI across the development lifecycle
The closing platform overview places these architectures within Google Cloud’s Vertex AI. Sen presents it as a full-lifecycle development platform for calling models, fine-tuning them and building agents. Its development options range from no code and low code to highly customized implementations.
At the time of the talk, Sen reports access to more than 150 models. This is catalog breadth, not a performance measure or a promise of identical APIs and deployment conditions. The offerings she names include Google’s models, Anthropic models, Llama 2, Llama 3 and other open-source models. Her description of the Anthropic offering as including all its models is broader than the contemporary announcement establishes. API access is presented as a way to start experimenting and building with less setup.
For deployment, Sen emphasizes enterprise readiness, security and data privacy, connecting the platform pitch to the responsibilities raised earlier. She also describes compute orchestration as a way to avoid unnecessary spending. These are platform capabilities and intended benefits in the overview; the talk does not demonstrate a deployment procedure or quantify cost savings.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choosing models and supporting builders
Model Garden is the place Sen points to for choosing and fine-tuning a model. Evaluation workflows help match the model to a particular use case. Agent Builder then supplies options across the no-code-to-full-code spectrum for constructing agents. These names describe the platform as presented in the talk; the linked Model Garden documentation now uses Gemini Enterprise Agent Platform branding.
Sen closes by treating the developer experience as a continuous path: training and resources to get started, tools to build with, and support through deployment. The final slide puts developer resources and integrations alongside Vertex AI, Colab, Flutter and Firebase. Her invitation for feedback covers both what users love and what they dislike—a request to shape the platform around the practical experience of the people building these interfaces.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
The original announcement of Llama 3 availability in Model Garden, including model sizes and evaluation options.
Further reading
A contemporary announcement covering Claude availability, the 150-plus-model catalog, evaluation tooling and enterprise controls.
Google’s historical release recap covering Agent Builder, Rapid Evaluation and AutoSxS.
Updates since the talk
Current guidance for discovering, customizing, evaluating and deploying models, with links to practical workflows.
Full transcript
Loading transcript…