AI Engineer World's Fair 2024

Level Up Your Automation Game: Agentic Workflows on Vertex AI

Rukma Sen· PMM, Google18:06

Loading slides…

Read the talk

Level Up Your Automation Game: Agentic Workflows on Vertex AI

Agents turn model capabilities into user-facing actions, but useful automation depends on how reasoning, tools, state and quality checks work together.

From a talk by Rukma Sen

Where do everyday users meet AI?

Where will developers, employees, parents and students actually interact with generative AI? Powerful models can still be intimidating or inaccessible to the people who could benefit from them. Rukma Sen’s starting proposition is that, for many everyday tasks, an agent will become the interface between the model and the user. The user describes what they need in natural language; the agent makes model capabilities accessible through that interaction. This is a prediction about interface design, not a claim that every AI application must become an agent.

Customer, employee, creative, data, code and security agents arranged around semicircular platform, models and infrastructure layers above a user icon.Open full source frame
Six agent categories surround layers labeled platform, models and infrastructure.

Developers occupy a useful position at that boundary: they understand both model capabilities and the people they are building for. Their work includes designing the interaction and setting limits that make it helpful, humane and safe. Sen compares this opportunity with earlier changes in how people used technology: household machines and appliances, followed by internet-era screens and the familiar gestures of swiping, zooming and scrolling. Agent interfaces offer another opportunity to shape those habits.

That opportunity brings responsibility. Sen invokes Spider-Man’s familiar maxim before turning to the practical obligations of designing systems that mediate people’s interactions with AI. The limits around an agent are part of its interface, just as much as the way a user asks it for help.

0:521:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:52 · section reference included

Responsibility extends from outputs to stored data

The first obligation is protecting users. Guardrails should limit misinformation and make clear the difference between generated material and information presented as factual. An interface that makes an answer easy to obtain also needs to help people understand what kind of answer they have received.

The next questions concern cybersecurity and privacy: where is the data used in model reasoning stored, and how is it safeguarded? Sen highlights wearables and unobtrusive computing because these systems place computation throughout everyday life. As the interface becomes less conspicuous, the handling of personal data becomes more consequential.

4:174:35
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:17 · section reference included

A model, tools and orchestration

For the architecture discussion, Sen uses a deliberately broad working definition: an AI agent is a system designed to achieve specific goals by interacting with its environment. The definition establishes a practical scope without trying to settle every disagreement about what counts as an agent.

Three components make that system work:

  • Model: Reasons about the goal, determines the next plan of action and guides behavior—the agent’s brain.
  • Tools: Let the agent act beyond reasoning or generating text—the agent’s hands. Examples include retrieving information from the internet and calling external APIs to book flights or process payments.
  • Orchestration: Maintains memory and state, keeps track of goals and coordinates the pieces—the nervous system connecting brain and hands.

The distinction between thinking and acting matters here. Tools give model decisions a way to affect the environment, while orchestration maintains the context in which those decisions occur.

Together, these components support autonomous task completion. But the category is broader than today’s generative systems: some kinds of agents existed long before the recent growth of generative AI.

6:326:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:32 · section reference included

Deterministic, generative and hybrid behavior

Sen separates agent behavior into three broad categories. A deterministic system follows fixed rules: a calculator given 2 + 2 returns 4. A generative system can produce varied responses and combine rules in new ways. Her example is a customer-service chatbot that uses company product information to compose a helpful answer. A hybrid system assigns different parts of the task to each approach.

TypeBehaviorExample
DeterministicFixed rules; consistent outputCalculator
GenerativeVaried, composed responsesProduct-support chatbot
HybridRules plus generated communicationFinancial-advisor workflow

In Sen’s illustrative financial-advisor workflow, deterministic agents perform market analysis and identify investments, while a generative agent communicates the strategy to customers. The useful architectural distinction is the separation of analysis from explanation; the example does not establish the accuracy of investment predictions.

Three columns describe deterministic, generative and hybrid agents beneath a horizontal spectrum, with benefits and challenges for each.Open full source frame
Deterministic, generative and hybrid agents compared by behavior, benefits and challenges.
8:368:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:36 · section reference included

From one model to specialized roles

A single-agent architecture puts reasoning, planning and action under one model. Give it instructions, a goal and a set of tools, and it handles the task. This is a straightforward architecture to implement, but obtaining an answer is not the same as obtaining a dependable result.

Sen illustrates the reliability problem by asking a model to count occurrences of the letter A in banana. In her anecdote, the model answers four; when asked to check, it changes the answer to two; asked again, it says one. Repeated prompting changes the output without establishing correctness. This is an illustrative failure, not a measured failure rate, but it exposes a production concern: self-checking does not necessarily produce a trustworthy check.

A multi-agent architecture distributes work among specialized roles, much as a company assigns different responsibilities to different people. Sen’s customer-service example follows three levels:

  1. Dispatcher: Assess the incoming customer issue and route the case.
  2. Subject-matter expert: Respond using expertise in a particular product line or region.
  3. Supervisor: Check the work against a predefined dataset and resolve some issues.

The architecture separates routing, answering and reviewing rather than asking one model to perform every responsibility within a single role.

Adding a supervisor creates another component whose behavior must be designed carefully. In one system Sen built, the supervisor was supposed to return “This is not good enough. Please try again” when dissatisfied. It kept rejecting the first agent’s work until she recreated the setup. No specific repair is demonstrated, but the anecdote supplies a consequential qualification: specialized review can itself become a source of failure rather than an automatic guarantee of quality.

10:2010:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:20 · section reference included

Applications by audience, expertise and modality

The application categories Sen describes use several different organizing principles. Customer and employee agents are defined by whom they serve; knowledge agents by their expertise; voice agents by how people interact with them.

  • Customer agents: Support, e-commerce, B2B interactions and travel services.
  • Employee agents: HR enrollment, benefits questions, sales, accounts payable and supply-chain work.
  • Knowledge agents: Specialized subject expertise, such as answering legal questions.
  • Voice agents: Spoken interactions, including fast-food drive-through ordering.

Sen reports seeing increased interest in voice agents as multimodal applications expand, without attaching an adoption figure to that observation.

The drive-through example makes the interface concrete. A customer speaks an order; the agent transcribes it and sends it to the ordering system; a person at the delivery window serves the customer. The agent connects a natural human input to an existing operational system, while the physical fulfillment remains with staff.

Four boxes list customer, employee, knowledge and voice agent applications, including support, HR, legal expertise and drive-through food ordering.Open full source frame
Customer, employee, knowledge and voice agents with example use cases.
13:0813:21
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:08 · section reference included

Vertex AI across the development lifecycle

The closing platform overview places these architectures within Google Cloud’s Vertex AI. Sen presents it as a full-lifecycle development platform for calling models, fine-tuning them and building agents. Its development options range from no code and low code to highly customized implementations.

At the time of the talk, Sen reports access to more than 150 models. This is catalog breadth, not a performance measure or a promise of identical APIs and deployment conditions. The offerings she names include Google’s models, Anthropic models, Llama 2, Llama 3 and other open-source models. Her description of the Anthropic offering as including all its models is broader than the contemporary announcement establishes. API access is presented as a way to start experimenting and building with less setup.

For deployment, Sen emphasizes enterprise readiness, security and data privacy, connecting the platform pitch to the responsibilities raised earlier. She also describes compute orchestration as a way to avoid unnecessary spending. These are platform capabilities and intended benefits in the overview; the talk does not demonstrate a deployment procedure or quantify cost savings.

14:5115:10
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:51 · section reference included

Choosing models and supporting builders

Model Garden is the place Sen points to for choosing and fine-tuning a model. Evaluation workflows help match the model to a particular use case. Agent Builder then supplies options across the no-code-to-full-code spectrum for constructing agents. These names describe the platform as presented in the talk; the linked Model Garden documentation now uses Gemini Enterprise Agent Platform branding.

Sen closes by treating the developer experience as a continuous path: training and resources to get started, tools to build with, and support through deployment. The final slide puts developer resources and integrations alongside Vertex AI, Colab, Flutter and Firebase. Her invitation for feedback covers both what users love and what they dislike—a request to shape the platform around the practical experience of the people building these interfaces.

Slide lists quick-start resources, developer labs, integrations, packages and extensions beside Vertex AI, Colab, Flutter and Firebase logos surrounding 'Interfaces for all developers.'Open full source frame
Vertex AI developer resources, integrations and interfaces.
16:3216:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:32 · section reference included

Resources

From the talk

  • Llama 3 on Vertex AIArticle15:32

    The original announcement of Llama 3 availability in Model Garden, including model sizes and evaluation options.

Updates since the talk

Transcript download
Full transcript

Loading transcript…