# How to build Enterprise-aware agents

Chau Tran

AI Engineer World's Fair 2025

[Talk page](https://ai.engineer/talks/hxFpUcvWPcU-build-enterprise-aware-agents) · [Video](https://www.youtube.com/watch?v=hxFpUcvWPcU)

## How to build enterprise-aware agents

Workflows capture how a company gets things done. Agents can discover, reuse and combine those procedures—provided they can find the right examples for the task and the person.

Familiarity with LLM prompts and tool calls is helpful; the article introduces workflows, agent loops and the adaptation methods it uses.

### Should you build a workflow or an agent?

[0:32](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=32s)

Should an AI system follow a predefined business process, or decide for itself how to accomplish a task? That is the opening problem in Chau Tran’s account of enterprise-aware agents: bringing model capabilities into the messy reality of how a business operates.

![An illustrated fork in a country road with signs pointing left to Workflows and right to Agents.](https://ai.engineer/api/talk-essay-media/7f12dcd030827f7342089180dcfb622b6dedf9466f0a5cf6477f5d8791db8887)

Workflows or agents: a fork in the road.

A **workflow** orchestrates LLMs and tools through predefined code paths. There are two common ways to express those paths:

- **Imperative code:** A program calls the LLM, reads its response and invokes tools. The programmer directly controls execution.
- **Declarative graphs:** Nodes represent steps such as model calls or tool calls, and edges connect them. The graph specifies the structure; a workflow framework handles execution.

Both provide structure and relative predictability. Running the workflow again should mostly follow the same process, even though individual model responses may vary.

An **agent** lets the LLM dynamically direct the process. Given a goal, it decides which steps to take and which tools to call. Its core loop is simple:

1. Receive a task or goal from a person.
2. Plan the next step.
3. Execute an action.
4. Observe the result from the environment.
5. Repeat until it has the results it needs, then respond.

The difference is who determines the path: the developer in a workflow, or the model during an agent’s execution.

### Predictability, autonomy and the moving boundary

[2:58](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=178s)

Tran compares workflows to a Toyota: predictable and useful for repetitive tasks or established business practices. They generally cost less and respond faster because the model does not need to make a planning call at every decision. Explicit code or graph steps also give developers somewhere to locate a failure. Humans can adjust the process to make a task work with an imperfect model.

Agents are his Tesla: more open-ended, capable of tackling unsolved research problems and benefiting from stronger models without requiring every new behavior to be programmed. Their simple core loop means less application logic to maintain, but dynamic planning generally adds cost and latency. They can also take an unexpected wrong turn—the moment when predictable execution becomes attractive again. These are architectural tendencies, not measured performance results.

![Two columns contrast predictable workflows with lower cost, easier debugging, and human control against open-ended agents with higher cost, less logic to maintain, and AI control.](https://ai.engineer/api/talk-essay-media/5447f189ae930fc214b76dd5cd38d6fb9825e3a08cc8a61f9d21dea61c270545)

Workflow and agent tradeoffs in predictability, cost, latency, maintenance, and control.

The boundary changes with model capability. A task that fails in an agent loop today may become practical with a later model. That makes a permanent choice between workflows and agents difficult—and raises a more useful question: can the two improve each other?

### An agent’s execution leaves behind a workflow

[5:21](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=321s)

Give an agent a task and it chooses an action, observes the result, then chooses another action. Once execution finishes, its trace contains a sequence of steps. **That completed sequence is a workflow.** Viewed this way, an agent takes a task and generates a workflow for accomplishing it, discovering the path as it goes.

This makes a library of **golden workflows** useful for evaluation. Each example pairs a task with the steps the company considers appropriate for solving it. Run the agent on that task, inspect what it actually did and compare its steps with the reference procedure. The evaluation asks whether the agent followed the right process, rather than judging only whether its final response looks convincing.

The same library can teach the agent. For known tasks, the aim is to reproduce established procedures. For new tasks, the model can use its reasoning to combine workflows or extend what it has learned. The library supplies company know-how; the agent supplies a way to apply that know-how beyond the original examples.

### Generate procedures, then learn from successful runs

[8:00](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=480s)

The relationship also works in the other direction: an agent can help build workflows. In Tran’s description of [Glean Agents](https://www.glean.com/blog/agents-product-blog), a user describes a task in natural language. An agent figures out the necessary steps and proposes a workflow in the builder. The user can then edit those steps. Generating a proposal and accepting it as the desired procedure are separate actions.

A broader possibility is to use agents as a **workflow discovery engine**:

1. Ship an agent that lets users attempt new tasks.
2. Let users identify executions that accomplished those tasks well.
3. Save the successful workflows as examples of how to do the work in that company.
4. Accumulate those examples as training data to improve the agent.

Tran presents this as a way to turn successful use into reusable organizational knowledge, not as evidence that every shipped agent automatically captures and retrains on its runs.

### Intelligence still needs onboarding

[9:20](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=560s)

Would these procedures still matter if models reached AGI? Tran’s thought experiment treats AGI as a highly intelligent new employee. Intelligence alone does not tell that employee how the company operates, whom to contact when blocked or which nuanced practices govern a task. Enterprise awareness means being fully onboarded into those ways of working.

Competitor analysis makes the distinction concrete. An agent can search Google, read outside material and produce an acceptable report. A great report must also follow the company’s protocols and address the metrics its executives care about. **A plausible answer is not necessarily the answer the organization needs.** The workflow library captures some of the process knowledge that separates the two.

### Teach workflows through fine-tuning

[11:03](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=663s)

Once task–workflow examples exist, how should they change agent behavior? Tran describes two approaches his team has experimented with. The first is **fine-tuning**, with two forms of supervision:

- **Supervised fine-tuning:** Supply an input and an expected output, then train the model to imitate that behavior.
- **RLHF:** In Tran’s description, supply a rating or reward indicating whether a workflow is good, rather than a golden output, and optimize the model using that feedback.

With enough task and workflow data, fine-tuning can support generalization across tasks and the combination of different workflows.

The maintenance cost is that the adapted model becomes a fork of its base model. A stronger frontier model may arrive by the time tuning finishes, creating pressure to repeat the process. Changes to the training examples also require another adaptation cycle: a new tool can make an old workflow obsolete, and new business priorities or processes can change what the correct workflow should be.

Personalization adds another difficulty. Two teams—or two employees—may need different procedures for the same task. A single learned behavior is less flexible when the optimal workflow depends on who is asking.

### Retrieve examples and compose a workflow at runtime

[13:18](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=798s)

The second approach is **dynamic prompting through search**. Keep the task–workflow pairs in a searchable library. When a new request arrives, find similar tasks and put their workflow representations into the LLM’s prompt as examples. The knowledge is supplied at runtime instead of being incorporated into model weights.

This creates a spectrum between familiar procedures and model creativity. If no workflow matches the task, the model has more room to devise a new one. If retrieval finds a high-confidence match, the examples encourage the model to produce a similar workflow. This is guidance through prompting, not a guarantee of deterministic execution.

Return to competitor analysis, this time with the request: what competitors have we been running into recently? The library can supply a procedure for analyzing a competitor and another for finding recent customer calls. The model uses these examples to compose a proposed workflow:

1. Read recent customer calls.
2. Read internal messages.
3. Extract the competitors mentioned.
4. Run an analysis for each competitor.

The task requires more than copying one retrieved procedure. It connects discovery of recent competitive encounters with an existing method for analyzing each competitor. The example illustrates the generated process; it does not report results from executing it.

![Three columns show collecting task–workflow pairs, retrieving similar tasks, and generating a workflow that searches customer calls and internal messages, extracts competitors, and analyzes each competitor.](https://ai.engineer/api/talk-essay-media/0ec9b8e8a97e9f26ad57e6b46c3343295acae0dfd3932fd0e467b5ab927644fd)

Dynamic prompting combines a workflow library, runtime retrieval, and a composed competitor-analysis workflow.

### Match adaptation to the stability of the behavior

[15:04](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=904s)

Fine-tuning and retrieval solve different adaptation problems. Tran favors fine-tuning for generalizing from substantial data and dynamic prompting for flexibility. Retrieval also makes the examples influencing an output available for inspection.

| Consideration | Fine-tuning | Dynamic prompting with search |
|---|---|---|
| Behavior | Generalized | Customized |
| Requirements | Stable across time and users | Change quickly or vary by user |
| Adaptation material | Training examples or rewards | Retrieved workflow examples |
| Inspectability | Behavior learned into the model | Prompt examples can be inspected |

The distinction matters most in the last gap between broadly competent behavior and the specific process a team needs.

Tran’s analogy is custom hardware versus software. Fine-tuning resembles building specialized hardware for a well-understood task whose requirements stay stable: specialization can be valuable, but changing the requirements is costly. Dynamic prompting resembles software: it may not offer the same degree of specialization, but its instructions and examples can change quickly.

### Find the relevant workflow—and the authoritative one

[16:24](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=984s)

Workflow search begins much like document search: compare a new task with task descriptions in the library. Tran names hybrid lexical and vector search, reranking and late interaction as techniques for textual matching. Together, these address the first question: which stored tasks resemble this request?

Enterprise search has a second question: which of the plausible matches should this person use? Once users can create their own documents and workflows, a library may contain hundreds or thousands of similar-looking entries. That is Tran’s illustration of the selection problem, rather than a measured corpus size. Textual similarity alone cannot resolve it.

**Authoritativeness** adds organizational context to relevance. Tran proposes using knowledge-graph relationships and signals such as whether a workflow’s creator works closely with the requester, whether the workflow has a strong record of success and whether people discuss it on Slack. Recommender-system techniques can help select among these candidates.

These signals are difficult to encode directly into an LLM. That motivates a separate workflow-search system: it can account for relationships and evidence of usefulness when choosing the examples the model receives. Enterprise awareness therefore depends on retrieving the appropriate procedure for the user, not merely finding one whose description sounds similar.

![Slide titled How to build Workflow Search lists lexical keywords, vector embeddings, late interaction, and reranking, followed by knowledge graph, popularity, and RecSys signals for authoritativeness.](https://ai.engineer/api/talk-essay-media/f609e75da89d985382c426c1e8327b09fc2fb39578b0690df9a05f9011cf8ecc)

Workflow search uses textual similarity and authoritativeness signals.

### Can verifiable rewards reduce the data requirement?

[19:02](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=1142s)

The closing question asks whether RLVR could reduce the amount of data needed for fine-tuning, suggesting a reward model as the source of feedback. There is a terminology distinction here: in the original [**Tülu 3** formulation](https://allenai.org/blog/tulu-3-technical), **reinforcement learning with verifiable rewards** uses a verification function rather than a learned reward model. It still needs suitable tasks and outcomes that can be checked.

Tran reframes the question as how much fine-tuning data is needed with RLVR. His answer is conditional: it depends on how far the task falls outside the distribution of the model’s existing knowledge. He gives no numerical threshold or demonstrated reduction in enterprise training-data needs, leaving that question for further discussion.

## Resources

- [Building agentic reasoning for the modern workspace](https://www.glean.com/blog/agentic-reasoning-modern-work) — Chau Tran explains enterprise context, editable workflow generation, and the tradeoffs between general and specialized agents.
- [Glean Agents and its natural-language builder](https://www.glean.com/blog/agents-product-blog) — An introduction to Glean Agents covering natural-language building, workflow branches and loops, templates, and enterprise search.
- [Building Effective AI Agents](https://www.anthropic.com/engineering/building-effective-agents) — Architectural patterns for predefined workflows and autonomous agents, with guidance on complexity, cost, and latency.
- [Tülu 3: The next era in open post-training](https://allenai.org/blog/tulu-3-technical) — A technical account of supervised tuning, preference optimization, and reinforcement learning using verifiable rewards.

## Slides

### How to build enterprise-aware agents

Chau Tran introduces the challenge of bringing AI into the complex realities of how businesses operate.

[0:32](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=32s) [Title slide for a talk by Chau Tran on building enterprise-aware agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/c078af748782221c7ba6a02bdaa8e0bd956d918dc201c7c4a153c963ee68152a.webp)

How to build
enterprise-aware
agents

glean

Chau Tran
AI Engineer World’s Fair

Confidential | © 2025 Glean Technologies, Inc.

AIE
aws

### Workflows and agents

A fork in the road frames Chau Tran’s opening question: should AI builders choose workflows or agents?

[0:41](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=41s) [A fork in a road with signs pointing toward workflows and agents.](https://assets.ai.engineer/sites/aiecode2025/sha256/df7ea041b9d47e01e14a97f2c2953924e8fff9a9ab2aa786f1801129cfa623df.webp)

WORKFLOWS
AGENTS

Confidential | © 2025 Glean Technologies, Inc.
2

### Workflows: Structured and predictable

Workflows follow predefined code paths. Imperative code controls execution directly; declarative graphs define steps and connections while a framework orchestrates execution.

[0:54](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=54s) [A definition of workflows and a comparison of imperative code-based and declarative graph-based approaches.](https://assets.ai.engineer/sites/aiecode2025/sha256/89e22ea9e9f3542df85a8ecbeb388c10c98ed883274c3f92c74d2311c003451f.webp)

Workflows: Structured & Predictable

Definition: Systems where LLMs and tools are orchestrated through predefined code paths

Imperative (Code-Based)
• Traditional programming flow
• if/else, loops, functions
• Direct control over execution

Declarative (Graph-Based)
• Visual node-and-edge representation
• Define structure, not execution
• Framework handles orchestration

Confidential | © 2025 Glean Technologies, Inc.
3

### Agents: Dynamic and autonomous

An agent chooses its steps and tools, then loops through actions and environmental feedback until it completes the task and responds to the user.

[2:15](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=135s) [An agent definition, core loop steps, and a diagram of an LLM interacting with a human and environment.](https://assets.ai.engineer/sites/aiecode2025/sha256/8f05ae0f1a6df1d4ddecb8a4c1eb93066c80f0c8024c858cf5f0b52c1fbcf584.webp)

🤖 Agents: Dynamic & Autonomous

Definition: Systems where LLMs dynamically direct their own processes and tool usage, maintaining control on how to accomplish the task

Core Agent Loop
• Receive task from human or system
• Plan approach using reasoning capabilities
• Execute actions using available tools
• Assess progress from environmental feedback
• Iterate or complete based on results

Human ↔ LLM Call
Action: LLM Call → Environment
Feedback: Environment → LLM Call
LLM Call → Stop

Building effective agents
https://www.anthropic.com/engineering/building-effective-agents

Confidential | © 2025 Glean Technologies, Inc.
4

### Workflows and agents: Trade-offs

Workflows keep humans in control with predictable behavior, lower cost and easier debugging. Agents tackle open-ended problems with AI in control, trading higher cost and latency for less logic to maintain.

[2:58](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=178s) [Side-by-side comparison of workflow and agent behavior, cost, debugging, and control.](https://assets.ai.engineer/sites/aiecode2025/sha256/98744721f2a7a0f6319d2c61bbe869c8832cc87d008070e9608aa5875a0659f2.webp)

Workflows

• Predictable behavior
  ◦ Automating repetitive tasks
  ◦ Encoding existing best practices and business processes
• Lower cost & latency
• Easier debugging
• Human in control
  ◦ Control your own destiny, solve quality problems given current LLMs

Agents

• Open-ended problems
  ◦ Researching unsolved problems
  ◦ Taking advantage of complex LLM reasoning
• Higher cost & latency
• Less logic to maintain
• AI in control
  ◦ Better take advantage of future LLM capabilities

Confidential | © 2025 Glean Technologies, Inc.
5

### Do we need to choose?

An agent can generate a workflow as it acts: it chooses and executes one step, then figures out the next. Different tasks can produce different workflows.

[5:11](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=311s) [An agent generates a workflow for a task, potentially iteratively and differently for each task.](https://assets.ai.engineer/sites/aiecode2025/sha256/60f669acb9e62b9a367aace319331fc77a9bd3088f2335a38968139f4a62cd73.webp)

But do we really need to choose?

def Agent(task: str) -> Workflow

Agent generates the workflow to accomplish a task

Fine print:
• Iterative generation
• Can generate totally different workflows for different tasks

Confidential | © 2025 Glean Technologies, Inc.
6

### Workflows as evaluation for agents

Compare an agent’s steps with a golden workflow for the task. In the diagram, Agent C’s workflow passes; Agents A and B fail. The evaluation checks the steps taken, rather than only the final response.

[6:06](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=366s) [A task is sent to three agents; their workflows are compared, with one marked correct.](https://assets.ai.engineer/sites/aiecode2025/sha256/8476585958b4d44d8b5d3267ee009cd7da598b8a5a28befca632dae2c29fa81a.webp)

Workflows as evaluation for agents

• Collect a dataset of labeled data task → Workflow
• Run your Agent and evaluates if it executes the right workflow

Task
Agent A → Workflow A ✗
Agent B → Workflow B ✗
Agent C → Workflow C ✓

Confidential | © 2025 Glean Technologies, Inc.
7

### Workflows as training data for agents

Labeled task–workflow pairs can train agents to execute known workflows and combine them with their own reasoning for new tasks.

[7:09](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=429s) [Labeled workflows train an agent to reproduce known workflows and generalize to new tasks.](https://assets.ai.engineer/sites/aiecode2025/sha256/32edc5ba7a4ca3a4601981496e301bff73b52f9b491ee75e47b43928b6c19c95.webp)

Workflows as training data for agents

• Collect a dataset of labeled data task → Workflow
  ◦ or (task, Workflow) → reward
• Use this to train your agents
• Best of both worlds:
  ◦ Data Fitting:
    ▪ Executing the provided workflows for the known tasks
  ◦ Generalization:
    ▪ Composing workflows together to achieve more complex tasks
    ▪ Blending provided workflows with internal reasoning capabilities

Task A → Workflow A
Task B → Workflow B
Task C → Workflow C
training → Agent
New Task → Agent → New Workflow

8

### Agents as workflow generators

Glean uses an agent to turn a natural-language task description into a proposed workflow. The example searches for an on-call playbook, selects and reads it, then suggests a resolution; users can edit the proposed steps.

[7:58](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=478s) [An example shows natural-language instructions alongside a generated on-call workflow.](https://assets.ai.engineer/sites/aiecode2025/sha256/42da67efd21d40529fc7b84e1d937d308ebdd4f7d65600a8a6f07f100ebae9e4.webp)

Agents as Workflows generator

Workflows Building Platforms use Agents to generate Workflows

Write instructions
Describe steps this agent should follow to complete this task as well as how the final output should be generated.

Suggest how to resolve the eng oncall issue in Jira ticket

• Search for playbooks that are related to the content of the ticket
• Think about which playbook is most relevant to the ticket
• Read the full content of that playbook and summarize how to resolve the ticket

Tips
• Write steps in a bullet list format
• Use “@” to tag specific documents and files
• Use [[ double brackets ]] to create fields in your instructions

Resolve an eng oncall issue
START
0. Trigger
Agent starts when user fills out fields:
Jira ticket

1. Company search
Search for playbooks relevant to Jira ticket

2. Think
Which of these search results are most relevant to Jira ticket?

3. Read document
Read the playbook that is most relevant to Jira ticket

4. Respond
Suggest how to resolve Jira ticket using the playbook you found.

Confidential | © 2025 Glean Technologies, Inc.
9

Interface text:
Back
Generate agent

### Agents as a workflow discovery engine

Agents try new tasks; successful steps can be saved as golden workflows and used as training and evaluation data, feeding a cycle of workflow discovery.

[8:44](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=524s) [A cycle links agents generating workflows with workflows providing training and evaluation data.](https://assets.ai.engineer/sites/aiecode2025/sha256/7079b594b9e6ccfb85e69dbb9fa356ef05d9e4927defafdd2ceb41db2ae38e60.webp)

Agents as Workflow discovery engine

Use Agents for try accomplishing new tasks
• If the task is accomplished, save the steps it took as a golden workflow to train Agents

Agents
generation, discovery
Workflows
training and evaluation data

Confidential | © 2025 Glean Technologies, Inc.
10

### Do we still need this post-AGI?

The speaker compares AGI to a brilliant new employee who still needs onboarding. Enterprise-aware AGI would know company practices, helping bridge the gap between an acceptable output and a great one.

[9:25](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=565s) [Slide compares AGI needing company onboarding with enterprise-aware AGI and its last-mile quality challenges.](https://assets.ai.engineer/sites/aiecode2025/sha256/06f3915e3650c67ebd377a4fe39a98419424f9b435c584d2f7ca1e172f25f043.webp)

Do we still need this post-AGI?

Maybe?

• AGI:
  ○ 👏 a super intelligent employee
  ○ 🫠 who just joined your company and need “onboarding”
  ○ 👏 but can accomplish most tasks given enough trial and error
  ○ 🫠 but might not do it the “right” way
• Enterprise-aware AGI:
  ○ 💪 fully onboarded, knows the ways your company do things
  ○ “Last mile” quality problems:
    ▪ There are infinitely “acceptable” outputs of a “task”
    ▪ But there’s a gap between “acceptable” and “great”

Confidential | © 2025 Glean Technologies, Inc.
11

### How to Train Your Agents

The talk turns to training agents using tasks and golden workflows as data.

[11:03](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=663s) [A movie-poster-style image of a rider on a dragon, with the word Agents substituted into the title.](https://assets.ai.engineer/sites/aiecode2025/sha256/8c967b589d61f6d0c7e24ae6419ea4f71ff475c536a23082001f6cd55b71bf2d.webp)

FILMED FOR
IMAX
DreamWorks
HOW TO TRAIN YOUR
AGENTS
JUNE 13

Confidential | © 2025 Glean Technologies, Inc.
12

### Option 1: Finetuning

Supervised fine-tuning learns from task-to-workflow examples; RLHF uses rewards. The slide notes strong learning with ample data, but says changes to tools or workflows require retraining.

[11:17](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=677s) [Slide describes supervised fine-tuning and RLHF, then lists benefits of abundant data and drawbacks of retraining as workflows change.](https://assets.ai.engineer/sites/aiecode2025/sha256/8fcb737f255c217098a0fd1f56f32eef5b5a3f5c2e2c90ab72c9c2ee93fe74d1.webp)

Option 1: Finetuning

• Supervised finetuning:
  ◦ Collect labeled data
    ▪ task → workflow
• RLHF:
  ◦ Collect rewards data
    ▪ (task, workflow) → reward
• Run your favorite optimization algorithms to finetune the LLM

Pros:
• Can learn really well when you have a lot of data
  ◦ “The Bitter Lesson”

Cons:
• Creates a fork from frontier LLM
• Any change to training data requires re-training
  ◦ add/remove/edit tools
  ◦ changing business processes/adding new workflows
  ◦ personalized workflows for different teams/employees

Confidential | © 2025 Glean Technologies, Inc.
13

[11:57](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=717s) [Fine-tuning methods, benefits, and drawbacks for task workflows.](https://assets.ai.engineer/sites/aiecode2025/sha256/155a9982175f4ffd11d99603761c0865877a5544eff33f47fa4387bae7afd9be.webp)

Option 1: Finetuning

• Supervised finetuning:
  ◦ Collect labeled data
    ▪ task → workflow
• RLHF:
  ◦ Collect rewards data
    ▪ (task, workflow) → reward
• Run your favorite optimization algorithms to finetune the LLM

Pros:
• Can learn really well when you have a lot of data
  ◦ “The Bitter Lesson”

Cons:
• Creates a fork from frontier LLM
• Any change to training data requires re-training
  ◦ add/remove/edit tools
  ◦ changing business processes/adding new workflows
  ◦ personalized workflows for different teams/employees

Confidential | © 2025 Glean Technologies, Inc.
13

Interface text:
AIE

### Option 2: Dynamic Prompting through Search

For a new task, search labeled task–workflow pairs for similar tasks and give their workflows to the LLM as examples. The slide places high-confidence matches toward determinism and no match toward creativity.

[13:22](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=802s) [Dynamic prompting uses similar task workflows as examples, ranging from high-confidence matches to creative generation.](https://assets.ai.engineer/sites/aiecode2025/sha256/5c7456984cb1ea90df98111beaa8398f02c4d45ed204f0a6866030fdc073231a.webp)

Option 2: Dynamic Prompting through Search

• Collect labeled data
  ◦ task → workflow
• Build a really good search engine for tasks
  ◦ task -> similar tasks
• To accomplish new task, find similar tasks in the training data
  ◦ Feed the representation of those workflows to the LLM as examples

determinism ↔ creativity
high confidence match — no match

Confidential | © 2025 Glean Technologies, Inc.
14

### Dynamic Prompting through Search

A competitor question retrieves similar workflows for competitor analysis and customer calls. The LLM combines them into a workflow that searches calls and messages, extracts competitors, and analyzes each one.

[14:37](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=877s) [Library building, runtime search, and LLM workflow generation illustrated with a competitor-analysis example.](https://assets.ai.engineer/sites/aiecode2025/sha256/e37195b24a93308ad7033e5687a181ad30449234e545c4c74e8a7f82cdf6c7ca.webp)

Option 2: Dynamic Prompting through Search

1. Library Build (offline) 📚

Collect a library of Task → Workflow pairs

Example:
• COMPETITOR_ANALYSIS
• FIND_CUSTOMER_CALL
• OUTBOUND_EMAIL
• PULL_REQUEST_REVIEW
• ACCOUNT_SNAPSHOT
…

2. Runtime Search + Prompt 🔍

Similarity search on user task → Retrieve top-k examples

Example Task:
• “What competitors have we been running into lately”

Similar tasks:
• COMPETITOR_ANALYSIS
• FIND_CUSTOMER_CALL

3. LLM Generates New Workflow ⚡

Output composed workflow given input task, tool descriptions, and retrieved examples as demonstrations

Output workflow:
• Search for customer call transcripts
• Search for internal messages about competition
• Extract competitors mentioned from the above
• For each competitor, run competitor analysis

Confidential | © 2025 Glean Technologies, Inc.
15

### Fine-tuning and Dynamic Prompting with Search

Fine-tuning suits stable, generalized behaviors when data is plentiful. Dynamic prompting makes examples easier to change and inspect, helping adapt workflows to changing requirements and users.

[15:07](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=907s) [Comparison of generalization through fine-tuning with flexible, customized behavior through dynamic prompting.](https://assets.ai.engineer/sites/aiecode2025/sha256/2d058473623c9f3708b28e9e806a4ea6b96f69e07b69a1119f2ce9fa12c4287e.webp)

Finetuning/RLHF

Pros:
• Stronger generalization when you have a lot of data

Good for:
• Learning generalized behaviors
  ◦ Ground truth labels don’t change across time/users

Analogy: HARDWARE

Dynamic Prompting w Search

Pros:
• More flexible
  ◦ Easily add/remove/modify examples
  ◦ Quickly swap out LLMs
• Better interpretability

Good for:
• Learning customized behaviors (“last mile” quality gap)
  ◦ Changing requirements (tool set changes, business process changes)
  ◦ Personalized behavior

Analogy: SOFTWARE

Confidential | © 2025 Glean Technologies, Inc.
16

### How to build Workflow Search?

Workflow search finds similar tasks through text matching, then uses authoritativeness signals to choose among similar workflows. Those signals are hard to encode directly in an LLM.

[16:23](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=983s) [Workflow search combines textual similarity methods with authoritativeness signals such as knowledge graphs, popularity and recommendation systems.](https://assets.ai.engineer/sites/aiecode2025/sha256/aa5a48051881fcefaed0ee673328a76e2bedd674e6c49e928218eb1150790664.webp)

How to build Workflow Search?

• Very similar to Document Search
• Textual similarity:
  ◦ Lexical key words
  ◦ Vector embeddings
  ◦ Late interaction
  ◦ Reranking
  ◦ ...
• Authoritativeness: which relevant sounding workflows are the best
  ◦ Knowledge graph
  ◦ Popularity
  ◦ RecSys
  ◦ ...
• Authoritativeness signals are very hard to encode directly into an LLM

Confidential | © 2025 Glean Technologies, Inc.
17

## Full transcript

[0:00](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=0s) [upbeat music] Uh, thanks Alex for the introduction.

[0:16](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=16s) That was a very impressive LLM-generated summary of me. Uh, I've never heard it [chuckles] before, but, uh, nice. Um, so, um, today I'm gonna talk to you about something that has been keeping me up at night, uh, probably some of you too.

[0:32](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=32s) So how to build enterprise-aware agents. How to bring the brilliance of AI into the messy, complex realities of, uh, how your business op-operated. So let's jump straight to the hottest question of the month for AI builders: Uh, should I build workflows or should I build agents?

[0:52](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=52s) So what are workflows? Workflows are a system where LLMs and tools are orchestrated through predefined code paths. So there are two main ways where you can, um, represent the workflows.

[1:06](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=66s) The first way is through, uh, imperative code base. So these are the workflows where you, you know, write a program that calls LLMs, uh, read the response, and then call tools and sort of like, uh, do this in a traditional programming flow.

[1:22](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=82s) And then here you get-- have like direct control of the execution of, um, all the steps. The second way to represent workflow is through, uh, declarative graphs. So in this way, you sort of, um, represent your workflow as like a graph of-- where nodes are sort of like steps, where you can call tools or call LLMs, and

[1:44](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=104s) then there's sort of edge between nodes. Um, so you kind of define the structure but not the execution, and the execution of this is usually handled by some framework, um, workflow frameworks.

[1:56](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=116s) So I'm, I'm not gonna go into the details of pros and cons for these two approaches, but, um, the main point here is, like, for workflows, you get structure and predictability.

[2:07](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=127s) So if you run a workflow today, it will mostly behave the same way, uh, if you run it tomorrow.

[2:15](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=135s) On the other hand, um, we have agents, which are systems where LLM sort of dynamically direct their own processes of, like, decide how to achieve a task. Like decide what tools to call, uh, what step to take, depends on the task itself.

[2:33](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=153s) Um, so the core agent loop is pretty simple. So it receive a task or, like, a goal from a human, and then it, uh, sort of enter this iterative loop, where it, uh, plan what to do and then execute the action, and then read the results from the environment, and sort of iterate until, uh, it, uh, get

[2:53](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=173s) all the result it wants and then, uh, respond to the user.

[2:58](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=178s) So what are the trade-offs between workflows and agents? Um, workflows are sort of like the Toyota of AI systems. Uh, it's very predictable. Um, it's good for when you wanna automate, uh, repetitive tasks, uh, or, like, encode existing best practice or, or, like, know-how in your bu- in your business.

[3:20](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=200s) It's usually lower cost and lower latency 'cause you don't have to spend time on this, all these LLM calls to decide what to do. And they're also, also easier to debug 'cause, like, you have this code or this graph that you can manually pinpoint, uh, at which step is going wrong in, in the execution.

[3:38](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=218s) And in building workflows, uh, humans are sort of in control. Like, you can control your destiny, like, given, even given, uh, imperfect LLMs. Uh, you can sort of do tweaks and engineering so that your tasks work right now.

[3:55](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=235s) On the other hand, agents are sort of like the Tesla of AI systems. Like, it's more, uh, you know, open-ended. It's good for, like, researching unsolved problems. Uh, it's also usually good at taking advantage of, um, better and better LLM capabilities because here the AI is in control.

[4:13](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=253s) Um, generally, it's higher cost and latency because you need LLM to, like, figure out what to do and then... But the, uh, upside is, like, there's less logic to maintain, the core loop is very simple, and, um, sometimes you get, like, these, uh, hints of brilliance that always feels like, you know, everything is gonna be automated in

[4:32](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=272s) a few months. Um, the problem is, like, your, your Tesla, like, it's-- works very well most of the time, but sometimes it still take the wrong exit on the highway, and that's when you kinda miss your Toyota.

[4:45](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=285s) So, and the decision to build workflows or agent is a pretty tricky one because it depends highly on the state of the LLM. Um, so some workflows that doesn't work in an agentic loop now might start to work later in a few months when a new model has come out.

[5:07](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=307s) So it's, uh, it's a really huge dilemma. Um, but recently, one thought, um, that's sort of really changed how I think about it is, what if you don't really have to choose, right?

[5:21](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=321s) So if you think of agent, what they do is when you give the agent a task, it will figure out the steps that needs to be done to achieve that task, right?

[5:35](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=335s) So, um, you give it a task, it figure out the, the one step, take the action, figure out the next step, and then at the end when the agent finish the execution and then you look at the trace of what happened, all those se- series of steps is a workflow.

[5:51](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=351s) So if I represent this in, like, a, um,

[5:55](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=355s) a programming kinda way, then agent takes a sta- a task and then generate a workflow to achieve that task. Um-

[6:04](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=364s) So if we think of th-it this way, agent take a task and generate a workflow, then you can sort of see, like, there are really good synergies between workflows and agent.

[6:17](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=377s) So the first thing is you can actually use workflows as, uh, evaluation for your agents, right? So, uh, let's say in, in your company, you can collect a huge amount of, um, golden workflows, like given a task, this is the steps that, uh, needs to be done to solve that task, and you, you have a huge list

[6:38](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=398s) of, uh, of those, uh, sort of handbook on, on how, um, to do things in your company. Then you can actually evaluate your agents by, uh, you know, like give it a task, see what it did, and compare it to the, the golden workflow.

[6:53](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=413s) Like, did it actually figure out the right steps? So this is a little bit different from evaluating end-to-end. You are not judging agent by, uh, the end response, but, like, by, uh, whether it actually did the right step to get to that end response.

[7:10](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=430s) Um, the second and, uh, even better way, uh, for workflows to help, help agents is, you know, given that same golden, uh, workflows library, you can also use it to train your agents.

[7:23](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=443s) Um, so here you truly get the best of both worlds, where, you know, with the data feeding, you can, uh, your agents will be able to execute the exact workflow that you have in your library for the known task.

[7:39](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=459s) Um, but then it co-or co- um, it can also rely on its own, uh, internal reasoning capabilities to sort of compose different workflows together to, uh, achieve new tasks, and even use its own reasoning to kind of extend, uh, what you teach it, but, like, make it better.

[8:00](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=480s) Um, and then agents can also help workflows as well. Uh, one way to do that is, um, for workflow building platforms, uh, you can use an agent to generate the workflows.

[8:14](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=494s) Um, so this is sort of how, uh, Glean agents work under the hood, where, uh, the user can give, uh, the workflow builder like a sort of natural language, uh, a description of the task it, it is trying to achieve, and then we run an agent implementation to figure out the steps that are needed to achieve that

[8:31](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=511s) workflow. Then the user can sort of like, uh, make edit or like add change, uh, the workflow that, that the g- the agent was, uh, proposing.

[8:44](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=524s) Um, and lastly, and, and I think is like, uh, the most powerful, um, synergy is you can use agents as a workflow discovery engine, right? So you ship an agent, uh, users try to accomplish new tasks with your agent, and then when they find that the agent did a good job, then you can sort of save that

[9:09](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=549s) workflow as like, "Okay, this is how you do this task in my company." And then over time, you can use this, um, as like training data to help agents get better.

[9:20](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=560s) Cool. Um, so that was the main points of my talk. Um, I guess maybe some of you are thinking, do we st-still need this kind of stuff in a world where we have AGI?

[9:34](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=574s) Um, so here's, here's my thought experiment and, uh, why I think this may be still needed after AGI. So AGI is gonna be a super intelligent employee, right? Um, but if they, if AGI doesn't know about, uh, how your company works, it's sort of like, uh, a really good employee who just joined and doesn't know about all

[9:57](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=597s) the business practices and still needs onboarding, needs to know, like, who to talk to, to get unblocked and, like, uh, all the very nuanced ways of doing things in the enterprise.

[10:09](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=609s) Um, so what is enterprise-aware AGI? So enterprise a-aware AGI is fully on-onboarded, very intelligent, knows the ways your company do things. And, um,

[10:23](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=623s) one, one key kind of insight I, um, I think is like, so there are many acceptable ways to achieve a task, um, but there's a gap between an acceptable output versus a great output.

[10:37](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=637s) Um, one example is like, you know, competitor analysis. Like, sure, it can do some basic Google search and, like, uh, read some, uh, notes out-outside to, like, do some competitor analysis, but does it actually follow, uh, the protocols or the processes that your company define?

[10:57](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=657s) And does it actually address all the key metrics that your executive, uh, really care about?

[11:03](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=663s) So, um, given all this data, you know, like tasks and golden workflows, how do you actually train your agents, um, using those data? So this is, uh, the second part of my talk.

[11:18](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=678s) Um, so there are two main ways we have, um, experimented with. The first one is through fine-tuning. Um, there are sort of two main flavor of fine-tuning here. One is, uh, you know, supervised fine-tuning, where you give, uh, give an input and an expected output, and you train your model to just, um, mimic that, uh, behavior.

[11:43](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=703s) The second way is through RL-RLHF, where you don't have a golden label, but you sort of have a, a rating or a reward when you know, like, this task, this workflow, is it a good one or is it a, a bad one?

[11:55](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=715s) So then you can sort of run f-your, uh, favorite optimization algorithms to fine-tune the LLM.

[12:02](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=722s) So the pros of this method is that it can learn really well when you have a lot of data, um- Um, if you have a huge amount of, uh, tasks and workflows, it can really learn, um, like sort of generalize across different tasks and like combine workflows.

[12:20](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=740s) Um, the problem here is, one, uh, you kinda have to create a fork from the, from the frontier LLM, right? So you start with some LLM, you do some fine-tuning, and then by the time the fine-tuning finishes, maybe there's a be- new and better model already come out, and you have to like redo this whole process again.

[12:40](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=760s) And the second is like any change to your training data, uh, like you need to do retraining, right? So if you have a new tool, then maybe some of the existing workflow is outdated, then you have to retrain.

[12:52](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=772s) Uh, if you do s- change some business priorities or business processes, then you have to like redo the training a- again. And it also not super flexible for personalization.

[13:04](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=784s) Um, so given the same task, maybe different teams or different employees might actually have a different optimal workflows to, to do those tasks, and fine-tuning is not super well-suited for, for those use cases.

[13:18](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=798s) Um, then comes the second option, uh, which is dynamic prompting through search. So, um, given the same label data, uh, from task to a golden workflow, you build a really good search engine for tasks, um, so that you can find similar tasks given a new task.

[13:36](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=816s) So then at runtime, uh, to accomplish a new task, we'll find the most similar task in the training data, and then you feed the representation of those workflows to the LLM as the examples, right?

[13:50](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=830s) So here you really have a spectrum of, uh, determinism and creativity. So when there's no workflow that sort of match your input task, then the LLM are in control.

[14:03](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=843s) Like it can, uh, use its creativity to generate a new workflow. But when there's a high confidence match of something that, that you have done before, then the LLM will sort of give you a work- workflow that's very similar to what was in the training data.

[14:20](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=860s) Um, so one very concrete example, uh, come back to the competitor analysis be-- uh, example before. So you collected this huge list of task workflow, um, and then when a new task, like say what, what competitors have we been running into recently, then it will retrieve, you know, how to analyze each competitor, and then you will find

[14:45](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=885s) a workflow on how to find, uh, your recent customer calls. And then the LLM will take those example and then sort of generate a composed workflow where it read customer calls, read, uh, internal messages, extract competitors, and then run analysis for each of them.

[15:04](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=904s) Um, okay, so comparison time. Um, fine-tuning RLHF is very strong, uh, when you have a lot of data that you wanna generalize. Dynamic prompting with search is more flexible, uh, also gives you better interpretability, uh, that you can sort of look into the exact e- examples that was affecting your outputs.

[15:27](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=927s) And, um, fine-tuning is good for learning generalized behaviors, uh, where the ground truth labels don't change over time or like across different users. Um, dynamic prompting with search is better for learning customized behaviors or like the last mile quality gap where, you know, uh, requirements are changing quickly.

[15:47](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=947s) Um, one, one sort of analogy I think about fine-tuning versus dynamic prompting is, um, fine-tuning is very similar to like building customized hardware. So when you know-- When you have a sort of task that you really want to optimize for and the requirements don't change over time, like you can really build custom hardware that do it very

[16:09](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=969s) well. Uh, but it's sort of costly when you, uh, change the requirements, compared to dynamic prompting is more like writing software. Um, not as, uh, optimized, but like you can just change them very quickly.

[16:24](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=984s) Um, last point, uh, so how do we actually build this workflow search, right? So how do you give it a task, like find similar tasks? Uh, I would say it's very similar to building document search, right?

[16:38](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=998s) Um, and there are two main components to this. The first one, uh, is what everyone usually think of when they think of search, which is a textual similarity, right?

[16:49](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=1009s) Um, given this task, what are some of the similar sounding tasks that are in the training data? Um, and here the sort of, uh,

[17:00](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=1020s) golden recipe is like le- like hybrid search between lexical vector embeddings, uh, re-ranking, late interaction, all that.

[17:07](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=1027s) But, uh, what I found is in, in the enterprise settings, uh, pure textual similarity is not enough. When, uh, when you give users the choice to create workflows and write documents,

[17:21](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=1041s) when you s- wanna search for something, there will be like hundreds or thousands of similar looking documents or workflows, and, uh, the problem becomes how do you choose the right one, uh, right?

[17:32](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=1052s) So, uh, which is what I call as, uh, authoritativeness here. And to solve this problem, uh, then you kinda have to go into a knowledge graph, right? So if this workflow is created by someone who I work closely with, uh, it has high success rate, and like people post about it, um, on Slack, then it's more likely

[17:54](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=1074s) to be the right one. So all the tricks in, uh, the recommender system,

[17:59](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=1079s) uh, um, world also applies here for, for workflow search. And, um, this kind of authoritativeness signals are very hard to encode directly into an LLM, which is why we sort of have to have the, like a separate system that does the, the search for workflows.

[18:18](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=1098s) Cool. Um, so key takeaways, uh, workflows, good for determinism, human are in control. Agents, more open-ended, AI is in control. And, um, the synergy between a- agents and workflows is workflows can be used for agents evaluation, uh, workflows used for agents training, and agents is used for workflows discovery.

[18:43](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=1123s) Um, fine-tuning is good for generalized behaviors. Dynamic prompting with search is good for personalized behaviors. All right, uh, I still have one minute and, uh, thirty seconds, uh, maybe time for one question.

[18:58](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=1138s) Exactly one question.

[19:01](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=1141s) Yeah.

[19:02](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=1142s) Um, I'm kind of curious on the fine-tuning, and I think earlier you mentioned that we may need to offer data, right? How about with the new, like RLVR? Can we set up a reward model, um, so we don't need to provide as much data for the fine-tuning process?

[19:16](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=1156s) So the question was, uh, wait, I'll try to re- re-interpret it. Let me know if it's wrong. Uh, how much data do we need to do fine-tuning given the new-

[19:27](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=1167s) RLVR

[19:29](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=1169s) ... RLVR? That's a very difficult question to answer because, uh, it really depends on

[19:36](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=1176s) how out- out of distribution your task is compared to the internal, uh, knowledge of the LLM. Um, but I'll catch you after and we can talk more.

[19:45](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=1185s) Yes.

[19:45](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=1185s) Thank you.

[19:45](https://www.youtube.com/watch?v=hxFpUcvWPcU&t=1185s) Too difficult of a question- [outro music]
