# Optimizing LLMs in Insurance with DSPy: Beyond Manual Tuning

Jeronim Morina, AXA

AI Engineer World's Fair 2024

[Talk page](https://ai.engineer/talks/IAdZxqjZ45U-optimizing-llms-in-insurance-dspy-beyond-manual) · [Video](https://www.youtube.com/watch?v=IAdZxqjZ45U)

## Optimizing LLMs in Insurance with DSPy: Beyond Manual Tuning

A useful insurance chatbot needs domain examples, visible execution traces, and trustworthy evaluations before modular programs and DSPy optimizers can improve it.

Basic familiarity with LLM prompts and Python classes is useful; no prior DSPy experience is required.

### When a plausible answer becomes the acceptance test

[0:19](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=19s)

What problem are we actually solving when we spend the day trying new models, tools, and prompts? Jeronim Morina opens with a provocation: AI engineers have become too comfortable tinkering with tools and HotpotQA instead of solving real problems. His proposed way out is to return to engineering: define the problem and program an AI system around it.

![Dark slide with the opening claim in white and the fully revealed response in red; speaker inset at lower left.](https://ai.engineer/api/talk-essay-media/aebecf5b9d0a85bf3513929d53c4ddf84060e59d198e48facaef3394ec7862ad)

“We’re all bad AI Engineers”—“But there is a way out.”

A neural network becomes useful through the larger system that surrounds it. Yet excitement about model capabilities often precedes a clear account of what that system must achieve. Handwritten prompts and prompt libraries then accumulate around a vaguely defined goal, with the hope that another API will supply the missing solution.

Morina makes this concrete with an engineer’s morning routine: brew specialty coffee, settle into focused work, tweak prompts, and try another tool to make the output JSON. Then inspect the answers and approve them using the joke metric “Looks good to me” at ten. That may feel like progress, but it leaves the acceptance criterion in the engineer’s head. **A prompt that seems stable is not yet a measured system.** Moving beyond that routine starts with a problem worth solving.

### The workload behind the chatbot

[2:50](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=170s)

Insurance supplies a concrete setting for that work. Morina describes it as a business built around data and intangible protection: customers pay for a promise of financial support when something happens. Climate change brings increasing claims and more work serving those customers. AXA Germany founded its Data Innovation Lab in 2017; its machine learning engineers collaborate with business-unit data scientists to help customer service agents handle that work.

![Forest-canopy background behind text about insurers’ data history, AI investment, and a cross-functional group of machine learning experts.](https://ai.engineer/api/talk-essay-media/c55b6f07864821260fdc17eb963930266df656b58fca19ad866463c5aa975cdf)

AXA’s “Pioneers in Data Mastery” slide introduces its Data Innovation Lab.

The organizational pace differs from a startup’s. Morina compares Silicon Valley companies to speedboats and large insurers to tankers—then refines the latter image to icebreakers, moving slowly but opening a path. He dates AXA’s work with OpenAI and its internal, Azure-hosted Secure GPT platform to early 2023. At the time of the talk, he also describes a Mistral AI collaboration announced that Monday. Expanding the platform’s usefulness had to preserve customer data security.

### Define the chatbot through easy and hard examples

[5:23](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=323s)

The walkthrough begins with an illustrative insurance problem: a customer-facing chatbot that helps people navigate policy terms and conditions. Before choosing its implementation, the team needs to define what useful behavior means. Domain experts guide that process by crafting examples at both extremes: questions that should be straightforward and questions that are unusually difficult. Those examples expose the domain’s boundaries and the expectations for an eventual production system.

Morina reports starting with about a dozen examples and immediately using them to prompt internal Secure GPT. This first round is an exploration of feasibility: can the model produce useful answers to the questions the domain experts care about? The examples make that investigation concrete before the system acquires more machinery.

### Look inside the structured-output machinery

[6:26](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=386s)

The next challenge is getting clean, structured output without treating prompting as an incantation. Morina rejects pleading for JSON or threatening to quit if the model fails to produce it. He describes a technique attributed to Jason Liu: begin an assistant response with an opening code fence and the `json` language label, steering the model to continue in that format. The mechanism is continuation from a supplied prefix, rather than another instruction asking for JSON.

Chain-of-thought, zero-shot and few-shot prompting, and libraries such as LangChain, Guardrails, [DSPy](https://dspy.ai/), and Instructor offer different ways to shape behavior. The desired properties include safer, more deterministic, structured, and resilient output, sometimes optimized against a chosen metric. But finding a good prompt remains laborious when templates are hard-coded and library behavior is opaque.

That opacity makes request inspection part of the engineering work. Morina recommends Hamel Husain’s [prompt-inspection article](https://hamel.dev/blog/posts/prompt/) and its use of `mitmproxy` to examine what frameworks actually send to the model. Inspect the generated template and outgoing request before attributing an improvement to a library. **The tool’s abstraction does not remove responsibility for understanding its prompt.**

### A feasible task inside a fragile system

[9:01](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=541s)

Minor prompt changes caused large downstream problems. The team responded with chaining libraries and additional error handling, but those repairs accumulated into an overly complex, fragile system. Morina judged it unsuitable for production. It still established something useful: the task appeared achievable, and the prototype could serve as a baseline.

![Batman slapping Robin meme with “SHOW ME THE PROMPT,” beside a matching slide heading and a hamel.dev link.](https://ai.engineer/api/talk-essay-media/a11e587728db2d314e2082c2ab08ca11f0a596ac01be24c50b2cbbf135ac279b)

“Show me the prompt” challenges claims of library magic.

Inspecting individual requests did not explain the whole execution. The team was not logging traces, so it lacked visibility into how related calls fit together. Adding the open-source [Arize Phoenix](https://github.com/Arize-ai/phoenix) library helped reveal which calls were bundled together and what happened behind the abstractions. Request inspection exposed prompt contents; tracing exposed relationships among calls.

### Turn production data into an evaluation

[10:17](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=617s)

Visibility alone did not replace informal approval. The next requirement was a basic evaluation, which needed labels as well as inputs. Morina reports spending several months collecting and preparing existing production data into clean input-output pairs. The expensive part was establishing usable examples of expected behavior, not adding another model call.

Those pairs also needed protection against **data leakage**. If an input exposes information that already reveals its target label, an evaluation can reward access to the answer instead of the intended task. A modern API does not repair that defect in the dataset. Morina recommends Hamel Husain’s writing on evaluations, then makes the prerequisite explicit: understand the data and gather enough examples before moving to DSPy or another optimization tool.

### DSPy does not skip the foundations

[12:12](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=732s)

Skipping those foundations produces the caricature Morina warns against: prompt engineering, then DSPy, then profit. His own interest came from meeting Omar the previous November. That encounter prompted the team to try DSPy, but enthusiasm did not make the transition effortless.

The starting point was a system with huge prompts and extensive error handling. Replacing it with DSPy seemed as though it should simplify everything; instead, the team encountered a steep learning curve. Morina encourages experimentation without offering an unconditional recommendation. The framework still requires learning how to express the problem in a form it can optimize.

![Slide headed “Steep learning curve” with a butterfly meme labeled “DSPY” and “WOW IS THAT LEARNING CURVE STEEP.”](https://ai.engineer/api/talk-essay-media/7c7fcbab7605edc45119ff0c1c81ed303916328ad4a49f05061172af77110cba)

DSPy’s steep learning curve.

### Give the optimizer a program it can work with

[13:13](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=793s)

The manual workflow already contained several distinct jobs: break the problem into steps, make each step work in isolation, adjust the steps to work together, and generate examples for tuning. Those examples might eventually support fine-tuning smaller models. DSPy retains the need for decomposition. Without meaningful modules, the optimizer has too little structure and signal to guide improvement.

The team’s attempt to express everything as one huge `ReAct` program was unsuccessful. A more useful design makes the flow explicit: retrieve information, perform additional retrieval if the question requires multiple hops, and combine the relevant pieces into a final answer. Each operation becomes a module with a defined role.

An optimizer then uses a supplied metric to tune the program’s prompts or weights. For this chatbot, Morina describes the objective as finding the right answer; he supplies neither a formal scoring rule nor a measured improvement. Compilation can produce different instructions, few-shot examples, or fine-tuning from the same program, depending on the optimization approach. The important separation is between the program’s structure and the choices being tuned inside it.

The PyTorch analogy makes that separation tangible. In `__init__`, declare components such as `ChainOfThought` or `ReAct`, much as a neural-network module declares convolution or dropout layers. In `forward`, specify how data moves through them, including retrieval. This Python example expresses that structure for the insurance chatbot: a supplied retriever returns policy passages, and an answer module consumes those passages with the customer’s question.

```python
import dspy


class PolicyAnswer(dspy.Signature):
    """Answer the question using the supplied policy passages."""

    context: list[str] = dspy.InputField()
    question: str = dspy.InputField()
    answer: str = dspy.OutputField()


class InsuranceQA(dspy.Module):
    def __init__(self, retrieve):
        super().__init__()
        self.retrieve = retrieve
        self.answer = dspy.ChainOfThought(PolicyAnswer)

    def forward(self, question: str):
        passages = self.retrieve(question)
        return self.answer(context=passages, question=question)
```

The constructor defines the parts; `forward` defines their composition. Retrieval and answer generation now have explicit places in the program instead of being hidden inside a single enormous prompt.

### German text and task-specific modules need care

[15:47](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=947s)

Two customization needs emerged in the team’s DSPy experience:

- **German evaluation:** Morina found `answer_exact_match` and `answer_passage_match` unsuitable out of the box for the team’s German text. They needed their own metrics and evaluators, particularly for exact and passage matching.
- **Typed behavior:** The version he describes offered `TypedPredictor` and `TypedChainOfThought`, but lacked the typed `ReAct` component they wanted. More specific requirements called for a custom module.

These are historical implementation observations. The Python example above uses the current typed-signature style; current DSPy also documents standard modules including `ReAct`. English-oriented normalization remains a reason to inspect matching behavior, rather than assuming either that a built-in metric fits German or that every German comparison necessarily fails.

The larger architectural lesson survived those API details: the team’s attempt at one big model did not work. A framework cannot supply useful boundaries for a task the engineer has not yet decomposed.

### Make the next improvement small enough to measure

[16:56](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=1016s)

The practical workflow begins before automated optimization:

1. **Write prompts by hand** to determine whether the task is achievable at all.
2. **Prepare annotated examples and a clear goal metric** so an apparent improvement has an objective test.
3. **Start with basic evaluations**, such as regex checks or string comparisons, before introducing LLM-as-judge.
4. **Make very small adjustments** so the system does not become harder to understand with every repair.

This sequence keeps evaluation proportional to the question being tested. A straightforward check should not become another elaborate AI subsystem without a reason.

A short talk cannot provide complete DSPy training. Morina points readers toward Connor Shorten’s learning materials and the DSPy website for that next step. His own curiosity grew through meeting the people who built these tools; the intended result is the same curiosity paired with first-principles engineering. The closing challenge is to find worthwhile problems that AI now makes possible to solve—and let those problems determine what to build.

## Resources

- [Fuck You, Show Me The Prompt.](https://hamel.dev/blog/posts/prompt/) — Hamel Husain shows how to inspect outgoing LLM requests with mitmproxy and understand what prompting frameworks actually send.
- [Your AI Product Needs Evals](https://hamel.dev/blog/posts/evals/) — A practical guide to domain-specific evaluation, including scoped tests, trace review, human evaluation, and model-based evaluation.
- [Phoenix](https://github.com/Arize-ai/phoenix) — The Phoenix project provides tools for observing and evaluating LLM applications.
- [DSPy documentation](https://dspy.ai/) — The current DSPy learning hub covers signatures, modular programs, evaluation, and optimization.
- [DSPy metrics and evaluation](https://github.com/stanfordnlp/dspy/blob/main/docs/docs/diving-deeper/metrics-and-evaluation.md) — Current documentation explains custom metrics and the normalization behavior behind built-in string and passage matching.
- [DSPy ReAct API reference](https://dspy.ai/api/modules/ReAct/) — The current API reference for constructing and using DSPy's ReAct module.
- [AXA's Secure GPT launch announcement](https://www.axa.com/en/press/press-releases/axa-offers-securegenerative-ai-to-employees) — AXA's July 2023 announcement describes Secure GPT's Azure OpenAI foundation and its initial employee rollout.
- [AXA and Mistral AI](https://mistral.ai/customers/axa/) — Mistral describes its collaboration with AXA and the integration of its models into Secure GPT.

## Slides

### Presenting Sponsor

Microsoft is the presenting sponsor.

[0:05](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=5s) [Microsoft logo beneath the presenting sponsor heading.](https://assets.ai.engineer/sites/aiecode2025/sha256/5d4030caf16e58685083cf890102e4ce670906b2134c951c4a8b4af50c4a66b2.webp)

PRESENTING SPONSOR

Microsoft

### Innovation and Platinum Sponsors

AWS is the innovation sponsor; MongoDB, Google Cloud, and Neo4j are platinum sponsors.

[0:10](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=10s) [AWS is the innovation sponsor; MongoDB, Google Cloud and Neo4j are platinum sponsors.](https://assets.ai.engineer/sites/aiecode2025/sha256/39ac6a24ef33988c4681c9e663c0a79ba5a68bf6849f50ea6f2021833e9a4029.webp)

INNOVATION SPONSOR
aws

PLATINUM SPONSORS
MongoDB®
Google Cloud
neo4j

### We're All Bad AI Engineers

The speaker argues that AI engineers spend too much time tinkering with tools and datasets instead of solving real-world problems. His proposed way out: start programming AI systems again.

[0:42](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=42s) [Title slide says, “We're all bad AI Engineers,” followed by “But there is a way out.”](https://assets.ai.engineer/sites/aiecode2025/sha256/848ea9cf9a5d9b9863360e5c50c64edf368a9de7656ae677e0fce928db369df4.webp)

We’re all bad AI Engineers
But there is a way out

AXA
AIE
Microsoft
smol ai

### From a Model to a Compound AI System

A single neural network gives way to a compound AI system, where models connect with data and tools such as a calculator and terminal.

[1:13](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=73s) [A neural network points toward a system of interconnected models, data and tools.](https://assets.ai.engineer/sites/aiecode2025/sha256/5d0290c460396a7ea3f3f9491c7f8eeb550bb51e3d7f04a2f852a9817af431d9.webp)

AXA
AIE
https://bair.berkeley.edu/blog/2024/02/18/compound-ai-systems
aws

### The struggle is real

Excitement about newer models meets a practical problem: teams often start with only a vague idea of the outcomes they want.

[1:21](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=81s) [Slide describing the challenges of finding LLM use cases and defining desired outcomes.](https://assets.ai.engineer/sites/aiecode2025/sha256/f3eaa7c65c94eeab99ae98c6fb69c1b405e32a4132ebbae7f49f04a26eedaba8.webp)

The struggle is real [emoji]

• Daily struggle with new use cases in the LLM space
• Excitement about capabilities of newer models
• Often starting with only a vague idea of desired outcomes

AXA
AIE
Microsoft
smol ai

### Common Pitfalls

The speaker questions whether tinkering with prompts and eyeballing outputs is enough, and urges engineers to tackle real problems rather than expect an API to solve everything.

[2:06](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=126s) [Slide lists ill-defined problems, reliance on prompts, hopes for a magical solution, and the misconception that an API resolves every issue.](https://assets.ai.engineer/sites/aiecode2025/sha256/aaa28de8c15a0f71f8bbd1f0856c9cee63929e7af145deeb8abff304ec2f4a1a.webp)

Common Pitfalls

• Using tools on ill-defined problems
• Relying on hand-written or library-based prompts
• hoping for a "magical solution“
• The misconception that simply using an API will resolve all issues

AIE
AXA
Microsoft
smol ai

[2:30](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=150s) [Slide lists pitfalls including ill-defined problems, reliance on prompts, hoping for a magical solution, and expecting an API to resolve every issue.](https://assets.ai.engineer/sites/aiecode2025/sha256/7c89042cfc27ff74b6170a57d9bab5ee9a5d07fe072f5de8744fe0dcc5d152c7.webp)

Common Pitfalls

• Using tools on ill-defined problems
• Relying on hand-written or library-based prompts
• hoping for a "magical solution“
• The misconception that simply using an API will resolve all issues

AIE
AXA
Microsoft
smol ai

### AXA and Mistral AI Partnership

Mistral AI says its models are being integrated into AXA’s GenAI platform on Microsoft Azure, which serves nearly 150,000 users.

[4:44](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=284s) [AXA slide showing a social media announcement about integrating Mistral AI models into AXA’s generative AI platform.](https://assets.ai.engineer/sites/aiecode2025/sha256/e15d750b8077ba85a0f41cefea07702a282d171af33fa9bcdc2ae733affc404e.webp)

AIE
AXA

Jeronim Morina • You
Sr. MLOps Engineer | driving AI & LLM Innovation at AXA | co-Founder @ bloo...
2d • 🌐

Hello World, we have the data, the talent and the problems everybody is craving for [illegible emoji] We even have two major internal engineering conferences with hundreds of ML and software engineers every year. And now even one #AXAGenAISummit

Mistral AI
214,517 followers
2d • 🌐

[illegible emoji] We are pleased to announce an important partnership with AXA. As part of our ongoing commitment to providing cutting-edge AI solutions, our models is being integrated into AXA's GenAI platform, serving nearly 150,000 users on the Microsoft Azure Cloud.

Microsoft
smol ai

[5:03](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=303s) [AXA slide showing a post announcing the integration of Mistral AI models into AXA’s generative AI platform.](https://assets.ai.engineer/sites/aiecode2025/sha256/c5b58b84f19172c7d5fec1a10624da4fab54f8e548d70568b8ca3b41872cca17.webp)

AIE
AXA

Jeronim Morina • You
Sr. MLOps Engineer | driving AI & LLM Innovation at AXA | co-Founder @ bloo...
2d • 🌐

Hello World, we have the data, the talent and the problems everybody is craving for [illegible emoji] We even have two major internal engineering conferences with hundreds of ML and software engineers every year. And now even one #AXAGenAISummit

Mistral AI
214,517 followers
2d • 🌐

[illegible emoji] We are pleased to announce an important partnership with AXA. As part of our ongoing commitment to providing cutting-edge AI solutions, our models is being integrated into AXA's GenAI platform, serving nearly 150,000 users on the Microsoft Azure Cloud.

Microsoft
smol ai

### Imaginary Day in the Life of an AI Engineer

For an imagined insurance chatbot, engineers develop examples with domain experts, experiment with prompts and tools, then evaluate outputs against preferred metrics and question their stability.

[5:24](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=324s) [Slide describing prompt and tool experimentation and evaluation of output metrics and stability.](https://assets.ai.engineer/sites/aiecode2025/sha256/a13f0d2cb67343ab2a8b72b8366d32303132f7348e3a5ba08a7848ac852382a3.webp)

Imaginary Day in the Life of an AI Engineer 👩‍💻

• Experimenting with various prompts and tools to guide system outputs
• Evaluating outputs using preferred metrics, questioning the stability of results

AXA
AIE
Microsoft
smol ai

[5:51](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=351s) [Slide describing experimentation with prompts and tools, and evaluation of outputs and result stability.](https://assets.ai.engineer/sites/aiecode2025/sha256/db5a2ec40c7a28dc5081bf01db0a698efd05ef57597a0fcefca49c0fd16ff465.webp)

Imaginary Day in the Life of an AI Engineer 👩‍💻

• Experimenting with various prompts and tools to guide system outputs
• Evaluating outputs using preferred metrics, questioning the stability of results

AXA
AIE
Microsoft
smol ai

### Prompting for JSON

The example starts an assistant message with a JSON code fence to steer the model toward producing structured JSON.

[6:52](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=412s) [Slide showing a code example that primes an assistant response to produce JSON.](https://assets.ai.engineer/sites/aiecode2025/sha256/63431d99138b40bdb0134bae0af4b9cedfaa2618ef0ec9f664335f1ebd6df302.webp)

AXA

JSON ?

completion = openai.ChatCompletion.create(
    model="gpt-3.5-turbo",
    temperature=0,
    max_tokens=100,
    messages=[
        {
            "role": "user",
            "content": "create a fake json object for a ecommerce product",
        },
        {"role": "assistant", "content": "```json\n"},
    ],
)

AIE
Microsoft
smol ai

### Show me the prompt

Inspect the prompt templates and requests LLM tools send. The speaker recommends Hamel Husain’s post and using mitmproxy to see those requests.

[8:06](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=486s) [Slide with a comic urging users to inspect the prompt behind an LLM tool.](https://assets.ai.engineer/sites/aiecode2025/sha256/e3c9badd6c7f77c2325017d22c47352549f7ca3c7e48f97510e67fb77635b17d.webp)

Show me the prompt 👀

Our library allows you to get the best output from your LLMs. pip install llm-magic.

SHOW ME THE PROMPT

https://hamel.dev/blog/posts/prompt/

Interface text:
AXA
AIE
imgflip.com
Microsoft
smol ai

### Giving DSPy a chance

The meme mocks the leap from prompt engineering through DSPy to profit. The speaker urges teams to understand their data and collect enough examples before using DSPy or similar tools.

[11:29](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=689s) [A meme presents prompt engineering, DSPy and profit as three phases, with a question mark over the middle phase.](https://assets.ai.engineer/sites/aiecode2025/sha256/4dc72d570c7667606c59f528b08ab8e6e2a20c421668ff398be98f149c54cfe2.webp)

AXA
AIE

Giving DSPy a chance ✔

PHASE 1    PHASE 2    PHASE 3
?
Profit
PROMPT ENGINEERING | DSPY | PROFIT
imgflip.com

aws

### Giving DSPy a chance: conference photo

The speaker says meeting Omar the previous year prompted him to give DSPy a chance. The slide pairs that decision with a conference photo.

[12:25](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=745s) [The slide pairs the phrase “Giving DSPy a chance” with a photo of two people at a conference.](https://assets.ai.engineer/sites/aiecode2025/sha256/919ee1539fdcafb5127e7a062449ce8d4d3a548e8aba38443047fa2e5555cd6c.webp)

Giving DSPy a chance ✔

AXA
AIE

### Steep learning curve

DSPy appealed to the speaker, but its steep learning curve tempered expectations that it would make a complex system easy. He encourages trying it without an unconditional recommendation.

[12:32](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=752s) [A meme illustrates the claim that DSPy has a steep learning curve.](https://assets.ai.engineer/sites/aiecode2025/sha256/8ca4481c15acc66b3ef327d2cdc365e5440bb1fe673c31b17bb0c05f6e43d910.webp)

Steep learning curve 📈

WOW IS THAT
LEARNING CURVE STEEP
DSPY

AXA
AIE
aws
imgflip.com

### Without DSPy

Without DSPy, the team tuned each step in isolation, then coordinated the steps. They also generated examples for tuning and potentially fine-tuning smaller models to cut costs.

[13:13](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=793s) [A list describes breaking a problem into steps, prompting and coordinating them, generating examples, and fine-tuning smaller models.](https://assets.ai.engineer/sites/aiecode2025/sha256/2904a08222b3e9b618c68d015d720f2042237d751421c75fd4dce29c1e385162.webp)

Without DSPy ✖

• break down problem down into steps
• prompt LLM well, until each step works well in isolation
• tweak steps to work well together
• generate synthetic examples to tune each step
• use these examples to finetune smaller LMs to cut costs

AXA
AIE
aws

[13:32](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=812s) [Steps for manually developing and tuning a multi-stage language model workflow.](https://assets.ai.engineer/sites/aiecode2025/sha256/1d492a5b119ad8c0d7c8333ceb945752de398c68b63e98313ba3658a8c477d07.webp)

Without DSPy ✖

• break down problem down into steps
• prompt LLM well, until each step works well in isolation
• tweak steps to work well together
• generate synthetic examples to tune each step
• use these examples to finetune smaller LMs to cut costs

AXA
AIE
aws

### With DSPy

DSPy separates program flow from parameters and uses a metric to tune prompts or weights. It compiles the same program into instructions, few-shot prompts and fine-tunes for each language model of interest.

[13:33](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=813s) [DSPy separates program modules from parameters and uses optimizers and metrics to tune them.](https://assets.ai.engineer/sites/aiecode2025/sha256/13d1bcbd7583cac32644aad6d88eda62e2b760e205e593aba7a62502b59b5819.webp)

With DSPy 🧩

• Separating Flow of your program modules from parameters
• Adding optimizers to tune prompts / weights given a metric to maximize
• DSpy compiles the same program into different instructions, few-shot prompt and finetunes for each LM in interest
• Instead of prompting one LM with a huge prompt trying to accomplish your task, use DSPy to break down the problem into distinct modules, that can be reused

AXA
AIE
Microsoft
smol ai

### Analogy to Neural Networks

DSPy modules such as ChainOfThought and ReAct play a role analogous to PyTorch layers. Optimizers can recompile the LLM program as its code, data, assertions or metrics change.

[15:11](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=911s) [DSPy modules and optimizers are compared with neural network layers and training.](https://assets.ai.engineer/sites/aiecode2025/sha256/bd0ad85045905dbee2b5d01c01a244202adc88340949b3dd286c00e975a2d47e.webp)

Analogy to Neural Network 🧠

• Like Convolution and Dropout Layers in PyTorch, DSPy uses modules like ChainOfThought or ReAct to replace string-based prompting tricks
• optimizers like BootstrapFewShotWithRandomSearch or MIPRO. These can be used to recompile our LLM program when code, data, assertions or metrics change.

AXA
AIE
aws

### DSPy Learnings

The speaker says German optimization required custom metrics and evaluators, while a typed ReAct required a custom module. DSPy’s placeholders need adapting to the task’s architecture.

[15:46](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=946s) [Four columns discuss metrics, ReAct, evaluations, and adapting DSPy to the task.](https://assets.ai.engineer/sites/aiecode2025/sha256/4957242178486045d783885543d067033781b43f2695675aa45e4e0041620be1.webp)

DSPy Learnings

Metrics
answer_exact_match and answer_passage_match only works for English texts

ReAct
there is no TypedReAct as there is TypedPredictor or TypedChainOfThought, thus there is room for open source contributions ✔

Evaluations
Start writing basic evaluations. Start with fast evaluations based on Regex or String comparisons. Only then try to use techniques like LLM as a Judge.

DSPy
It's not about the one majestic model, as it isn't about particular layers in a neural network. It's really about building the right architecture for the task at hand

DSPy comes with a bunch of good placeholders, but needs to be adapted heavily

AXA
AIE
aws

### Takeaways

Establish feasibility with handwritten prompts, then measure progress with annotated data and a goal metric. Start evaluations with regex or string comparisons before considering LLM judges or DSPy.

[16:53](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=1013s) [Recommendations to establish a prompt baseline, define metrics, write evaluations, and avoid overcomplicating DSPy.](https://assets.ai.engineer/sites/aiecode2025/sha256/34fbb524663ab81fb2c60ee0ec2c8ff648bfbee3aad19aea74165e9dcf52830c.webp)

Takeaways

Start with Prompts

You should definitely write prompts by hand and thus create a baseline of what's achievable and prove the task is possible.

LGTM@10

You can't improve what you don't measure. Avoid Metrics like LGTM@10. Without annotated data and a clear goal metric, there is no clear objective, to validate if results are getting better.

Evaluations

Start writing basic evaluations. Start with fast evaluations based on Regex or String comparisons. Only then try to use techniques like LLM as a Judge.

DSPy

Don’t ever start with DSPy without doing all of the things to the left. And please don’t overcomplicate things.

AXA
AIE
aws

[17:04](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=1024s) [Four recommendations: start with prompts, define meaningful metrics, build evaluations, and use DSPy after establishing the basics.](https://assets.ai.engineer/sites/aiecode2025/sha256/3bad03d21c0e5535476fa46981ec0d162a5d25686681914afa786412759e4523.webp)

Takeaways

Start with Prompts

You should definitely write prompts by hand and thus create a baseline of what's achievable and prove the task is possible.

LGTM@10

You can't improve what you don't measure. Avoid Metrics like LGTM@10. Without annotated data and a clear goal metric, there is no clear objective, to validate if results are getting better.

Evaluations

Start writing basic evaluations. Start with fast evaluations based on Regex or String comparisons. Only then try to use techniques like LLM as a Judge.

DSPy

Don’t ever start with DSPy without doing all of the things to the left. And please don’t overcomplicate things.

AXA
AIE
aws

## Full transcript

[0:00](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=0s) [upbeat music] Welcome to, uh, Optimizing LLMs with DSPy: Beyond Manual Tuning.

[0:19](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=19s) And, um, I hate to break it to you, but we're all bad AI engineers. [laughs]

[0:26](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=26s) Yes, you are. [laughs] So why, why, why is that? Um, because we don't care enough about solving real world problems. Um, instead we all love keep tinkering around with tools and HotpotQA data sets.

[0:42](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=42s) But let-- bear with me, there's a way out. So let's start with first principles thinking again and reminding ourselves what we are. We are engineers, you know? We have to reconsider the thing we are doing these, these days, which is like prompt engineering, and start programming AI systems again.

[1:01](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=61s) I mean, maybe you've seen this, uh, already, like left we have this huge neural network model, but usually neural networks on themselves and being like language models or any other, uh, AI model are not useful on their own, but are only useful if you're considering them, uh, in a huger system.

[1:19](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=79s) So the struggle is real when it comes to use cases in the LLM space. We're all thrilled by the excitement around capabilities of ever newer models. Uh, but we have only a vague idea of what we really want to achieve.

[1:33](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=93s) We throw a bunch of tools of them at the not well-defined problems. We write some handwritten prompts, use some prompt libraries in the hope that that will lead us to a magical solution of our problems.

[1:43](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=103s) I know myself how much I love to tinker around with all these tools and see how they work. But here I'm telling you to stop hoping that using API just works and solves all your problem.

[1:55](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=115s) So let's imagine you're an AI engineer, and you start your day by brewing some specialty coffee, sitting down, delving into focus mode. And you fiddle around with a bunch of prompts and, uh, try the next tool to guide system output to-- for JSON.

[2:11](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=131s) So you evaluate the system with your most loved metric. The answers are all like, "Looks good to me" at ten. So I have just look at all of the outputs, and I was like, "Yeah, that's fine enough.

[2:23](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=143s) I can, I can do this." But is this really the end? And, uh, is it really that we fiddle around with our prompts until they seem stable enough? So the question is: How do we become like great AI engineers again?

[2:36](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=156s) I think we should stop, start, uh, stop working on toy problems, all these HotpotQA data sets and everything, and start working on real problems again. So why am I telling you this?

[2:50](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=170s) So a company dude like me is going to explain to you how to solve real world problems. I mean, yes, uh, we have to. We have to solve real world problems every day because we can, and it's really about the data and helping our customers every day.

[3:06](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=186s) So we are data-driven company since, like ever. We don't have any assets. I mean, insurance is what are they? They are selling you a good feeling. You give us money, and, uh, we give you money back in case something happens to you.

[3:18](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=198s) Uh, so there's no real asset, but we always have been a data, data-focused, um, company and only providing non-touchable assets. So, um, to, to cape, to keep with the ever-- So one of the huge problems we are facing today is climate change, and with that, the ever-increasing like incoming, uh, claims, for example, and more problems

[3:43](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=223s) that we are seeing with our customers. So there's a huge rise of, um, of labor we have to do. And so that's the reason like why AXA Germany started the Data Innovation Lab in two thousand and seventeen.

[3:56](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=236s) And we are like a group of machine learning engineers working with the data scientists of the business units, and we are trying like to work cr-cross-colla- uh, collaboratively with the business units to, to really make an impact on the customer service agents to help them make the customers happy every day.

[4:15](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=255s) So I know you think you, you are like in Silicon Valley and, uh, you're like these amazing speedboats. You're like super fast. You can change your ideas quickly every day.

[4:25](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=265s) And we are like rather tankers. And in a way, you are right. So we are these tankers, but more like icebreakers often. So it's like slowly, but we are like making progress forward and, um, like pushing things forward.

[4:39](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=279s) And you can see this also in, uh, things that just made the announcement on Monday. So in the forefront already in two thousand twenty-three, the beginning of the year, we started collaborating with OpenAI and created this Secure GPT GenAI platform internally, uh, which is hosted on Azure platform.

[4:57](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=297s) And even now, we are now like announcing, uh, on Monday that we're working together with Mistral AI, so our internal GenAI platform gets like more usable every day and like especially not compromising our data security or anything our customers don't want to.

[5:14](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=314s) So that's super cool. So that's the reason like the tanker, uh, has some impact and, uh, makes fun. But

[5:23](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=323s) w-what does it, what does it take us to, to get bad-- good engineers? So h-how do we make this path, um, better? So let's me, let's take me, let me take you on a path on imaginary real world problem insurance.

[5:38](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=338s) So we're creating a customer-facing chatbot to help them navigate our insurance fair terms and conditions. So how would we tackle this? I mean, yeah, we kind of have like everything, but the main issue is like you really need to define your problem really clearly.

[5:57](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=357s) So we went to the domain experts and let us guide through the problem space, especially crafting examples. And this is especially tr-- important for prompt engineering, which are either super simple or like super hard to solve, so we can understand the, the domain space better.

[6:14](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=374s) And, um, that-

[6:16](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=376s) Guided us to, to learn a lot about what is expected in our production system in the end. So we came up with a dozen of examples and immediately started prompting our internal Secure GPT.

[6:26](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=386s) And, um, uh, we, we tried to come up with some useful prompts. But prompt engineering is like quite of black magic, so we didn't use phrases like, "If you don't output JSON, uh, I'm going to quit," or, "Please, please, please output JSON."

[6:41](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=401s) And, uh, [chuckles] there are these prompts where you fiddle around, you try to make your output better, cleaner, more structured. And, um, one, one way, uh, to do this is, uh, what Jason Liu already proposed last year, um, was to, to hijack the prompt and like say, "Okay, I'm an assistant here," and we start the prompt by just

[7:06](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=426s) giving in triple backticks and JSON, and that then the, the model, uh, push forward into, uh, just creating JSON. But, uh, there are also other, a dozen other of prompting techniques like chain-of-thought, zero- few-shot learning, uh, tools like LangChain, Guardrails, DSPy, Instructor, and to help make use of these techniques,

[7:31](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=451s) um, to make the output safer, more deterministic, more structured, more resilient, and optimized for maybe even arbitrary metrics. And so we, we fiddled around with all these tools and played around, and to be honest, that was a lot of work.

[7:47](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=467s) Like finding a good prompt is hard and error-prone, as these prompt templates are often hard-coded and the usage is quite intransparent. So please don't fall into these tools will solve the problem for me trap.

[8:02](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=482s) So get your hands dirty, inspect what these tools are really doing for you, and I highly recommend, um, this amazing blog post from Hamel Husain, so

[8:14](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=494s) which is called, uh, Please Show Me the Prompt. [chuckles] It has another title, but bear with me. So, um, uh, the idea is just to use mitmproxy and to really inspect like what are all these tools using I've been mentioning before, what are re- they really sending?

[8:32](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=512s) So this is about first principles thinking. I'm not advocating here to, to use tools this and that and there, and I will talk definitely about DSPy, but

[8:44](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=524s) please do first principles thinking. Look at what these tools create, what, what kind of prompt templates they use, what, what they send to your LLMs. Don't just believe that it will happen or it will do something great.

[9:01](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=541s) So one issue we had been experiencing a lot was sensitivity to minor prompt changes. So like very small prompt changes led to very high problems in the end. So what we, what we had was, what we tried to do was using chaining libraries to, to fix these issues and throwing more and more error handling code at that.

[9:22](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=562s) So, um, we tried to make this a bit more achievable and but this lead led into the end to an overly complex and fragile system. So we had the gut feeling that was by no means something we want to put in production, but we could show that the task was at least achievable, um, or could serve as

[9:43](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=583s) a baseline. So before continuing further, we knew we were missing some key pieces to fully understand what's happening. And, uh, apart from inspecting our calls, that was we were not logging any traces or something.

[9:56](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=596s) Um, for that, we started using an open source library called Arize Phoenix, and, uh, that definitely helped a lot, um, in understanding like which calls are bundled together, which helped to see what,

[10:12](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=612s) what, what is doing, what is happening behind the curtains.

[10:17](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=617s) So a- again, here, once again, I'm advocating for first principles thinking. You cannot improve anything what you don't measure. And we were still, at that point in time, we were still at the level looks good to me at 10, right?

[10:34](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=634s) Which was our main driver. So we need to definitely write some basic evaluations. So preparing the data to match. So we-- If you want to evaluate something, you need to not only have your input data, but also to have some labels.

[10:48](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=648s) And that was the hard part, to be honest. That took us some several months to really get scrape the production data we were already getting and, uh, looking at it and preparing it, so we had like clean input-output pairs.

[11:00](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=660s) And one main thing is, please don't stumble upon this, data leakage is still a thing. So it doesn't matter if you have like the coolest APIs. If you're using data which is, has compromised data leakage in it, and you already can deduce the output label from your input data, that's really bad.

[11:22](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=682s) So if you're creating your, uh, your evaluation data sets, really, really, really pay attention to, uh, to not, um, to not, uh,

[11:37](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=697s) destroy your, uh, evaluation. So there's another blog post I would definitely recommend from, uh, Hamel Husain, uh, which is, uh, evals, called E- Evals is all you need or evals for your, um, for your, uh, LLMs.

[11:53](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=713s) Please get into that. That is amazing and will so much help you. So feeling quite confident. So this is the base setting, and I encourage you, please don't do anything with DSPy or any other tools when you don't have like this base setting, when you don't understand your data, when you don't have enough examples.

[12:12](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=732s) Because otherwise, like what? You, you're just saying prompt engineering, DSPy, profit? So what, what's, what's the case? The please, please don't do this. And To, to be fair, I, I just went into DSPy because I met Omar last year at, uh, [REDACTED:location] in, uh, in November.

[12:29](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=749s) And I was like, "Okay, cool. So we are now giving this a chance." But

[12:35](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=755s) please bear with me. This has a steep learning curve. So I'm-- I love the idea of DSPy. I love how, how it enabled us to...

[12:48](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=768s) Now we have this overly complex system with these huge prompts and huge error handling code and everything, and we all thought, "Okay, we're going to throw, uh, DSPy at it, and it's going to be super easy."

[12:58](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=778s) But to be honest, uh, the s- the learning curve is quite steep, and it's maybe a bit too much overall. So this is not a 100% recommendation, go to DSPy.

[13:08](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=788s) But, uh, please give it a try if you can.

[13:13](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=793s) Okay. So without DSPy, we had to break down our problems into step. We have to prompt well. Like, each step had to work well in isolation. We had to tweak the steps to work together.

[13:25](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=805s) We had to generate examples to tune each step and to use these examples to maybe even fine-tune some smaller models. Uh, and what we now could do with DSPy was taking this base idea, and you need to do that.

[13:39](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=819s) I encourage you, please f- uh, is-- tear apart your problem into separate modules, and we're talking about modules in a second. Um, otherwise, you won't be able to take advantage of DSPy.

[13:54](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=834s) And also, DSPy is not going to take-- to be able to take advantage of your, uh, of your DSPy program because it doesn't get any hints, any, any signal on how to improve, uh, the program you're providing it.

[14:10](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=850s) So don't come up with one huge, uh, ReAct when we did this, and it's bad. Uh, but rather split. Try to understand, um, how your flow of your program is.

[14:21](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=861s) Like, do you, do you have a retriever step first? Uh, do you have, uh, do you want to have, like, multi-hop, uh, question answering? Do you want to

[14:32](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=872s) bring together different pieces and answer them in one question in the end? This, this is really important. So what do you do? You have to separate your program into modules, which are kind like the prompts and weights.

[14:46](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=886s) You have to add optimizers to tune your prompts and weights, giving a metric. And for us, the metric was quite clear. Um, for the chatbot, we needed to find, like, the right answer.

[14:58](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=898s) And, um, DSPy compiles the same program then into different instructions, few-shot prompting, and fine-tuning. So instead of prompting one language model with a huge prompt, as I told you, uh, please break it apart.

[15:12](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=912s) So there's this analogy to neural networks for maybe for you to better understand how DSPy works. Um, we have this separation into an init and, and forward step, where you define in your init, um, how like in a, in a DSP-- um, how you define like a convolution or dropout layers in PyTorch.

[15:34](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=934s) You use then things like ChainOfThought or ReAct, and, uh, you describe in your forward pass how you do the things you want to do, like retrieving and other stuff.

[15:47](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=947s) Okay. So what were the key learnings? Um,

[15:53](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=953s) so if you want to do this for, like, German texts, answer_exact_match, answer_passage_match only work for English text. So metrics are really not working out of the box.

[16:02](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=962s) We have-- You have to come up with your own metrics. You have to create your own metrics. Um, if you want to use, um, something like ReAct, so there's nothing like a typed ReAct.

[16:12](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=972s) So we-- DSPy comes with these typed, um, typed things like TypedPredictor, um, or TypedChainOfThought. But if you want to have something more specific for your case, you need, uh, you need to come up with your own, uh, module and improve that.

[16:31](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=991s) And evalu-evaluation is also quite fixed on English language, to be honest. Uh, in case you want to use DSPy for German optimization, you need also to create like your own evaluators, especially for the exact and passage match.

[16:46](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=1006s) Okay. So we tried to come up with one big model, and, uh, we found out like this is really not, not working.

[16:56](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=1016s) Okay. Whoops. Sorry. Okay. So what are your key takeaways? Uh, you should definitely write prompts by hand and start with that.

[17:12](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=1032s) Otherwise, uh, you cannot say if your task is achievable at all. Uh, you can't improve what you don't measure. Avoid metrics like looks good to me at 10 without any annotated data or clear goal metric.

[17:24](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=1044s) There's no clear objective to validate if the results are getting even better. You definitely should write br- basic evaluations. Start with some small evaluations like regex or string comparisons, and then only then use techniques like LLM-as-judge.

[17:38](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=1058s) Please don't overcomplicate things. It's already super complicated and, uh, just make very, very small adjustments.

[17:50](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=1070s) So, okay. Um, what do you take home from this? Um, so am I really thinking that you've learned how to use DSPy like from a twenty minutes corporate ti- guy talk?

[18:03](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=1083s) If yes, cool. You're amazing. Wow. [laughs] Uh, but to be honest, um, if you want to learn DSPy, go to all these learning resources from Connor Shorten, from the, uh, DSPy website it-itself.

[18:16](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=1096s) Um, there are all these people like presenting it in a lot better fashion than I did. But what, what I wanted to do is like I wanted to spark some interest in you.

[18:27](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=1107s) The same spark like I got when I, when I saw this conference last year where I learned all the people, uh, where I learned all the guys who created these toolings.

[18:37](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=1117s) And I want to spark that same curiosity in you and to see if you can get like better AI engineers by using not only the tools, but also first principles thinking.

[18:51](https://www.youtube.com/watch?v=IAdZxqjZ45U&t=1131s) And I think we are meant to become like great AI engineers. So this conference is about you getting inspired. Get out and find like some good problems where you can try this out and which are like worth solving with AI that wasn't possible before. [upbeat music]
