AI Engineer Summit 2025

The LLM Triangle: Engineering Principles for Robust AI Applications

Almog Baku26:19

Loading slides…

Read the talk

The LLM Triangle: Engineering Reliable Applications

Reliable LLM applications start with an explicit procedure, then match models, software and contextual data to each step of the work.

From a talk by Almog Baku

Before you start: Basic familiarity with LLM prompts and API calls is helpful; the Python example only constructs prompt text.

Why does the proof of concept break?

What separates an impressive LLM prototype from an application that works reliably for customers? Small workflow enhancements and wrappers are an early use of a much broader computing resource. Almog Baku suggests thinking of LLMs alongside CPUs and GPUs: a new resource whose capabilities should influence how we design software. His quantum-computing analogy pushes this toward changing programming languages, but that part is too categorical—Cirq already supports quantum programming in Python. The useful distinction is that LLM applications combine conventional languages such as Python, JavaScript and Go with instructions written in natural language.

Iceberg slide with the text “Think of LLMs as a new computing resource, just like CPUs” and a circular presenter inset.Open full source frame
LLMs as a new computing resource: the tip of the iceberg.

That combination makes a proof of concept easy to build and consistent production quality difficult to achieve. The prototype works on the cases its builder has tried; then a real customer supplies the exceptional input that breaks it. The customer sees a failed application, regardless of how impressive the underlying model is. Baku describes LLM-native applications as 10% sophisticated models and 90% experimentation and data-driven engineering, immediately acknowledging that he made up the split. The practical thesis is reliability comes from engineering the application around the model.

Baku grounds the framework in experience founding companies, including one in AI infrastructure, and founding GenAI Israel, which he describes as Israel’s largest such community. At the time of the talk, Baku reports 5,000 engineers, researchers and data scientists in the community. He also reports helping customers build dozens of applications over the preceding two years. The framework distills that application-building experience into a repeatable way to approach the work.

0:411:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:41 · section reference included

Define the procedure before choosing the implementation

The LLM Triangle connects three choices: the model, the engineering techniques and the contextual data. A fourth element guides all three: the standard operating procedure, or SOP. Borrowed from manufacturing, an SOP specifies how to carry out routine work while maintaining a required level of quality. It gives the application a process to implement rather than merely a desired outcome.

Consider a fulfillment center. A package that comes apart creates support costs, lost goods and reputational damage. The center cannot depend on every packer independently discovering the right method. Instead, it provides a sequence:

  1. Check that all the items are present.
  2. Mark them in the system.
  3. Apply the label.
  4. Inspect the package.

Baku’s analogy is to treat the LLM as a smart, agreeable intern who is inexperienced at this particular job. Intelligence does not supply knowledge of your organization’s procedure. Give the model an explicit recipe for the task.

For a landing-page builder, writing that recipe starts with an expert interview. Ask how the expert approaches a new page and which elements matter. Then describe the process back to them. Their corrections reveal where your description has skipped something they consider obvious. This decomposition reduces the cognitive load of the task while exposing the decisions the application must reproduce.

A first description might jump directly from gathering information to writing the page. The expert, however, constructs a compelling story between those two steps. That story determines what the landing page should communicate. If the SOP omits it, the implementation asks the model to make an important reasoning jump without guidance. Experts may not notice that they perform these intermediate steps, so extracting them requires specific questions. Once the steps are explicit, draw the process as a graph when loops or recursion make a simple list inadequate.

4:284:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:28 · section reference included

Implement each step with the right tool

Once the SOP describes what to build, engineering techniques provide ways to implement it. Baku groups them by where the work happens:

LayerTechniques
PromptChain of thought, role assumption, few-shot examples
SoftwareChains, tools, fine-tuning
Across bothReAct, structured output

The point of this grouping is to choose a technique according to the step it must accomplish. A prompt technique alone need not carry the entire process.

LLM-native architecture is the arrangement of those steps into a working system. Flow engineering, chains and workflows describe much the same architectural activity here: split the SOP into components, each responsible for a particular transformation. Some components call an LLM; others are ordinary code.

A Wikipedia-style article writer makes the division concrete. Its SOP begins by identifying the subject, researching the literature, writing a table of contents and expanding each section. The implementation adds the necessary intermediate operations:

  1. Normalize and distill the user’s request.
  2. Categorize the subject.
  3. Search Google for relevant information.
  4. Convert the retrieved HTML into Markdown using code.
  5. Give the processed material to an LLM component that writes the table of contents.
  6. Expand the sections.

Baku motivates the HTML conversion with his observation that LLMs work better with Markdown than HTML. The architectural lesson does not require every operation to use a model: use software to prepare the information before asking the model to reason or write with it.

An LLM’s capabilities do not make it the best implementation for every step. Where a transformation can be handled deterministically, conventional code provides consistent behavior and a more direct way to inspect what happened. Reserve model calls for the parts that benefit from them.

9:039:16
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:03 · section reference included

Choose where the agent gets discretion

In this framework, an agent is a standalone component of the architecture that calls an LLM: a model plus a prompt template. That is a deliberately broad definition. An agent may be ephemeral, reusable or recursive; its basic responsibility is to take the process from one state to another. An autonomous agent adds discretion over how to accomplish the task, using tools to plan and act.

An assistant can use either design. In the autonomous version, give the model weather and retrieval-augmented generation, or RAG, tools and let it decide how to complete the request. In the crafted version, the application first asks an LLM to classify the message into an explicit enum: emotional support, weather, trip planning or support. That selection determines the next process, such as retrieving knowledge or fetching weather information. A subsequent LLM component composes the response; only that final response is shown to the user.

The autonomous version looks cleaner because the developer writes functions, exposes how to call them and delegates the sequence. It is quick to build and leaves room for creative solutions. The cost appears when behavior varies: an excellent solution on one run may become a failed customer demonstration on another, with a difficult debugging trail. A crafted flow makes the next step explicit, improving predictability and making failures easier to locate, but takes more development time and resources.

DesignMain advantageMain cost
Autonomous tool useFast to build; flexible solutionsVariable behavior; harder debugging
Crafted flowExplicit sequence; easier inspectionMore implementation effort

These are trade-offs in how the application controls its work, not a reason to choose one design for every task.

Baku recommends scoping autonomy inside clear boundaries. Give an autonomous component an area in which it can be creative while keeping the surrounding process constrained. If that component produces a poor result, the consequences remain limited to its assigned area. The architecture can therefore combine predictable orchestration with discretion where discretion is useful.

12:0612:28
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:06 · section reference included

Match the model to the task and its constraints

Different steps need different capabilities. Summarization and entity extraction generally demand less from a model than complex reasoning. Model selection must also account for the environment in which the step runs:

  • Infrastructure and performance: Self-hosting an open model brings infrastructure work; proprietary services from Anthropic or OpenAI offer a different operational arrangement. Latency matters alongside capability.
  • Cost-effectiveness: Compare the model’s price with the value the task creates and the budget available.
  • Data availability: Labeled examples may support few-shot prompting or fine-tuning. A task that instead depends heavily on pretrained knowledge places different demands on the model.

The decision is therefore about a task, its data and its operating constraints—not just a model ranking.

Factors in Model Selection slide listing four categories with examples covering reasoning, deployment, latency, pricing, budgets, fine-tuning, and in-context learning.Open full source frame
Model selection factors: task complexity, infrastructure and performance, cost-effectiveness, and data availability.

Baku’s model-selection heuristic favors large models for broad capability, with o1 among his examples, while recognizing that capability does not imply low cost or low latency. Smaller models are candidates for simpler tasks, especially when useful task data is available. They are more likely to struggle with complex work; a large model is a stronger starting point for a complex task with a limited dataset.

SituationStarting heuristic
Simple task with useful examplesConsider a smaller model
Complex task with little task dataStart with a larger model
Broad capability neededExpect cost and latency trade-offs

This is a selection guide, not a measured comparison of the models named in the talk.

The optimization sequence follows from that uncertainty:

  1. Start with a large model and establish how the application behaves.
  2. Collect data from the task.
  3. Add useful few-shot examples, potentially selected dynamically.
  4. Optimize incrementally, considering fine-tuning or a model change once the accumulated evidence supports it.

Baku places fine-tuning later—potentially after months of learning about the application—rather than making it a prerequisite on day zero. Early model capability buys room to understand the task before taking on that additional work.

16:2916:43
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:29 · section reference included

Build the input the model actually needs

Data is the triangle’s central practical concern. Baku explains model input through a simplified picture: application software assembles instructions, user messages and context into an input that the model consumes. That picture is useful for thinking about prompt construction, but it is not a portable API contract. The current Claude Messages API, for example, uses user and assistant messages with a separate top-level system parameter; its documentation does not establish the hidden serialization described in the talk.

A prompt template separates reusable instructions from the values supplied for a particular request. Substitute different variables and the application produces a different concrete input while retaining a maintainable template. The template itself does not create memory: a stateless call needs the relevant information included in its input. Organizing data into variables helps the application manage that input; it does not remove the need to decide what the model should receive.

That decision is more demanding than finding one fact in a long document. A needle-in-a-haystack test asks whether the model can recover a hidden item. Baku questions whether success on that test captures application needs: real work may require many relevant facts from a much larger collection. Too much broad context makes the task harder to focus; too little leaves the model without the information needed for a correct answer. Supply enough context to solve the task, and select it for relevance.

19:4620:03
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

19:46 · section reference included

Use examples to specify the intended categories

Consider the instruction to classify a product by its name. Given PlayStation, the model returns gaming console. That is a reasonable answer, but the instruction has not specified the level or vocabulary of classification the application wants. Few-shot examples can communicate that convention through in-context learning, without fine-tuning.

Keep the instruction the same and add TV → consumer good and printer → business equipment. In Baku’s example, the answer for PlayStation changes to consumer electronic. Both answers describe the product, but the second is closer to the intended vocabulary. The examples supply information that the original instruction left ambiguous.

The same prompt construction can be expressed directly in Python. The operation below adds the example pairs while preserving both the instruction and the product being classified:

python

instruction = "Classify the product by its name."
product = "PlayStation"
examples = [
    ("TV", "consumer good"),
    ("printer", "business equipment"),
]


def classification_prompt(product, examples):
    lines = [instruction]
    lines.extend(f"{name} -> {category}" for name, category in examples)
    lines.append(f"{product} ->")
    return "\n".join(lines)


prompt_without_examples = classification_prompt(product, [])
prompt_with_examples = classification_prompt(product, examples)

These strings are inputs for a model call; the code does not itself assign a category. The teaching mechanism is to show the desired convention through examples when describing it in words is difficult. Baku compresses that principle into “show, don’t tell.”

Adding examples without changing the request

Constructed example: Arrow formatting and object IDs are teaching representations matching the adjacent Python code. The comparison shows prompt construction, not an executed model call.

Product — unchanged
PlayStation

Operation: Insert the two demonstration pairs into the classification prompt while preserving the instruction and product.

Instruction

Before: Prompt without examples
Classify the product by its name.
After: Prompt with examples · Unchanged
Classify the product by its name.

TV example

Before: Prompt without examples
Not present
After: Prompt with examples · Added
TV -> consumer good

Printer example

Before: Prompt without examples
Not present
After: Prompt with examples · Added
printer -> business equipment

Classification request

Before: Prompt without examples
PlayStation ->
After: Prompt with examples · Unchanged
PlayStation ->
The added pairs communicate a classification convention; the product and instruction stay unchanged.
22:5123:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

22:51 · section reference included

Select examples for the current request

A large example collection does not need to become one large prompt. Dynamic few-shot selection chooses examples for the current use case. Baku suggests selecting five or seven examples rather than supplying the entire collection. Similarity-based selection can use vector search, much like RAG; rule-based selection can use keywords or categories. In either case, the application retrieves demonstrations of how to perform the task, choosing the subset most useful for this request.

This returns the triangle to its organizing relationship: the model, engineering techniques and contextual data all operate under the SOP. The procedure specifies the work; engineering techniques implement and specialize it; the data gives the model what it needs at each step.

Green triangle on concentric circles with vertices labeled Model, Engineering Techniques, and Contextual Data, a center labeled Optimal LLM Usage, and Standard Operating Procedure around the top.Open full source frame
The LLM Triangle connects model, engineering techniques, and contextual data within a standard operating procedure.

Baku closes by putting data first. Inspect the data available to the application, then use the SOP and engineering techniques to shape what reaches the model. Better task-specific input is the practical route toward better output. He ends by inviting questions through LinkedIn and pointing viewers to the full white paper; his website lists his publications.

24:3924:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

24:39 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…