AI Engineer Summit 2025
Lessons from building GenAI based applications — Juan Peredo
Loading slides…
Read the talk
Lessons from building GenAI based applications
Generated code is only the beginning: reliable GenAI applications need deliberate hosting, evaluation, prompt management, agent orchestration, cost modeling, and traces.
From a talk by Juan Peredo
Before you start: Basic familiarity with web application development, API calls, and LLM prompts will help; retrieval and agent workflows are explained as they appear.
A prompt can start a website. Who finishes the application?
Give an LLM a prompt and get a website in a minute: the promise hides the difference between generating code and delivering an application. Juan Peredo opens with that gap. The model can produce a useful starting point, but the remaining work does not disappear.
Codeium and Cursor bring that assistance into an IDE; copying code into an LLM and asking a question is another workable interface. The appropriate model depends on the task. Routine website boilerplate asks less of a model than complex application logic or a recently released framework such as Svelte 5. For the latter, Peredo names DeepSeek V3 and Claude as stronger options. Faster coding helps, but integrating AI into the application’s user experience introduces its own engineering responsibilities.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
AI adds a second set of engineering decisions
The conventional application still needs its frontend, backend, and infrastructure as code. Adding AI brings another set of choices: which model to use, whether to fine-tune it, how to write its prompts, and whether retrieval-augmented generation should supply information for its answers. Those choices also create a validation problem. A model’s willingness to answer does not establish that its answer is correct.
Model selection becomes an ongoing lifecycle, not a one-time dependency choice. Newly released models may perform well on benchmarks without working well inside the application already built around another model. Evaluate both the deployed model and potential replacements against the application’s actual needs. Hosting adds another dimension: CPU and memory planning may now include GPUs, with a choice between cloud and on-premises infrastructure.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Choose where experimentation ends and operations begin
For exploration, Ollama lets you download models and interact with them locally without paying a hosted inference API for each request. The constraint moves to the machine: it must have enough capacity for the chosen model. The same category of tooling can support on-premises hosting. In the cloud, the question expands from which provider to use to whether suitable GPUs are available and what they cost. Peredo names Google Cloud, AWS, and Azure as familiar options.
Deployment tools can reduce the amount of infrastructure work around the model:
| Option | Mechanism | Useful when |
|---|---|---|
| Modal | Python decorators and CLI deployment; platform builds and pushes containers | Moving Python workloads into the cloud |
| SkyPilot | Configured cloud choices and workload placement | Seeking lower infrastructure costs across clouds |
| Hugging Face Transformers | Run models directly through a local library | Experimenting without an additional hosting tool |
Peredo describes Modal as a way to take Python code into cloud execution with little deployment ceremony. He describes SkyPilot as spreading clusters and containers across configured clouds to reduce cost. Transformers offers the more direct local path. These options simplify different parts of the job; none removes the need to choose a hosting arrangement that fits the model.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A working chatbot can still answer three times three with thirty-three
Peredo estimates that a basic chatbot built with existing frameworks can take one or two hours. Correctness is the harder problem. He reports asking an LLM what three times three was and receiving thirty-three. The interface worked; the answer failed. Validating and moderating what reaches the user therefore belongs to the application, even when a framework makes the chat interface easy to assemble.
Prompt engineering adds instructions and context to the user’s question. Those instructions need to be precise enough for the model to follow, but precision is not enforcement: the model may still disregard them.
A guardrail classifier adds a separate decision around the main model. In Peredo’s example, a secondary LLM classifies incoming questions as safe or problematic, including attempts to evade restrictions or requests involving criminal activity. The same pattern can classify outgoing answers. This creates another opportunity to catch an unwanted interaction, at the cost of another model call, additional latency, and another fallible judgment.
Retrieval-augmented generation, or RAG, supplies relevant company information when the model answers:
- Split documents into smaller chunks. Paragraphs are a useful illustration, not a required chunk format.
- Store the chunks in a vector database for retrieval.
- Use the incoming question to perform a similarity search.
- Put the retrieved information into the model’s context as material for answering the question.
The answer is now dependent on what the retrieval system contains and returns. If the database supplies outdated information, the model can produce an outdated answer while following the provided context. Data freshness is part of answer quality.
Fine-tuning changes the model through additional training. Peredo describes it as more expensive and slower than the preceding techniques, with no guarantee of improved responses. Poorly chosen training information can degrade behavior: the model may forget prior information or struggle with conflicts between old and new material. Prompting, guardrails, retrieval, and fine-tuning can be combined, but none makes chatbot correctness automatic.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Evaluate candidates, providers, and recorded runs
Evaluation starts while the application is still an idea. Ollama supports local model trials; Hugging Face provides models and Spaces containing applications whose code and behavior can be inspected. These are ways to explore whether a model or interaction pattern fits before committing to an implementation.
OpenRouter extends that exploration to open and closed models through a common service. Peredo describes sending a prompt to Llama 3 and having the service choose the cheapest provider offering that model. For present-day implementations, that description needs one qualification: the documented default uses price-weighted load balancing with uptime considerations, rather than invariably selecting the cheapest endpoint; explicit sorting and provider controls are available.
Peredo reports depositing about $50 into OpenRouter six months earlier and having about $40 remaining despite daily exploratory use. He does not specify the workload. His production concern is more consequential than that personal spending example: an application should have safeguards around routing and visibility into which provider receives its requests. Convenient experimentation does not settle production provider policy.
LangSmith, from the creators of LangChain, extends evaluation into recorded application behavior. Recording LLM runs makes it possible to compare performance over time and see what the application and its models have actually done. Evaluation should continue through development and operation, rather than ending when a candidate model looks promising.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Make prompts accessible to the people who understand the task
Hard-coding a prompt is convenient for the developer writing the first version. Externalizing it makes the behavior accessible to other people. Peredo uses LangChain Hub to illustrate a workflow in which an education expert, without programming experience, can edit a prompt, run it against an LLM, inspect the output, and commit the version they want the application to consume. The application must be wired to consume those externally managed prompts for that workflow to reach users.
External prompts also make model changes easier to handle. A prompt that works with one Llama release may produce different results with the next. Keeping prompts accessible for editing and testing shortens the adaptation loop. Peredo’s recommendation is to keep prompts out of embedded application code so that domain collaboration and model-specific iteration do not require editing the application’s logic.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From answering a request to choosing functions
Agents extend the interaction beyond a chat response. Given the text of a book, an application might produce audio and translations into multiple languages. Peredo presents this as a promising direction still being developed: the model’s output can guide work that produces an artifact or changes something outside the conversation.
The simple agent diagram has a start, an assistant, tools, and an end. The assistant contains an LLM that analyzes the incoming request and chooses tool calls—function calls—until it has enough information or results to finish. In the book example, the assistant calls audio creation, then translation, then ends. The useful capability is choosing those actions without requiring the developer to hard-code every possible path through them.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Parallelize independent work to shorten the critical path
The freedom to choose actions introduces waiting time. In Peredo’s sequential example, four or five model calls taking three seconds each produce twelve to fifteen seconds of model latency. That wait conflicts with the responsiveness people expect from web applications; he warns that users may leave rather than wait.
LangChain, LlamaIndex, and LangFlow offer ways to organize branching and concurrent calls; the accompanying slide specifically names LangGraph. Return to the book request: node a can dispatch audio creation and translation independently, then node d can combine their results. The branches share the original book text. If the desired audio were instead a reading of the translation, that dependency would require a different ordering.
Python’s asyncio.gather expresses the independent branch operation directly:
python
import asyncio
from collections.abc import Awaitable, Callable
async def create_book_outputs(
book_text: str,
create_audio: Callable[[str], Awaitable[bytes]],
translate: Callable[[str], Awaitable[str]],
) -> tuple[bytes, str]:
audio, translation = await asyncio.gather(
create_audio(book_text),
translate(book_text),
)
return audio, translation
The split-and-rejoin diagram contains four work nodes: a, b, c, and d. With two seconds per node, running b and c concurrently reduces the illustrated duration from eight seconds to six. The critical path is a, then the slower branch, then d:
Concurrency helps where work is independent; it does not remove the time spent on the shared steps before and after the branches.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Price the complete customer interaction
Agent costs accumulate across the steps needed to serve one user. Peredo considers a hypothetical call center handling 3,000 customer calls per day, with 15 tool or function calls per customer call. He chooses OpenAI o1 for the illustration, quoting $15 per million input tokens and $60 per million output tokens, and assumes 1,500 input tokens and 3,000 output tokens per model request. The relevant unit is the whole customer interaction, not the price of a single token or request.
Peredo reports $3.24 per customer call, $9,270 per day, and almost $300,000 per month, but those figures do not reconcile. Treating each of the 15 steps as one identical billable model request, the stated rates and token counts instead yield $3.0375 per customer call, $9,112.50 per day, and $273,375 over 30 days.
The arithmetic correction preserves the lesson: apparently small token rates can become a substantial operating expense when multiplied across an agent’s repeated requests and the application’s traffic.
For provider endpoints billed by usage, additional users bring additional inference expense. Under a fixed workload per user, that expense scales linearly with traffic. Product pricing needs to cover the work the agent actually performs, including the repeated model interactions behind a single customer-facing action.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Buy sufficient capability at a workable speed
Renting GPUs changes the billing model but does not eliminate infrastructure expense. Peredo gives an illustrative comparison of an A100 at potentially $4–5 per hour against a traditional EC2 instance costing fractions of a cent or a few cents per hour. These are broad examples without matching provider, region, instance configuration, or billing terms; they are not an equivalent-capacity comparison.
Peredo describes a spread of more than twentyfold between the endpoints of his model-price chart. That motivates a task-specific question: does this application require the expensive model’s capability, or could one or two cheaper models complete the same task? The evaluation needs to establish adequacy before the price comparison becomes actionable.
Output speed is another part of provider selection. For Llama 3.1 70B, Peredo reports Cerebras and Groq above 1,000 output tokens per second, Cerebras above 2,000, and unnamed providers at 31 and 29. Those are his chart comparisons, not established end-to-end application timings; the workload and measurement conditions are unspecified, and the Groq figure remains uncorroborated. The related Cerebras historical speed report provides context for the Cerebras claim.
Generation speed affects how long users wait, and model size also affects inference time. Peredo’s selection principle is to use a smaller model when it can do the job, associating that choice with faster responses, lower expense, and lower environmental impact. Capability that the task does not need can still impose a cost on every request.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Revisit the call center with a cheaper model
The call-center example returns with Llama 3.3 70B replacing o1, provided the task does not require o1’s capabilities. Peredo reports the price falling from above $3 to $0.52 per customer call and from almost $300,000 to almost $50,000 per month. The slide identifies the replacement as Llama 3.3 70B on Groq and shows headline totals of $0.52 per call, $1,560 per day, and $46,800 per month. Those headline totals are consistent with 3,000 daily calls over 30 days, but the underlying replacement token rates are not supplied and the slide’s smaller calculation notes contain inconsistent totals. This is a price illustration, not a demonstration that the two models produce equivalent results.
The decision still turns on the use case. If the cheaper model meets the application’s requirements, its lower cost applies repeatedly across the workload. If it does not, the attractive price cannot make it an adequate replacement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Trace the failure all the way back to the prompt
Multiple interacting model nodes make an agent difficult to debug. Unlike a fixed sequence of application operations, a probabilistic component introduces variation into the path and the values passed between steps. Peredo illustrates the surprise as a process that works repeatedly and then fails on a later attempt.
His concrete example is a database lookup for a user named John. The lookup had been working, then began failing. Traces revealed that the incoming name was lowercase john rather than John. Because the prompt had already been externalized, the team could add the instruction “ignore case.” Peredo reports that the lookup then worked again. The useful chain of evidence was the failing input, the behavior it triggered, and the prompt controlling that behavior.
The same lowercase input after a prompt repair
Constructed example: The display labels and isolated instruction field organize the anecdote for comparison. The complete original prompt, database schema, returned record, and trace payload were not supplied.
john
Operation: Add the instruction “ignore case” to the externalized lookup prompt.
Incoming name
john
john
User name being sought
John
John
Added prompt instruction
Not present
ignore case
Reported lookup outcome
Lookup fails
Lookup works again
In this multi-model agent, Peredo expects ordinary debugger sessions and print statements would have made diagnosis much harder. LangSmith is one way to obtain the needed visibility; custom instrumentation is another. What matters is having the execution evidence available when the intermittent failure occurs.
The closing trace example makes the required evidence concrete. The agent calls Llama Guard, then Llama 3.1, then Mistral. A useful trace preserves that order together with each call’s metadata, inputs, and outputs. When a step fails, it also retains the error and the information entering and leaving the model, so the developer can reconstruct what happened rather than infer it from the final response.
Observability is part of building the agent. The application needs a record of the path it actually took, especially when that path or its intermediate values can vary between requests.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Introduction with a GPU inference example, Python decorators, container environments, and CLI execution.
Evaluate application outputs using datasets, compare experiments, and monitor production traces.
Model specifications and token pricing for the o1 model used in the call-center illustration.
Further reading
Historical report of 2,100 output tokens per second, including speculative decoding and expected speed variation.
Updates since the talk
Current controls for provider selection, price and performance sorting, and fallbacks.
Full transcript
Loading transcript…