AI Engineer World's Fair 2024
Build, Evaluate and Deploy a RAG-based retail copilot with Azure AI
Loading slides…
Read the talk
Build, Evaluate and Deploy a RAG-based retail copilot with Azure AI
Follow a retail RAG backend from product retrieval and customer lookup through prompt assembly, deployment and an evaluation that catches a convincingly invented product.
From a talk by Cedric Vidal, David Smith and Miguel Martinez
Before you start: Basic Python, GitHub and API familiarity will help; reproducing the cloud workflow also requires an Azure account and configured model, search and database resources.
What information does a retail chatbot need?
How do you build a chat backend whose answers come from a retailer’s databases rather than just the model’s training? The workshop starts with that engineering problem: connect an OpenAI model to ordinary database records and a vector search system, then assemble the information into an answer. The production-oriented work is in that surrounding workflow—retrieving evidence, supplying context and checking the result.
The hands-on environment separates instructions, development and cloud resources. Participants need a laptop and a GitHub account. The Windows virtual machine holds the paginated workshop guide; its green text inserts URLs and credentials into the VM’s browser. A participant’s own browser also works, with manual copying from the guide. GitHub Codespaces supplies VS Code, Python dependencies and a terminal, avoiding differences between local installations.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A tent recommendation, then a purchase-history question
The application is an outdoor retailer’s storefront, with a small catalog of camping equipment, backpacks and hiking clothing. The workshop builds its backend; frontend code and data are also available. Product pages contain much more than names and prices: the Trailblaze hiking pants page includes features, reviews, FAQs, return information, cautions, technical specifications and care instructions. Sarah Lee is already logged in.
A basic capabilities question can be answered from the model’s role and context. A product recommendation needs more. When David Smith asks which tent goes with the Trail Walker shoes, the copilot recommends the TrailMaster X4 tent and refers to an upcoming trip to Andalusia. A subsequent purchase-history question produces an answer naming Sarah’s Cozy Night sleeping bag and Trailblazer hiking tent. The backend therefore needs two distinct sources: product documentation for recommendations and customer records for personalization. Customer context can include a name, loyalty status, location and previous orders.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Two retrieval paths meet in one prompt
Azure AI Studio manages the project, model endpoints, connections and flows. Azure AI Services supplies the OpenAI models. Azure AI Search holds the searchable product information, while Cosmos DB holds customer records and purchase histories. Cosmos DB’s role here is the customer-record store; AI Search performs the product vector retrieval.
The request moves through the backend in a definite sequence:
- Receive the question as a string, together with chat history.
- Use Azure OpenAI to embed the question into the same vector space as the product documentation.
- Search for nearby product vectors and retrieve the corresponding product information.
- Look up customer information and purchase history in Cosmos DB alongside product retrieval.
- Assemble instructions, retrieved products, customer context and the question into a prompt.
- Generate an answer and return it to the website.
An embedding makes semantic similarity searchable: nearby points identify potentially relevant products. The generation step then uses the retrieved information as context. Prompt flow implements this retrieval–augmentation–generation sequence as connected components.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Launch the lab, authenticate and inspect the data
The historical lab uses the Contoso Chat repository and its ms-build-lab-322 branch. That branch skips infrastructure deployment because attendees receive preprovisioned Azure resources; the recording directs at-home users to main for deployment instructions. The repository’s archived main branch now uses Prompty rather than the demonstrated Prompt flow implementation, so it is no longer an exact copy of this lab. In the recorded setup, GitHub single sign-on can require using the participant’s own browser rather than the VM browser.
Launch Codespaces from the selected branch, then sign in to the supplied Azure account in the portal and from the development terminal. The temporary workshop account and its resources disappear after the session. The terminal authentication uses az login and a device-login code; completing browser authentication connects subsequent CLI commands to the chosen Azure account. Desktop VS Code can also attach to the Codespace.
Several setup details matter when reproducing the sequence:
- Preserve the lab branch: When forking, uncheck “Copy the main branch only,” then select the lab branch before creating the Codespace.
- Use the prepared environment: Its container supplies the command-line dependencies, so a shell is ready once startup completes.
- Recover the existing session: Confusion between macOS browser tabs and VM browser tabs does not require creating another Codespace; reopen the one already created.
- Finish terminal authentication: After device login, return to the terminal and confirm the subscription when prompted.
These are separate states: a browser login, an open editor and an authenticated CLI are not interchangeable.
The post-provisioning script populates Cosmos DB and AI Search. David inspects the resource group, opens Cosmos DB’s Data Explorer and finds customer records with purchase histories. He then opens Search Service → Search management → Indexes and finds no indexes. Permission errors appear during troubleshooting, and he recognizes that he has skipped configuration. A deployed search service is not evidence that its index has been populated. Inspect the data and index themselves before debugging the chatbot.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
What a flow owns
An audience question about ReAct agents exposes an important distinction: a directed acyclic graph does not loop back through its own nodes. A Python component can nevertheless contain loops or interactive logic. The graph describes dependencies between components; it does not restrict every computation inside a component to a single operation.
Meanwhile, the setup failure narrows to the missing .env creation step. Another question asks whether Enterprise ChatGPT Actions could call the flow through middleware rather than through the retail website. The intended deployment provides precisely that boundary: publish the flow as an endpoint, then let a client call it. Before data setup can succeed, step six must create .env with resource endpoints and keys, plus config.json for AI Studio access. Only then should the provisioning script be rerun.
The product indexing unit is deliberately simple: one Markdown file per product becomes one embedding. David initially agrees that the data is chunked, then corrects himself: this sample does not split each product document into chunks. He contrasts it with AI Studio’s own-data workflow, which performs chunking. That choice determines the retrieval granularity: a match brings in a product document rather than a smaller passage.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Prompt flow, application orchestration and the playground
The framework comparison reflects the presenters’ choices at recording time:
| Tool | Role in the discussion | Main emphasis |
|---|---|---|
| Semantic Kernel | Application orchestration | Enterprise versioning and API lifecycle |
| AutoGen | Experimental agent orchestration | Research and rapidly changing techniques |
| Prompt flow | A connected LLM workflow | Flow development, evaluation and deployment |
These roles can compose. Semantic Kernel can call multiple deployed Prompt flow endpoints. Cedric Vidal also corrects an overly narrow endpoint-only description: a flow can be embedded as a Python library for integration tests, evaluation or a native application.
The graph is useful for explaining dependencies and inspecting data, but the underlying YAML defines node types, inputs, outputs and connections. The CLI can run that definition with supplied inputs. Cedric describes a Python configuration alternative with a more code-first experience, while expressing uncertainty about its availability at the time. The workshop itself stays with the YAML-backed graph and a single deployed endpoint. The presenters also note overlapping capabilities: Semantic Kernel can build agentic applications, and embedding Prompt flow is useful for reusable Python snippets and tests.
AI Studio’s advantage in this workflow is integration. Model catalog endpoints, Cosmos DB and AI Search connections, evaluation and endpoint management live together. A React application or another orchestrator can consume the resulting endpoint. The playground supplies a route from experimentation to code: configure a RAG application, export it to Prompt flow, then customize the generated flow. The lab project’s playground exposes GPT-4 and GPT-35 Turbo endpoints for experimentation and connection testing.
Copilot Studio sits at a different level in the presenters’ comparison: assembling a complete chatbot or integrating it into an application such as Teams. A custom AI Studio endpoint is an optional extension when the application needs more control. David directs evaluation questions to the AI Studio endpoint, without establishing Copilot Studio’s full evaluation capabilities. The simplest playground export makes the boundary visible: input → chat model node → output. Adding retrieval and customer context is additional application behavior around that model call.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Connections, persistence and search quality
The retrieval component is replaceable. When an attendee proposes Chroma, David points to the product-retrieval node: its AI Search connection could be replaced by another vector database or an endpoint. The surrounding flow still needs the returned product information in the expected form. A separate startup question highlights the operational cost of the Azure approach. David emphasizes authentication, security, scaling and monitoring, while acknowledging the setup burden of resource groups and supporting resources. Miguel Martinez describes the startup program’s tiered credits, mentors and learning sessions as assistance with that burden.
As setup continues, the presenters return to two kinds of state. Resource configuration must precede the search-index loading script. Conversation state, meanwhile, travels through the flow as chat history; retaining it beyond the interaction requires database storage. Retrieving product information and carrying conversation history are separate responsibilities, even though both eventually influence the answer.
A connection-name mismatch causes another concrete failure. The lab already has an AOAI-named Azure OpenAI connection selected. A separate connection named default points to a different resource set; switching to it breaks the lab. Keep the working model connection, then create the Cosmos DB connection separately through Connected resources → New connection → Custom connection. Its key-value pairs come from .env. Standard connectors handle supported services, while custom connections carry configuration for other services.
An audience scenario asks about nearest-neighbor search over a million vectors: how do approximation, pruning and search parameters affect recall compared with brute force? David explains why indexed vector stores exist—to make similarity retrieval practical at scale—but does not provide a measured recall–latency curve or tuning recipe. His practical recommendation is broader: in the team’s application experience, combining keyword and vector retrieval has worked better than either alone. This workshop intentionally uses vector-only retrieval. Hybrid search requires both text and vector query components; it should not be assumed to happen automatically for every vector query. Product keyword metadata can also be updated.
David does not know the embedding-dimension ceiling when asked. For present-day configuration, use the documented service limit rather than treating that uncertainty as unlimited support: Azure AI Search limits, checked in August 2026, specify a maximum of 4,096 dimensions per vector field. That is a current configuration constraint, not a measurement from the workshop.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From lookup results to a rendered prompt
A question about dynamic routing leads into the prompt itself. The customer prompt contains Jinja loops that insert matching product documents and previous orders. Before the customer’s shoe question reaches OpenAI, it has been surrounded by instructions identifying the Contoso Outdoor role, asking for references for factual statements and supplying the relevant documentation. The model receives that assembled context plus the question, not the question alone.
Three different kinds of control are easy to confuse:
- Template control: Jinja loops and conditions determine which strings appear in the rendered prompt.
- Python control: A Python node can branch, loop or perform other internal work.
- Graph control: Node activation determines whether an entire component runs.
Cedric describes the demonstrated DAG as executing every node, with conditional work inside nodes. His blanket statement that conditional node execution is impossible should not be generalized: Prompt flow’s conditional-control documentation supports activation through activate.when and activate.is. That does not create graph-level cycles or change what this sample executes.
The binding between the lookup and the prompt is ordinary data passing. A database node returns customer information; the next node receives it as customer; Jinja iterates over customer.orders. This compact Python example makes that rendering step explicit, using Sarah and the purchase names shown earlier:
python
from jinja2 import Environment, StrictUndefined
customer = {
"name": "Sarah Lee",
"orders": [
"Cozy Night sleeping bag",
"Trailblazer hiking tent",
],
}
question = "What have I already purchased from you?"
environment = Environment(undefined=StrictUndefined)
template = environment.from_string(
"""Customer: {{ customer.name }}
Previous purchases:
{% for order in customer.orders %}
- {{ order }}
{% endfor %}
Question: {{ question }}
"""
)
prompt = template.render(customer=customer, question=question)
print(prompt)
The collection is available before rendering begins. Input/output bindings connect that returned value to the template; graph arrows visualize those dependencies. Selecting a node opens its editable definition, while the graph itself is especially useful as an inspection view.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Run locally, package the flow and understand its limits
The VS Code Run button launches Python against the YAML flow. During development, orchestration runs in Codespaces, which substitutes for a laptop; the databases and OpenAI endpoints remain in Azure. This separation gives a fast place to change prompts, test connections and replace model nodes without first deploying a hosted application. The next deployment step packages the same flow into a container-backed Azure endpoint. Packages can be added to the Docker environment for custom processing, and the presenters describe platform-managed deployment and scaling. For a new implementation, the current Prompt flow lifecycle notice matters: retirement is announced for April 20, 2027, and runtime images no longer receive package or security updates.
The sample always performs both retrieval paths. Even a question asking what else the customer purchased is embedded and sent to product search, while the customer lookup retrieves order history. The model must attend to the useful part of the combined context. That simplicity has a limit: asking what was purchased during the last month does not cause this flow to generate a new database query. It receives a predetermined slice of order history; Cedric tentatively recalls it as the last ten orders.
Natural-language-to-SQL is a different implementation: interpret a request, construct a query with the required filters and execute it. David warns that an intermediate model does not eliminate malicious influence over generated SQL. Cedric points to RAG on PostgreSQL as related work. Its documented implementation combines hybrid retrieval with optional model-generated query filters; it should not be treated as verification of unrestricted SQL generation.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A fluent answer can still invent a product
Evaluation asks questions that ordinary pass/fail assertions do not answer easily: is the response relevant, grounded in the supplied information, coherent and fluent? Step thirteen uses a Python notebook to expose the evaluation code and prompts. The application generates with GPT-35 Turbo, while GPT-4 judges the resulting question–context–answer combination. David’s model choice reflects the recording’s cost and latency trade-offs; he plans to move a later workshop to GPT-4o.
The judge receives the original question, the retrieved context and the generated answer, then scores particular properties. David describes the notebook’s scale as zero to five. Documented built-in relevance, coherence and fluency evaluators use one to five, so the spoken scale should be read as this demonstration’s description. For the jacket question, the answer addresses Sarah and mentions the Summer Breeze jacket. David reports a groundedness score of 5 for the jacket answer on the stated zero-to-five scale.
The more revealing test asks for a bright toothbrush. Contoso sells camping equipment, not toothbrushes, but nearest-neighbor retrieval still returns documents that are comparatively close to the query. A nearest match does not establish that the requested product exists. GPT-35 Turbo nevertheless recommends a Fresh Breeze travel toothbrush—a product absent from the catalog. The response sounds plausible enough that language quality alone would miss the failure.
| Evaluation dimension | Reported toothbrush score, maximum 5 |
|---|---|
| Coherence | 4 |
| Fluency | 5 |
| Groundedness | 1 |
| Relevance | 1 |
These are GPT-4’s reported scores for this single generated toothbrush answer. The low groundedness score captures the missing catalog support, while high fluency captures the answer’s polished language. The test separates qualities that a quick read could mistakenly combine.
Turn that contrast into a regression suite. Include questions that should produce useful recommendations and requests that should not produce a recommendation at all. AI Studio’s evaluation workflow accepts question–response–context records and automates scoring across them. When the catalog changes or the application moves from GPT-35 Turbo to GPT-4o, rerun the suite to assess the resulting behavior rather than assuming the change is an improvement.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Use failures to change the system
Evaluation is an input to development, not its final ceremony. David describes a cycle that starts with a use case, exploratory work and an initial flow, then runs that flow against sample data. When scores fall short, change the Jinja instructions, retrieval process, chunking or presentation of context. Test again, then involve user cohorts and red teams before committing the application to production. A hallucinated recommendation can therefore lead to changes in both what the model is told and what evidence it receives.
After deployment, sample live questions and answers and track their evaluation scores. David corrects an initial reference to real-time charts to the more practical example of daily charts. Falling groundedness can signal changes in the catalog, unexpected interactions with product wording or adversarial activity. Investigate the affected examples, revise the system and evaluate the next deployment. Monitoring closes the same loop that began with the development notebook.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Uploaded PDFs need a retrieval lifetime
The closing questions extend the application beyond typed text. An attendee wants to upload a purchase-order PDF. David first describes persistent document retrieval: index PDFs in AI Search, retrieve relevant extracted content, insert it into the prompt and provide source references that authorized users can inspect. The sample itself converts everything into text. Multimodal models offer another path, but the discussion of GPT-4o image and audio input is explicitly tied to the rollout then underway. Cedric proposes Florence-2 as an image-to-text stage whose output can feed a text model.
The attendee clarifies that the document is transient. It is needed for the current conversation, not as a permanent addition to the retailer’s knowledge base. Cedric describes a smaller retrieval design:
- Upload the PDF and divide its extracted content into chunks.
- Embed those chunks and retain their vectors in Python memory.
- Compare the question embedding with the chunk vectors.
- Put relevant chunks into the conversation’s prompt.
- Discard the temporary data when its use is over.
A long-lived vector database is not required for that interaction. Temporary PostgreSQL or Cosmos DB storage followed by deletion is another option when memory is not the desired storage boundary.
The final exchange distinguishes model capability from application capability. A presenter points to image input in the Azure OpenAI playground and the example of generating website code from a screenshot. The attendee already knows that feature; their own RAG interface still accepts neither images nor PDFs. Supporting those uploads requires application work around the model and retrieval path, and the workshop does not implement that interface. Participants leave with their repository fork to continue independently; the closing slide provides repository access for anyone who missed that step.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
From the talk
Retail RAG sample with product retrieval, customer records, evaluation and Azure deployment guidance. The archived main branch uses Prompty and differs from the recorded Prompt flow workshop.
Python and React chat application using PostgreSQL hybrid search and optional model-generated query filters.
Further reading
Configure conditional execution and bypass behavior with node activation conditions.
How keyword and vector retrieval run together and combine results through reciprocal rank fusion.
Configure Microsoft Entra identities and distinguish database data access from resource-management permissions.
Updates since the talk
- Prompt flow documentation and retirement noticeDocumentation
Legacy flow guidance and the announced April 20, 2027 retirement, including migration guidance and runtime-image support limitations.
Full transcript
Loading transcript…