AI Engineer World's Fair 2024

Build, Evaluate and Deploy a RAG-based retail copilot with Azure AI

Loading slides…

Read the talk

Build, Evaluate and Deploy a RAG-based retail copilot with Azure AI

Follow a retail RAG backend from product retrieval and customer lookup through prompt assembly, deployment and an evaluation that catches a convincingly invented product.

From a talk by Cedric Vidal, David Smith and Miguel Martinez

Before you start: Basic Python, GitHub and API familiarity will help; reproducing the cloud workflow also requires an Azure account and configured model, search and database resources.

What information does a retail chatbot need?

How do you build a chat backend whose answers come from a retailer’s databases rather than just the model’s training? The workshop starts with that engineering problem: connect an OpenAI model to ordinary database records and a vector search system, then assemble the information into an answer. The production-oriented work is in that surrounding workflow—retrieving evidence, supplying context and checking the result.

The hands-on environment separates instructions, development and cloud resources. Participants need a laptop and a GitHub account. The Windows virtual machine holds the paginated workshop guide; its green text inserts URLs and credentials into the VM’s browser. A participant’s own browser also works, with manual copying from the guide. GitHub Codespaces supplies VS Code, Python dependencies and a terminal, avoiding differences between local installations.

Two-column slide lists a laptop and GitHub account under requirements, and a temporary Azure account, resources and take-home instructions under provisions. A box links to https://skill.info/workshop.Open full source frame
Workshop prerequisites and provided Azure resources.
0:000:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

A tent recommendation, then a purchase-history question

The application is an outdoor retailer’s storefront, with a small catalog of camping equipment, backpacks and hiking clothing. The workshop builds its backend; frontend code and data are also available. Product pages contain much more than names and prices: the Trailblaze hiking pants page includes features, reviews, FAQs, return information, cautions, technical specifications and care instructions. Sarah Lee is already logged in.

A basic capabilities question can be answered from the model’s role and context. A product recommendation needs more. When David Smith asks which tent goes with the Trail Walker shoes, the copilot recommends the TrailMaster X4 tent and refers to an upcoming trip to Andalusia. A subsequent purchase-history question produces an answer naming Sarah’s Cozy Night sleeping bag and Trailblazer hiking tent. The backend therefore needs two distinct sources: product documentation for recommendations and customer records for personalization. Customer context can include a name, loyalty status, location and previous orders.

7:077:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:07 · section reference included

Two retrieval paths meet in one prompt

Azure AI Studio manages the project, model endpoints, connections and flows. Azure AI Services supplies the OpenAI models. Azure AI Search holds the searchable product information, while Cosmos DB holds customer records and purchase histories. Cosmos DB’s role here is the customer-record store; AI Search performs the product vector retrieval.

The request moves through the backend in a definite sequence:

  1. Receive the question as a string, together with chat history.
  2. Use Azure OpenAI to embed the question into the same vector space as the product documentation.
  3. Search for nearby product vectors and retrieve the corresponding product information.
  4. Look up customer information and purchase history in Cosmos DB alongside product retrieval.
  5. Assemble instructions, retrieved products, customer context and the question into a prompt.
  6. Generate an answer and return it to the website.

An embedding makes semantic similarity searchable: nearby points identify potentially relevant products. The generation step then uses the retrieved information as context. Prompt flow implements this retrieval–augmentation–generation sequence as connected components.

10:5511:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:55 · section reference included

Launch the lab, authenticate and inspect the data

The historical lab uses the Contoso Chat repository and its ms-build-lab-322 branch. That branch skips infrastructure deployment because attendees receive preprovisioned Azure resources; the recording directs at-home users to main for deployment instructions. The repository’s archived main branch now uses Prompty rather than the demonstrated Prompt flow implementation, so it is no longer an exact copy of this lab. In the recorded setup, GitHub single sign-on can require using the participant’s own browser rather than the VM browser.

Launch Codespaces from the selected branch, then sign in to the supplied Azure account in the portal and from the development terminal. The temporary workshop account and its resources disappear after the session. The terminal authentication uses az login and a device-login code; completing browser authentication connects subsequent CLI commands to the chosen Azure account. Desktop VS Code can also attach to the Codespace.

Several setup details matter when reproducing the sequence:

  • Preserve the lab branch: When forking, uncheck “Copy the main branch only,” then select the lab branch before creating the Codespace.
  • Use the prepared environment: Its container supplies the command-line dependencies, so a shell is ready once startup completes.
  • Recover the existing session: Confusion between macOS browser tabs and VM browser tabs does not require creating another Codespace; reopen the one already created.
  • Finish terminal authentication: After device login, return to the terminal and confirm the subscription when prompted.

These are separate states: a browser login, an open editor and an authenticated CLI are not interchangeable.

The post-provisioning script populates Cosmos DB and AI Search. David inspects the resource group, opens Cosmos DB’s Data Explorer and finds customer records with purchase histories. He then opens Search Service → Search management → Indexes and finds no indexes. Permission errors appear during troubleshooting, and he recognizes that he has skipped configuration. A deployed search service is not evidence that its index has been populated. Inspect the data and index themselves before debugging the chatbot.

16:2216:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:22 · section reference included

What a flow owns

An audience question about ReAct agents exposes an important distinction: a directed acyclic graph does not loop back through its own nodes. A Python component can nevertheless contain loops or interactive logic. The graph describes dependencies between components; it does not restrict every computation inside a component to a single operation.

Meanwhile, the setup failure narrows to the missing .env creation step. Another question asks whether Enterprise ChatGPT Actions could call the flow through middleware rather than through the retail website. The intended deployment provides precisely that boundary: publish the flow as an endpoint, then let a client call it. Before data setup can succeed, step six must create .env with resource endpoints and keys, plus config.json for AI Studio access. Only then should the provisioning script be rerun.

The product indexing unit is deliberately simple: one Markdown file per product becomes one embedding. David initially agrees that the data is chunked, then corrects himself: this sample does not split each product document into chunks. He contrasts it with AI Studio’s own-data workflow, which performs chunking. That choice determines the retrieval granularity: a match brings in a product document rather than a smaller passage.

40:1340:18
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

40:13 · section reference included

Prompt flow, application orchestration and the playground

The framework comparison reflects the presenters’ choices at recording time:

ToolRole in the discussionMain emphasis
Semantic KernelApplication orchestrationEnterprise versioning and API lifecycle
AutoGenExperimental agent orchestrationResearch and rapidly changing techniques
Prompt flowA connected LLM workflowFlow development, evaluation and deployment

These roles can compose. Semantic Kernel can call multiple deployed Prompt flow endpoints. Cedric Vidal also corrects an overly narrow endpoint-only description: a flow can be embedded as a Python library for integration tests, evaluation or a native application.

The graph is useful for explaining dependencies and inspecting data, but the underlying YAML defines node types, inputs, outputs and connections. The CLI can run that definition with supplied inputs. Cedric describes a Python configuration alternative with a more code-first experience, while expressing uncertainty about its availability at the time. The workshop itself stays with the YAML-backed graph and a single deployed endpoint. The presenters also note overlapping capabilities: Semantic Kernel can build agentic applications, and embedding Prompt flow is useful for reusable Python snippets and tests.

AI Studio’s advantage in this workflow is integration. Model catalog endpoints, Cosmos DB and AI Search connections, evaluation and endpoint management live together. A React application or another orchestrator can consume the resulting endpoint. The playground supplies a route from experimentation to code: configure a RAG application, export it to Prompt flow, then customize the generated flow. The lab project’s playground exposes GPT-4 and GPT-35 Turbo endpoints for experimentation and connection testing.

Copilot Studio sits at a different level in the presenters’ comparison: assembling a complete chatbot or integrating it into an application such as Teams. A custom AI Studio endpoint is an optional extension when the application needs more control. David directs evaluation questions to the AI Studio endpoint, without establishing Copilot Studio’s full evaluation capabilities. The simplest playground export makes the boundary visible: input → chat model node → output. Adding retrieval and customer context is additional application behavior around that model call.

49:4850:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

49:48 · section reference included

Connections, persistence and search quality

The retrieval component is replaceable. When an attendee proposes Chroma, David points to the product-retrieval node: its AI Search connection could be replaced by another vector database or an endpoint. The surrounding flow still needs the returned product information in the expected form. A separate startup question highlights the operational cost of the Azure approach. David emphasizes authentication, security, scaling and monitoring, while acknowledging the setup burden of resource groups and supporting resources. Miguel Martinez describes the startup program’s tiered credits, mentors and learning sessions as assistance with that burden.

As setup continues, the presenters return to two kinds of state. Resource configuration must precede the search-index loading script. Conversation state, meanwhile, travels through the flow as chat history; retaining it beyond the interaction requires database storage. Retrieving product information and carrying conversation history are separate responsibilities, even though both eventually influence the answer.

A connection-name mismatch causes another concrete failure. The lab already has an AOAI-named Azure OpenAI connection selected. A separate connection named default points to a different resource set; switching to it breaks the lab. Keep the working model connection, then create the Cosmos DB connection separately through Connected resources → New connection → Custom connection. Its key-value pairs come from .env. Standard connectors handle supported services, while custom connections carry configuration for other services.

An audience scenario asks about nearest-neighbor search over a million vectors: how do approximation, pruning and search parameters affect recall compared with brute force? David explains why indexed vector stores exist—to make similarity retrieval practical at scale—but does not provide a measured recall–latency curve or tuning recipe. His practical recommendation is broader: in the team’s application experience, combining keyword and vector retrieval has worked better than either alone. This workshop intentionally uses vector-only retrieval. Hybrid search requires both text and vector query components; it should not be assumed to happen automatically for every vector query. Product keyword metadata can also be updated.

David does not know the embedding-dimension ceiling when asked. For present-day configuration, use the documented service limit rather than treating that uncertainty as unlimited support: Azure AI Search limits, checked in August 2026, specify a maximum of 4,096 dimensions per vector field. That is a current configuration constraint, not a measurement from the workshop.

1:04:311:04:51
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:04:31 · section reference included

Customer context is not customer authorization

The security questions move from signing in to controlling what a request can retrieve. An attendee wants Entra External ID and social logins to establish a user-specific scope across the application, AI Search and Cosmos DB. The presenters distinguish developer access to AI Studio from application access to the deployed endpoint, which can use tokens or managed identity. They do not establish an end-to-end implementation of the requested external-user delegation.

First, clarify what crosses the retrieval boundary. The language model receives document text, not just a vector. The vector identifies relevant Markdown documentation; the retrieved text replaces a variable in the prompt template. Any authorization mistake before that point can therefore expose actual document content to generation.

The next question makes the threat concrete: a broadly privileged flow might retrieve another user’s data, including after malicious prompting. The attendee asks whether a calling user’s delegated token can be propagated to downstream services. Cedric describes passing JWTs through a flow in general terms. Miguel explains the intended separation between identity and conversation: establish who the user is before generation, rather than allowing a message such as a changed name to select someone else’s records.

David then corrects the sample’s actual behavior. Sarah’s customer ID is an endpoint parameter, not an identity derived from a delegated token. The app sends that ID and a purchase-history question; a privileged flow queries Cosmos DB for the corresponding orders and returns generated text. The browser has no direct database access, but that alone does not make the lookup user-authorized. The attendee explicitly wants the flow to inherit the caller’s restricted access instead of retaining broad privilege. That implementation remains unresolved in the discussion; Cosmos DB identity and data-plane authorization require explicit configuration beyond supplying a customer ID.

1:17:251:17:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:17:25 · section reference included

From lookup results to a rendered prompt

A question about dynamic routing leads into the prompt itself. The customer prompt contains Jinja loops that insert matching product documents and previous orders. Before the customer’s shoe question reaches OpenAI, it has been surrounded by instructions identifying the Contoso Outdoor role, asking for references for factual statements and supplying the relevant documentation. The model receives that assembled context plus the question, not the question alone.

Three different kinds of control are easy to confuse:

  • Template control: Jinja loops and conditions determine which strings appear in the rendered prompt.
  • Python control: A Python node can branch, loop or perform other internal work.
  • Graph control: Node activation determines whether an entire component runs.

Cedric describes the demonstrated DAG as executing every node, with conditional work inside nodes. His blanket statement that conditional node execution is impossible should not be generalized: Prompt flow’s conditional-control documentation supports activation through activate.when and activate.is. That does not create graph-level cycles or change what this sample executes.

The binding between the lookup and the prompt is ordinary data passing. A database node returns customer information; the next node receives it as customer; Jinja iterates over customer.orders. This compact Python example makes that rendering step explicit, using Sarah and the purchase names shown earlier:

python

from jinja2 import Environment, StrictUndefined

customer = {
    "name": "Sarah Lee",
    "orders": [
        "Cozy Night sleeping bag",
        "Trailblazer hiking tent",
    ],
}
question = "What have I already purchased from you?"

environment = Environment(undefined=StrictUndefined)
template = environment.from_string(
    """Customer: {{ customer.name }}
Previous purchases:
{% for order in customer.orders %}
- {{ order }}
{% endfor %}
Question: {{ question }}
"""
)

prompt = template.render(customer=customer, question=question)
print(prompt)

The collection is available before rendering begins. Input/output bindings connect that returned value to the template; graph arrows visualize those dependencies. Selecting a node opens its editable definition, while the graph itself is especially useful as an inspection view.

VS Code shows flow inputs and outputs beside connected graph nodes with green completion indicators, with text output in the terminal below and workshop instructions on the right.Open full source frame
Prompt flow graph with completed nodes and terminal output in VS Code.
1:26:511:27:19
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:26:51 · section reference included

Run locally, package the flow and understand its limits

The VS Code Run button launches Python against the YAML flow. During development, orchestration runs in Codespaces, which substitutes for a laptop; the databases and OpenAI endpoints remain in Azure. This separation gives a fast place to change prompts, test connections and replace model nodes without first deploying a hosted application. The next deployment step packages the same flow into a container-backed Azure endpoint. Packages can be added to the Docker environment for custom processing, and the presenters describe platform-managed deployment and scaling. For a new implementation, the current Prompt flow lifecycle notice matters: retirement is announced for April 20, 2027, and runtime images no longer receive package or security updates.

The sample always performs both retrieval paths. Even a question asking what else the customer purchased is embedded and sent to product search, while the customer lookup retrieves order history. The model must attend to the useful part of the combined context. That simplicity has a limit: asking what was purchased during the last month does not cause this flow to generate a new database query. It receives a predetermined slice of order history; Cedric tentatively recalls it as the last ten orders.

Natural-language-to-SQL is a different implementation: interpret a request, construct a query with the required filters and execute it. David warns that an intermediate model does not eliminate malicious influence over generated SQL. Cedric points to RAG on PostgreSQL as related work. Its documented implementation combines hybrid retrieval with optional model-generated query filters; it should not be treated as verification of unrestricted SQL generation.

1:33:501:34:05
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:33:50 · section reference included

A fluent answer can still invent a product

Evaluation asks questions that ordinary pass/fail assertions do not answer easily: is the response relevant, grounded in the supplied information, coherent and fluent? Step thirteen uses a Python notebook to expose the evaluation code and prompts. The application generates with GPT-35 Turbo, while GPT-4 judges the resulting question–context–answer combination. David’s model choice reflects the recording’s cost and latency trade-offs; he plans to move a later workshop to GPT-4o.

The judge receives the original question, the retrieved context and the generated answer, then scores particular properties. David describes the notebook’s scale as zero to five. Documented built-in relevance, coherence and fluency evaluators use one to five, so the spoken scale should be read as this demonstration’s description. For the jacket question, the answer addresses Sarah and mentions the Summer Breeze jacket. David reports a groundedness score of 5 for the jacket answer on the stated zero-to-five scale.

The more revealing test asks for a bright toothbrush. Contoso sells camping equipment, not toothbrushes, but nearest-neighbor retrieval still returns documents that are comparatively close to the query. A nearest match does not establish that the requested product exists. GPT-35 Turbo nevertheless recommends a Fresh Breeze travel toothbrush—a product absent from the catalog. The response sounds plausible enough that language quality alone would miss the failure.

Evaluation dimensionReported toothbrush score, maximum 5
Coherence4
Fluency5
Groundedness1
Relevance1

These are GPT-4’s reported scores for this single generated toothbrush answer. The low groundedness score captures the missing catalog support, while high fluency captures the answer’s polished language. The test separates qualities that a quick read could mistakenly combine.

Turn that contrast into a regression suite. Include questions that should produce useful recommendations and requests that should not produce a recommendation at all. AI Studio’s evaluation workflow accepts question–response–context records and automates scoring across them. When the catalog changes or the application moves from GPT-35 Turbo to GPT-4o, rerun the suite to assess the resulting behavior rather than assuming the change is an improvement.

1:41:201:41:44
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:41:20 · section reference included

Use failures to change the system

Evaluation is an input to development, not its final ceremony. David describes a cycle that starts with a use case, exploratory work and an initial flow, then runs that flow against sample data. When scores fall short, change the Jinja instructions, retrieval process, chunking or presentation of context. Test again, then involve user cohorts and red teams before committing the application to production. A hallucinated recommendation can therefore lead to changes in both what the model is told and what evidence it receives.

After deployment, sample live questions and answers and track their evaluation scores. David corrects an initial reference to real-time charts to the more practical example of daily charts. Falling groundedness can signal changes in the catalog, unexpected interactions with product wording or adversarial activity. Investigate the affected examples, revise the system and evaluate the next deployment. Monitoring closes the same loop that began with the development notebook.

LLM Lifecycle Considerations diagram groups steps into ideating/exploring, building/augmenting and operationalizing, with evaluation and modification loops followed by deployment, monitoring and application integration.Open full source frame
LLM lifecycle: exploration, iterative evaluation and operational deployment.
1:49:331:49:47
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:49:33 · section reference included

Uploaded PDFs need a retrieval lifetime

The closing questions extend the application beyond typed text. An attendee wants to upload a purchase-order PDF. David first describes persistent document retrieval: index PDFs in AI Search, retrieve relevant extracted content, insert it into the prompt and provide source references that authorized users can inspect. The sample itself converts everything into text. Multimodal models offer another path, but the discussion of GPT-4o image and audio input is explicitly tied to the rollout then underway. Cedric proposes Florence-2 as an image-to-text stage whose output can feed a text model.

The attendee clarifies that the document is transient. It is needed for the current conversation, not as a permanent addition to the retailer’s knowledge base. Cedric describes a smaller retrieval design:

  1. Upload the PDF and divide its extracted content into chunks.
  2. Embed those chunks and retain their vectors in Python memory.
  3. Compare the question embedding with the chunk vectors.
  4. Put relevant chunks into the conversation’s prompt.
  5. Discard the temporary data when its use is over.

A long-lived vector database is not required for that interaction. Temporary PostgreSQL or Cosmos DB storage followed by deletion is another option when memory is not the desired storage boundary.

The final exchange distinguishes model capability from application capability. A presenter points to image input in the Azure OpenAI playground and the example of generating website code from a screenshot. The attendee already knows that feature; their own RAG interface still accepts neither images nor PDFs. Supporting those uploads requires application work around the model and retrieval path, and the workshop does not implement that interface. Participants leave with their repository fork to continue independently; the closing slide provides repository access for anyone who missed that step.

Thank-you slide displays https://aka.ms/aitour/contoso-chat beside a QR code, with a speaker inset and sponsor logos below.Open full source frame
Source code link for continuing independently.
1:52:021:52:04
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:52:02 · section reference included

Resources

From the talk

  • Retail RAG sample with product retrieval, customer records, evaluation and Azure deployment guidance. The archived main branch uses Prompty and differs from the recorded Prompt flow workshop.

  • Python and React chat application using PostgreSQL hybrid search and optional model-generated query filters.

Updates since the talk

Transcript download
Full transcript

Loading transcript…