AI Engineer World's Fair 2025

Data is Your Differentiator: Building Secure and Tailored AI Systems

Loading slides…

Read the talk

Data Is Your Differentiator: Building Secure and Tailored AI Systems

A useful generative AI application needs more than a model: it needs application-specific data, controlled retrieval, observable behavior, and a repeatable path from evaluation to updates.

From a talk by Mani Khanuja

Before you start: Familiarity with language-model prompts and basic API calls will help; retrieval-augmented generation and its main components are explained as they appear.

Dancing coconuts and a deeper foundation

Dancing coconuts open Mani Khanuja’s presentation, courtesy of the Amazon Nova models. They are a playful example of what generative AI can produce—and a memory aid that will return when the application is ready to scale. First comes the less visible work that makes an AI application useful to a business.

The more ambitious the application, the deeper its foundation must be. That foundation is data: the material that represents the company, its brand, and its organization. The iceberg slide makes the supporting work visible, placing mindset, people, process, and technology beneath the business value above the surface.

Iceberg diagram places generative AI business value above water and mindset, people, process, and technology below the surface.Open full source frame
Get your data foundation right: mindset, people, process, and technology.

Data preparation is familiar from earlier machine learning systems, but generative AI changes the requirements according to the application and its business purpose. Transformation, parsing, and loading still matter. So do the interactions between data, technology, and people. If useful information remains trapped in organizational silos, an application that could use it cannot deliver the intended experience. Getting the foundation right therefore includes how information becomes available, not just how it is formatted.

0:000:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:00 · section reference included

The application determines the data requirements

Consider a travel agent. It needs a customer profile to personalize the conversation; learning the relevant details too late is like remembering someone’s name after they have left the room. That personalization also creates a responsibility to prevent disclosure of personally identifiable information, or PII. The agent then needs company information: whether a customer qualifies for a ticket refund depends on travel policies and the relevant airline’s rules. A fluent answer without those inputs cannot reliably resolve the request.

An employee chatbot has a different boundary. It needs company knowledge, but each employee must receive only information they are authorized to access. It may appear in Slack or a custom application, with information drawn from several sources. A marketing application adds another purpose: producing content that fits the brand. Khanuja introduces that third case without developing a separate implementation.

ApplicationRequired dataGoverning concern
Travel agentCustomer profiles and travel policiesPersonalization without PII disclosure
Employee chatbotCompany knowledge from multiple sourcesPreserve employee access permissions
Marketing applicationBrand-relevant informationKeep generated content on brand

There is no single data requirement for generative AI. The intended interaction determines which information the application needs and which boundaries it must preserve.

Slide with three panels titled A more personal virtual agent, More relevant conversational search, and Content that’s on brand, alongside phone, chat, and tablet images.Open full source frame
Three generative AI applications and their data needs.
2:543:11
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:54 · section reference included

Data enters through the prompt, context, and model

Following the travel agent into a model interaction reveals several distinct uses of data. First, the system prompt supplies instructions and the user query supplies the immediate request. Both are data. A prompt catalog or set of templates can provide different instructions for different business requirements, with the agent selecting the appropriate template.

Next comes context, which is no longer necessarily static text bundled with the application. It can arrive from changing data services and sources. Finally, there is the model itself: an out-of-the-box model may be sufficient, while a fine-tuned model requires training data that represents the company. Data is involved before meaningful output appears, whether it supplies instructions, current knowledge, or model customization. Responsible generation adds another requirement, addressed later through guardrails.

5:105:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:10 · section reference included

From source material to a retrieval corpus

Amazon Bedrock supplies several parts of this application architecture. Alongside model choice, customization, and evaluation, Bedrock Data Automation handles data processing. Bedrock Knowledge Bases provides managed retrieval-augmented generation, or RAG, and Bedrock Guardrails adds configurable safeguards. The security and privacy requirements apply across agentic RAG, summarization, classification, fraud detection, and contextual chatbots. To examine the pieces more closely, Khanuja narrows the example to a contextual chatbot.

Its source material may include text, videos, and images. Understanding an image here means more than describing a portrait: a financial document may contain charts and line graphs whose business meaning must survive processing. Khanuja describes Data Automation as offering a single API for this processing. In the launch-era workflow, that means a unified asynchronous invocation, InvokeDataAutomationAsync, with configuration and S3 result handling around it—not an entire pipeline without setup.

Knowledge Bases natively integrates Data Automation, then gives the application choices about how to prepare the resulting material for retrieval:

  1. Choose a chunking strategy. Use hierarchical or semantic chunking, or supply custom chunking logic when the documents require it. Complex tables are one reason the choice deserves attention.
  2. Choose an embedding model. Convert the chunks into vector representations used for similarity search.
  3. Choose a vector store. Store those embeddings for retrieval.
  4. Maintain ingestion and updates. Knowledge Bases manages ingestion and incremental processing. The application still needs an appropriate synchronization trigger; the companion chatbot implementation uses an S3-triggered Lambda to call StartIngestionJob.

The result is a maintained retrieval corpus, rather than a one-time upload that can be assumed to remain current.

6:376:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:37 · section reference included

Retrieve first, or retrieve and generate

Once ingestion is configured, the application needs a way to retrieve relevant information. The Retrieve API returns similar content, leaving the application to augment a prompt and invoke a model. Search is also configurable: hybrid search combines semantic similarity with text matching. In the current vector-search configuration, HYBRID requires a supported store and a filterable text field; it is not available for every storage configuration.

For a refund question, a Python retrieval call can expose the policy passages before any answer is generated. This function accepts the configured knowledge base ID and lets the caller choose the search mode:

python

import boto3


def retrieve_policy(knowledge_base_id, question, search_type="SEMANTIC"):
    client = boto3.client("bedrock-agent-runtime")
    response = client.retrieve(
        knowledgeBaseId=knowledge_base_id,
        retrievalQuery={"text": question},
        retrievalConfiguration={
            "vectorSearchConfiguration": {
                "overrideSearchType": search_type,
            }
        },
    )
    return response["retrievalResults"]

The returned retrieval results are evidence for the next step, not a completed refund decision. The application can inspect that evidence before adding it to the model’s prompt.

Retrieval quality can be improved through reranking, post-processing, and query decomposition. For an integrated path, RetrieveAndGenerate retrieves context and generates the response, with parameters for controls such as reranking and query decomposition. Decomposition is useful when a complex question needs to be broken into smaller retrieval requests.

APIReturnsApplication responsibility
RetrieveRetrieved contentInspect context, augment the prompt, invoke the model
RetrieveAndGenerateA generated response grounded in retrievalConfigure the managed retrieval and generation path

The choice is about where to retain control. A retrieval-only call leaves an explicit boundary for application logic; the combined API packages more of the workflow behind query and generation controls.

Diagram connects a user query to Amazon Bedrock and knowledge bases, then an augmented prompt, a list of models, and an answer.Open full source frame
Amazon Bedrock Knowledge Bases retrieval and generation workflow.
9:5810:14
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:58 · section reference included

Add safeguards without losing the access boundary

A working RAG path still needs responsible response generation. For the contextual chatbot, Khanuja highlights preventing PII disclosure and filtering unwanted words. Bedrock Guardrails lets teams define policies, specify examples of information that should not be shared, and apply grounding checks to help reduce hallucinations. Guardrail interventions can also reveal patterns in how users interact with the application, provided the relevant telemetry is collected.

Data Automation, Knowledge Bases, and Guardrails now form an assembled chatbot, with native integration between the retrieval-and-generation path and its safeguards. One consequential boundary remains: current AWS documentation states that guardrails cover input and generated responses, not retrieved references. Guardrails do not replace access control or source-data protection. Sensitive retrieved material needs its own controls, especially if the interface exposes references directly.

With documents ingested, the application can search and return material in image form as well as text. That brings the presentation back to the dancing coconuts. Getting responses is the starting point for the operational practices that follow, many of which apply beyond RAG. The first is chunking: the way information is divided affects generated-answer accuracy, but it is only one step in improving the application.

11:0511:22
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:05 · section reference included

Optimize accuracy, cost, and latency

Optimization offers many possible interventions: better parsing, reranking, hybrid search, query reformulation, and decomposition. Choosing among them requires knowing what is wrong. Evaluation identifies the weakness; optimization addresses it. The objective also extends beyond answer accuracy to the three dimensions Khanuja emphasizes: performance, latency, and cost.

Semantic caching targets repeated work. Users do not always express the same question in identical words, so an exact-string cache can miss a reusable answer. A semantic cache looks for a sufficiently similar question that has already been answered. When a cached result can be reused, the application avoids another foundation-model invocation and the time needed to synthesize a fresh response.

14:4815:01
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

14:48 · section reference included

Observe the application, then evaluate the right stage

To diagnose failures, capture the chain from user query to retrieval hits to model response. Those records let a team investigate a complaint and determine whether the problem began with the request, the retrieved evidence, or the generated answer. Khanuja’s advice is explicit: “Please don't go into production or even do a pilot without observability.”

Telemetry requires deliberate configuration. Bedrock model invocation logging is disabled by default and requires a destination such as CloudWatch Logs or S3. Enabling it does not establish complete coverage of retrieval events or guardrail interventions; those need to be accounted for in the application’s instrumentation. Full request and response logs can themselves contain sensitive data, so the privacy requirements also apply to this new dataset. Once collected, it becomes material for evaluation.

For RAG, start with context relevance. If search returns irrelevant material, placing it in the prompt spends generation budget on a poor input. Evaluate the search results first; only then use suitable evidence to augment the prompt and ask the model for an answer. This separates a retrieval problem from a generation problem instead of treating every bad answer as a model failure.

Evaluation must fit the task. A summarization application needs summarization-specific metrics, rather than automatically inheriting the criteria for retrieval. Khanuja does not prescribe particular summarization metrics here; the broader practice is to choose evaluation according to the behavior the application is meant to deliver.

16:4116:58
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:41 · section reference included

Update, test again, and scale

Never go blind: evaluation findings must lead to action. Sometimes the data needs refreshing, because stale data produces stale answers. Sometimes the retrieval or optimization strategy needs to change. The evaluation result should determine which part of the application gets updated.

An update is not the end of the loop. A defined test suite should automatically rerun evaluations after changes. That makes testing a repeatable part of maintenance, rather than a manual exercise performed only before the first release. Khanuja presents this as a way to reduce production failures and improve application quality, without supplying a measured improvement.

Repeated testing leads to the final coconut callback: crack the coconut, then scale to coconuts. The complete COCONUTS memory aid connects chunking strategy, optimization, caching, observability, never going blind, updating, testing again, and scaling. Production is the point at which this working and tested application can bear fruit—not the point at which the first generated answer appears.

Table lists chunking strategy, optimize, cache results, observability, never go blind, update, test again, and scale; the speaker inset partially overlaps the bottom-left row.Open full source frame
COCONUTS best practices for RAG applications.
15:4318:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:39 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…