AI Engineer World's Fair 2025
"Data readiness" is a Myth: Reliable AI with an Agentic Semantic Layer — Anushrut Gupta, PromptQL
Loading slides…
Read the talk
"Data readiness" is a Myth: Reliable AI with an Agentic Semantic Layer
PromptQL’s demonstrations show how inspectable plans, user corrections, and versioned business context can help AI work with ambiguous schemas and company-specific definitions.
From a talk by Anushrut Gupta
Before you start: Basic familiarity with database tables, SQL queries, and LLM-generated plans will help; no prior knowledge of PromptQL is required.
What does CST_NM actually mean?
You want to deploy AI on production data, but what must the system understand before it can answer a question? Anushrut Gupta, who introduces himself as the lead of applied research at PromptQL, begins with the gap between having data and knowing what it means. Clean names and complete annotations are the exception in the audience’s experience. Waiting for them makes reliability a prerequisite that deployment never reaches.
Consider CST_NM: customer name, or custom nomenclature? A revenue field called rev_amount_USD looks more informative, but an active-status field still leaves questions about whether its values are Boolean, numeric, or nullable. Another system has organization names and total revenue. Matching those records requires understanding both identity and units: cents versus dollars, and the representations used to store them. A schema exposes structure; it does not settle business meaning.
Gupta traces a familiar sequence of attempted fixes. In his illustrative timeline, warehouse consolidation into Snowflake or Databricks begins in 2019 but remains partly complete. Master data management is still being implemented. Semantic layers arrive as the next answer in 2023, then struggle as tables, schemas, and workflows change. By 2025, the requirement has become perfect data for AI, with the AI still waiting. These are examples of an ongoing maintenance problem, not measured completion rates or a universal quarterly failure schedule.
Gupta cites McKinsey for an average annual loss of $250 million at a Fortune 500 company due to poor data quality. That attribution needs a distinction: the matching McKinsey estimate concerns managerial wages associated with ineffective decision-making, not losses specifically established to result from poor data quality. The narrower engineering problem remains: standardizing storage does not remove the need to interpret changing business practices.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A formula still leaves business decisions unresolved
The audience discussion makes maintenance concrete. One exchange describes a flexible model built with Django that needs substantial pruning. When Gupta assumes information is being added manually, the response says the updates are automatic. Automation can therefore be part of maintaining the model; the unresolved question is how much context that model captures.
Customer acquisition cost looks easy to define:
But neither term identifies a unique calculation without further decisions.
- Spending: Does the numerator include brand marketing, performance marketing, or both?
- Customer eligibility: Does a first purchase count differently from a reactivated customer? What happens to failed trials?
- Time: Which reporting period applies, and how should seasonality affect the interpretation?
Each choice can change the answer while leaving the formula intact. Gupta’s objection is to expecting people to enumerate every such edge case in advance. A useful definition starts the conversation; it does not finish it.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A graph can connect a deal without defining its risk
A sales knowledge graph connects a deal to its stage, date, and owner. Now ask for deals at risk. The relationships provide places to look, but they do not determine the risk rule. A deal might be endangered because its internal champion has left, or because it has remained in one stage for two months. Those are business interpretations that must be supplied or learned.
Gupta also challenges the practicality of representing a billion-row Snowflake table in a graph. That is his objection to the proposed architecture, not a demonstrated capacity limit for graph databases. The more general point applies to knowledge graphs and approaches such as GraphRAG: representing relationships does not, by itself, establish the local meaning of a question.
The same ambiguity appears in ordinary language. GM can mean gross margin in finance and general manager in HR. Conversion needs an event definition; quarter needs a calendar; active customer needs an eligibility rule that may differ by team. A capable general LLM does not automatically possess the tacit knowledge that employees accumulate while working inside a company.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Learn business context like a newly hired analyst
An analyst traditionally bridges this gap. A business user brings a question; the analyst combines company knowledge with SQL or code and access to the underlying systems. The database might be relational, NoSQL, or something else. What matters is that the analyst translates intent into operations and explains those operations so colleagues can inspect the work. Gupta describes this idealized analyst as completely reliable; it is an analogy for trusted collaboration, not a measured claim about human accuracy.
An agentic semantic layer applies that apprenticeship model to AI. Start with a capable new hire who lacks company context. As the analyst works, someone corrects a misunderstanding and explains what a business term means. Those corrections accumulate into reusable knowledge. The proposed AI follows the same progression: use, explanation, steering, correction, and learning.
This places requirements on the system before learning can help. It must be correctable, explainable, and steerable, and it must use what it already knows accurately. A user needs to see where an interpretation went wrong and have a way to change it. Otherwise, repeated use merely repeats the same hidden assumptions.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Generate a plan, then execute it separately
PromptQL implements the analyst role by having a foundational LLM generate a plan in a domain-specific language, or DSL. Gupta says this does not require fine-tuning. The language covers three kinds of work: retrieving data, computing and aggregating it, and performing semantic operations.
The architectural separation is between generating that plan and executing it:
- The LLM translates the user’s request into a PromptQL plan.
- A deterministic runtime executes the plan.
- A distributed query engine accesses the relevant sources, retrieves data, and supports the composition required by the plan.
- The resulting answer goes directly to the user instead of returning to an LLM for another answer-generation pass.
Gupta contrasts this with a RAG loop that feeds retrieved data back into a model to produce the final answer. Removing that final generation step reduces one opportunity for the model to alter a computed result. Deterministic execution does not establish that the generated plan expresses the right business meaning. That still depends on context, inspection, and correction.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
An empty result exposes the wrong status assumption
The first demonstration asks for the top five customers by revenue. PromptQL interprets revenue through invoice items and performs the calculation, but the first attempt returns no results. The failure is not complicated arithmetic: the plan has made an incorrect assumption about the data.
Inspection reveals that the query searched for succeeded, while the actual invoice statuses are paid and pending. After revising the status interpretation, PromptQL produces the customer ranking. The example demonstrates recovery from a mistaken assumption, rather than correctness on the first attempt. An inspectable plan makes that distinction visible: an empty result can trigger investigation instead of being presented as a business conclusion.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
From email domains to support sentiment and a proposed credit
The next request makes the dependencies explicit. Organization IDs are unreliable, so the system must construct an alternative way to identify customers before it can rank them or inspect their support history:
- Retrieve individual users and identify unique organizations from their email domains, ignoring the unreliable organization IDs.
- Rank those organizations by revenue and select the third highest.
- Retrieve that organization’s latest 30 Zendesk support tickets, including their comments, and summarize each ticket.
- Use the summaries to classify feelings toward the product into five categories, from bad to great, and explain what is going well and what could improve.
- Issue a credit to the organization’s highest-usage project, with $5,000 for bad sentiment and $1,000 for great sentiment.
The requested workflow crosses databases, Zendesk, and an API such as Stripe. It combines identity inference, aggregation, document summarization, classification, and a financial action. Summarization and sentiment classification are semantic operations; the description of a deterministic runtime does not make them ordinary arithmetic or establish how every semantic operation was implemented.
As the request runs, the displayed plan describes operations such as retrieving users and extracting their email domains. A pencil control lets the user edit that plan: remove a step, or replace it with several other steps. This is a concrete point of control over how the task will be carried out. Gupta says those interventions also teach the system how the company does business, so future work can reuse the correction.
The demonstration reports that it is about to issue $3,000, which Gupta calls a refund and tentatively interprets as neutral sentiment. He notes that he could have requested intermediate results from the beginning, then reviews details connecting the revenue ranking, ticket summaries, and sentiment classification to the third-ranked customer. The financial action remains a proposed outcome here: the demonstration does not establish that a credit or refund completed, or show an approval mechanism. The visible value is the traceable chain of analysis leading to that proposal.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Persist the corrections with AutoGraph
Recovering within one conversation is only the first stage. PromptQL’s learning layer is intended to improve a semantic graph from ongoing interactions, gradually building a language specific to the company. Gupta calls the illustrative result AcmeQL: the system can reuse Acme’s definitions instead of rediscovering them in every task. This is a label for accumulated company context, not a separate language implementation demonstrated in the talk.
The next example asks which employees work in departments with budgets above US $10,000. The deliberately opaque tables—morg, plug, and zorp in Gupta’s narration—initially give the system too little information to answer. The user asks it to sample rows and identify the employee table. Sampling lets it infer that zorp contains employees, the department table contains departments, and morg is a junction table. The AutoGraph interface spells the department table plugh.
The resulting answer still has a unit error. The user explains that department budgets are stored in cents, not dollars, and asks the system to divide by 100. For the question about department budgets above US $10,000, correcting cents to dollars changes the reported answer from five employees to two. Discovering the tables was therefore insufficient: the budget’s unit was necessary context for the filter.
The next operation turns those conversational corrections into metadata. Gupta manually asks for improvements based on recent interactions and the hints he supplied; he says this process can also run agentically in the background. The suggestions add table meanings and record that department budgets are in cents. Applying them creates a new, version-controlled semantic-layer build, with the ability to return to a previous build. The selected frame shows the updated descriptions and confirmation that the changes were applied.
With the improved context supplied by AutoGraph, Gupta repeats the employee-budget question and reports that the system now produces the correct plan and answer. The learning loop is more specific than remembering an earlier response: it retains the table meanings and unit interpretation needed to plan the task correctly next time.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
A quarter can start in February
A final query extends the same mechanism from schema interpretation to a business calendar. In an anti-money-laundering example, the request is to find accounts with the maximum suspicious outgoing amounts in the first quarter and return their IDs and names. When the connection is slow, Gupta switches to a preloaded conversation.
The correction is simple but consequential: this company’s quarter starts in February, not January. A query can aggregate every row correctly and still answer the wrong question if its date boundaries are wrong. Gupta describes retaining the February start in the semantic layer for future queries, just as it retains table meanings and relationships. The reusable knowledge is the calendar convention, not merely the account list returned by this particular query.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Start with incomplete context, then make corrections durable
The closing comparison returns to the newly hired analyst. On day zero, the AI lacks definitions of an enterprise customer, mappings between customer IDs across systems, and the start of the financial quarter. Gupta’s day-thirty illustration claims 47 learned business terms, relationships mapped across six systems, 12 calculation variants, and 100% accuracy on complex tasks. No evaluation protocol accompanies those figures; they illustrate the intended progression rather than establish a reproducible accuracy result.
The deployment recommendation is to begin before exhaustive preparation is complete, then improve the semantic layer through use. Instead of treating every missing definition as a reason to postpone deployment for months, the system encounters ambiguity, exposes its interpretation, accepts corrections, and retains useful context. Gupta presents eventual complete accuracy as something customers report, rather than providing a test that establishes a universal guarantee.
Gupta says an unnamed Fortune 500 food-chain company evaluated 100 vendors before finding PromptQL fully reliable. He also claims 100% accuracy on an unnamed high-growth fintech company’s hardest questions. These are attributed customer anecdotes: the talk does not provide task counts, scoring rules, or independent validation. The concrete engineering contribution is the combination demonstrated along the way—plans users can inspect and edit, execution separated from final answer generation, and versioned business context that lets a correction outlive the conversation in which it was made.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Further reading
- The data-readiness mythArticle
Anushrut Gupta explains how user corrections can develop company-specific business context.
A companion demonstration of AutoGraph, schema introspection, and learning business definitions from feedback.
PromptQL explains its accuracy messaging through predictability, explainability, and steerability.
Documentation and getting-started material for Microsoft's graph-based retrieval project.
Updates since the talk
A July 2025 vendor report on CRMArena-Pro database-querying and numerical-computation tasks, with architectural explanations and worked examples.
Full transcript
Loading transcript…