Contents
  1. Purpose and development
    1. Answers built from external information
    2. From retrieved snippets to generated answers
  2. Evidence entering the answer
    1. The retrieval-to-generation handoff
    2. Enough evidence for the question
    3. Applicable sources and unresolved conflicts
  3. Preserving and expressing support
    1. Select context without losing meaning
    2. Construct warranted claims
    3. Citations attached to claims
  4. Checking and limiting answers
    1. Check support, not just references
    2. Answer only what the evidence allows
  5. Evaluation and improvement
    1. Test the complete evidence task
    2. Measure delivery, support, and completeness
    3. Locate lost support and test the repair
  6. Check understanding
  7. Open questions
  8. Selected talks
  9. References
  10. Talk library
← All topics

Retrieval-augmented generation

Retrieval-augmented generation (RAG) lets a language model answer using information fetched from external documents or databases. This is useful when the answer depends on a particular collection or on information that changes. Finding related material is only the first step: the system must select enough information to answer the question, preserve its qualifications, and let the reader inspect the passages supporting the response.

Purpose and development

Answers built from external information

In a retrieval-augmented generation system, the retrieval component returns documents, passages, or records relevant to a request. The application includes selected material alongside the question in the model's input, and the generator composes a response from that input. An existing documentation search service or SQL database can supply the information; RAG does not prescribe a particular search method. The sequence can be fixed, or the model can decide when to request additional information.

This supplies information without necessarily teaching the model anything permanently. Learned parameters are numerical values fitted during training. Model context is the input available for a particular call. Ordinary RAG changes that input, not the saved parameters: an answer can change because a newly retrieved document describes a different policy. See Parameters and inductive assumptions and The input the model sees for those separate mechanisms.

Search and answer generation serve different needs. Search exposes material for a person to inspect; generation can combine that material into a direct response. IBM's 2024 documentation-system report describes an interface that retains ordinary search results alongside a concise generated answer and source links. Keeping both gives readers access to the underlying material instead of making the generated summary their only view.

Grounding ties an answer's claims to identifiable evidence. A claim is an assertion that can be assessed, interpreted with enough surrounding context to know what it means. Citation support asks whether the identified material warrants the attached claim. Neither definition says that the source is true: an answer can accurately repeat a mistaken source, while an independently correct answer can lack support in the supplied evidence.

These properties require different comparisons.
PropertyComparison targetWhat can still fail
RelevanceThe questionRelated material may omit the requested fact.
Contextual supportThe supplied evidenceThe evidence itself may be wrong.
Factual correctnessThe applicable factsA correct statement may have no supplied support.
CompletenessThe requested informationA complete-looking response may contain unsupported additions.

Define the answer contract before choosing components. A documentation assistant might require source support for every product-specific assertion while allowing ordinary language explanations and explicitly marked deductions. Retrieval adds processing and source-maintenance work; synthesis adds opportunities to omit conditions or join incompatible facts. Its benefit is therefore a system-level proposition to test, not a consequence of attaching a search service.

From retrieved snippets to generated answers

Open-domain question answering answers across a broad collection rather than a passage selected in advance. This introduces two coupled problems: locate the information and produce an answer from it. Their separation predates modern generative models. Finding a correct answer somewhere in a collection does not establish that the system retrieved its support, and producing the right words does not repair that omission.

Selected developments changed how answers were produced while retaining the need to find external information.
DevelopmentContribution
TREC Question Answering, 1999NIST introduced a track returning answer-bearing snippets with document identifiers. Later strict scoring rejected correct answers whose cited documents did not support them.
DrQA, 2017Danqi Chen and colleagues at Stanford and Facebook AI Research combined term-based Wikipedia retrieval with a neural reader that selected answer spans. Relevant passages were no longer assumed to be supplied.
REALM, February 2020 preprintKelvin Guu and colleagues at Google Research learned retrieval during pretraining: predicting missing text supplied feedback about which documents helped. The downstream reader extracted answer spans.
RAG, May 2020 preprintPatrick Lewis and colleagues combined a pretrained generator with a learned retriever and an external passage index. The architecture examined passage conditioning across whole answers or individual generated tokens.
REPLUG, January 2023 preprintWeijia Shi and colleagues explored augmentation when a model was accessible only as a frozen black box. It supplied retrieved documents through inputs and combined document-conditioned output probabilities.

These are complementary architectural choices, not successive replacements. Extraction retains value when an answer is a source span. Trained retrieval-generation systems can learn how external information contributes to a task. Application-level RAG often instead retrieves text and passes it to an unchanged model. REPLUG illustrates frozen-model augmentation, but its probability combination is more specific than an ordinary single-call retrieve-then-prompt application.

The original RAG paper also tested changing answers by replacing the external index without retraining its generator. On 82 leadership positions that changed between 2016 and 2018, the same model achieved 70% and 68% accuracy with the corresponding historical indexes; mismatched indexes performed substantially worse. This demonstrated an update mechanism, not automatic freshness or removal of everything remembered in model parameters.

Evidence entering the answer

The retrieval-to-generation handoff

The corpus is the searchable collection. A retrieval unit is what comes back: perhaps a whole document, a passage, or a database record. A candidate is a returned unit considered for inclusion in the model input. Ranking orders candidates under a retrieval criterion; it does not certify their truth or ability to answer the question. Reranking a Fixed Set explains the ranking boundary.

An inspectable application passes more than strings between components. Provenance records where an artifact came from and how it was transformed. The W3C PROV model distinguishes source entities, transformation activities, and responsible people or software. Applied here, a source revision, extracted text, selected passage, and generated summary are separate artifacts connected by derivation records. That history establishes an asserted origin, not semantic correctness.

A useful application-level handoff contract preserves the following information; it need not adopt a particular framework's types.
BoundaryInformation to preserve
Retrieval requestQuestion, intended entity or version, relevant time, permitted sources, and access context.
Retrieved candidatesContent; source, revision, and passage identity; source location; score meaning; execution and coverage status.
Generation inputThe exact selected evidence, its references, answer requirements, permitted deductions, and known gaps.
Answer outputClaims with evidence references, unresolved requirements, and whether the response is complete, partial, or withheld.

Do not infer execution success from the result count. Elasticsearch's Search API, for example, reports timeouts and failed shards separately from returned hits. An empty hit list after a timeout differs from a completed search returning no hits. Even completed execution says nothing by itself about whether the query or cutoff covered all required evidence.

Returned evidence is not always model-visible

Example

Only selected evidence reaches this generation call; retrieval execution status remains a separate property of the returned response.

Candidates: P1, P2Passage content, source revision, and location. Execution status belongs to this response: hits can accompany incomplete execution.
Exact model input: P1Question, answer requirements, and selected P1 text with its source/revision reference. P2 is absent.
Answer recordGenerated claims, source references, and unresolved requirements. Assess claim support separately.
P2 omittedRetained outside model input with its omission reason; unavailable to this generation call.
P2 was returned but omitted from model input. Source references survive selection; neither returned hits nor valid references establish sufficient support.
Read the diagram as text
  • Candidates: P1, P2. Passage content, source revision, and location. Execution status belongs to this response: hits can accompany incomplete execution.
  • Exact model input: P1. Question, answer requirements, and selected P1 text with its source/revision reference. P2 is absent.
  • Answer record. Generated claims, source references, and unresolved requirements. Assess claim support separately.
  • P2 omitted. Retained outside model input with its omission reason; unavailable to this generation call.
  • Candidates: P1, P2 → Exact model input: P1: Select P1 + reference.
  • Candidates: P1, P2 → P2 omitted: Record omission.
  • Exact model input: P1 → Answer record: Generation.

Authorization determines whether the material may be used for this request. Enforce it before model access and preserve it across citations and derived answers. Indexed permission metadata can lag source revocations; cached answers likewise need dependency-aware checks or invalidation. These are application obligations, not properties conferred by grounding. Enforce current authority develops the controls.

RAG does not require vector search. For an exact error code, a lexical retriever may supply a mapping that a semantic match misses; for a paraphrased symptom, the reverse can occur. Compare candidate evidence before changing the generator. Search and Retrieval explains lexical, dense and hybrid methods. Whatever supplied the passage, the answer must still use the right source and preserve the conditions attached to its claim.

Enough evidence for the question

Evidence sufficiency means that the supplied information permits the requested answer under the task's scope and allowed assumptions. It is a property of an evidence set, not merely of each passage's topical relevance. Sufficient material can contain an incorrect source answer; insufficient material can still prompt a correct answer from the model's prior knowledge. Those outcomes do not erase the distinction.

Translate the request into information requirements before inspecting candidate answers. A comparison needs facts about both subjects plus compatible definitions, units, and circumstances. A connected-fact question needs the intermediate relationship, not just two relevant endpoints. HotpotQA, introduced in 2018, made this distinction concrete: comparison questions combine facts about two entities, while bridge questions use an intermediate entity to reach another needed fact.

Consider a small invented documentation example. One passage gives Plan A's log-retention period as 7 days, another gives Plan B's as 30 days, and a shared heading establishes that both describe calendar-day retention under the current policy. The two values are necessary for a comparison; the heading makes them comparable. Retrieving more copies of the Plan A passage supplies neither Plan B's value nor the shared conditions. Sufficiency depends on covering those requirements, not on filling a result list.

Apply eligibility-first selection: exclude material that is not permitted or applicable before asking what the remaining set supports. Then distinguish three locations for an omission. The fact may be absent from the eligible corpus, present there but missed by retrieval, or retrieved but removed before generation. A generator cannot inspect a discarded passage merely because the search service returned it.

Support need not have one canonical route. FEVER's evidence format permits alternative evidence sets, including single-sentence and multi-page combinations. Evaluate whether at least one valid set establishes the required content, rather than requiring one exact passage identity.

Exhaustive and negative answers impose an additional obligation. A bounded list of relevant results does not establish every qualifying item or prove that none exists. The open-world assumption allows an unstated fact to be true; a closed-world interpretation treats absence as false within a defined collection. Neither makes that collection a complete account of reality. To report a total or a negative conclusion, justify the search boundary and its completeness for the question.

Applicable sources and unresolved conflicts

Before resolving disagreement, determine whether the statements concern the same thing. Align entity identity, product or policy version, operating conditions, and effective time—when the statement applies. Effective time differs from when the system obtained it. A correction received today may describe an earlier period, while a newly published policy may take effect next month. Time in XTDB explains this distinction; Refresh, supersede, remove applies it to context maintenance.

A real documentation example shows why scope comes first. Stripe describes different idempotency behavior for its v1 and v2 API namespaces. Idempotency here is a request-repetition contract intended to avoid duplicate operations. Combining conditions from the two namespaces would create a policy that neither document establishes.

Different scope explains these differences; publication recency alone would not select the applicable rule.
Documented scopeReplay conditions and behavior
Stripe's v1-style referenceReuse requires matching parameters. Keys may be removed after at least 24 hours; a pruned key starts a new request.
Stripe API v2Replay requires the same key, API, and account or sandbox within 30 days.

Source authority concerns responsibility or competence for the particular fact. It is distinct from permission to read the source. An official specification can define supported behavior; an incident report can describe what happened during an outage. Neither should automatically replace the other. Likewise, current code describes implementation, while an approved design may describe intended future behavior. Select using the question's purpose, and expose disagreements that those distinctions do not resolve.

Several agreeing pages may repeat one upstream assertion. Corroboration is stronger when reports supply independent support rather than copies. Dong, Berti-Equille, and Srivastava's 2009 source-dependence research showed why counting copied values as independent votes can favor false information. Agreement alone does not prove copying either. Preserve known origins, and distinguish selecting an applicable source from reporting unresolved, attributed alternatives.

The same obligations apply to graph results. A knowledge graph represents identifiable entities and typed relationships; GraphRAG uses such relationships on the retrieval path, sometimes alongside text search. Preserve each needed relationship's direction, identity, time, and source before composing a conclusion. A graph connecting a sales deal to its stage and owner does not define whether that deal is at risk: the business rule is another required premise. See A connected path is not proof and Graphs complement search and generation.

Preserving and expressing support

Select context without losing meaning

A good search unit is not necessarily a sufficient interpretation unit. A matching sentence may depend on a heading, table header, exception, or preceding definition. Parent-passage expansion searches smaller passages but returns their identified parent sections or documents. The ParentDocumentRetriever documentation describes this separation explicitly. It restores surrounding material without requiring that the same large unit perform the initial matching.

Expansion trades specificity for additional context. Passing whole documents can restore missing conditions, but also adds irrelevant text and processing. Select complementary evidence for the answer, remove redundant material, and expand only where interpretation requires it. LlamaIndex's production guidance likewise separates representations used to find information from material used to synthesize an answer. In the fictional export section S1, the heading “Paid plans” governs “Administrators may export audit logs.” A child match carries S1’s identifier so the stored section can supply both pieces.

The same sentence and source revision survive the lookup. The parent heading restores the paid-plan condition; the returned parent can be a section rather than a full document. Expansion retrieves existing text and still requires selection within the context budget.

Extractive compression selects source text; abstractive compression generates a shorter restatement. Both can omit necessary information, and abstraction can additionally introduce unsupported content. These are different failures. Faithfulness, in contextual-support evaluations, asks whether the statements in the result follow from the source material. Comprehensiveness asks whether the result retains enough information to answer. The RECOMP study assessed both: a summary containing only supported statements can still leave the generator without a necessary fact.

In this invented documentation fragment, exact copying can still remove an essential condition.
RepresentationTextEffect
SourceOn paid plans, administrators may export audit logs.Permission depends on both plan and role.
Qualification-losing extractAdministrators may export audit logs.No longer carries the paid-plan restriction.
Meaning-preserving restatementAudit-log export is available to administrators on paid plans.Preserves both conditions while changing wording.

Selection is followed by placement in the model's context. Lost in the Middle, published initially in 2023, varied the position of an answer-bearing document among distractors while holding the question fixed. Tested models often used evidence better near the beginning or end than in the middle; another experiment found answer performance saturating before retrieval recall. The lesson is to test evidence use, not adopt a universal ordering rule. General ordering and request budgets belong to context engineering.

Keep source references attached through every reduction, and keep retrieved content in its role as data. A document's instruction to ignore the user's request does not become application authority because retrieval selected it. The separate untrusted-content boundary explains why evidence selection is not instruction delegation.

Construct warranted claims

Answer production can perform several different operations. Extraction selects source wording. Paraphrase restates its meaning. Synthesis combines information from several places. Deduction derives a conclusion from premises. Each operation needs a different check: quotation matching can assess extraction, but combining sources also requires compatible subjects and circumstances, and deduction requires justified connecting assumptions.

Resolve references before assessing meaning. A pronoun or short answer may express a claim only in conversational context. The Attributable to Identified Sources framework separates that interpretation from checking support. It also allows evidence-based inference across passages, such as calculating an age from dates, provided the connecting assumptions are valid.

Preserve the source's logical force: negation, quantities, units, conditions, attribution, and uncertainty. In the export example, replacing “may export” with “automatically exports” introduces behavior the source never states. Likewise, “approval is expected” must not become “approval was received.” These changes can arise during repeated rewriting even when the surrounding prose remains coherent.

Multi-passage generation makes evidence combination possible, not automatically valid. Fusion-in-Decoder, first released as a preprint in July 2020 and published at EACL 2021, processed each retrieved passage separately into a numerical representation. Its decoder—the component producing the answer—could then use information across those representations. In the tested tasks and passage-count range, supplying more passages improved the rate of exact matches to reference answers. That result does not establish citation support or show that adding passages always helps other architectures.

Organize a synthesis around the requested information, not around one paragraph per source. Attribute source claims where needed, mark deductions as deductions, and check calculations separately from the prose that describes them. When a connecting premise is missing, preserve the gap rather than completing it with a plausible transition. Narrower subquestions can help a long synthesis retain important detail.

Instructions should specify these obligations and permit insufficient-information responses. They guide generation rather than prove compliance. Turn a request into a task contract explains the prompting side; the resulting claims still need inspection.

Citations attached to claims

A useful citation combines an address with a support relationship. An evidence span is the particular text or regions needed to assess a claim, possibly including separated qualifications. Attach references close enough that readers can identify which assertion each supports. A link to a long document leaves substantial verification work if the supporting passage is not located.

Citation interfaces can preserve these locations. The documented Claude citation interface uses request-local document indices with character, page, or block ranges. Valid pointers do not establish semantic support, and request-local indices do not supply durable revision identity. Store the mapping back to the particular source representation.

Locations can break when text changes. W3C's Web Annotation Data Model distinguishes text-position selectors from quoted-text selectors and records representation state. A position in extracted text is not interchangeable with a byte offset or PDF region. Re-resolve changed evidence or mark the citation unresolved instead of silently attaching it elsewhere. Locate supporting source material covers these mappings.

The export example can be extended with a second passage stating that exports use CSV. Together they support a claim about administrators exporting CSV on paid plans. Neither establishes immediate completion. A citation can therefore resolve correctly yet fail to support an attached timing claim. Similarly, a retrieved search snippet supports only what was inspected, not every assertion an unseen full page might contain.

Valid references, different support

Example

Two passages can jointly support one claim while failing to establish an additional qualifier in another.

E1: Export ruleExample manual, revision 1, paragraph 1: On paid plans, administrators may export audit logs.
E2: Export formatExample manual, revision 1, paragraph 2: Audit-log exports use CSV.
Combined claim · requires E1 + E2Administrators on paid plans can export audit logs as CSV. References: E1 and E2.
Immediate-completion claimAudit-log exports finish immediately. References: E1 and E2 resolve, but neither supplies timing evidence.
Fictional export documentation: E1 supplies role and plan; E2 supplies format. Both are required for the combined claim. Dashed connections show absent timing evidence, not proof that immediate completion is false.
Read the diagram as text
  • E1: Export rule. Example manual, revision 1, paragraph 1: On paid plans, administrators may export audit logs.
  • E2: Export format. Example manual, revision 1, paragraph 2: Audit-log exports use CSV.
  • Combined claim · requires E1 + E2. Administrators on paid plans can export audit logs as CSV. References: E1 and E2.
  • Immediate-completion claim. Audit-log exports finish immediately. References: E1 and E2 resolve, but neither supplies timing evidence.
  • E1: Export rule → Combined claim · requires E1 + E2: Supports role and plan.
  • E2: Export format → Combined claim · requires E1 + E2: Supplies required format.
  • E1: Export rule → Immediate-completion claim: Does not establish timing.
  • E2: Export format → Immediate-completion claim: Does not establish timing.

ALCE, introduced by Gao and colleagues in 2023, separated answer correctness from citation quality and tested support through entailment judgments. This makes two checks necessary: whether cited passages support the attached statements, and whether statements requiring support actually receive it.

Even a supporting citation does not prove causal reliance. The 2025 study Correctness is not Faithfulness in RAG Attributions distinguished evidence use from post-hoc citation matching. Controlled modifications to documents changed citation behavior in the tested system. Its causal meaning of faithfulness differs from the contextual-support meaning used by many evaluation tools. For practical review, Build for the Memo, Not the Demo emphasizes direct claim-to-passage inspection rather than a detached bibliography.

Checking and limiting answers

Check support, not just references

Start with checks whose conclusions are narrow and explicit. Code can verify that a returned reference belongs to the supplied evidence set and that a quoted string matches the identified representation. Entailment asks a different question: whether the proposed claim follows from the supplied premises. A semantically useful assessment distinguishes supported, contradicted, and insufficient-information outcomes.

Choose the check that matches the obligation.
CheckInputsWhat passing does not establish
Reference validationReturned identifier and supplied source mapThat the source supports the claim.
Quotation matchingQuoted text and identified source representationThat omitted context leaves the meaning unchanged.
Contextual supportInterpreted claim and necessary source passagesThat the source is factually correct.
Independent factual verificationClaim and authoritative external evidenceThat the generated answer used or cited its original evidence properly.

Assess material claims individually without stripping away their conditions. Splitting “administrators on paid plans can export” into “administrators can export” changes what is being checked. Compound sentences may also hide an unsupported clause beside a supported one. A whole-answer judgment can overlook that asymmetry.

Natural language inference is the task of judging whether text entails a proposed statement. The 2019 HANS study exposed failures from word-overlap heuristics: reversing who paid whom can preserve nearly all words while changing the claim. Its results concern the tested inference models, but the diagnostic pattern remains useful when testing a support checker.

A separate verification call can focus attention on whether a particular finding follows from a source. It does not make the verifier independent or infallible. The generator and checker may share misunderstandings, and a confident explanation is not additional evidence. Compare checker decisions with reviewed examples, including missing qualifiers and reversed relationships. Choose checks that match the requirement and Validate the model judge develop that assessment discipline.

Answer only what the evidence allows

Abstention means declining to supply an unsupported answer or claim. It does not require discarding independently useful supported content. But a partial answer is acceptable only when its boundaries remain clear: listing some prerequisites must not look like a complete authorization to proceed. Missing a decisive condition can prevent the requested conclusion even when several surrounding facts are known.

The following is an application policy to adapt to the task, not a universal confidence threshold.
Evidence conditionResponse obligation
Sufficient and applicableAnswer within the established scope and attach supporting references.
Only independent subparts supportedAnswer those subparts if useful; identify what remains unresolved.
Question underspecifiedRequest the missing entity, version, period, or condition.
Substantive conflict remainsPresent attributed alternatives without inventing a resolution.
Required evidence missingState the support limit; do not convert non-discovery into nonexistence.
Retrieval incomplete or unavailableReport the execution limit rather than implying a completed negative search.
Required information not permittedRespect the access boundary; do not expose restricted content while explaining the limitation.

UAEval4RAG, introduced by Xiangyu Peng and Salesforce Research colleagues in December 2024, evaluates reason-specific nonanswers, including clarification, correction of false assumptions, and acknowledgment of missing knowledge. Its contribution is to assess the appropriateness of the response, not merely whether the model answered or refused.

Further retrieval is useful when it targets an identifiable gap and is permitted by the task. A relevance check can trigger another source lookup rather than immediate generation, as in the demonstrated Corrective RAG flow. Reassess the resulting evidence and bound the additional work; another search is not proof of progress. Decide whether to continue covers the broader stopping decision.

Acceptance also changes the population being evaluated. Coverage is the fraction of cases answered; selective risk is error among accepted answers. Lower error can result from answering fewer cases. Retrieval scores and verbal confidence are not validated acceptance probabilities. Measure the policy on held-out, deployment-like work and examine its behavior when the domain changes. Evaluate deferral as a policy explains the general tradeoff.

Evaluation and improvement

Test the complete evidence task

A question and reference answer are not a complete RAG test. The case must identify the information the system may use and what that information permits it to conclude. A corpus snapshot fixes the collection state for the assessment. KILT, published in 2021, aligned knowledge-intensive tasks to an August 2019 Wikipedia snapshot and separately evaluated answers, retrieved evidence, and combined outcomes. Mapping the sources was itself necessary work: some earlier evidence no longer mapped adequately.

An example case for the invented retention comparison separates the answer requirements from their possible supporting material.
Case fieldExample meaning
Input and scopeCompare Plan A and Plan B's log retention under the specified current-policy snapshot.
Required contentBoth retention periods, compatible units and conditions, and the resulting comparison.
Acceptable evidence setsThe two plan passages plus their shared heading, or a separately verified comparison table covering the same requirements.
Response policyA full comparison requires both sides. An independently useful one-plan answer must explicitly remain partial.
Run artifactsActual candidates, selected context, answer, citations, execution status, and assessment outcomes.

References are not necessarily exhaustive. A newly discovered passage may provide valid support absent from the annotations; review it instead of automatically marking it wrong. Include unanswerable cases whose passages are highly relevant but lack the decisive fact. SQuAD 2.0 used such examples, including questions with tempting answer-shaped distractors. Paragraph-level insufficiency, however, is different from absence across an entire corpus.

Synthetic questions are generated test inputs. Document-grounded generation can bootstrap cases, provided people audit them, but it favors questions that the selected passage conveniently answers. IBM's documentation report illustrates the mismatch: a credentials definition encourages a question about what credentials are, while users may need to know where to obtain them. Counts, complete lists, conflicts, scope mismatches, and damaged extraction need deliberate coverage when they occur in the intended work.

Keep development cases separate from independent assessment. Retrieving source facts is legitimate when that is the capability under test; retrieving leaked benchmark answers changes the test. Use case design, workload sampling, and independent-assessment controls to specify that boundary rather than declaring every external lookup contamination.

Measure delivery, support, and completeness

Measure evidence delivery separately from what the answer says. For each required fact or acceptable evidence set, record whether support exists in the eligible collection, reaches the candidates, and survives into final context. Then assess the emitted claims. RAGChecker distinguishes retrieved reference claims, their use in the answer, and answer faithfulness. These relationships help localize omissions without treating a single score as complete diagnosis.

Declare these units and denominators before comparing systems.
MeasureUnit and denominatorInterpretation limit
Evidence deliveryRequired information supported at a named boundary, divided by assessed required information; alternatively, cases with a complete acceptable set.Incomplete annotations leave coverage uncertain.
Claim supportSupported emitted claims divided by assessed emitted claims.Says nothing about omitted requirements or source truth.
Answer completenessCredited required information divided by all required information under the declared rubric.Addressing a requirement does not establish correctness.
Unsupported additionsUnsupported emitted claims, reported as a count and a fraction of assessed emitted claims.Unassessed claims must remain visible separately.
Citation qualityAssess support of claim–citation mappings and support coverage of citation-requiring claims separately.Valid locations are a separate structural check.
Factual correctnessCorrect assessed answers or claims under an explicit reference and scope.Reference errors limit the result.
Coverage and selective riskAccepted cases / all eligible cases; erroneous accepted cases / accepted cases.Error among accepted cases can fall as more cases are declined.

Ragas Faithfulness implements the emitted-claim denominator: it decomposes a response and judges support from retrieved context. By contrast, AutoNuggetizer constructs an inventory of answer-relevant facts and measures their presence in the response. Its November 2024 report permits full, partial, or zero coverage credit; strict scoring accepts only full matches. The report excludes citation-support evaluation, so its coverage judgments must not be read as citation entailment.

This distinction has older roots. NIST's TREC 2003 question-answering track used information nuggets for definition answers because exact factoid matching could not measure longer responses adequately. Assessors distinguished essential from optional facts and counted repeated facts once. The enduring idea is to measure requested information rather than reward verbosity or repeated wording.

For a tiny calculation, suppose a request has two equally weighted requirements. An answer states one supported fact and omits the other. Its claim support is 1/1, while its completeness is 1/2. A citation on that sole fact does not repair the omission. ALCE's terminology also needs care: its citation recall assesses support for statements, while citation precision identifies irrelevant citations—not ordinary retrieval precision and recall.

Specify empty-denominator behavior: no emitted claims should not silently become perfect support, and selective risk is undefined when nothing is accepted. State whether permitted partial answers count as accepted, and use the same acceptance set for coverage and selective risk. Keep unjudged evidence and failed assessments separate from negative judgments. Report partial answers against the original requirements, not a denominator reduced to what the system attempted. Retrieval metrics, retrieval evaluation, and metric discipline explain the corresponding measurement rules.

Locate lost support and test the repair

Start with the failed answer and trace each missing or unsupported claim backward. Check source availability and extraction first. Then inspect candidates, final context, synthesis, citations, and acceptance. Support present among candidates but absent from final context establishes an assembly loss. Sufficient context followed by a deficient answer identifies a generation-side failure under that input. Different claims in one answer can fail at different boundaries.

A trace records the operations and artifacts of an execution. Retain actual candidate passages and the exact model-visible evidence, not merely URLs fetched later. Source revisions, selected order, and omitted qualifications can change the diagnosis. Record meaning-changing boundaries explains the broader instrumentation contract.

Choose the boundary to change, then compare its recorded inputs and outputs with the baseline. Replaying the same candidates through different assembly tests preservation; replacing the final evidence bypasses retrieval and assembly together.

Substitute source passages, never the reference answer. Hold query, corpus and permission snapshot, prompt, budgets, and evaluator fixed. Keep model configuration fixed except when it is the declared intervention. Record passage identities and order, preserving them at unchanged boundaries; disclose changed length, position, or grouping. These are applications of blocking principles, which hold nuisance variation constant to make a comparison interpretable.

Repeat matched conditions and report uncertainty. A successful final-context replacement does not prove retrieval was the sole fault; unsuccessful replacement does not exonerate retrieval if the generator also fails. Test repairs individually and together when their effects may interact. Compare changes on matched work develops this design. An idealized, sufficient-evidence condition—often called oracle context—is diagnostic, not a deployable production ceiling.

Change one boundary, narrow the diagnosis

Question: Compare Plan A and Plan B log retention under the current policy, in calendar days.

Intervention

Fixed: question, eligible corpus / permission / time snapshot, prompt template, input budget and assessment. G0 is fixed except for the generation comparison.

One source snapshot: Retention manual · revision 1
BoundaryBaselineIntervention
Candidate passages
In recorded order
  1. H S1 / headingCurrent-policy calendar-day retention.
  2. A S1 / Plan APlan A retains logs for 7 days.
  3. B S1 / Plan BPlan B retains logs for 30 days.
  1. H S1 / headingCurrent-policy calendar-day retention.
  2. A S1 / Plan APlan A retains logs for 7 days.
  3. B S1 / Plan BPlan B retains logs for 30 days.
Assembly ruleOriginal: remove standalone headingsChanged: retain the heading with both values
Exact evidence block
Other prompt fields fixed
  1. A S1 / Plan APlan A retains logs for 7 days.
  2. B S1 / Plan BPlan B retains logs for 30 days.
  1. H S1 / headingCurrent-policy calendar-day retention.
  2. A S1 / Plan APlan A retains logs for 7 days.
  3. B S1 / Plan BPlan B retains logs for 30 days.
GeneratorG0 · recorded model and decoding settingsG0 · recorded model and decoding settings

Changed: Assembly only; candidate text, IDs, revision and order are identical.

Path: No stage bypassed; replay the recorded candidates.

If repeated trials improved: Improvement would support this evidence-preserving assembly change under the fixed setup. It would not establish that every other stage is fault-free.

The source section S1 contains H, A and B in that order. S supplies that existing section as one passage; the original rule removes only standalone headings. All displayed evidence fits the fixed budget. Retaining H changes input length and the positions of A and B; record those changes when interpreting a trial.

These are experimental setups using the retention example, not measured outcomes. Restoring source context does not guarantee a successful answer; repeat matched trials and assess support and completeness.

Compare credible alternatives for the actual job: search results, extracted answers, selected-context generation, and a small complete document set. In a 2024 long-context comparison, Zhuowan Li and colleagues tested nine English question-based datasets with three models. Direct long context achieved higher average task scores, while retrieval reduced model input substantially. The default retrieval condition used five 300-word chunks, with additional retrieval variations. These results concern bounded collections and task scores; input usage is not total operating cost, and the study did not measure claim-level citation support or human verification effort.

Workflow outcomes can differ from answer scores. A LinkedIn customer-service study preserved relationships between historical ticket problems and solutions, then supplied retrieved information to a generator. Its production comparison randomly divided the service team between tool use and traditional manual work: median resolution time was five hours with the tool versus seven without it. Group sizes and uncertainty intervals were not reported. This is a whole-tool result, not an isolated graph effect or a measurement of citation-verification time.

Keep the improvement decision equally specific. Measure supported completeness and review effort alongside latency, resource use, and answer coverage. Add complexity when inspected failures justify it, and turn reviewed production failures into regression cases. Collecting those cases does not improve the system by itself; the gain comes from a tested repair that still works when the full evidence-to-answer path runs.

Open questions

  1. Reliable sufficiency assessment must recognize missing conditions without assuming the answer in advance. Broad and exhaustive requests are especially difficult because a plausible subset can look complete. Progress would combine reviewed evidence requirements with tests that deliberately remove decisive facts and measure both unsupported answers and unnecessary abstentions.

  2. Partial answers need evidence of usefulness as well as claim support. A correct fragment can help a reader or mislead them into acting on an incomplete account. Progress would compare explicit partial-answer policies on mixed-support tasks, measuring misunderstanding, retained completeness, and review effort—not merely fewer unsupported sentences.

  3. Citation support and causal reliance remain different targets. Post-hoc matching can produce a defensible reference without showing that the source influenced generation. Progress would use controlled evidence changes alongside semantic citation assessment, while preserving the distinction between observed behavioral dependence and inaccessible internal computation.

  4. The practical value of generated answers depends on the review workflow. Search, extraction, and synthesis expose different amounts of work to the user, and answer benchmarks do not measure all of it. Progress would compare those interfaces on the same documentation tasks, including supported completeness, verification effort, errors, and completion time.

Follow the curated reading path through the speakers and demonstrations behind this entry.

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

640 matching talks

TalkSpeakerEventYear
Douwe KielaAI Engineer Summit 20252025
Emil EifremAI Engineer World's Fair 20242024
Daniel ChalefAI Engineer World's Fair 20262026
Abed MatiniAI Engineer World's Fair 20262026
Leonie MonigattiAI Engineer Europe 20262026
AI Engineering 101

Cited in this entry

Noah HeinAI Engineer Summit 20232023
Garrett GalowAI Engineer Europe 20262026
Yohei NakajimaAI Engineer World's Fair 20262026
Anton TroynikovAI Engineer Summit 20232023
Cedric ClyburnAI Engineer World's Fair 20262026
Sam JulienAI Engineer World's Fair 20252025
Philipp KrennAI Engineer World's Fair 20252025
David KaramAI Engineer World's Fair 20252025
Paul CopplestoneAI Engineer Summit 20232023
Simrat HanspalAI Engineer Summit 20232023
Dex HorthyAI Engineer Code 20252025
Louis-François Bouchard, Omar Solano, Samridhi VaidAI Engineer World's Fair 20262026
Sally-Ann DeLuciaAI Engineer Europe 20262026
Ari HeljakkaAI Engineer Summit 20252025
Building security around ML

Transcript reviewed

Dr. Andrew DavisAI Engineer World's Fair 20242024
Harshil AgrawalAI Engineer Europe 20262026
Sarah Sachs, Carlos Esteban, Doug GuthrieAI Engineer World's Fair 20252025
Prompt Engineering is Dead

Cited in this entry

Nir GazitAI Engineer World's Fair 20252025
Mahmoud MabroukAI Engineer Europe 20262026
Divakar KumarAI Engineer World's Fair 20262026
Sandipan BhaumikAI Engineer Europe 20262026
Nishant GuptaAI Engineer World's Fair 20262026
Zach BlumenfeldAI Engineer World's Fair 20252025
Jonathan LarsonAI Engineer World's Fair 20252025
Kyle CorbittAI Engineer World's Fair 20242024
Tengyu MaAI Engineer World's Fair 20252025
Michael Hunger, Stephen Chin, Jesús BarrasaAI Engineer World's Fair 20252025
Stephen ChinAI Engineer World's Fair 20262026
Peter Werry, BrandonAI Engineer Europe 20262026
Karina NguyenAI Engineer Summit 20232023
Dat Ngo, Aman KhanAI Engineer World's Fair 20252025
Building an Agentic Platform

Transcript reviewed

Ben KusAI Engineer World's Fair 20252025
Yuval Belfer, Niv GranotAI Engineer World's Fair 20252025
Laurie VossAI Engineer Europe 20262026
Andreas KolleggerAI Engineer World's Fair 20252025
Tim AingeAI Engineer World's Fair 20262026
Julia Neagu, Deanna Emery, Maitar AsherAI Engineer World's Fair 20252025
Waseem AlshikhAI Engineer Summit 20252025
Stephen Chin, Jonathan LoweAI Engineer Summit 20252025
Anushrut GuptaAI Engineer World's Fair 20252025
Mike ConoverAI Engineer Summit 20252025
Stephen ChinAI Engineer World's Fair 20252025
Brendan RappazzoAI Engineer World's Fair 20262026
Kuba RogutAI Engineer Europe 20262026
Arjun Chintapalli, Bhavani KalisettyAI Engineer Summit 20252025
Louis-François Bouchard, Paul Iusztin, Samridhi VaidAI Engineer Europe 20262026
Will BrykAI Engineer World's Fair 20252025
Jerry LiuAI Engineer World's Fair 20252025
Dan MasonAI Engineer World's Fair 20252025
Stephen ChinAI Engineer Europe 20262026
Stephen ChinAI Engineer Code 20252025
Andreas Kollegger, Zaid ZaimAI Engineer Europe 20262026
Kevin HouAI Engineer World's Fair 20242024
Ofer MendelevitchAI Engineer Code 20252025
Benjamin FletcherAI Engineer World's Fair 20242024
Richard SocherAI Engineer World's Fair 20262026
Nina Lopatina, Rajiv ShahAI Engineer World's Fair 20252025
Jerry LiuAI Engineer World's Fair 20242024
Phoebe KlettAI Engineer World's Fair 20242024
Vaibhav Page, Infant VasanthAI Engineer World's Fair 20252025
How Deep Research Works

Metadata candidate

Mukund Sridhar, Aarush SelvanAI Engineer Summit 20252025
Raia HadsellAI Engineer Europe 20262026
Vinesh GudlaAI Engineer World's Fair 20252025
Hanna Lichtenberg, Aamir ShakirAI Engineer World's Fair 20262026
Amol KapoorAI Engineer World's Fair 20262026
Mitesh PatelAI Engineer World's Fair 20252025
Intro to GraphRAG

Metadata candidate

Zach BlumenfeldAI Engineer World's Fair 20252025
Andreas Kolleger, Zach Blumenthal, Michael Hunger, TomaszAI Engineer World's Fair 20242024
Tom SmokerAI Engineer World's Fair 20252025
Ben HolmesAI Engineer World's Fair 20262026
Joe FiotiAI Engineer World's Fair 20252025
Stefania DrugaAI Engineer World's Fair 20262026
Ola MabadejeAI Engineer World's Fair 20252025
On AI and Knowledge

Metadata candidate

Pablo CastroAI Engineer World's Fair 20262026
Phil NashAI Engineer Europe 20262026
Ben FlastAI Engineer World's Fair 20242024
Pablo CastroAI Engineer World's Fair 20242024
RAG for VPs of AI

Metadata candidate

Jerry LiuAI Engineer World's Fair 20242024
Kuba RogutAI Engineer Europe 20262026
Eugene YanAI Engineer World's Fair 20252025
Vaidas RazgaitisAI Engineer World's Fair 20262026
Calvin Qi, Chang SheAI Engineer World's Fair 20252025
Stop Using RAG as Memory

Metadata candidate

Daniel ChalefAI Engineer World's Fair 20252025
Ofer MendelevitchAI Engineer Summit 20252025
William LyonAI Engineer World's Fair 20252025
Jonathan FernandesAI Engineer World's Fair 20252025
Frank LiuAI Engineer World's Fair 20252025
Sonam PankajAI Engineer World's Fair 20262026
Philipp KrennAI Engineer World's Fair 20252025
Nico AlbaneseAI Engineer Summit 20252025
Suman DebnathAI Engineer World's Fair 20252025
Chin Keong LamAI Engineer World's Fair 20252025
Rafael LeviAI Engineer Europe 20262026
Cedric Vidal, David Smith, Miguel MartinezAI Engineer World's Fair 20242024
Ivan LeoAI Engineer Code 20252025
Tomas ReimersAI Engineer World's Fair 20252025
Zhengyao JiangAI Engineer World's Fair 20262026
Benoit SchillingsAI Engineer World's Fair 20262026
Ritvik PandyaAI Engineer World's Fair 20262026
AI Engineer Summit 20252025
Stephen BatifolAI Engineer Europe 20262026
Diego CarpinteroAI Engineer Europe 20262026
Shelby HeineckeAI Engineer World's Fair 20242024
A Song of Types and Agents

Metadata candidate

Roberto StagiAI Engineer World's Fair 20262026
Sharmila Chokalingam, ShubhiAI Engineer World's Fair 20242024
Barry Zhang, Mahesh MuragAI Engineer Code 20252025
Agents Building Agents

Metadata candidate

Alfonso GrazianoAI Engineer World's Fair 20262026
Ian Butler, Nick GregoryAI Engineer World's Fair 20252025
Rajat ShahAI Engineer World's Fair 20262026
Anita KirkovskaAI Engineer Summit 20252025
Varsha ShahAI Engineer World's Fair 20262026
Charles FryeAI Engineer Summit 20232023
Zach Blumenfeld, Ben Squire, Ryan KnightAI Engineer World's Fair 20262026
Apoorva JoshiAI Engineer World's Fair 20262026
Charlie GuoAI Engineer World's Fair 20252025
Richmond AlakeAI Engineer World's Fair 20252025
Corey CooperAI Engineer World's Fair 20252025
Sina ShahandehAI Engineer World's Fair 20262026
Parth AsawaAI Engineer World's Fair 20262026
Łukasz GandeckiAI Engineer World's Fair 20252025
Paul HenryAI Engineer World's Fair 20242024
Paige BaileyAI Engineer Europe 20262026
Eliza Cabrera, Jeremy SilvaAI Engineer World's Fair 20252025
Varun Badrinath Krishna, Petro Junior Milan, Rachelle MatternAI Engineer World's Fair 20242024
Dr. Sajjan KanukolanuAI Engineer World's Fair 20262026
Raj NavakotiAI Engineer Europe 20262026
Julián Duque, Anush DSouzaAI Engineer World's Fair 20252025
Jeff NgAI Engineer World's Fair 20262026
Mahesh MuragAI Engineer Summit 20252025
Sherwood Callaway, Satwik SinghAI Engineer World's Fair 20252025
Eugene YanAI Engineer Summit 20232023
Thor Schaeff, PaulAI Engineer World's Fair 20252025
Shaan DesaiAI Engineer Summit 20252025
Kat Kampf, Ammaar ReshiAI Engineer Code 20252025
Marlene Mhangami, Liam HamptonAI Engineer Europe 20262026
David KaramAI Engineer World's Fair 20252025
Adam TerlsonAI Engineer Summit 20252025
Apoorva JoshiAI Engineer World's Fair 20252025
Building Reactive AI Apps

Metadata candidate

Matt WelshAI Engineer Summit 20232023
Gergely Orosz, Simon EskildsenAI Engineer World's Fair 20262026
Eno ReyesAI Engineer World's Fair 20242024
Building Self-Coding Agents

Metadata candidate

Colin FlahertyAI Engineer Summit 20252025
Sandra KublikAI Engineer World's Fair 20242024
Anant ShankhdharAI Engineer World's Fair 20262026
Thariq ShihiparAI Engineer Code 20252025
Derek BinghamAI Engineer World's Fair 20242024
Rachna SrivastavaAI Engineer World's Fair 20252025
Cohere for VPs of AI

Metadata candidate

Vivek MuppallaAI Engineer World's Fair 20242024
Context Is the New Code

Metadata candidate

Patrick DeboisAI Engineer Europe 20262026
Karina NguyenAI Engineer Summit 20252025
Hanchi WangAI Engineer World's Fair 20242024
Dmytro (Dima) DzhulgakovAI Engineer World's Fair 20242024
Mani KhanujaAI Engineer World's Fair 20252025
Devendra Chaplot, Devendra Singh ChaplotAI Engineer World's Fair 20242024
Shawn Wang (swyx)AI Engineer World's Fair 20252025
Dan ShipperAI Engineer Code 20252025
Abi AryanAI Engineer Summit 20232023
Laurie VossAI Engineer World's Fair 20252025
Sylendran ArunagiriAI Engineer World's Fair 20252025
Aparna Dhinkaran, Aparna DhinakaranAI Engineer Summit 20252025
Garry TanAI Engineer World's Fair 20262026
Katelyn LesseAI Engineer Code 20252025
Mehedi HassanAI Engineer Europe 20262026
Joel HronAI Engineer World's Fair 20252025
Antje Barth, Mike ChambersAI Engineer World's Fair 20242024
Omri Bruchim, Tomer AstAI Engineer World's Fair 20262026
Jason LiuAI Engineer World's Fair 20262026
Ilan BigioAI Engineer Summit 20252025
Cassidy HardinAI Engineer Europe 20262026
Git push, get an AI API.

Metadata candidate

Ryan Fox-TylerAI Engineer World's Fair 20242024
Anirban ChatterjeeAI Engineer World's Fair 20262026
Tanmai GopalAI Engineer World's Fair 20242024
Phil HetzelAI Engineer Europe 20262026
Niels RoggeAI Engineer World's Fair 20262026
Jaspreet SinghAI Engineer World's Fair 20252025
Vinoo GaneshAI Engineer World's Fair 20262026
Patrick DoughertyAI Engineer Summit 20252025
Chau TranAI Engineer World's Fair 20252025
How to Build Trustworthy AI

Metadata candidate

Allie HoweAI Engineer World's Fair 20252025
David MyttonAI Engineer World's Fair 20252025
Zhou YuAI Engineer Summit 20252025
Jeff Huber, Jason LiuAI Engineer World's Fair 20252025
Patricija ŽemaitytėAI Engineer World's Fair 20262026
Hypermode Launch

Metadata candidate

Kevin Van GundyAI Engineer World's Fair 20242024
Nicolas SchlaepferAI Engineer World's Fair 20242024
Radek SienkiewiczAI Engineer Europe 20262026
Lachlan Ainley, Humza IqbalAI Engineer World's Fair 20242024
Yu SuAI Engineer World's Fair 20262026
Judging LLMs

Metadata candidate

Alex VolkovAI Engineer World's Fair 20242024
Juan PeredoAI Engineer Summit 20252025
Xiaofeng WangAI Engineer Summit 20252025
Shlok KhemaniAI Engineer World's Fair 20262026
Kam LasaterAI Engineer Summit 20252025
Rachelle Mattern, Petro Milan, Varun KrishnaAI Engineer World's Fair 20242024
Danilo CamposAI Engineer Europe 20262026
LLM Evals That Work IRL

Metadata candidate

Aparna Dhinkaran, Aparna DhinakaranAI Engineer World's Fair 20242024
Daniel WhitenackAI Engineer World's Fair 20242024
Hubert MisztelaAI Engineer World's Fair 20242024
Lin Qiao, Dmytro (Dima) DzhulgakovAI Engineer World's Fair 20242024
Eashan SinhaAI Engineer World's Fair 20252025
Ronan McGovernAI Engineer World's Fair 20252025
Drasko ProfirovicAI Engineer World's Fair 20262026
Mark Bain, Vasilije Markovic, Daniel Chalef, Alex GilmoreAI Engineer World's Fair 20252025
Minimax M2

Metadata candidate

Olive SongAI Engineer Code 20252025
Ilan BigioAI Engineer World's Fair 20252025
Ahmed MenshawyAI Engineer World's Fair 20242024
Shafik QuoraisheeAI Engineer World's Fair 20252025
Omar KhattabAI Engineer World's Fair 20252025
Maggie AppletonAI Engineer Europe 20262026
Yesu FengAI Engineer World's Fair 20252025
Frank CoyleAI Engineer World's Fair 20262026
DottaAI Engineer Europe 20262026
Diego RodriguezAI Engineer World's Fair 20252025
Ishan AnandAI Engineer World's Fair 20262026
Randall HuntAI Engineer World's Fair 20252025
Sander SchulhoffAI Engineer World's Fair 20252025
Jason LiuAI Engineer World's Fair 20242024
Raza HabibAI Engineer World's Fair 20242024
Idan GazitAI Engineer World's Fair 20262026
Will BrownAI Engineer World's Fair 20262026
Shashi JagtapAI Engineer World's Fair 20262026
Max RyabininAI Engineer Europe 20262026
Pamela Fox, Harald Kirschner, Gabriela de QueirozAI Engineer World's Fair 20242024
Scaling Compute on Context

Metadata candidate

Jack MorrisAI Engineer World's Fair 20262026
Bobby Tiernay, Kam SweenAI Engineer World's Fair 20252025
Joshua SnyderAI Engineer Europe 20262026
Raahul Singh, Vanč LevstikAI Engineer World's Fair 20262026
Arek BoruckiAI Engineer World's Fair 20262026
Reid MayoAI Engineer Summit 20232023
Peter BarAI Engineer World's Fair 20252025
Marc KlingenAI Engineer Europe 20262026
Yogendra MirajeAI Engineer World's Fair 20262026
The New Code

Metadata candidate

Sean GroveAI Engineer World's Fair 20252025
Charles PackerAI Engineer Summit 20252025
Josh PurtellAI Engineer World's Fair 20252025
Elizabeth Fuentes LeoneAI Engineer World's Fair 20262026
Brandon WaselnukAI Engineer Europe 20262026
Annabell SchäferAI Engineer World's Fair 20262026
Lars GrammelAI Engineer Summit 20232023
Taylor Jordan SmithAI Engineer World's Fair 20252025
Jack MorrisAI Engineer Code 20252025
Rob CheungAI Engineer World's Fair 20242024
Jim BennettAI Engineer World's Fair 20252025
Devansh TandonAI Engineer World's Fair 20252025
Sohail Shaikh, Ankush RastogiAI Engineer World's Fair 20262026
Barr YaronAI Engineer World's Fair 20252025
Apoorva Joshi, Ben PerlmutterAI Engineer World's Fair 20242024
Sumit AgarwalAI Engineer World's Fair 20242024
Chris White, Bryan Bischof, Brittany WalkerAI Engineer Summit 20232023
Travis FrisingerAI Engineer World's Fair 20252025
Kieran KlaassenAI Engineer World's Fair 20262026
Rushabh DoshiAI Engineer World's Fair 20262026
Linus LeeAI Engineer Summit 20232023
Chang She, Noah ShpakAI Engineer World's Fair 20242024
Omer PrimorAI Engineer World's Fair 20262026
Filip MakraduliAI Engineer World's Fair 20252025
Emil EifremAI Engineer World's Fair 20262026
Roy DerksAI Engineer Summit 20252025
Leo PekelisAI Engineer World's Fair 20242024
Dr. Sarah BuchnerAI Engineer World's Fair 20242024
Paul Iusztin, Louis-François BouchardAI Engineer World's Fair 20262026
Philip RathleAI Engineer World's Fair 20242024
James LeAI Engineer World's Fair 20262026
Alex AlbertAI Engineer World's Fair 20242024
Mukuntha Narayanan, Han WangAI Engineer World's Fair 20252025
Luis Romero-SevillaAI Engineer World's Fair 20262026
Nupur SharmaAI Engineer Europe 20262026
Subbiah Sethuraman, Abhilash AsokanAI Engineer World's Fair 20262026
Diane LinAI Engineer World's Fair 20262026
Jesús BarrasaAI Engineer World's Fair 20252025
Zach BlumenfeldAI Engineer Europe 20262026
Ameya BhatawdekarAI Engineer World's Fair 20262026
Ben BurtenshawAI Engineer Europe 20262026
Dan BjornnAI Engineer World's Fair 20262026
Sachin KumarAI Engineer World's Fair 20262026
Mike PhippsAI Engineer World's Fair 20262026
Harrison ChaseAI Engineer World's Fair 20252025
Sanja GrbicAI Engineer World's Fair 20262026
Shawn "swyx" WangAI Engineer World's Fair 20262026
A Genius With Amnesia

Metadata candidate

Victor SavkinAI Engineer World's Fair 20262026
Nathan LambertAI Engineer World's Fair 20252025
Logan KilpatrickAI Engineer World's Fair 20252025
Sara HookerAI Engineer World's Fair 20262026
Brendan O'LearyAI Engineer Europe 20262026
Hubert MisztelaAI Engineer World's Fair 20252025
Uday Kiran Medisetty, Adam HudaAI Engineer World's Fair 20262026
Shawn "swyx" WangAI Engineer Europe 20262026
Armanas PovilionisAI Engineer World's Fair 20262026
Sam BhagwatAI Engineer World's Fair 20252025
Yegor Denisov-BlanchAI Engineer World's Fair 20252025
swyxAI Engineer World's Fair 20242024
Clay Cockrell, Tony FabrikantAI Engineer World's Fair 20262026
Tomas ReimersAI Engineer World's Fair 20252025
Josh AlbrechtAI Engineer World's Fair 20252025
Sunny MadraAI Engineer World's Fair 20242024
Samuel DentonAI Engineer World's Fair 20262026
Angel Ortmann LeeAI Engineer World's Fair 20262026
Nick Ung, Akshay SharmaAI Engineer World's Fair 20262026
Du’An Lightfoot, Banjo ObayomiAI Engineer World's Fair 20252025
Bruno Passos, Beyang LiuAI Engineer Summit 20252025
Soumya Gupta, Jai ChopraAI Engineer World's Fair 20262026
Cedric VidalAI Engineer World's Fair 20252025
Sander DielemanAI Engineer Europe 20262026
Anju KambadurAI Engineer Summit 20252025
Nathan SoboAI Engineer World's Fair 20252025
Sunil PaiAI Engineer Europe 20262026
Jacob KahnAI Engineer Code 20252025
Copilots Everywhere

Metadata candidate

Thomas Dohmke, Eugene YanAI Engineer World's Fair 20242024
#define AI Engineer

Metadata candidate

Greg Brockman, swyx, Jensen HuangAI Engineer World's Fair 20252025
Satya NittaAI Engineer World's Fair 20242024
Sayash KapoorAI Engineer Summit 20252025
Gaurav MishraAI Engineer World's Fair 20262026
Samuel ColvinAI Engineer Code 20252025
Frontier Feud

Metadata candidate

Barr Yaron, Mihir, John, Tina, Shresta, Paige, Colin, Petra, StevenAI Engineer Summit 20252025
Dave BurnisonAI Engineer World's Fair 20242024
Dave Burnison, Alex Malebranche, Dimitrios Philliou, Christina Warren, HaraldAI Engineer World's Fair 20242024
GitHub Next Explorations

Metadata candidate

Rahul PanditaAI Engineer World's Fair 20242024
Luke HarriesAI Engineer Europe 20262026
Mike BursellAI Engineer World's Fair 20252025
Iman MakaremiAI Engineer World's Fair 20252025
Dr Bryan Bischof, Dr Bryan BischofAI Engineer World's Fair 20242024
KP Sawhney, Ian BallantyneAI Engineer Europe 20262026
Yogendra MirajeAI Engineer World's Fair 20252025
Kwindla Hultman KramerAI Engineer World's Fair 20242024
How to Kill the Code Review

Metadata candidate

Ankit JainAI Engineer World's Fair 20262026
Yegor Denisov-BlanchAI Engineer Code 20252025
Muktesh MishraAI Engineer World's Fair 20252025
Joe ReeveAI Engineer Europe 20262026
Kyle CorbittAI Engineer World's Fair 20252025
Jake NationsAI Engineer Code 20252025
Gabriel Jorge MenezesAI Engineer World's Fair 20262026
Heath BlackAI Engineer Summit 20252025
Vibhu SapraAI Engineer World's Fair 20252025
Kent C. DoddsAI Engineer World's Fair 20252025
Joel BeckerAI Engineer Code 20252025
Kelvin MaAI Engineer World's Fair 20252025
Shirsha ChaudhuriAI Engineer Summit 20252025
Will BrownAI Engineer World's Fair 20262026
Jess Grogan-Avignon, Jack WangAI Engineer Europe 20262026
Martin Harrysson, Natasha ManiarAI Engineer Code 20252025
Cedric VidalAI Engineer World's Fair 20242024
Andres MarafiotiAI Engineer Europe 20262026
Harald Kirschner, Christopher HarrisonAI Engineer World's Fair 20252025
Jon PeckAI Engineer World's Fair 20252025
Recursive Model Improvement

Metadata candidate

Lee RobinsonAI Engineer World's Fair 20262026
Respect The Process

Metadata candidate

Andrew DumitAI Engineer World's Fair 20262026
Sachin GuptaAI Engineer World's Fair 20262026
RL Environments at Scale

Metadata candidate

Will BrownAI Engineer Code 20252025
Gabriela de Queiroz, Aishwarya Srinivasan, Pamela FoxAI Engineer World's Fair 20242024
Michael YuanAI Engineer World's Fair 20252025
Second Order Effects

Metadata candidate

Cheng LouAI Engineer World's Fair 20242024
Arjun Desai, Rohit TalluriAI Engineer World's Fair 20252025
Eno ReyesAI Engineer World's Fair 20252025
Jared JoselowitzAI Engineer World's Fair 20262026
Denys LinkovAI Engineer World's Fair 20252025
Brian BalfourAI Engineer World's Fair 20252025
Ibragim BadertdinovAI Engineer Europe 20262026
Nan JiangAI Engineer World's Fair 20262026
Nuno CamposAI Engineer Europe 20262026
Brendan O'DonoghueAI Engineer Europe 20262026
Ronan McGovernAI Engineer World's Fair 20252025
Patrick DeboisAI Engineer World's Fair 20252025
Quinn SlackAI Engineer World's Fair 20242024
Vincent ChenAI Engineer Europe 20262026
Ado KukicAI Engineer Summit 20232023
Jerry Wu, Wyatt MarshallAI Engineer World's Fair 20252025
Ahmad OsmanAI Engineer World's Fair 20262026
Beyang LiuAI Engineer World's Fair 20252025
Addy OsmaniAI Engineer World's Fair 20262026
Itamar FriedmanAI Engineer World's Fair 20262026
Stefania DrugaAI Engineer World's Fair 20242024
Kwindla Kramer, Kwindla Hultman KramerAI Engineer World's Fair 20262026
Ahmed AhresAI Engineer World's Fair 20262026
Beyang LiuAI Engineer World's Fair 20242024
Itamar FriedmanAI Engineer Code 20252025
Kathryn Grayson NanzAI Engineer World's Fair 20262026
The Weekend AI Engineer

Metadata candidate

Hassan El MghariAI Engineer Summit 20232023
Thinking Deeper in Gemini

Metadata candidate

Jack RaeAI Engineer World's Fair 20252025
MuhtesemAI Engineer Summit 20252025
tldraw computer

Metadata candidate

Steve RuizAI Engineer World's Fair 20252025
Ayush BhardwajAI Engineer World's Fair 20262026
Training Agentic Reasoners

Metadata candidate

Will BrownAI Engineer World's Fair 20252025
Angelos PerivolaropoulosAI Engineer Europe 20262026
Uri Rolls, Thom WolfAI Engineer World's Fair 20262026
Sangwu LeeAI Engineer World's Fair 20262026
Kathleen KenealyAI Engineer World's Fair 20242024
Victor DibiaAI Engineer World's Fair 20252025
Peter RobicheauxAI Engineer World's Fair 20252025
Eddie SiegelAI Engineer Summit 20252025
Jyh-Jing HwangAI Engineer World's Fair 20252025
Rajkumar SakthivelAI Engineer World's Fair 20262026
Bilge YücelAI Engineer Europe 20262026
Nicholas ArcolanoAI Engineer Code 20252025
What RL Means for Agents

Metadata candidate

Will BrownAI Engineer Summit 20252025
Why Agent Engineering

Metadata candidate

swyx (Shawn Wang)AI Engineer Summit 20252025
Joel BeckerAI Engineer Code 20252025
Mark BissellAI Engineer World's Fair 20252025
Eugene CheahAI Engineer Summit 20252025
Dan FarrellyAI Engineer World's Fair 20262026
Joel Allou, Ornella BahidikaAI Engineer World's Fair 20262026
Dex HorthyAI Engineer World's Fair 20252025
Joseph NelsonAI Engineer Summit 20232023
2026: The Year the IDE Died

Metadata candidate

Steve Yegge, Gene KimAI Engineer Code 20252025
Hamed Firooz, Maziar SanjabiAI Engineer World's Fair 20252025
Steve YeggeAI Engineer World's Fair 20262026
Kevin HouAI Engineer Summit 20252025
Jesse HuAI Engineer Code 20252025
Gabe De MesaAI Engineer World's Fair 20262026
Agents Need Feature Flags

Metadata candidate

Sachin GuptaAI Engineer World's Fair 20262026
Armanas PovilionisAI Engineer World's Fair 20262026
Charles FryeAI Engineer Summit 20232023
Philipp SchmidAI Engineer World's Fair 20252025
Phlo YoungAI Engineer World's Fair 20242024
Nick Nisi, Zack ProserAI Engineer World's Fair 20252025
Nagkumar Arkalgud, Keiji KanazawaAI Engineer World's Fair 20252025
AI SDK v6

Metadata candidate

Nico AlbaneseAI Engineer Europe 20262026
Vasuman MozaAI Engineer World's Fair 20262026
Beyang LiuAI Engineer Code 20252025
Frank CoyleAI Engineer World's Fair 20262026
Patrick LöberAI Engineer Europe 20262026
Henry MaoAI Engineer World's Fair 20252025
Ali KhialAI Engineer World's Fair 20262026
Filip KozeraAI Engineer World's Fair 20252025
Siddharth AhujaAI Engineer World's Fair 20252025
Angus J. McLeanAI Engineer Europe 20262026
Aparna DhinakaranAI Engineer World's Fair 20252025
SallyAnn DeLucia, Fuad AliAI Engineer Code 20252025
Paige Bailey, Guillaume Vernade, Ian BallantyneAI Engineer Europe 20262026
Build Systems, Not Code

Metadata candidate

Angie JonesAI Engineer World's Fair 20262026
Michael HablichAI Engineer Europe 20262026
Tom RedmanAI Engineer World's Fair 20242024
Harrison ChaseAI Engineer Summit 20232023
Nishant GuptaAI Engineer World's Fair 20262026
Cornelia DavisAI Engineer Code 20252025
Matt PocockAI Engineer World's Fair 20262026
Michael FesterAI Engineer World's Fair 20252025
Tom MoorAI Engineer World's Fair 20252025
Dominik KundelAI Engineer World's Fair 20252025
Jamie Neuwirth, Zack WittenAI Engineer World's Fair 20242024
Liam McGarrigleAI Engineer Europe 20262026
Andrew ThompsonAI Engineer World's Fair 20252025
Cat Wu, Thariq Shihipar, Simon WillisonAI Engineer World's Fair 20262026
Lance MartinAI Engineer World's Fair 20262026
Pedro RodriguesAI Engineer Europe 20262026
Yusuf OlokobaAI Engineer Code 20252025
Dylan PatelAI Engineer World's Fair 20242024
Val Bercovici, Callan FoxAI Engineer Code 20252025
Convex Launch

Metadata candidate

Jamie TurnerAI Engineer World's Fair 20242024
James ShiAI Engineer World's Fair 20262026
Develop at Idea Velocity

Metadata candidate

Jeffrey Lee-ChanAI Engineer World's Fair 20262026
Max Kanat-AlexanderAI Engineer Code 20252025
Scott WuAI Engineer World's Fair 20252025
Phil HetzelAI Engineer Europe 20262026
Ara KhanAI Engineer Europe 20262026
Arthur ObjartelAI Engineer Summit 20252025
Philipp SchmidAI Engineer World's Fair 20262026
Kevin MaduraAI Engineer Code 20252025
Ishita DagaAI Engineer World's Fair 20262026
Evals Are Not Unit Tests

Metadata candidate

Ido PesokAI Engineer World's Fair 20252025
Akele Reed, Dave Revere, Doug KellerAI Engineer World's Fair 20262026
Carlos Esteban, DougAI Engineer World's Fair 20252025
Sam BhagwatAI Engineer World's Fair 20262026
Maxime LabonneAI Engineer Europe 20262026
Sarah ChiengAI Engineer Europe 20262026
fighting slop with slop

Metadata candidate

Vaibhav GuptaAI Engineer World's Fair 20262026
Ankur GoyalAI Engineer World's Fair 20252025
Craig WattrusAI Engineer World's Fair 20252025
Kevin BaiAI Engineer World's Fair 20262026
Rustem FeyzkhanovAI Engineer World's Fair 20262026
Samir ModyAI Engineer Code 20252025
Chris NoringAI Engineer Europe 20262026
Alex CheemaAI Engineer Europe 20262026
Rachel Lee Nabors (RL Nabors)AI Engineer World's Fair 20262026
Alex AtallahAI Engineer World's Fair 20252025
Florina Muntenescu, Oli GaymondAI Engineer Europe 20262026
Giving a Voice to AI Agents

Metadata candidate

Scott StephensonAI Engineer World's Fair 20242024
Matija SosicAI Engineer Summit 20232023
Rashi AgrawalAI Engineer World's Fair 20262026
Ryan Lopopolo, Vibhu SapraAI Engineer Europe 20262026
Mithun HunsurAI Engineer Summit 20232023
Vasant KearneyAI Engineer World's Fair 20262026
How Claude Code Works

Metadata candidate

Jared ZoneraichAI Engineer Code 20252025
Alex BauerAI Engineer World's Fair 20262026
Ishan AnandAI Engineer World's Fair 20252025
Ash Prabaker, Andrew WilsonAI Engineer Europe 20262026
Hamel Husain, Greg CeccarelliAI Engineer Summit 20252025
Hamel Husain, Emil SedghAI Engineer World's Fair 20242024
Ian ButlerAI Engineer World's Fair 20252025
Rene BrandelAI Engineer World's Fair 20252025
Mustafa Ali, Kyle CorbittAI Engineer Summit 20252025
Samuel ColvinAI Engineer World's Fair 20252025
Kyle Jaejun LeeAI Engineer World's Fair 20262026
Erik MeijerAI Engineer World's Fair 20262026
Vivek TrivedyAI Engineer World's Fair 20262026
Mahmoud AbdelwahabAI Engineer Code 20252025
Suman DebnathAI Engineer World's Fair 20252025
Ian WebsterAI Engineer World's Fair 20242024
Robert ChandlerAI Engineer World's Fair 20252025
Chip HuyenAI Engineer Summit 20252025
Dat NgoAI Engineer Europe 20262026
2025 in LLMs so far

Metadata candidate

Simon WillisonAI Engineer World's Fair 20252025
Daniel HanAI Engineer World's Fair 20242024
Matthias LoiblAI Engineer World's Fair 20252025
Pietro ZulloAI Engineer World's Fair 20262026
MCP is all you need

Metadata candidate

Samuel ColvinAI Engineer World's Fair 20252025
Mentoring the Machine

Metadata candidate

Eric HouAI Engineer World's Fair 20252025
Amy Boyd, Nitya NarasimhanAI Engineer Europe 20262026
Rami AlhamadAI Engineer World's Fair 20252025
Mark HenningsAI Engineer Summit 20232023
Rémi LoufAI Engineer World's Fair 20242024
Sharif ShameemAI Engineer World's Fair 20252025
Simon WillisonAI Engineer World's Fair 20242024
Simon WillisonAI Engineer Summit 20232023
Saoud RizwanAI Engineer World's Fair 20262026
OpenAI for VPs of AI

Metadata candidate

Prashant Mital, Toki SherbakovAI Engineer Summit 20252025
Lech KalinowskiAI Engineer World's Fair 20262026
OpenLLMetry is all you need

Metadata candidate

Nir GazitAI Engineer Summit 20252025
Jeronim MorinaAI Engineer World's Fair 20242024
Antje BarthAI Engineer World's Fair 20262026
Shivam VermaAI Engineer World's Fair 20252025
Kwindla Hultman KramerAI Engineer World's Fair 20252025
Samuel ColvinAI Engineer Europe 20262026
Dmitry KuchinAI Engineer World's Fair 20252025
Pragmatic AI With TypeChat

Metadata candidate

Daniel RosenwasserAI Engineer Summit 20232023
Prompt Engineering Tactics

Metadata candidate

Dan ClearyAI Engineer Summit 20232023
Chris ParsonsAI Engineer Europe 20262026
Yuval BelferAI Engineer World's Fair 20252025
Recursive Coding Agents

Metadata candidate

Raymond WeitekampAI Engineer World's Fair 20262026
David GomesAI Engineer Europe 20262026
Mozhgan Kabiri ChimehAI Engineer Europe 20262026
Fouad MatinAI Engineer World's Fair 20252025
Scaffold Wisely

Metadata candidate

Rahul SengottuveluAI Engineer Summit 20252025
Preeti SomalAI Engineer World's Fair 20252025
Sam MorrowAI Engineer Europe 20262026
Ryan DahlAI Engineer World's Fair 20262026
Merve NoyanAI Engineer Europe 20262026
Aman KhanAI Engineer World's Fair 20252025
Shreya Rajpal, Aman GuptaAI Engineer World's Fair 20262026
Skills are the New SDKs

Metadata candidate

Elvin AghammadzadaAI Engineer World's Fair 20262026
Skills at Scale

Metadata candidate

Nick Nisi, Zack ProserAI Engineer Europe 20262026
Vikas ParuchuriAI Engineer World's Fair 20252025
Asaf BordAI Engineer Code 20252025
Ishan AnandAI Engineer World's Fair 20242024
Sarah GuoAI Engineer World's Fair 20252025
Nader Khalil, Alex Cheema, Matthew Berman, Ahmad Osman, Joseph NelsonAI Engineer World's Fair 20262026
Karan GoelAI Engineer World's Fair 20242024
Manish SanwalAI Engineer Summit 20252025
Thiyagarajan MaruthavananAI Engineer World's Fair 20262026
Isadora Martin-DyeAI Engineer World's Fair 20262026
Pydantic is all you need

Metadata candidate

Jason LiuAI Engineer Summit 20232023
Kobie CrawfordAI Engineer Europe 20262026
The Agent-Native Company

Metadata candidate

Rick BlalockAI Engineer World's Fair 20252025
Tara AgyemangAI Engineer Europe 20262026
Ramesh Raskar, Maria GorskikhAI Engineer World's Fair 20262026
Kevin Madura, Mo BhasinAI Engineer World's Fair 20252025
Corey J. GallonAI Engineer Code 20252025
Natalie MeurerAI Engineer World's Fair 20262026
The End of Apps

Metadata candidate

KitzeAI Engineer Europe 20262026
Ben HylakAI Engineer World's Fair 20242024
Armin Ronacher, Cristina Poncela CubeiroAI Engineer Europe 20262026
Ankur GoyalAI Engineer World's Fair 20252025
The Future of Work

Metadata candidate

Toran Bruce Richards, Silen Naihin, PootsAI Engineer Summit 20232023
Allie Howe, Dex Horthy, Geoffrey Huntley, Ian Livingstone, Greg PstruchaAI Engineer World's Fair 20262026
The Intelligent Interface

Metadata candidate

Samantha Whitmore, Jason YuanAI Engineer Summit 20232023
Almog BakuAI Engineer Summit 20252025
The Log Is The Agent

Metadata candidate

Ishaan SehgalAI Engineer World's Fair 20262026
The Making of Devin

Metadata candidate

Scott WuAI Engineer World's Fair 20242024
Jacob E. ThomasAI Engineer World's Fair 20262026
Hassan El MghariAI Engineer World's Fair 20262026
Dan ClearyAI Engineer Summit 20252025
Diego Rodriguez, Eugene, Jonas Bauer, Shijia Liao, David Vorick, Alex AtallahAI Engineer World's Fair 20252025
Ted JohnsonAI Engineer World's Fair 20262026
Alex Volkov, Benjamin EckelAI Engineer World's Fair 20252025
Walden, Carter, Tanay, Alex Atallah, NavAI Engineer World's Fair 20262026
Alberto RomeroAI Engineer Code 20252025
Aparna DhinakaranAI Engineer Code 20252025
Maxime Rivest, Isaac MillerAI Engineer World's Fair 20262026
Gregory BrussAI Engineer World's Fair 20252025
Trust, but Verify

Metadata candidate

Shreya RajpalAI Engineer Summit 20232023
Eric AllamAI Engineer World's Fair 20252025
Useful General Intelligence

Metadata candidate

Danielle PerszykAI Engineer World's Fair 20252025
Jeff SchomayAI Engineer Summit 20232023
Eugene YanAI Engineer World's Fair 20262026
Anna Marie BenzonAI Engineer World's Fair 20262026
Harald KirschnerAI Engineer World's Fair 20252025
Dippu Kumar SinghAI Engineer Europe 20262026
Lucas PalmaAI Engineer World's Fair 20262026
Sai Krishna RallabandiAI Engineer World's Fair 20262026
Peter GostevAI Engineer Europe 20262026
Aditya BhargavaAI Engineer World's Fair 20262026
What the Best Agents Share

Metadata candidate

Mardu SwanepoelAI Engineer Europe 20262026
Dmitry PetrovAI Engineer World's Fair 20262026
Nick HeinerAI Engineer World's Fair 20262026
Fryderyk Wiatrowski, Peter AlbertAI Engineer World's Fair 20242024
Phil HetzelAI Engineer Europe 20262026
Tom Shapland, PhDAI Engineer World's Fair 20252025
Sunil Pai, Matt CareyAI Engineer Europe 20262026
Philipp SchmidAI Engineer Europe 20262026
Manu GoyalAI Engineer World's Fair 20252025
Ahmad AwaisAI Engineer World's Fair 20252025
Prukalpa SankarAI Engineer World's Fair 20262026
Erik HanchettAI Engineer World's Fair 20262026
Rustin BanksAI Engineer World's Fair 20252025
Tun Shwe, Jeremy FrenayAI Engineer Europe 20262026
Jeremiah LowinAI Engineer Code 20252025
Yuxuan ZhangAI Engineer Code 20252025
Will BrykAI Engineer World's Fair 20262026
Anant SrivastavaAI Engineer Code 2025Unknown
Marina PetzelEvent unknownUnknown
Edo LibertyAI Engineer World's Fair 20262026
Hiral Shah, Sean SodhaEvent unknownUnknown
George HeEvent unknownUnknown
Dylan CouzonAI Engineer Code 2025Unknown
Gil FeigAI Engineer World's Fair 20262026
Pamela FoxAI Engineer World's Fair 20262026
Tejas KumarAI Engineer World's Fair 20262026
Yuval BelferAI Engineer World's Fair 20262026
Jeff Koss, Leo PlatzerAI Engineer World's Fair 20262026
Varun KrovvidiEvent unknownUnknown
Nina Žižakić, Eleonora DrykinaAI Engineer NYC 2026 Online Talks2026
Kamalakannan NandagopalEvent unknownUnknown
Peter WerryAI Engineer World's Fair 20262026
William LyonEvent unknownUnknown
Jerry LiuAI Engineer World's Fair 20262026
Vinoo GaneshEvent unknownUnknown
Andy Wong, Andrei Bocan, Shane WolfAI Engineer World's Fair 20262026
Matt LawlerAI Engineer World's Fair 20262026
Adit AbrahamAI Engineer World's Fair 20262026
Jacob Lauritzen, Simon EskildsenEvent unknownUnknown
Nyah MacklinAI Engineer World's Fair 20262026
Daksh GuptaEvent unknownUnknown
Sam Green, Aaron LinskyEvent unknownUnknown
Peter WerryAI Engineer World's Fair 20262026
Karthik Ranganathan, Heather DowningAI Engineer World's Fair 20262026
Maximilian-David RumpfAI Engineer World's Fair 20262026

References

Coverage and source review
Processed transcripts
51 processed in full · 5 in the curated path
Automated source review
Passed
Metadata candidates
594 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. LangChain Documentation: Retrieval

    https://docs.langchain.com/oss/python/deepagents/retrieval

  2. Optimizing LLMs for Speed and Memory

    https://huggingface.co/docs/transformers/main/en/llm_tutorial_optimization

  3. Optimizing and Evaluating Enterprise Retrieval-Augmented Generation (RAG): A Content Design Perspective

    https://arxiv.org/html/2410.12812

  4. Measuring Attribution in Natural Language Generation Models

    https://aclanthology.org/2023.cl-4.2.pdf

  5. Agent Evals: Finally, With The Map

    https://www.youtube.com/watch?v=y2Drx0SDZLo&t=244

  6. Evaluating AI Search: A Practical Framework for Augmented AI Systems

    https://www.youtube.com/watch?v=wRJD0inpmjU&t=842

  7. Building Production-Ready RAG Applications

    https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=319

  8. Specialized RAG Agents: Lessons learned from deploying complex AI systems in production

    https://www.youtube.com/watch?v=kPL-6-9MVyA&t=825

  9. Reading Wikipedia to Answer Open-Domain Questions

    https://aclanthology.org/P17-1171.pdf

  10. NIST TREC: Question Answering Collections

    https://trec.nist.gov/data/qa.html

  11. REALM: Retrieval-Augmented Language Model Pre-Training

    https://proceedings.mlr.press/v119/guu20a/guu20a.pdf

  12. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

    https://arxiv.org/abs/2005.11401

  13. REPLUG: Retrieval-Augmented Black-Box Language Models

    https://aclanthology.org/2024.naacl-long.463.pdf

  14. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks

    https://proceedings.nips.cc/paper/2020/file/6b493230205f780e1bc26945df7481e5-Paper.pdf

  15. Building Production-Ready RAG Applications

    https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=201

  16. Retrieval Augmented Generation in the Wild

    https://www.youtube.com/watch?v=FDEmbYPgG-s&t=541

  17. PROV-DM: The PROV Data Model

    https://www.w3.org/TR/prov-dm/

  18. Elasticsearch 8.19 Search API

    https://www.elastic.co/guide/en/elasticsearch/reference/8.19/search-search.html

  19. Document-level access control — Azure AI Search

    https://learn.microsoft.com/en-us/azure/search/search-document-level-access-overview

  20. Anthropic: Introducing Contextual Retrieval

    https://www.anthropic.com/engineering/contextual-retrieval

  21. Sufficient Context: A New Lens on Retrieval Augmented Generation Systems

    https://arxiv.org/html/2411.06037v3

  22. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering

    https://aclanthology.org/D18-1259.pdf

  23. Building performant RAG applications for production

    https://developers.llamaindex.ai/python/framework/optimizing/production_rag/

  24. FEVER Dataset

    https://fever.ai/dataset/fever.html

  25. OWL 2 Web Ontology Language Primer, Second Edition

    https://www.w3.org/TR/owl2-primer/

  26. Time in XTDB

    https://docs.xtdb.com/about/time-in-xtdb.html

  27. Stripe API Reference: Idempotent requests

    https://docs.stripe.com/api/idempotent_requests?lang=curl

  28. Stripe API v2 overview

    https://docs.stripe.com/api-v2-overview

  29. Build for the Memo, Not the Demo — Notes from 200 Investment Committees

    https://www.youtube.com/watch?v=tJFjeMBKbIY&t=585

  30. Mergeable by default: Building the context engine to save time and tokens

    https://www.youtube.com/watch?v=5ID22ACI7IM&t=1222

  31. Integrating Conflicting Data: The Role of Source Dependence

    https://lunadong.com/publication/dependence_vldb.pdf

  32. DRAGged into Conflicts: Detecting and Addressing Conflicting Sources in Search-Augmented LLMs

    https://arxiv.org/html/2506.08500v1

  33. GraphRAG: The Marriage of Knowledge Graphs and RAG

    https://www.youtube.com/watch?v=knDDGYHnnSI&t=306

  34. "Data readiness" is a Myth: Reliable AI with an Agentic Semantic Layer — Anushrut Gupta, PromptQL

    https://www.youtube.com/watch?v=1nOTQsfe1RU&t=256

  35. LangChain 0.0.349: ParentDocumentRetriever

    https://datastax.github.io/ragstack-ai/api_reference/0.3.1/langchain/retrievers/langchain.retrievers.parent_document_retriever.ParentDocumentRetriever.html

  36. Architecting and Testing Controllable Agents

    https://www.youtube.com/watch?v=ib-wTAvCZqg&t=4205

  37. RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation

    https://arxiv.org/pdf/2310.04408

  38. Lost in the Middle: How Language Models Use Long Contexts

    https://arxiv.org/html/2307.03172

  39. Trust, but Verify: High-Fidelity Reasoning in Agentic Workflows

    https://www.youtube.com/watch?v=MWTJIAwAAnk&t=876

  40. Build for the Memo, Not the Demo — Notes from 200 Investment Committees

    https://www.youtube.com/watch?v=tJFjeMBKbIY&t=899

  41. Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering

    https://aclanthology.org/2021.eacl-main.74.pdf

  42. Trust, but Verify: High-Fidelity Reasoning in Agentic Workflows

    https://www.youtube.com/watch?v=MWTJIAwAAnk&t=837

  43. Writing Principles for Task-Tuned Prompt Engineering

    https://www.youtube.com/watch?v=6d60zVdcCV4&t=2469

  44. AI Engineering 101

    https://www.youtube.com/watch?v=C0ZUdFg-iTo&t=4623

  45. Build for the Memo, Not the Demo — Notes from 200 Investment Committees

    https://www.youtube.com/watch?v=tJFjeMBKbIY&t=1087

  46. Claude Platform Docs: Citations

    https://platform.claude.com/docs/en/build-with-claude/citations

  47. Web Annotation Data Model

    https://www.w3.org/TR/annotation-model/

  48. Enabling Large Language Models to Generate Text with Citations

    https://arxiv.org/html/2305.14627

  49. Correctness is not Faithfulness in Retrieval Augmented Generation Attributions

    https://mariaheuss.com/assets/correctness.pdf

  50. Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inference

    https://aclanthology.org/P19-1334.pdf

  51. open-rag-eval: RAG Evaluation without "golden" answers.

    https://www.youtube.com/watch?v=1cQlnfwmIdU&t=104

  52. Trust, but Verify: High-Fidelity Reasoning in Agentic Workflows

    https://www.youtube.com/watch?v=MWTJIAwAAnk&t=551

  53. Unanswerability Evaluation for Retrieval Augmented Generation

    https://arxiv.org/html/2412.12300v1

  54. Architecting and Testing Controllable Agents

    https://www.youtube.com/watch?v=ib-wTAvCZqg&t=549

  55. Selective Question Answering under Domain Shift

    https://arxiv.org/pdf/2006.09462

  56. KILT: a Benchmark for Knowledge Intensive Language Tasks

    https://aclanthology.org/2021.naacl-main.200.pdf

  57. Know What You Don't Know: Unanswerable Questions for SQuAD

    https://nlp.stanford.edu/pubs/rajpurkar2018squad.pdf

  58. Architecting and Testing Controllable Agents

    https://www.youtube.com/watch?v=ib-wTAvCZqg&t=6189

  59. RAG Evaluation Is Broken! Here's Why (And How to Fix It)

    https://www.youtube.com/watch?v=Ywl4LsvHKzU&t=35

  60. Building Trust in Enterprise AI: Evaluating Domain-Specific LLMs for Real-World Financial Scenarios

    https://www.youtube.com/watch?v=pPvoLjYj_mY&t=275

  61. RAGChecker: claim-level retrieval and generation diagnosis

    https://arxiv.org/abs/2408.08067

  62. Ragas Faithfulness

    https://docs.ragas.io/en/stable/concepts/metrics/available_metrics/faithfulness/

  63. Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework

    https://arxiv.org/html/2411.09607v1

  64. Overview of the TREC 2003 Question Answering Track

    https://trec.nist.gov/pubs/trec12/papers/QA.OVERVIEW.pdf

  65. Selective Classification: Coverage and Conditional Risk

    https://arxiv.org/pdf/1705.08500

  66. Navigating RAG Optimization with an Evaluation-Driven Compass

    https://www.youtube.com/watch?v=DId2KP8Ykz4&t=202

  67. Evaluating AI Search: A Practical Framework for Augmented AI Systems

    https://www.youtube.com/watch?v=wRJD0inpmjU&t=956

  68. NIST randomized blocks: applying controlled comparisons to RAG evidence

    https://www.itl.nist.gov/div898/handbook/pri/section3/pri332.htm

  69. NIST experimental-design terminology: replication, interactions and confounding

    https://www.itl.nist.gov/div898/handbook/pri/section7/pri7.htm

  70. Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approach

    https://aclanthology.org/2024.emnlp-industry.66.pdf

  71. Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answering

    https://arxiv.org/html/2404.17723v2

  72. Navigating RAG Optimization with an Evaluation-Driven Compass

    https://www.youtube.com/watch?v=DId2KP8Ykz4&t=661

  73. Architecting and Testing Controllable Agents

    https://www.youtube.com/watch?v=ib-wTAvCZqg&t=6285

  74. Prompt Engineering is Dead

    https://www.youtube.com/watch?v=jvKf6zXrNO4&t=392

  75. CrabRAG: Why Automated Assistants Need Graph Memory, Not More Tokens

    https://www.youtube.com/watch?v=Q0VkgCyNVUg&t=502

  76. Building Production-Ready RAG Applications

    https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=603

  77. Navigating RAG Optimization with an Evaluation-Driven Compass

    https://www.youtube.com/watch?v=DId2KP8Ykz4&t=702

  78. Build for the Memo, Not the Demo — Notes from 200 Investment Committees

    https://www.youtube.com/watch?v=tJFjeMBKbIY&t=799

  79. open-rag-eval: RAG Evaluation without "golden" answers.

    https://www.youtube.com/watch?v=1cQlnfwmIdU&t=172

  80. open-rag-eval: RAG Evaluation without "golden" answers.

    https://www.youtube.com/watch?v=1cQlnfwmIdU&t=217

  81. Architecting and Testing Controllable Agents

    https://www.youtube.com/watch?v=ib-wTAvCZqg&t=5974