Contents
  1. The retrieval contract
    1. Finding useful information
    2. Need, Query, Corpus, and Result
  2. Building the searchable collection
    1. Acquisition Guarantees
    2. From Discovery to Searchable State
    3. Retrieval Turning Points
  3. Representing and retrieving candidates
    1. Choosing Searchable Units
    2. Terms and Posting Lists
      1. From matches to a candidate set
    3. Lexical Evidence and BM25
    4. Dense and Approximate Retrieval
    5. Web Signals Beyond Text
  4. Combining and refining rankings
    1. Hybrid Retrieval
    2. Reranking a Fixed Set
  5. Relevance evidence
    1. Judgments Define the Target
    2. Metrics and Denominators
    3. Offline and Live Evaluation
  6. Serving trustworthy results
    1. Authority on Every Path
    2. Freshness, Deletion, and Cutover
    3. Latency and Degraded Coverage
  7. Diagnosing the complete path
    1. One Trace, Separate Claims
  8. Check understanding
  9. Open questions
  10. Selected talks
  11. References
  12. Talk library
← All topics

Search & retrieval

Search turns a person’s information need into useful results from a larger collection. That requires more than choosing a similarity function: the right material must be collected, indexed, made eligible and ranked for the task. Exact identifiers, paraphrased questions and broad exploration call for different matching signals. This chapter connects lexical, dense and hybrid retrieval to the practical work of judging relevance, keeping results current and serving them reliably.

Part I — The retrieval contract

Finding useful information

A search system selects and orders useful information from a larger collection in response to a query. The underlying discipline is usually called information retrieval. Search may help someone reach a known page, investigate a subject, discover unfamiliar items, filter a catalog, supply recommendation candidates, or locate evidence. None of those purposes requires a generated answer. If a model later turns retrieved passages into an answer, that separate evidence-to-answer problem belongs to Retrieval-Augmented Generation.

Private and public-web search begin with different collection problems.
ConditionPrivate collectionPublic web
MembershipChosen by an organization, connector, or product boundaryDistributed across independently operated sites
AuthorityOften available from source identities and access-control listsPublic reachability is not permission for every use
DiscoveryDatabases, exports, connectors, or change feedsLinks, sitemaps, submitted URLs, and recrawling
ChangeSource-specific updates and revocationsPages appear, move, duplicate, redirect, and disappear
CoverageBounded only if the source interface supports defensible enumerationNo central registry makes complete discovery impossible to assume

Need, Query, Corpus, and Result

An information need is the task to be satisfied; a query is the submitted expression of that need. The corpus is the collection against which search runs, and the retrieval unit is what the system ranks—perhaps a document, passage, product, page, or database record. Relevance is therefore conditional: an item is useful for a particular need, corpus, unit, user context, and time. The same query can require one known destination in a navigational task but exhaustive coverage in an audit or counting task.

A top-k response contains the first kk items under a ranking rule. Ranking compares results with one another; even the highest-ranked result may not satisfy the information need. Nor does a shortlist establish completeness: returning five matching documents is not proof that only five matching entities exist. Raw search scores usually order items within one configured query. They need not be probabilities or remain comparable after the corpus, query, or scoring configuration changes.

A practical result record makes the search contract inspectable.
FieldWhat it establishes
sourceId and unitIdStable source identity and the particular ranked representation
rank and scoreTypeOrdering and the meaning of the score—not calibrated confidence unless separately established
sourceRevision and retrievedAtWhich source state the result refers to and when it was observed
eligibility constraintsFilters and authority applied to this request
coverage statusWhether execution completed, degraded, timed out, or omitted sources

Part II — Building the searchable collection

Acquisition Guarantees

Search can query an external provider or build its own collection. A local index copies or derives records from databases, exports, connectors, or change feeds. Federated search sends live queries to independently operated collections and merges their responses. A search API returns a bounded provider-ranked list through provider-specific semantics. These modes expose different knowledge about enumeration, source versions, failures, and authorization.

The acquisition boundary determines which claims can be justified.
ModeSystem can usually observeRemaining uncertainty
Local collectionRecord identities, processing state, index generation, and copied metadataWhether source enumeration, permission synchronization, and deletion propagation are complete
Federated queryWhich sources were contacted and which responses arrivedCross-source score comparability, partial failure, and each source's internal coverage
Opaque search APISubmitted parameters and returned ranked resultsProvider corpus coverage, ranking internals, hard caps, and unreturned matches

Pagination completion means the documented page sequence ended; it is not automatically proof that the source exposed every eligible record. A successful batch envelope may hide per-item throttling failures. When a provider returns HTTP 429, retry behavior and durable progress determine whether synchronization is incomplete or merely delayed. Every accepted record should enter the collection with source identity, source time, acquisition time, provenance, and eligibility metadata. The broader rules are developed in Origins, times and permitted use and Establish permitted uses.

From Discovery to Searchable State

A crawler is an automated client that schedules locations to fetch. Its crawl frontier is the pending work. Starting from seed URLs, it discovers links or sitemap entries, obeys scope and politeness rules, resolves redirects, normalizes addresses, detects repeated content, and schedules revisits. Different URLs can return the same content, while generated addresses and session identifiers can create effectively unbounded crawler traps. The Robots Exclusion Protocol communicates requested crawling rules; it is not access authorization.

Fetching and publication remain separate. A conditional request can revalidate a representation without downloading its body, but it does not discover new URLs or update an index by itself. A fetch may redirect, remain unchanged, fail, parse incorrectly, violate acceptance rules, or produce a version awaiting index refresh. Private polling and change feeds replace link discovery with source-specific enumeration, yet retain the same distinction among acquisition, processing, publication, and deletion.

Retrieval Turning Points

Search developed along several complementary lines: representing documents, comparing systems, and deciding which results deserve attention. These turning points explain why modern search combines methods rather than replacing every earlier technique.

Selected developments and the problems they addressed.
DevelopmentContribution
Statistical text searching — 1957Hans Peter Luhn proposed computing searchable representations from document language to help organize growing technical literature.
Cranfield comparative evaluation — 1966Cleverdon and Keen compared indexing approaches by retrieval performance, testing whether richer representations actually helped.
Probabilistic relevance weighting — 1976Stephen Robertson and Karen Spärck Jones used collection statistics and relevance information to weight discriminating query terms.
Okapi BM25 — 1994 experimentsThe Okapi team combined term-frequency and document-length weighting in experiments at the Text REtrieval Conference (TREC).
Google's link and anchor evidence — 1998Brin and Page supplemented page text with incoming-link importance and the words other pages used to link to a destination.
RankNet — 2005Learned ordering from query-document features and relevance judgments; later LambdaRank and LambdaMART addressed the importance of mistakes near the top.
Reciprocal Rank Fusion — 2009Cormack, Clarke, and Büttcher combined result positions from multiple rankings without training a fusion model.
BERT passage reranking — first submitted 2019Nogueira and Cho applied a pretrained language model to query-passage pairs on a BM25 shortlist, strengthening a later ranking stage.
Dense Passage Retrieval — 2020Separate learned question and passage encoders enabled candidate retrieval with reusable, precomputed passage vectors.

Part III — Representing and retrieving candidates

Choosing Searchable Units

The retrieval unit controls what can match and what evaluation counts. Whole documents preserve context but may mix several subjects. Chunking divides a document into smaller, independently searchable units. Those units can follow fixed sizes, sentences, sections, or combinations. Smaller units allow finer matches, but a statement may lose its heading, table header, or qualifying exception. Repeating headings or table headers preserves their meaning locally; overlapping adjacent chunks repeats boundary text to preserve continuity. The useful boundary depends on the document's structure and the questions search must answer.

Each chunk needs a key distinguishing it from other chunks and a parent identifier linking it to its source document. Retain the source revision and inherited metadata as well. A selector locates the chunk within that revision—for example, a character range measured against the extracted text. Without the revision, later edits can make the same range point to different content. Generated chunk keys may change when a parent changes, so they cannot replace stable source identity. Parent-child links also let deletion processing find the derived units to remove. Precise source coordinates are developed in Locate supporting source material.

In this fictional example, the two chunks contain different rows from one document revision, with repeated headings and column headers for context. Each structural row selector applies to revision r12 and excludes that repeated context; unit identifiers distinguish the chunks from their parent document.

Counting must use the intended unit. Twenty retrieved passages may belong to three documents; grouping by source can display one source with several passages, but grouping is not semantic deduplication. A question about the number of entities requires an enumerative or structured operation with population coverage, not a count of returned passages.

Terms and Posting Lists

An inverted index reverses the document-to-words relationship: each searchable term points to a posting list identifying the documents that contain it. Before building the index, an analyzer applies text-processing rules such as splitting text into tokens and lowercasing them. Index construction assigns document identifiers, sorts term-document pairs, and groups each term's entries into postings. A query can then look up document identities without scanning all source text. Query analysis must produce terms compatible with the indexed representation. Using identical analyzers is common, but compatibility means producing the intended matches, not merely choosing the same analyzer name.

A positional example shows why co-occurrence is weaker than a phrase match.
DocumentAnalyzed tokensPositions for “vector” and “search”Phrase “vector search”?
D1hybrid vector searchvector:2, search:3Yes: positions are adjacent
D2search combines vector methodssearch:1, vector:3No: order and distance differ
D3vector database searchvector:1, search:3No: another token intervenes

Positions belong to the indexed token stream; offsets locate matches in retained source text for highlighting. Term presence only produces candidates. It does not establish relevance, freshness, permission, or support for a later claim.

From matches to a candidate set

Posting-list matches can still be too numerous for expensive ranking. The eligible set contains the indexed units allowed by the request's mandatory filters and current authority. First-stage candidate generation selects a manageable subset of those units using a cheaper scoring rule. Its candidate depth is the maximum number passed onward, not the number ultimately displayed. To maintain a top-k shortlist, the engine keeps the best k scored items seen so far. Once that shortlist is full, its lowest score becomes the threshold a new item must beat to displace an existing item, subject to the declared tie rule.

Finding that shortlist need not require fully scoring every match. WAND uses upper bounds on query-term score contributions to skip documents that cannot beat the current threshold. Block-Max WAND tightens those bounds for blocks of postings. If a block's maximum possible score is strictly below the threshold, skipping it cannot change the top-k results. This is safe pruning: it reduces work while preserving the exact result under the supported scoring rule. Invalid bounds or more aggressive approximate pruning can instead omit competitive documents.

A valid bound can rule out competitive documents

A full two-item shortlist scores 9 and 7. Its current competitive threshold is 7.

Illustrative valid bounds; bars encode maximum possible document scores, not measured scores. The dashed line marks 7 on a common 0–10 scale.

Posting blockUpper boundDecision
A6Safe to skip: 6 < 7
B7Consider further: ties depend on the tie rule
C10Consider further: may beat 7

One bound comparison within Block-Max WAND, not the complete pivot-selection algorithm.

Only a strictly lower valid bound is safe to skip when the shortlist is full. This preserves first-stage top-k, not relevance under a later reranker.

Exactness here concerns the first-stage score, not relevance under every later model. Exhaustive scoring and safe pruning can produce the same shortlist; approximate execution can miss even that shortlist's best-scoring items. Either way, a relevant unit outside the admitted candidate set is unavailable to fusion or reranking. Increasing candidate depth can expose more items to those stages at additional cost, but cannot guarantee that the missing relevant unit appears. The next sections explain the lexical and vector scoring rules used to construct these shortlists.

Lexical Evidence and BM25

Lexical ranking goes beyond Boolean matching by weighting the evidence supplied by query terms. BM25 combines inverse document frequency, saturating term frequency, and document-length normalization. A rare query term can discriminate more strongly than a common one. Repetition adds evidence, but with diminishing returns. A match in a short field can be more concentrated than the same match in a long field. Parameters control saturation and the strength of length normalization and should be tuned on development judgments, not the final test collection.

Repetition saturates; length changes concentration

One query term; fixed IDF = 1 and average document length = 100 tokens. Document lengths stay at 50, 100, and 200 tokens.

Contribution = tf × (k₁ + 1) / [tf + k₁ × (1 − b + b × length / 100)]. Zero occurrences contribute zero.

BM25 single-term contribution by term frequency. Solid: 50 tokens; dashed: 100; dotted: 200. Numeric values at the selected frequency follow.0123405101520Single-term contribution (score units)Term occurrences
Solid: short (50)Dashed: average (100)Dotted: long (200)
Document lengthOccurrencesContribution
50 tokens31.7600
100 tokens31.5714
200 tokens31.2941

For the same frequency, length normalization changes how concentrated the match is. Increasing repetition has diminishing returns.

The plot isolates one term with fixed corpus statistics. Term frequency stays within every document’s token count. Scores are ranking contributions, not probabilities; no parameter setting is universally best.

Text relevance and eligibility are different operations. A Boolean filter excludes a document without contributing a score; a title boost, inventory signal, or popularity feature changes ordering among eligible results. An item cannot compensate for failing a required tenant, date, status, or authorization filter by scoring highly elsewhere.

Lexical retrieval remains especially useful for exact identifiers, product codes, names, quoted phrases, and vocabulary already shared by queries and documents. Analysis can also destroy useful signals: indiscriminate stop-word removal may erase a quotation or negation, and the wrong language analyzer applies inappropriate stemming and frequency statistics. Raw scores should remain ranking values rather than percentages.

Dense and Approximate Retrieval

Lexical search depends on matching indexed terms. Dense retrieval can bridge differences in wording by comparing learned representations instead. An embedding is a vector—an array of numbers—produced by an encoder for a query or corpus unit. Corpus embeddings are computed and indexed ahead of time; a request encodes its query and compares it with compatible stored vectors. Candidate retrieval, not complete search develops this representation. Query and document preprocessing must follow the encoder's conventions, and nearest still means best under the chosen representation, metric, collection, and selection policy—not necessarily useful for the task.

Exhaustive nearest-neighbor search compares the query with every eligible vector. Approximate nearest-neighbor (ANN) search avoids some comparisons but can miss vectors that exhaustive search would return. Keep two questions separate: how well the index recovers the exact nearest vectors, and how useful those vectors' source items are to a person. Exact search provides a baseline for the first question, not the second.

Hierarchical Navigable Small World (HNSW) organizes vectors in layers of proximity graphs. Search moves through sparse upper layers, then explores more candidates near the bottom. Increasing exploration spends more work on neighbor recovery. IVFFlat, an inverted-file index, instead groups vectors into lists and searches selected nearby lists; increasing the number of lists probed broadens the search at additional cost.

Compression addresses a related resource problem. Product quantization divides a vector into smaller subvectors and represents each with a short code. Search can estimate distances from those codes, reducing storage while potentially changing neighbor order. Index traversal and compression therefore introduce distinct ways to lose exact neighbors; benchmark their combined settings rather than comparing speed alone.

Filters interact with approximation too. In Azure AI Search's HNSW implementation, a prefilter constrains eligibility during traversal and may require additional exploration when few candidates qualify. A postfilter removes ineligible items from an unfiltered shortlist, so the response can be short or empty even when eligible neighbors exist outside it. Speed comparisons must disclose the distance function, index settings, filter placement, requested neighbors, and recovery against exact search.

A postfilter cannot recover truncated neighbors

Eight fixed vectors; query Q = (0, 0), Euclidean distance, requested k = 3. This example uses exact distances to isolate filtering loss.

Fixed vector positions. Circles: EU; squares: US. A gold outer ring marks membership in the unfiltered window. Filtered-out points are faded.00224466xyQ (0, 0)ABCDEFGH

Exact eligible top-3C, E, F

Unfiltered input windowA, B, C

Window → filter → top-3C

Circles: EU · squares: US. Gold rings mark window membership; faded points fail the filter. Ties would use document ID.

1/3 result slots filled. Exact eligible neighbors omitted by the window: E, F.

Inspect all coordinates and distances
ID(x, y)RegionDistanceEligibleIn window
A(1, 0)US1.000noyes
B(0, 2)US2.000noyes
C(2, 2)EU2.828yesyes
D(3, 0)US3.000nono
E(0, 4)EU4.000yesno
F(3, 4)EU5.000yesno
G(6, 0)US6.000nono
H(0, 7)EU7.000yesno
A deeper input window can recover eligible neighbors that postfiltering missed, even with exact distance ordering. This attribute filter is not a user-controlled authorization mechanism, and exact-vector recovery is not human relevance.

Web Signals Beyond Text

Open-web ranking adds evidence unavailable in an isolated document collection. In Google's 1998 system paper, PageRank weighted incoming links using properties of their source pages, while anchor text associated other pages' descriptions with the destination. A resource could therefore receive textual evidence even when its own page text was unavailable. These signals complemented lexical retrieval rather than replacing it.

Link-derived importance is a prominence signal, not proof of topical relevance, authority, trustworthiness, or freshness. Web search must also reconcile duplicates and revisions, choose canonical identities, account for language or location, resist adversarial manipulation, and avoid lists that repeat the same information. Link manipulation exploits ranking signals by creating bogus pages that point to a target. In TrustRank, Gyöngyi, Garcia-Molina, and Pedersen's 2004 countermeasure, experts identify reputable seed sites and a biased PageRank computation propagates dampened trust through outgoing links. That score can filter an index or supplement other ranking signals. It assumes reputable pages seldom link to spam, so it is not a guarantee of trustworthiness or a description of current search engines. Maximum Marginal Relevance addresses a different list-level problem: select items by balancing query relevance with similarity to items already selected. Diversification still cannot recover an absent candidate.

Part IV — Combining and refining rankings

Hybrid Retrieval

Lexical and dense retrievers fail differently. Exact product names or identifiers favor lexical evidence; paraphrases and behaviorally related files with different vocabulary may favor dense representations. Hybrid retrieval runs complementary candidate generators and combines their identities. The combined system needs its own evaluation because an extra branch may add useful candidates, duplicates, irrelevant neighbors, latency, or all four.

Reciprocal Rank Fusion (RRF) combines rankings by giving a document more credit when it appears near the top of an input list. A document appearing in several lists accumulates their contributions. For document dd and input lists LL, RRF(d)=∑ℓ∈L1c+rank⁡ℓ(d). \mathrm{RRF}(d)=\sum_{\ell\in L}\frac{1}{c+\operatorname{rank}_{\ell}(d)}. Here rank⁡ℓ(d)\operatorname{rank}_{\ell}(d) is the document's one-based position in list ℓ\ell. A missing document contributes nothing from that list. The constant cc controls how sharply contributions differ between ranks. The original 2009 experiments used c=60c=60 after pilot work; that choice is not a universal optimum.

RRF combines positions rather than pretending BM25 and vector-score magnitudes share a calibrated scale. Input windows and final output size remain distinct: increasing the displayed result count cannot restore candidates truncated before fusion. Weighted score fusion is another option, but its normalization range and weights become part of the retrieval contract.

Two ranked windows become one identity-based union

Lexical window (depth 3)1 A → 2 B → 3 C

Dense window (depth 3)1 B → 2 D → 3 A

Fixed example rankings; c = 60. Each present rank contributes 1/(60 + rank). Missing ranks contribute zero.

Fused rank / IDLexical contributionDense contributionSumOutput (size 2)
1 / Brank 2: 1/62rank 1: 1/610.032522returned
2 / Arank 1: 1/61rank 3: 1/630.032266returned
3 / Dabsent: 0rank 2: 1/620.016129outside final 2
4 / Crank 3: 1/63absent: 00.015873outside final 2
B accumulates the strongest combined rank support. Input depth three, union size four, and output size two describe different boundaries; increasing the output size cannot admit E.

Reranking a Fixed Set

A reranker spends more computation on a bounded candidate set. A dual encoder represents query and document separately, enabling corpus vectors to be reused. A cross-encoder processes a query-candidate pair jointly, exposing interactions that independent vector comparison cannot. Because joint scoring is expensive across a large corpus, the usual architecture retrieves broadly and reranks a smaller shortlist.

Document vectors can be prepared per corpus version and compared with a query vector. A cross-encoder instead jointly processes the query and each already-retrieved candidate passage. These are computation patterns, not measured quality or latency gains.

The candidate set is a hard ceiling. The 2019 BERT passage-reranking work improved ordering on a BM25 shortlist but also identified questions whose relevant passage never reached that shortlist. No second-stage model can score an item it never receives. Candidate depth therefore trades opportunity against reranking cost, while input truncation can hide the very passage detail the reranker needs.

A learned relevance score is still not necessarily a calibrated probability or a complete product objective. Feature-based rankers may combine text, structure, popularity, and application signals. Diversity, deduplication, freshness, fairness, and business constraints are list-level or policy decisions that should remain distinguishable from query-item relevance.

Part V — Relevance evidence

Judgments Define the Target

A relevance judgment records an assessor's decision for a specified query, corpus unit, task, time, and rubric. Binary labels distinguish relevant from nonrelevant; graded labels distinguish degrees of usefulness. Passage and document tasks may assign different meanings to the same numeric grade. The resulting query-relevance files, commonly called qrels, are recorded measurements rather than universal truth.

Large collections are rarely judged exhaustively. Pooling combines leading results from participating systems and sends that subset for assessment. This makes evaluation practical but leaves unjudged documents and can affect systems that retrieve candidates unlike those that formed the pool. A deeper sampled assessment in the TREC 2006 Terabyte Track ranked systems somewhat differently from a shallow pool.

Disagreement may expose ambiguous intent, insufficient context, or a defective rubric. Preserve topical relevance, utility, authority, freshness, and satisfaction as separate concepts when the product decision needs them. Label design and disagreement handling are developed in Specify the label and Disagreement is diagnostic.

Metrics and Denominators

Precision asks how much of the returned material is relevant. Recall asks how much of the relevant material was returned. For retrieved set RR and judged-relevant set GG, precision=∣R∩G∣∣R∣,recall=∣R∩G∣∣G∣. \mathrm{precision}=\frac{|R\cap G|}{|R|},\qquad \mathrm{recall}=\frac{|R\cap G|}{|G|}. The numerator is the same; the denominators answer different questions. Recall includes missed relevant items, so inspecting returned items alone cannot establish it. With incomplete judgments, the measured recall concerns the known relevant set, which may omit other relevant material.

Rank-sensitive measures also account for where useful results appear.

  • Precision@k and Recall@k — Apply the set measures to the first k results. With k results returned, precision divides relevant hits by k; recall divides them by the size of the relevant set.
  • Reciprocal rank and MRR — Reciprocal rank is 1/r, where r is the first relevant result's rank, or zero if none appears within the cutoff. Mean reciprocal rank (MRR) averages this value across queries. Later relevant hits do not change it.
  • Average precision — Rewards relevant results appearing consistently early, rather than considering only the first hit. Unlike precision at one cutoff, it assesses precision at successive relevant hits.
  • Normalized discounted cumulative gain (nDCG) — Assigns gain to each relevance grade, discounts gain at later ranks, sums it, and divides by the gain of an ideal ordering at the same cutoff. One common convention uses gain 2^g−1 for grade g and discount log2(i+1) at rank i. State the convention: linear gains and different ideal denominators measure something different.

For example, suppose a query has two relevant documents, found at ranks 2 and 4. At cutoff 3, precision is 1/3, recall is 1/2, and reciprocal rank is 1/2. At cutoff 4, precision rises to 2/4 and recall to 2/2, while reciprocal rank stays 1/2: the first relevant result has not moved. The measures disagree because they describe different properties of the same list.

The same judgments, different ranks and cutoffs

Four fixed documents. The known relevant set is always {B, D}; A and C are judged nonrelevant.

  1. 1. A · nonrelevant
  2. 2. B · relevant
  3. 3. C · nonrelevant
  4. 4. D · relevant · outside cutoff
Precision@31/3 = 0.333Relevant returned / returned
Recall@31/2 = 0.500Relevant returned / known relevant
Reciprocal rank@31/2 = 0.500Only the first included relevant rank matters
Binary nDCG@30.6309/1.6309 = 0.387Observed discounted gain / ideal gain
Inspect discounted gains
Rank / IDBinary gainDiscountIncluded gain
1 / A0log₂(2) = 1.00000/1.0000 = 0.0000
2 / B1log₂(3) = 1.58501/1.5850 = 0.6309
3 / C0log₂(4) = 2.00000/2.0000 = 0.0000
4 / D1log₂(5) = 2.32190: outside cutoff

Gain = 2ᵍ − 1 for binary grade g. Ideal gain places the two known relevant documents first, taking only the first k positions: 1 + 1/log₂(3) = 1.6309. Relevant documents outside the observed cutoff still belong in the ideal ordering.

Judgments and document identities stay fixed while rank and cutoff change. Recall refers to the known relevant set; nDCG uses its ideal ordering at the same cutoff.

Define how per-query results become an aggregate. Averaging query scores gives each included query equal weight; pooling retrieved and relevant counts can give larger result sets more influence. A zero denominator requires an explicit reporting convention. Reports must also name the retrieval unit, relevance scale, cutoff, corpus and judgment versions, tie handling, and treatment of unjudged items, timeouts, and unresolved runs. An unjudged item may count as nonrelevant in a scoring implementation without having been assessed as nonrelevant. General reporting discipline is developed in Keep denominators visible.

Keep operational measures separate from relevance measures. Execution coverage describes how much of the requested search completed. Exact-vector recall describes recovery of nearest vectors under a metric. Neither establishes how much useful information the user received.

Offline and Live Evaluation

An offline test collection binds a corpus snapshot, information needs, judgments, retrieval configurations, and per-query runs. Candidate recall and final ranking should be measured separately: a reranker should not be blamed for a passage absent from its input. Representative queries estimate behavior for the intended workload; challenge slices deliberately concentrate difficult or consequential conditions. Keep paired per-query changes visible rather than relying only on an aggregate.

Production feedback adds information but changes the observation process. Clicks depend on exposure and rank position; no click may mean dissatisfaction or successful completion directly from a result page. In Team-Draft interleaving, two rankers contribute to one displayed list and clicks are credited to contributors, supporting a relative comparison under shared exposure. It does not turn clicks into universal relevance labels.

Offline relevance, user satisfaction, and business outcomes can disagree. Pinterest, for example, reported human-assessed relevance separately from search fulfillment actions. A ranking gain should not automatically be translated into reduced user effort. Sampling and matched-comparison methods are covered in Sample the intended work and Compare changes on matched work; incomplete production feedback and live experiments belong to Account for incomplete feedback and Choose the live experiment.

Part VI — Serving trustworthy results

Authority on Every Path

Authentication establishes the requesting actor; authorization decides which operations that actor may perform on resources. Search must apply current authority to candidate generation or safely compensate for filtering losses, then preserve it across reranking, snippets, facets, counts, caches, logs, and alternate lookups. Indexed principal identifiers are data used by an enforcement mechanism, not authentication or authorization by themselves.

An access-control list (ACL) records which users or groups may perform operations on a resource. Copying that information into an index allows retrieval to filter results, but the copy can become stale. Every independently searchable chunk needs the applicable permissions. Synchronization must handle changes inherited from parent resources as well as changes on the document itself; connector support for those paths varies. Content and permission updates must also stay consistent: checking new content against an older ACL can disclose it to a user whose access was revoked.

Visible hits are not the only disclosure surface. Global relevance statistics, terms, field names, aggregates, counts, timing, and shared caches can reveal information about inaccessible documents. Every query path should fail closed under missing identity or permission state. The general rule is Enforce current authority; model instructions are not an enforcement point.

Hits are one disclosure surface. Current authority must also govern counts, direct lookups and cache reuse; diagnostics have a separate operator policy. This reference design leaves global-statistics and timing risks visible rather than claiming that filtering eliminates them.

Freshness, Deletion, and Cutover

Search freshness spans several clocks: source event, discovery, acquisition, parsing, index write, refresh, replica visibility, cache invalidation, and query time. A successful indexing request need not make a change immediately searchable. Waiting for refresh can establish visibility at that boundary; forcing refresh trades visibility latency against indexing and merge costs.

Out-of-order events can overwrite newer content or recreate a deleted result. External versions can reject stale writes. A tombstone retains a newer deletion state so delayed older work cannot treat absence as permission to recreate the item. Parent-to-child identities are required to remove derived chunks, and deletion markers must remain observable long enough for failed or delayed consumers.

Remember the deletion version before delayed work arrives

Source emission order
  1. Publish document v41
  2. Delete document v42
Consumer arrival order
  1. Delete document v42
  2. Delayed publish v41
Consumer stepRetained stateConsequence
Accept delete v42v42 tombstoneRetain deletion version; initiate parent and derived-child removal.
Compare delayed write v41v42 tombstone41 < 42: reject the stale write before it can recreate the record.
Verify serving stateVersion remains knownCheck child removal, search visibility, replicas, and caches separately.

Event positions express ordering, not elapsed time. Without retained version state, absence alone cannot distinguish a new item from this stale write.

The v42 tombstone rejects delayed v41 work. This is a reference mechanism: the cited GOV.UK decision described the recreation risk while postponing tombstones. Accepting deletion does not prove that serving copies or caches are already clear.

A stable alias can switch queries from an old index generation to a rebuilt one, but a multi-action update can partially succeed unless failure semantics are configured appropriately. The cutover also does not coordinate query encoders, rerankers, or application caches automatically. Verify source state, active index generation, derived-record removal, replica visibility, cache behavior, and restoration paths independently. Maintenance and deletion obligations are developed in Maintain fitness through change and Propagate correction and deletion.

Latency and Degraded Coverage

A distributed search can send one request to several sources or shards, partitions of an index served separately. This fan-out lets branches retrieve candidates in parallel before their results are merged. The full request also includes query parsing or rewriting, permission checks, reranking, metadata or snippet retrieval, and response serialization. Elapsed time follows the critical path, the chain of operations that must finish before the response can complete. Parallel durations overlap, so adding them overstates elapsed time; merging waits for the branches the request requires.

Tail latency describes the slower end of the request-time distribution: p95, for example, is the time at or below which 95% of requests finish. Fan-out makes occasional component delays consequential because a request may wait for its slowest required branch. Propagate the remaining deadline downstream, and stop spawned work when cancellation arrives rather than merely ending the client's wait. Capacity tests must combine candidate depth, ANN exploration, restrictive filters, rerank depth, memory use, update traffic, and concurrent queries. A setting that is fast on an idle, read-only index may behave differently under the application's actual load.

A system may intentionally return partial results near a deadline, but the response must expose coverage and the reason for degradation. Execution coverage distinguishes timeouts, missing nodes, and configured matching limits; it is not relevance recall. A ranked list without execution status cannot safely be interpreted as a completed search.

Parallel retrieval and the request critical path

Example timings

A single slow required branch extends elapsed request time, while overlapping branch durations cannot be summed.

Search request0–190 msDuration 190 ms
Plan identity, filters, deadline0–15 msDuration 15 msWithin Search request
Source A · required15–65 msDuration 50 msWithin Search request
Source B · required15–115 msDuration 100 msWithin Search request
Source C · optional15–80 msDuration 65 msWithin Search request
Merge candidates115–130 msDuration 15 msWithin Search request
Final authority recheck130–142 msDuration 12 msWithin Search request
Rerank shortlist142–180 msDuration 38 msWithin Search request
Serialize complete response180–190 msDuration 10 msWithin Search request
This illustrative request takes 190 ms: 15 planning + 100 for the longest required branch B + 15 merge + 12 final authority recheck + 38 reranking + 10 serialization. A and B are required; C is optional and also completes here, so all three requested sources are covered. Omitting C could preserve required-branch completion while reducing requested-source coverage. The final recheck supplements authority enforced in source branches.
Read the diagram as text
  • Search request. Parent operation from receipt through response. 0 to 190 ms; duration 190 ms.
  • Plan identity, filters, deadline. Establish authenticated identity, current authority context, source filters, and the shared deadline. 0 to 15 ms; duration 15 ms. Parent: Search request.
  • Source A · required. Required branch; enforce applicable authority during source retrieval. 15 to 65 ms; duration 50 ms. Parent: Search request.
  • Source B · required. Longest required branch; enforce applicable authority during source retrieval. 15 to 115 ms; duration 100 ms. Parent: Search request.
  • Source C · optional. Optional branch; enforce applicable authority. Completes in this depicted run. 15 to 80 ms; duration 65 ms. Parent: Search request.
  • Merge candidates. Begins after required retrieval branches complete. 115 to 130 ms; duration 15 ms. Parent: Search request.
  • Final authority recheck. Recheck current authority after merge; source branches have already enforced applicable authority. 130 to 142 ms; duration 12 ms. Parent: Search request.
  • Rerank shortlist. More expensive scoring on bounded eligible candidates. 142 to 180 ms; duration 38 ms. Parent: Search request.
  • Serialize complete response. All three requested source branches finished in this example; report complete execution coverage. 180 to 190 ms; duration 10 ms. Parent: Search request.

Part VII — Diagnosing the complete path

One Trace, Separate Claims

A retrieval trace should preserve the source and corpus snapshot, index generation, analyzer or encoder contract, query and filters, authenticated authority, first-stage candidates, fusion inputs, reranker version, stage timings, coverage status, and delivered identities. A distributed trace connects these operations through spans and events, but an error-free span establishes only that the instrumented operation completed—not that the requested search outcome was correct.

Diagnose the first boundary whose contract failed.
BoundaryClaim it can supportTypical counterevidence
AcquisitionThe source record was observed under a stated snapshotSource absent, throttled, or only partially enumerated
Index publicationThe intended representation became searchableStale generation, parse rejection, or refresh pending
EligibilityCurrent authority admitted the recordStale ACL, wrong tenant, or unsafe postfilter
Candidate generationThe relevant unit reached the shortlistWindow truncation, ANN miss, or filter starvation
RankingAvailable candidates were ordered for the stated relevance targetWrong feature, rubric, fusion, or reranker
DeliveryThe client received a complete or explicitly degraded resultTimeout, truncation, cache error, or serialization failure

Repair the first invalid boundary. A fluent downstream answer cannot establish upstream corpus coverage, current authority, candidate recall, or freshness. Conversely, a bad answer does not prove retrieval failed. Record retrieval's meaning-changing boundaries as described in Record meaning-changing boundaries, then evaluate any later evidence-to-answer synthesis separately.

Open questions

  1. How can permission-aware approximate retrieval preserve both strict authorization and high candidate recall when highly selective filters fragment an ANN index? Progress would include explicit freshness semantics for authority, adversarial side-channel tests, and workload-matched recall and latency results.

  2. How should a search service communicate corpus coverage when it combines local indexes, federated sources, and opaque web APIs? A useful solution would distinguish complete enumeration, sampled coverage, provider-bounded rankings, source failures, and unknown coverage without reducing them to one percentage.

  3. How can relevance judgments remain reusable as corpora, source revisions, and user intents change? Progress would require versioned judgment units, explicit temporal validity, principled handling of unjudged new material, and affordable reassessment strategies.

  4. Which hybrid and reranking policies adapt candidate depth to query difficulty without hiding latency or coverage degradation? Progress would show paired improvements across exact identifiers, paraphrases, restrictive filters, and out-of-domain queries under a fixed resource budget.

  5. How should deletion completion be proven across source connectors, derived chunks, replicas, caches, logs, and restoration paths? A credible answer needs stable identity, durable deletion state, stale-event rejection, and serving-boundary verification.

Follow the curated reading path through the speakers and demonstrations behind this entry.

Explore more talks

The rest of the library, beyond the curated path. Cited talks support this entry; reviewed transcripts were processed in full. Metadata candidates have not been reviewed as sources or verified as topic members.

66 matching talks

TalkSpeakerEventYear
Anton TroynikovAI Engineer Summit 20232023
Brandon WaselnukAI Engineer Europe 20262026
Shawn ChanAI Engineer World's Fair 20262026
Sohail Shaikh, Ankush RastogiAI Engineer World's Fair 20262026
Yuval Belfer, Niv GranotAI Engineer World's Fair 20252025
Lance MartinAI Engineer World's Fair 20242024
Stephen Chin, Jonathan LoweAI Engineer Summit 20252025
Frank LiuAI Engineer World's Fair 20252025
Jonathan FernandesAI Engineer World's Fair 20252025
Pablo CastroAI Engineer World's Fair 20242024
Paul CopplestoneAI Engineer Summit 20232023
Abed MatiniAI Engineer World's Fair 20262026
Mukuntha Narayanan, Han WangAI Engineer World's Fair 20252025
Kyle KranenAI Engineer World's Fair 20252025
Vivek MuppallaAI Engineer World's Fair 20262026
Andreas KolleggerAI Engineer World's Fair 20252025
Tengyu MaAI Engineer World's Fair 20252025
Julia Neagu, Deanna Emery, Maitar AsherAI Engineer World's Fair 20252025
Kevin HouAI Engineer World's Fair 20242024
Will BrykAI Engineer World's Fair 20252025
Jerry LiuAI Engineer World's Fair 20252025
Patricija ŽemaitytėAI Engineer World's Fair 20262026
Suman DebnathAI Engineer World's Fair 20252025
Zach BlumenfeldAI Engineer World's Fair 20252025
Nico AlbaneseAI Engineer Summit 20252025
Pamela Fox, Harald Kirschner, Gabriela de QueirozAI Engineer World's Fair 20242024
Emil EifremAI Engineer World's Fair 20242024
Rafael LeviAI Engineer Europe 20262026
Tim AingeAI Engineer World's Fair 20262026
William LyonAI Engineer World's Fair 20252025
Eugene YanAI Engineer World's Fair 20262026
Jerry LiuAI Engineer Summit 20232023
Phoebe KlettAI Engineer World's Fair 20242024
Mani KhanujaAI Engineer World's Fair 20252025
Raahul Singh, Vanč LevstikAI Engineer World's Fair 20262026
Simrat HanspalAI Engineer Summit 20232023
Jo Kristian BergumAI Engineer World's Fair 20262026
Will BrykAI Engineer World's Fair 20262026
Noyan TokgozogluEvent unknownUnknown
Vayum Arora, Zhengyao Jiang, Dixing Xu, Dhruv SrikanthAI Engineer World's Fair 20262026
Giedrius ŠteimantasAI Engineer World's Fair 20262026
George HeEvent unknownUnknown
Dylan CouzonAI Engineer Code 2025Unknown
Raghav SabooAI Engineer World's Fair 20262026
Ryan AhernEvent unknownUnknown
Bogdan GazaAI Engineer Code 2025Unknown
Jeff Vestal, James WilliamsAI Engineer World's Fair 20262026
Jess WangEvent unknownUnknown
Jakub HojsanAI Engineer Code 2025Unknown
Yuval BelferAI Engineer World's Fair 20262026
Eric SchwartzAI Engineer Code 2025Unknown
Nina Žižakić, Eleonora DrykinaAI Engineer NYC 2026 Online Talks2026
Joseph KimEvent unknownUnknown
Peter WerryAI Engineer World's Fair 20262026
Dan AdlerAI Engineer World's Fair 20262026
Benjamin ClaviéEvent unknownUnknown
Ryan AhernEvent unknownUnknown
Andy Wong, Andrei Bocan, Shane WolfAI Engineer World's Fair 20262026
Jesse HanAI Engineer Summit 20232023
Dhruv NathawaniAI Engineer World's Fair 20262026
Yoni Michael, Brandon CallenderAI Engineer World's Fair 20262026
Sam Green, Aaron LinskyEvent unknownUnknown
Stephanie JarmakEvent unknownUnknown
Peter WerryAI Engineer World's Fair 20262026
Nixon DinhAI Engineer World's Fair 20262026
Maximilian-David RumpfAI Engineer World's Fair 20262026

References

Coverage and source review
Processed transcripts
41 processed in full · 5 in the curated path
Automated source review
Passed
Metadata candidates
30 unreviewed; not verified topic membership
Corpus version
1bd8e407b26a07b33815594e1b2db5f41827119a2b3cb6fbf240f9fc571fc767

Automated review checks source support; it is not publication approval.

A synthesis of selected conference talks and technical references. Citations link to the source material; they do not imply that every talk on this subject is included.

  1. A taxonomy of web search

    https://sigir.hosting.acm.org/files/forum/F2002/broder.pdf

  2. Use the Microsoft Search API to search OneDrive and SharePoint content

    https://learn.microsoft.com/en-us/graph/search-concept-files

  3. In-depth guide to how Google Search works

    https://developers.google.com/search/docs/fundamentals/how-search-works

  4. Building a Smarter AI Agent with Neural RAG

    https://www.youtube.com/watch?v=xnXqpUW_Kp8&t=368

  5. RFC 9309: Robots Exclusion Protocol

    https://www.rfc-editor.org/rfc/rfc9309.html

  6. How To TREC

    https://trec.nist.gov/howto.html

  7. The Concept of Relevance in IR

    https://onlinelibrary.wiley.com/doi/10.1002/asi.10286

  8. RAG Evaluation Is Broken! Here's Why (And How to Fix It)

    https://www.youtube.com/watch?v=Ywl4LsvHKzU&t=35

  9. Retrieval Augmented Generation in the Wild

    https://www.youtube.com/watch?v=FDEmbYPgG-s&t=541

  10. Agentic GraphRAG: Simplifying Retrieval Across Structured & Unstructured Data — Zach Blumenfeld

    https://www.youtube.com/watch?v=CzM3cW6FdBs&t=224

  11. Information Retrieval from the Ground Up

    https://www.youtube.com/watch?v=4Xe_iMYxBQc&t=2755

  12. Web Annotation Data Model

    https://www.w3.org/TR/annotation-model/

  13. Graceful Query Coverage Degradation — Vespa

    https://docs.vespa.ai/en/performance/graceful-degradation.html

  14. Using Sampled Data and Regression to Merge Search Engine Results

    https://www.cs.cmu.edu/~callan/Papers/sigir02-lsi.pdf

  15. Search — Exa API reference

    https://exa.ai/docs/reference/search

  16. Microsoft Graph throttling guidance

    https://learn.microsoft.com/en-us/graph/throttling

  17. driveItem: delta — Microsoft Graph v1.0

    https://learn.microsoft.com/en-us/graph/api/driveitem-delta?view=graph-rest-1.0

  18. Building AI Agents that actually automate Knowledge Work

    https://www.youtube.com/watch?v=jVGCulhBRZI&t=254

  19. Mercator: A scalable, extensible Web crawler

    https://marc.najork.org/papers/wwwmerc.pdf

  20. Sitemaps XML format

    https://www.sitemaps.org/protocol.html

  21. RFC 9110: HTTP Semantics

    https://www.rfc-editor.org/rfc/rfc9110.html

  22. Downloader Middleware — Scrapy

    https://docs.scrapy.org/en/latest/topics/downloader-middleware.html

  23. A Statistical Approach to Mechanized Encoding and Searching of Literary Information

    https://web.stanford.edu/class/linguist289/luhn57.pdf

  24. Factors Determining the Performance of Indexing Systems, Volume 2

    https://sigir.org/files/museum/Factors_Determining_the_Performace_of_Indexing_Systems_Vol_2/pdfs/frontmatter.pdf

  25. Relevance Weighting of Search Terms

    https://www.staff.city.ac.uk/~sbrp622/papers/RSJ76.pdf

  26. Okapi at TREC-3

    https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/okapi_trec3.pdf

  27. The Anatomy of a Large-Scale Hypertextual Web Search Engine

    https://www.cs.cornell.edu/courses/cs530/2004sp/papers/BP98.pdf

  28. RankNet: A ranking retrospective

    https://www.microsoft.com/en-us/research/blog/ranknet-a-ranking-retrospective/

  29. Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods

    https://cormack.uwaterloo.ca/cormack/cormacksigir09-rrf.pdf

  30. Passage Re-ranking with BERT

    https://arxiv.org/pdf/1901.04085

  31. Dense Passage Retrieval for Open-Domain Question Answering

    https://arxiv.org/html/2004.04906v3

  32. Chunk large documents for RAG and vector search — Azure AI Search

    https://learn.microsoft.com/en-us/azure/search/vector-search-how-to-chunk-documents

  33. Chunking — Docling

    https://docling-project.github.io/docling/concepts/chunking/

  34. Information Retrieval from the Ground Up

    https://www.youtube.com/watch?v=4Xe_iMYxBQc&t=4707

  35. Define an index projection for parent-child indexing — Azure AI Search

    https://learn.microsoft.com/en-us/azure/search/search-how-to-define-index-projections

  36. Collapse search results — Elasticsearch

    https://www.elastic.co/docs/reference/elasticsearch/rest-apis/collapse-search-results

  37. Introduction to Information Retrieval: A first take at building an inverted index

    https://nlp.stanford.edu/IR-book/html/htmledition/a-first-take-at-building-an-inverted-index-1.html

  38. Index and search analysis — Elasticsearch

    https://www.elastic.co/docs/manage-data/data-store/text-analysis/index-search-analysis

  39. Information Retrieval from the Ground Up

    https://www.youtube.com/watch?v=4Xe_iMYxBQc&t=1347

  40. Introduction to Information Retrieval: Positional indexes

    https://nlp.stanford.edu/IR-book/html/htmledition/positional-indexes-1.html

  41. Query and filter context — Elasticsearch

    https://www.elastic.co/docs/reference/query-languages/query-dsl/query-filter-context

  42. Retrieve & Re-Rank — Sentence Transformers

    https://www.sbert.net/examples/sentence_transformer/applications/retrieve_rerank/README.html

  43. Magic WAND: Faster Retrieval of Top Hits in Elasticsearch

    https://www.elastic.co/blog/faster-retrieval-of-top-hits-in-elasticsearch-with-block-max-wand/

  44. Efficient Query Evaluation Using a Two-Level Retrieval Process

    https://www.researchgate.net/profile/David-Carmel-3/publication/221613425_Efficient_query_evaluation_using_a_two-level_retrieval_process/links/02bfe50e6854500153000000/Efficient-query-evaluation-using-a-two-level-retrieval-process.pdf

  45. Introduction to Information Retrieval: Okapi BM25

    https://nlp.stanford.edu/IR-book/html/htmledition/okapi-bm25-a-non-binary-model-1.html

  46. Information Retrieval from the Ground Up

    https://www.youtube.com/watch?v=4Xe_iMYxBQc&t=2380

  47. A statistical interpretation of term specificity and its application in retrieval

    https://www.staff.city.ac.uk/~sbrp622/idfpapers/ksj_orig.pdf

  48. Information Retrieval from the Ground Up

    https://www.youtube.com/watch?v=4Xe_iMYxBQc&t=4859

  49. Keyword type family — Elasticsearch

    https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/keyword

  50. Information Retrieval from the Ground Up

    https://www.youtube.com/watch?v=4Xe_iMYxBQc&t=885

  51. Information Retrieval from the Ground Up

    https://www.youtube.com/watch?v=4Xe_iMYxBQc&t=762

  52. E5-base-v2 model card

    https://huggingface.co/intfloat/e5-base-v2

  53. pgvector README

    https://raw.githubusercontent.com/pgvector/pgvector/master/README.md

  54. ANN-Benchmarks: A Benchmarking Tool for Approximate Nearest Neighbor Algorithms

    https://pure.itu.dk/ws/files/82351620/sisap2017.pdf

  55. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs

    https://arxiv.org/pdf/1603.09320

  56. Product Quantization for Nearest Neighbor Search

    https://paper-notes.zhjwpku.com/assets/pdfs/Product_Quantization_for_Nearest_Neighbor_Search.pdf

  57. Vector Query Filters — Azure AI Search

    https://learn.microsoft.com/en-us/azure/search/vector-search-filters

  58. Vector Search Benchmark[eting]

    https://www.youtube.com/watch?v=YrUBFXa1KUY&t=230

  59. Vector Search Benchmark[eting]

    https://www.youtube.com/watch?v=YrUBFXa1KUY&t=426

  60. Layering every technique in RAG, one query at a time

    https://www.youtube.com/watch?v=w9u11ioHGA0&t=570

  61. Introduction to Information Retrieval: Near-duplicates and shingling

    https://nlp.stanford.edu/IR-book/html/htmledition/near-duplicates-and-shingling-1.html

  62. Summarization: (1) Using MMR for Diversity-Based Reranking and (2) Evaluating Summaries

    https://aclanthology.org/X98-1025.pdf

  63. Combating Web Spam with TrustRank

    https://www.vldb.org/conf/2004/RS15P3.PDF

  64. Bypassing the Multimodal Tax: Framework-Free Hybrid RAG, Raw SQL RRF, and Live UI Telemetry

    https://www.youtube.com/watch?v=Akm1sqvWG4A&t=1800

  65. Benchmarking semantic code retrieval on Claude Code

    https://www.youtube.com/watch?v=zKk7sDMGDEQ&t=518

  66. Running AI Application in Minutes: Quick Start with AI Templates

    https://www.youtube.com/watch?v=1t-9-s1brcg&t=3940

  67. Reciprocal rank fusion — Elasticsearch

    https://www.elastic.co/docs/reference/elasticsearch/rest-apis/reciprocal-rank-fusion

  68. Normalization processor — OpenSearch

    https://docs.opensearch.org/latest/search-plugins/search-pipelines/normalization-processor/

  69. Relevance scoring in hybrid search using Reciprocal Rank Fusion — Azure AI Search

    https://learn.microsoft.com/en-us/azure/search/hybrid-search-ranking

  70. Layering every technique in RAG, one query at a time

    https://www.youtube.com/watch?v=w9u11ioHGA0&t=370

  71. The RAG Stack We Landed On After 37 Fails

    https://www.youtube.com/watch?v=2CXn-CByNoo&t=633

  72. MS MARCO Cross-Encoders — Sentence Transformers

    https://www.sbert.net/docs/pretrained-models/ce-msmarco.html?highlight=cross+encoder

  73. TREC 2023 Deep Learning Track

    https://trec.nist.gov/data/deep2023.html

  74. Cumulated Gain-Based Evaluation of IR Techniques

    https://faculty.cc.gatech.edu/~zha/CS8803WST/dcg.pdf

  75. A Comparison of Pooled and Sampled Relevance Judgments in the TREC 2006 Terabyte Track

    https://research.nii.ac.jp/ntcir/workshop/OnlineProceedings6/EVIA/11.pdf

  76. Introduction to Information Retrieval: evaluation of unranked retrieval sets

    https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-unranked-retrieval-sets-1.html

  77. Sentence Transformers InformationRetrievalEvaluator implementation

    https://github.com/huggingface/sentence-transformers/blob/main/sentence_transformers/sentence_transformer/evaluation/information_retrieval.py

  78. Introduction to Information Retrieval: Ranked evaluation

    https://nlp.stanford.edu/IR-book/html/htmledition/evaluation-of-ranked-retrieval-results-1.html

  79. TREC-COVID Round 5 Relevance Judgments

    https://ir.nist.gov/trec-covid/qrels5.html

  80. The Text REtrieval Conferences (TRECs)

    https://aclanthology.org/anthology-files/anthology-files/pdf/X/X96/X96-1046.pdf

  81. Building Production-Ready RAG Applications

    https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=319

  82. Detecting Good Abandonment in Mobile Search

    https://www.microsoft.com/en-us/research/wp-content/uploads/2017/05/williams_www2016_good_abandonment.pdf

  83. Learning Diverse Rankings with Multi-Armed Bandits

    https://www.microsoft.com/en-us/research/wp-content/uploads/2010/07/fp146-radlinski.pdf

  84. Unbiased Learning-to-Rank with Biased Feedback

    https://www.cs.cornell.edu/people/tj/publications/joachims_etal_17a.pdf

  85. Improving Pinterest Search Relevance Using Large Language Models

    https://arxiv.org/html/2410.17152v1

  86. User Performance versus Precision Measures for Simple Search Tasks

    https://www.researchgate.net/publication/221301359_User_performance_versus_precision_measures_for_simple_search_tasks

  87. Security filters for trimming results — Azure AI Search

    https://learn.microsoft.com/en-us/azure/search/search-security-trimming-for-azure-search

  88. Use a SharePoint indexer to ingest permission metadata — Azure AI Search

    https://learn.microsoft.com/en-us/azure/search/search-indexer-sharepoint-access-control-lists

  89. Zanzibar: Google’s Consistent, Global Authorization System

    https://www.usenix.org/system/files/atc19-pang.pdf

  90. Controlling access at the document and field level — Elasticsearch

    https://www.elastic.co/docs/deploy-manage/users-roles/cluster-or-deployment-auth/controlling-access-at-document-field-level

  91. The refresh parameter — Elasticsearch

    https://www.elastic.co/docs/reference/elasticsearch/rest-apis/refresh-parameter

  92. The shard request cache — Elasticsearch

    https://www.elastic.co/docs/reference/elasticsearch/rest-apis/shard-request-cache

  93. Decision record: use versioning to handle race conditions when updating the govuk index

    https://docs.publishing.service.gov.uk/repos/search-api/arch/adr-002-handle-race-conditions-in-govuk-index.html

  94. Changed and Deleted Blobs — Azure AI Search

    https://learn.microsoft.com/en-us/azure/search/search-how-to-index-azure-blob-changed-deleted

  95. Aliases — Elasticsearch

    https://www.elastic.co/docs/manage-data/data-store/aliases

  96. The Tail at Scale

    https://users.cs.utah.edu/~stutsman/cs6450/public/papers/tail-at-scale.pdf

  97. Phased Ranking — Vespa

    https://docs.vespa.ai/en/ranking/phased-ranking.html

  98. Deadlines — gRPC

    https://grpc.io/docs/guides/deadlines/

  99. Vector Search Benchmark[eting]

    https://www.youtube.com/watch?v=YrUBFXa1KUY&t=678

  100. OpenTelemetry: Traces

    https://opentelemetry.io/docs/concepts/signals/traces/

  101. Data is Your Differentiator: Building Secure and Tailored AI Systems

    https://www.youtube.com/watch?v=ROfHHJmumcc&t=1001

  102. Navigating RAG Optimization with an Evaluation-Driven Compass

    https://www.youtube.com/watch?v=DId2KP8Ykz4&t=771

  103. Building Production-Ready RAG Applications

    https://www.youtube.com/watch?v=TRjq7t2Ms5I&t=201

  104. NIST randomized blocks: applying controlled comparisons to RAG evidence

    https://www.itl.nist.gov/div898/handbook/pri/section3/pri332.htm

  105. Data is Your Differentiator: Building Secure and Tailored AI Systems

    https://www.youtube.com/watch?v=ROfHHJmumcc&t=962

  106. Data is Your Differentiator: Building Secure and Tailored AI Systems

    https://www.youtube.com/watch?v=ROfHHJmumcc&t=1128

  107. What We Learned from Using LLMs in Pinterest

    https://www.youtube.com/watch?v=XdAWgO11zuk&t=677