AI Engineer World's Fair 2025

Vector Search Benchmark[eting]

Loading slides…

Read the talk

Vector Search Benchmark[eting]

Contradictory vector-search rankings often begin with different workloads, configurations and quality targets. A useful benchmark makes those conditions explicit.

From a talk by Philipp Krenn

Before you start: Familiarity with vector search and basic latency measurements will help; the article explains the approximate-search quality tradeoff.

How can every vendor be faster?

Why are vector-search benchmarks so hard to compare? When Philipp Krenn asks the audience about their experience, that is the problem they raise. One benchmark puts a product ahead; another reverses the ranking. In Krenn’s telling, almost any vendor can occupy either side of X is faster than Y.

Those contradictory winners pose a more useful question than whether everyone is lying: what changed between the comparisons? Workloads, settings and reporting choices can change the outcome. When a selected result becomes a broad sales claim, benchmarking becomes benchmarketing.

0:530:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:53 · section reference included

Similar conditions can still favor one system

A glossy chart does not establish a fair comparison. Start with the use case: does the test represent what your system must do? Krenn illustrates the problem with a cartoon in which a squid and a housecat face puzzles in matching water tanks. The surroundings are similar, but their suitability for the participants is not. A benchmark can apply the same conditions to two systems while making one system’s strengths decisive.

Cartoon of a scientist observing a squid and a submerged housecat in adjacent tanks, with a caption claiming the squid is more intelligent under similar conditions.Open full source frame
A squid and a housecat face puzzles in matching water tanks.

The next step is selective publication. Run several scenarios, reject the ones where your system looks ordinary or weak, and keep the one where it wins. Then generalize that narrow result into a claim about everything. Krenn’s example of being forty percent faster is an invented marketing headline, not a measured finding presented in the talk. The missing information is the set of scenarios discarded along the way.

1:281:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:28 · section reference included

A read-only dataset leaves out operational work

Read-only tests are attractive because they are easier to repeat and compare. They can also use an optimized static representation of the data. Krenn observes that these tests dominate the vector-search benchmarks he encounters, even though many real workloads continue to change while serving queries.

Once writes enter the workload, the benchmark needs more than a dataset and a query set. It needs a representative read/write ratio. That adds parameters and makes the experiment harder to design, but omitting them changes the question being answered: performance against an optimized, stationary dataset may not represent performance while that dataset is being updated.

3:073:24
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:07 · section reference included

Filtering can increase the search work

Filtering introduces a counterintuitive case. In a conventional datastore, a restrictive filter often reduces the work needed for subsequent ranking. Krenn asks us to imagine a filter that admits twenty percent of the records. For HNSW, an approximate nearest-neighbor graph index, fewer eligible records can instead mean more candidate exploration.

The search still needs to find enough acceptable neighbors. If candidates encountered during exploration fail the filter, it may have to examine more candidates before collecting the requested results.

Search situationEffect of a restrictive filter
Filter records, then rank survivorsFewer records to rank
Explore an approximate-neighbor graphMore exploration may be needed to find eligible neighbors

The distinction is between the size of the eligible result set and the work required to discover it. A smaller set does not guarantee a cheaper search.

This slowdown is conditional: implementations have filtering optimizations, and different filter scenarios exercise them differently. A benchmark publisher can select a filter ratio that suits its implementation or choose a workload that hits a recently added optimization. Read-only state, read/write mix and filter behavior therefore belong in the explanation of the result, not outside it.

Slide reading: “Example: Optimized read-only data, read-write or filter ratio, hitting optimizations.”Open full source frame
Optimized read-only data, read-write or filter ratio, and hitting optimizations.
3:353:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:50 · section reference included

Check which releases actually competed

A benchmark can change its winner simply by updating one side. Krenn reports seeing his company’s software represented by a roughly eighteen-month-old version while the publisher uses its own release from the preceding one or two months. This is an attributed example of unequal software freshness, not a comparison between identified releases.

Keeping competitors current requires work: someone must follow their releases, understand changed behavior and learn the appropriate configuration. It is natural for a team to concentrate on its own improvements. But leaving the competing systems untouched also conveniently improves the appearance of the publisher’s progress. A refreshed benchmark needs scrutiny of every product version, not just its publication date.

4:485:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:48 · section reference included

Familiar defaults are not neutral defaults

Bias does not require deliberate manipulation. Engineers know how their own architecture works, which workloads suit it and which settings make it perform well. That familiarity shapes scenario selection before anyone sees the competing results. Defaults can carry the same bias.

Several choices determine whether a benchmark fits a system:

  • Shards: How the data is split, how shards are allocated and how large each shard becomes.
  • Hardware and memory: Instance size, instance configuration and memory allocation.
  • Data representation: Total dataset size, whether the working data fits in memory and how it is compressed.
  • Access patterns: Which parts of the data the workload repeatedly touches.

A configuration chosen because it works well for one product may be a poor fit for another, even if nobody intended that outcome. Equal settings are not sufficient evidence of equally suitable settings.

5:385:50
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:38 · section reference included

Fast results must still be comparable results

There is also outright cheating. Krenn invokes Volkswagen’s emissions scandal as an analogy for looking good under test conditions, then recalls a joke CI project that makes tests pass. The satirical Volkswagen package captures that idea: improving the reported outcome without improving what the test is meant to establish.

Vector search has a subtler way to produce an impressive speed result. With approximate nearest-neighbor search, search breadth and implementation parameters affect which neighbors are recovered. Changing those parameters can change both execution time and result quality. A speed comparison needs a quality condition. Otherwise, two fast-looking numbers may describe different search outcomes.

Report precision and recall alongside performance, and make their definitions explicit. For nearest-neighbor approximation, recall measures recovery of the exact nearest neighbors; it does not automatically measure relevance to a human reader. A simple per-query recall_at_k check compares returned IDs with the exact top k IDs:

python

def recall_at_k(exact_ids, returned_ids, k):
    if k <= 0 or len(exact_ids) < k:
        raise ValueError("Provide at least k exact neighbors, with k > 0")
    expected = set(exact_ids[:k])
    if len(expected) != k:
        raise ValueError("Exact neighbor IDs must be unique")
    recovered = expected.intersection(returned_ids[:k])
    return len(recovered) / k

This makes the missing dimension concrete: returning quickly is only part of the result; recovering the intended neighbors is another.

Omitting quality is not necessarily intentional. Krenn recounts an internal benchmark that nearly reached publication without precision and recall checks. People accustomed to Boolean filters—where a record either passes or fails—had overlooked the additional quality tradeoff introduced by approximation. Adding those checks materially changed the comparison.

6:396:45
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:39 · section reference included

One exceptional query can dominate the headline

Even when individual measurements are defensible, their presentation can exaggerate the scope of an advantage. Krenn recalls roughly twenty measurements: eighteen or nineteen showed similar performance, while one optimized use case was about ten times faster. Aggregating the results made the systems appear much farther apart overall.

He tentatively remembers the exceptional case as a change in sort direction on a particular data type. The exact query and aggregation formula are not specified, so the anecdote cannot establish a general speedup. Its useful lesson is to inspect the distribution: does an advantage appear across the workload, or does one special case carry the claim?

The marketing destination is a simple headline, such as being five times faster than a competitor. That is another illustrative slogan, not a verified overall result here. Once the headline replaces the per-query evidence, readers can no longer see whether the improvement applies to the work they actually do.

8:358:46
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:35 · section reference included

Repeated measurements expose gradual regressions

A benchmark is difficult to challenge or use if its publisher leaves out the pieces required to reproduce it. The constructive response is to make tests automated and repeatable. Krenn describes Elastic’s nightly benchmarks: a broad set of tests run against the system’s own evolving performance.

The displayed chart shows a sharp latency drop after an optimization. The repeatable practice matters more than assigning that example a speedup multiplier: each night’s run measures the combined changes made during the day, allowing the team to see when performance changes.

Chart of latency across late May and early June, with percentile lines near 1,200 ms before dropping sharply; a June 3 tooltip lists values from 13.578 to 18.772 ms.Open full source frame
A nightly latency chart shows a sharp drop on June 3.

Krenn uses the slow-boiling-frog story as a metaphor for accumulated regressions. An illustrative one-percent slowdown today may escape notice; another two-percent slowdown the following week may also seem unremarkable. Over time, those small losses can become substantial. A continuous history makes that drift visible instead of leaving the team to discover it after performance has already deteriorated.

9:419:49
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

9:41 · section reference included

Turn your workload into a repeatable evaluation

To make a purchasing or architecture decision, the benchmark must represent your own workload. Specify the data size and structure, read/write ratio, actual queries, acceptable latency and available hardware. An external benchmark may match some of those conditions without matching the combination that determines your result. Running the local experiment takes work, but skipping it leaves you hoping that someone else’s conditions transfer to yours.

Krenn’s concrete tool is Rally, an Elasticsearch benchmarking framework. A track describes the data and operations to run. That gives the evaluation a repeatable structure:

  1. Define the data and queries that represent the workload.
  2. Run that workload against the system.
  3. Change settings or hardware and repeat the same workload to assess the effect.

Krenn describes using this approach to compare their own system with itself, including nightly testing.

For a cross-product evaluation, use the corresponding tools for the other systems and reproduce the same workload conditions. Rally is not being presented as a universal harness for every vector database. The question is what each system can deliver for your data, queries and hardware—not which product owns the most impressive unrelated chart.

11:1811:33
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

11:18 · section reference included

A flawed benchmark can still tell you where to look

Finding a flaw makes it tempting to discard the entire benchmark. Krenn recommends asking a second question before doing that: what can still be learned from the scenario the publisher chose? Even an unfair comparison can reveal where a vendor believes its system performs especially well.

Purple slide with a see-no-evil monkey beside “Find flaw, call results garbage” and a bandaged heart beside “Find something useful.”Open full source frame
“Find flaw, call results garbage” versus “Find something useful.”

The chosen workload can point toward a system’s strengths, preferred access patterns or recently improved behavior. Treat those as leads to investigate rather than accepting the headline—or dismissing every observation because one part of the comparison is weak.

That leaves a practical ending: use published benchmarks to discover what to test, then use your own workload to establish what to trust. Without that work, the decision still rests on benchmarketing.

12:4613:00
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:46 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…