AI Engineer World's Fair 2026

Serving 2 Million Models Without Melting: Scaling the Hugging Face Hub

Arek Borucki· Machine Learning Platform & Database Engineer, Hugging Face21:39

Loading slides…

Read the talk

Serving 2 Million Models Without Melting: Scaling the Hugging Face Hub

How Hugging Face separates metadata from artifacts, prepares names for search, distributes database work, and scales Kubernetes capacity as its model catalog grows.

From a talk by Arek Borucki

Before you start: Familiarity with database queries, replication, and Kubernetes pods will help; the article explains the search and scaling mechanisms as they appear.

What happens when the model catalog outgrows search?

How do you keep a model hub responsive while its catalog grows into the millions? Developers still need to publish, discover, and download models without thinking about the infrastructure behind those actions. Arek Borucki, a machine learning platform and database engineer at Hugging Face, describes the decisions behind that experience. Although the recording’s title says two million models, his account uses a spoken figure of three million.

At the time of the talk, Borucki reports more than 14 million users, three million public models, one million datasets, and 50,000 organizations. He also reports that more than 30% of Fortune 500 companies use Hugging Face in their AI workflows. The opening slide labels the public-model count more precisely as 2.9M+.

Slide lists 14M+ users, 2.9M+ public models, 1M+ datasets, 50K+ organizations, and adoption by 30%+ of the Fortune 500.Open full source frame
Hugging Face’s scale: users, public models, datasets, and organizations.

Borucki puts the model catalog’s growth at roughly 150×, from 20,000 models three years earlier to three million. Major releases such as Llama and DeepSeek generate thousands of derivative models, so a successful release also creates a wave of new repositories to manage.

Datasets follow the same trajectory: Borucki reports 10,000 in 2022, 100,000 in 2024, 500,000 less than a year before the talk, and one million at the time of the talk. Every addition must be stored, indexed, and made searchable. Storage growth therefore becomes a discovery problem, not just a capacity problem.

0:330:57
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:33 · section reference included

The slow end of search matters

At 20,000 models, Borucki recalls, even queries without an index seemed fast enough. At three million, that approach broke down. Users expect search results immediately; a slow catalog gives them a reason to leave. His illustrative calculation is that 1% of 14 million users would mean 140,000 people experiencing slow search. That is a hypothetical affected population, not a measured incident or a conversion from request percentiles into distinct users.

Tail latency becomes a design priority. P50 describes the median request; P99 describes the latency boundary below which 99% of requests fall. Borucki emphasizes P99 because a healthy median can conceal a painful slow tail.

Three slide panels describe expectations for instant results, growth from 20K to 3M models, and the importance of p99 over p50 for 14+M users.Open full source frame
At three million models, search latency and p99 matter.

The response spans several layers: precomputed search tokens, a denormalized MongoDB collection optimized for reads, Lucene-backed search, and Kubernetes autoscaling. Database sharding is the next planned step. Each addresses a different source of work or contention rather than expecting one larger machine to solve everything.

4:094:23
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

4:09 · section reference included

Separate compute, metadata, and model files

A request travels from the frontend to the Hub API, which runs on Kubernetes. The Horizontal Pod Autoscaler adds application pods during traffic spikes and removes them as demand falls. The request then reaches MongoDB Atlas, the source of truth for Hub metadata.

MongoDB stores information about models, not the model binaries themselves. Its responsibilities include users, repositories, model and dataset records, buckets, Spaces information, configuration data, billing, and access control. Model artifacts, tokenizer files, card assets, and configuration files live separately in cloud object storage such as AWS S3.

LayerResponsibility
KubernetesRun and scale Hub application compute
MongoDB AtlasStore and query metadata
Cloud object storageStore model files and other artifacts

This separation lets the team scale metadata independently of binary storage, and compute independently of both. Each component can be optimized for its own workload.

5:415:54
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

5:41 · section reference included

Prepare model names before anyone searches

Consider someone typing llama into the Hub search bar. The request reaches a dedicated MongoDB read collection, not the main repository collection containing all repository data. This separate, denormalized copy is built for reads and listings.

The key optimization happens earlier: model names are tokenized at insertion time rather than query time. For the example identifier meta-llama/llama3.1-8b, the stored array includes tokens such as meta, llama, 3.1, and 8b. Atlas Search uses Lucene-backed autocomplete to match against that prepared representation. The query does not have to start by decomposing every model name.

Borucki inspects the stored representation with findOne, selecting that model identifier and projecting only its search-token array. The useful distinction is between the repository’s full identifier and the smaller searchable pieces stored alongside it: the readable model name remains intact while search gets a representation suited to matching.

7:558:12
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:55 · section reference included

Replace regex matching while keeping trending order

The earlier search implementation used MongoDB’s find method with a regex against searchTokens, then sorted matches by descending trendingScore. The query’s structure is compact:

javascript

db.models
  .find({ searchTokens: { $regex: "llama" } })
  .sort({ trendingScore: -1 });

Borucki says the trending score is recalculated every five minutes using downloads and likes; he recalls a seven-day activity window. Matching determines which models qualify, while the trending score determines their order.

MongoDB query with arrows highlighting find, the regex search for llama in searchTokens, and descending trendingScore sorting.Open full source frame
The existing regex search sorts results by trending score.

That approach worked while the dataset was small. As the catalog grew, regex search developed latency problems, prompting the move to Atlas Search and Apache Lucene. In the architecture Borucki describes, a separate process, mongot, wraps Lucene while exposing search through MongoDB’s query interface. This is distinct from MongoDB’s native text indexes and $text queries; core MongoDB does have text-search capabilities, but those are not the Lucene-backed autocomplete path used here.

The replacement uses an aggregation pipeline with $search, names the search index, and applies autocomplete to the token field. It still sorts by trending score. Expressed as a JavaScript helper taking the configured index name, the query shape is:

javascript

function searchLlamaModels(models, indexName) {
  return models.aggregate([
    {
      $search: {
        index: indexName,
        autocomplete: {
          query: "llama",
          path: "searchTokens"
        }
      }
    },
    { $sort: { trendingScore: -1 } }
  ]);
}

The change is in the matching engine, while the product’s popularity-based ordering remains in place.

Borucki reports that the replacement had scaled well so far and resolved the team’s search-bar latency issues; he provides no latency distribution or benchmark workload. The two displayed results have trending scores of 33 and 14, illustrating the retained descending order.

10:0310:20
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:03 · section reference included

Keep unnecessary work off the primary

Search is only one database consumer. Borucki describes hundreds of Hugging Face services using MongoDB and a seven-node cluster handling millions of queries, without specifying a time interval. Distributing queries across machines prevents one node from becoming the read bottleneck. All inserts, updates, and deletions still go to the single primary; reads can be spread across other members.

One member is a hidden analytics node. It replicates data from the primary but is invisible to ordinary application routing: the MongoDB driver does not send normal application queries there. The team connects directly to it for reporting and heavy queries. Like the other secondaries, it stays synchronized by following the primary’s operation log, or oplog.

Routing then follows the workload’s requirements:

  • Freshness-sensitive reads: In the described policy, queries requiring strong consistency stay on the primary. Reads that do not require the latest data go to secondaries.
  • Complex aggregations: Pipelines that scan large amounts of data, sort, group, or transform records run on secondaries rather than competing with primary-only work.
  • Change streams: Cache invalidation, synchronization to systems such as AWS Redshift, and event-driven workloads react to database changes. This work also belongs on secondaries where possible.
  • Ad hoc analysis: Reporting, experiments, and exploratory queries go to the hidden member, isolated from ordinary production traffic.

The primary should concentrate on work only the primary can do. Moving the rest elsewhere preserves its capacity for writes and the reads that need it.

12:5613:08
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:56 · section reference included

The next database step is partitioning

Read distribution extends the useful life of a replica set, but it does not remove the limits of keeping the entire dataset on each member and sending writes to one primary. Borucki presents sharding as a planned next step, not an already completed migration. The dataset would be divided among shards, each retaining its own primary and secondaries.

PropertySingle replica setPlanned sharded cluster
Data placementFull dataset on each memberA portion on each shard
ReplicationOne set of replicasReplication within each shard
Write destinationOne primaryThe primary of the relevant shard
Horizontal expansionMore read-serving membersMore shards sharing the dataset

Adding shards introduces more places to store and process data. MongoDB’s balancer redistributes data across them, but the distribution depends on a consequential design choice: the shard key. Borucki explicitly leaves that nontrivial choice outside the talk’s scope. The intended expansion covers CPU, memory, storage, reads, and writes.

16:3816:52
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:38 · section reference included

Scale application replicas and the capacity beneath them

At the application layer, the Hub’s HPA configuration responds when CPU or memory rises above its target. Borucki reports that the Hub deployment can scale from 10 to 500 pods depending on traffic. It scales back down as demand drops, reducing the need for manual intervention and permanent overprovisioning.

More desired pods do not guarantee more usable capacity. If the Kubernetes cluster has no room for them, the scheduler leaves them pending. CAST AI supplies the second scaling layer:

  1. HPA increases the deployment’s requested pod count.
  2. Pods remain pending when existing nodes cannot accommodate them.
  3. CAST AI adds nodes.
  4. The Kubernetes scheduler places the pending pods on the new capacity.

Pod scaling changes how many application replicas should run. Node scaling supplies the infrastructure on which those replicas can run.

The next planned change is adoption of KEDA, Kubernetes Event-Driven Autoscaling, to make scaling decisions from application signals such as requests per second and event-loop utilization. A pod can have low CPU utilization while requests accumulate in a queue; resource utilization alone can therefore miss demand that matters to users. The CPU/memory limitation here describes the Hub’s presented HPA configuration, not HPA in general: HPA also supports custom and external metrics, and KEDA integrates with it rather than replacing its underlying control loop.

Kubernetes diagram connects users, a Hub Deployment labeled 10 to 500 pods, KEDA, and horizontal scaling, with panels for application-aware autoscaling and workload signals.Open full source frame
KEDA autoscaling driven by requests and event-loop signals.

The user-facing test of all this machinery is simple: pushing a model, searching the Hub, or downloading a model should work without requiring the user to understand it. The internal architecture can become more elaborate as the catalog grows, while those everyday actions remain straightforward.

18:1918:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:19 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…