AI Engineer World's Fair 2026

How Web Data Infrastructure Powers the Next Generation of AI

Patricija Žemaitytė· Product Manager, Oxylabs19:03

Loading slides…

Read the talk

How Web Data Infrastructure Powers the Next Generation of AI

Video collection, subsecond search and large-scale scraping expose the same engineering challenge: keeping web data usable as requirements and access conditions change.

From a talk by Patricija Žemaitytė

Before you start: Familiarity with APIs, web scraping and latency metrics will help; the talk focuses on infrastructure decisions rather than code.

How does fresh web data reach a model?

Public web data may be available to anyone, but how does it become reliable input for a model, an agent or a database? Availability does not supply the infrastructure needed to collect, structure and deliver it. That gap is the starting point for Patricija Žemaitytė, a product manager at Oxylabs, which builds infrastructure for extracting public web data at scale.

Training still matters, but a model’s stored knowledge cannot provide fresh information by itself. Live search and external data require a working connection to the world outside the model. Web data becomes a runtime dependency, with collection and delivery requirements that are different from assembling a static training dataset.

Slide with four points: static training data is not enough; RAG and grounding changed expectations; AI systems need fresh, live information; web data became part of runtime, not just training.Open full source frame
AI changed the role of web data: fresh information is needed at runtime, not just during training.
0:190:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:19 · section reference included

Two weeks to build a video pipeline

Before moving into product management, Žemaitytė led engineering teams. One squad, called UX, regularly handled unusual customer requests for which no ready-made product existed. Tight deadlines made adaptation part of the job. The first video request arrived through the sales team after a trip to San Francisco: a customer wanted a video API for AI training.

The customer required at least five petabytes per month, with a two-week deadline. That was a requirement, not a measured throughput result. The team had never built this kind of service, and the volume changed the meaning of the request: downloading videos was only one part of a pipeline that also needed collection, transfer, storage and delivery reliable enough for training workloads.

A multimodal pipeline also needs the context surrounding the video: metadata, transcripts, subtitles and other structured information. For the first release, however, the immediate work was a dedicated scraper with new extraction logic, new storage integrations and a delivery flow the team had not implemented before. Žemaitytė reports that the team shipped that first version within the two-week deadline.

Customer testing immediately exposed the next requirement. The customer asked for transcripts, the team added transcript support, and the requests failed. Discussion revealed a mismatch: the customer actually needed subtitles. The team added subtitle support, then received another request—search to discover videos in the languages the customer needed—and then a request for metadata. Each addition addressed a different stage of obtaining usable training material, from discovery to extraction to context.

The downloader grew into a family of capabilities: transcripts, subtitles, metadata and channel information, connected by an internal library. Žemaitytė reports that the full video API suite took roughly three months to emerge. The commercial coda was less tidy: in her 2026 account, the customer had gathered 30 petabytes while payment was still outstanding. The engineering lesson was that the first delivery started an ongoing adaptation process; customers needed the team to keep adjusting the product as their actual workflow became clearer.

1:301:41
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

1:30 · section reference included

Search becomes part of the response path

The next constraint was latency. Search engine results page data—SERP data—had long served analytics, SEO, monitoring and market intelligence. In AI systems it also feeds retrieval pipelines, grounds answers and gives assistants and agents access to current information. Žemaitytė points to Google’s grounding documentation as an example of this shift: search connects model responses to current public knowledge. Retrieval now sits alongside the model in the application’s response path.

In 2024, a customer requested subsecond SERP delivery against a regular scraper baseline of roughly four seconds average latency. The team built a version that Žemaitytė says met the request, but the customer never tested it. With little market uptake, the work went on the shelf. The useful discovery was that faster delivery required reconsidering what the search product collected.

ProductCollection scope
Regular search scraperAds, widgets, rich results, AI-generated results and varied layouts
Fast Search APIPrimarily organic results, top stories and news

The regular scraper aimed to retrieve as much information as possible. Fast Search reduced that scope and removed heavy layout handling to concentrate on results useful to AI workflows. A narrower output contract made lower latency more attainable before any individual optimization entered the picture.

6:236:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

6:23 · section reference included

The fast prototype meets a live block

A new customer arrived in 2025 with a more specific contract. The service needed zero data retention, support for different geolocations and query parameters, and readiness for customer testing in less than two weeks. The delivery requirement was under 800 milliseconds. Starting from the earlier scraper’s latency, the team treated this as a redesign.

Žemaitytė reports that the first version reached around 650 milliseconds P90 in less than two weeks. Then, during the customer test call, the service was blocked badly enough that the team had to reconsider the implementation. A successful development run and a good latency measurement had not established that the system could survive real access conditions.

Slide titled “Reality hits:” showing a Critical Alert notification with a red warning icon and the status FIRING.Open full source frame
“Reality hits”: a critical alert marked FIRING.

The second iteration depended much more heavily on browsers. They supplied capabilities the service needed, but brought latency, cost and complexity. The customer still wanted subsecond delivery; the browser-dependent approach pulled latency back toward seconds. The team began reviewing layouts, parsers, sessions and proxies, looking for savings throughout the request path. The improvement came from accumulated decisions across those components, rather than one breakthrough.

Žemaitytė reports a later Fast Search API average latency of 550 milliseconds. This is an average, whereas the prototype’s result was P90; the talk does not specify benchmark workloads or measurement conditions, so the figures are not a controlled before-and-after comparison. She also reports growth from 400 million daily requests to almost six billion daily requests, without defining that volume as Fast Search traffic alone.

That growth changes the operating model: costs, observability and failure domains all become larger engineering concerns. The latency change has a separate product consequence. A slow retrieval pipeline can supply data in the background; subsecond retrieval can participate in an interactive AI workflow while a user waits for a response. Speed determines which workflows the infrastructure can support.

8:429:07
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:42 · section reference included

Scale the whole scraping job

After adapting product scope and then architecture for latency, the next challenge was system capacity. Web Unblocker, described here as scraper-like infrastructure with proxy integration, faced a steep increase in demand. The requirement was to grow from around 10,000 to 60,000 requests per second in less than two months.

The unit of work matters. Here, a request means an end-to-end scraping job, encompassing routing, rendering, proxy handling, browser execution, parsing, retries, normalization and delivery. Counting these jobs as if they were simple HTTP requests hides the work behind each arrival. Žemaitytė argues that even adding another 2,000 servers would not solve the problem by itself. Capacity also depends on the architecture and the components that coordinate the work.

The accompanying slide brings component rewrites into the same picture as additional machines. Reliable central components, observability that still reflects actual system behavior, and tests that resemble customer workloads are all necessary to make the additional capacity useful. A larger fleet cannot by itself establish that the whole pipeline will behave correctly under load.

Web Unblocker scaling slide with three panels reading “10k req/s → 60k req/s,” “Rewrote components,” and “+2000 machines.”Open full source frame
Web Unblocker scaling: 10k to 60k req/s, rewritten components, and 2,000 additional machines.
12:1712:36
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

12:17 · section reference included

When testing becomes a capacity constraint

The next bottleneck appeared in load testing. Generating synthetic traffic was relatively straightforward; generating and processing traffic that behaved enough like real customer usage was harder. Žemaitytė calls this organic-data testing. The goal was to learn whether the system could handle the actual workload, not simply whether it could accept a large volume of requests.

One load test hit a wall at around 20,000 requests per second. The unresolved question was whether the system could go further, rather than whether it worked at all. The account does not establish a specific root cause for that ceiling. It establishes a limit on the team’s confidence in capacity beyond the tested load.

Telemetry is part of the workload. Generating, collecting and processing logs consumes resources, and metrics introduce their own collection and processing demands. At this scale, the machinery used to understand the system also adds load and complexity to it. Observability must continue to reveal useful behavior while the observed system and the telemetry pipeline are both under pressure.

The team scaled gradually and ultimately relied on production traffic for final validation. Žemaitytė reports that this phase went well. The capacity effort remained open-ended: Project Sixty named the original target, and Project One Hundred Fifty became the next ambition. She reports emerging scale of about 100,000 requests per second, not an achieved 150,000. Reaching one capacity target created the starting point for the next round of work.

13:5914:06
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:59 · section reference included

The product includes continuing maintenance

Proxies are one component of this infrastructure. The larger service includes reaching the web, collecting reliably, handling anti-bot systems, using browsers where needed, and structuring and delivering results in a form AI systems can consume. That work creates a division of responsibility: the customer builds the intelligence layer, while the infrastructure provider maintains the data-access layer beneath it.

The maintenance burden continues because targets, page layouts, detection systems, markets and customer requirements all change. The video product needed new capabilities as the customer’s workflow became clearer. Search needed a different scope and then another implementation when live blocking invalidated the first approach. Scaling needed realistic testing and gradual production validation as capacity demands rose. Each form of adaptation becomes part of the infrastructure that customers rely on.

Žemaitytė’s closing thesis is that better infrastructure around models will be essential to the next generation of AI. Its practical role is to deliver live web data into pipelines, databases, agents and AI tools while absorbing the continuing work of adapting products, architecture and systems. The model gains a bridge to the world—and someone remains responsible for keeping that bridge usable.

16:1616:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:16 · section reference included

Resources

From the talk

Transcript download
Full transcript

Loading transcript…