AI Engineer World's Fair 2026
The Missing Layer in Agentic AI — Giedrius Šteimantas, Oxylabs
Loading slides…
Read the talk
The Missing Layer in Agentic AI
Giedrius Šteimantas walks through a shopping agent’s redesign: search for candidates, validate product content before spending model tokens, ask the user to approve, and reserve browser automation for checkout.
From a talk by Giedrius Šteimantas
At a glance
Ideas worth remembering
Choose web-access tools by task: search for candidate URLs, extract validated product content, and use a browser for interactive checkout after user approval.
HTTP
200and response size cannot establish that a page contains useful product information. Explicit retrieval errors keep known blocked responses out of model input.Validate before compression: shrinking invalid content does not make it useful, and asking the model to reject it still spends tokens.
An extraction API can hide browser orchestration while retaining conditional browser rendering for dynamic pages.
Align geolocation across product verification and checkout so regional stock, sizes, and options are evaluated in the same context.
A personal shopper meets the open web
The shopping chatbot could discuss a customer’s style and turn their preferences into prompts. A second agent was supposed to take those prompts, find suitable items online, and purchase them. Getting from a plausible recommendation to an actual product page proved harder: requests often returned CAPTCHAs, and browser automation throughout the workflow made the product slow, expensive, and unreliable.
Giedrius Šteimantas of Oxylabs introduces this friend’s project as an example of the infrastructure layer agents need to work on the open web. Oxylabs had spent ten years helping companies obtain data, including companies training large language models. The same web-access infrastructure could now support agents.
The redesign starts with three scraping principles:
- Use a browser selectively. Pay for browser execution when the task requires it.
- Validate content. HTTP
200means the request received a successful response; it does not establish that the response contains the requested product page. - Send lighter content. JavaScript, CSS, and HTML can contribute many bytes without helping the model choose a product.
The operating constraint behind these choices is simple: “cost matters.”
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Discovery: search for candidates without opening every store
The workflow separates four jobs. Discovery finds product URLs. Decision checks whether the products have the right price, stock, description, and details. The user accepts or rejects the proposed purchase. Execution makes the purchase. Keeping these jobs separate allows each web-access step to use a tool suited to what it actually needs to accomplish.
Originally, discovery opened the search pages of a predefined list of major retailers in an automated browser. Sites challenged or blocked that browser, interrupting the flow. Retries added time and expense without guaranteeing access, making the final cost per transaction difficult to predict. The fixed retailer list also constrained the agent’s choices to stores its developer had selected.
Heavy JavaScript made those searches costly even when they succeeded. Location introduced another failure: an item could appear available early in the workflow and become unavailable at checkout. Retailers can show different stock, sizes, and options depending on the user’s location, and the original discovery stage lacked geolocation controls. A usable listing therefore needed more than a product match; its availability had to apply to the customer’s region.
The replacement, Fast Search API, gives the agent search results as compact JSON. The agent formulates fan-out queries—several searches for the shopping task—and selects relevant URLs from the results. Search engines supply the indexed candidates, removing the need to navigate each retailer’s search interface or restrict discovery to a hardcoded list.
Šteimantas reports responses below 2,000 tokens and average response times below 700 milliseconds, alongside high success rates and predictable low pricing. These are reported service characteristics rather than a controlled comparison of the two shopping agents. Small responses also reduce the need for a complicated model at this stage: the immediate job is to select promising links, with detailed product verification still to come.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Decision: validate the page before the model reads it
Selected URLs now need inspection. The decision stage reads the product pages to confirm price, stock, description, and other details before choosing an item. The original agent ran many browsers in parallel to accelerate this work. Parallelism helped with speed, but it could not make blocked pages useful. Failed access left the agent with few options and excluded many popular retailers.
Observability let the friend notice these failures. Other implementations can miss them by checking only response size and HTTP status before sending the HTML to an LLM. A substantial CAPTCHA page can pass both checks. The model may recognize it as invalid product content, but the application has already spent tokens to find that out.
Consider the talk’s concrete example: ten attempted pages return three valid product pages and seven invalid responses, yet all ten go to the model. Šteimantas estimates that this wastes 70% of the tokens. Seven out of ten establishes the fraction of invalid responses; the token fraction also depends on their lengths. The practical failure is the same: blocked content consumes model input while supplying no products to compare.
Compression initially looked like the fix. Smaller HTML would cost fewer tokens. But compressing a CAPTCHA still leaves a CAPTCHA, so validation has to come first. Once a blocked response becomes an explicit retrieval error, the application can exclude it before inference. Better retrieval can expand the usable choice set; filtering keeps unsuccessful retrievals from becoming model work.
Where does the wasted model work disappear? The diagram follows the ten-response example through the original path, then shows the redesigned routing. Previously, valid and blocked pages traveled together into the model. The new service sends useful content onward and reports blocks separately. Validation changes which data reaches inference, rather than merely shortening every response.
Blocked responses stop before inference
Read the diagram as text
- Ten responses. Original path — Three valid product pages and seven invalid responses.
- Model reads all ten. Original path — Tokens are spent recognizing CAPTCHAs as well as comparing products.
- Web Scraper API. Redesigned path — Retrieves content and reports CAPTCHAs or other blocks as explicit errors.
- Valid product content. Redesigned path — Markdown can replace raw HTML as model input.
- Explicit retrieval error. Redesigned path — The application excludes the blocked response from model input.
- Model compares products. Redesigned path — Product content reaches the decision stage without known blocked responses.
- Ten responses → Model reads all ten: Send valid and invalid HTML.
- Web Scraper API → Valid product content: Successful retrieval.
- Web Scraper API → Explicit retrieval error: CAPTCHA or other block.
- Valid product content → Model compares products: Send product content.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Move retrieval complexity behind an extraction service
The rebuilt decision stage uses Oxylabs Web Scraper API through a lightweight REST interface. Explicit errors establish which responses should stay out of the LLM call. The service then addresses several other costs independently:
- Concurrent requests: Hundreds of requests can run in parallel without the shopping application orchestrating a browser for each one.
- Markdown output: Product content can reach the model in a lighter form than raw HTML.
- Dynamic rendering: When a site requires it, the service runs a full browser internally to render the content.
- Geolocation: Requests can retrieve localized product information, including the regional context relevant to availability.
- Successful-result billing: Under the described pricing model, failed scraper requests incur no scraper charge.
A browser may still perform some retrieval work. The important change is that the application calls an extraction service, and that service handles rendering when needed. This preserves access to dynamic pages while removing browser management from the agent’s ordinary product-reading step. Failure also becomes a visible outcome with a clear billing rule: “No cure, no pay.”
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Checkout: keep the browser and align its location
With product information available, the agent presents its choice to the user. Only an affirmative answer starts the purchase stage. Here the browser earns its cost: checkout requires processing inputs and interacting with highly dynamic content. Extracted Markdown can help choose an item, but it cannot select a size or add the item to a cart.
Both implementations retain Playwright MCP, a browser, and an LLM for execution. The remaining obstacle is familiar: the original automated browser gets challenged repeatedly and cannot complete the flow. Šteimantas replaces it with Oxylabs Headless Browser, described as a drop-in replacement because it supports Playwright MCP.
The replacement supplies three access capabilities:
- Browser-level stealth: Changes at the browser source-code level address detection of automation.
- Residential proxy: A residential proxy comes attached to the browser.
- Consistent geolocation: Checkout uses the same location context as product verification, addressing the earlier mismatch between displayed availability and what can be purchased.
Return to the item that appeared available and then vanished at checkout. The redesigned flow checks localized product content, presents the choice for approval, and opens the interactive purchase flow using the same location. Playwright can then select the size specified in the prompt, add the item to the cart, and proceed through purchase. Šteimantas describes this as the intended working flow; the account does not establish an observed completed transaction or guarantee that inventory cannot change between verification and checkout.
The ending brings the implementation choices back to their purpose. Search handles discovery, extraction handles product reading, and a browser handles the interaction that remains. Infrastructure absorbs the web-access work so builders can focus on the shopping experience. The rules apply throughout: use browsers when necessary, validate content before feeding it to an LLM, and keep cost in view.
Suggest correction
This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.
Resources
Related talks
- How Web Data Infrastructure Powers the Next Generation of AI
A related talk on the web-data infrastructure that forms this recording’s central subject.
- The Agentic Commerce Stack — Ahnaf Prio, Best Buy
A related commerce-focused talk for readers interested in the shopping-agent setting.
Full transcript
Loading transcript…