Building AI Agents with Real-Time Web Data — Yohan Raju, Bright Data

All right, let's begin. So nice to meet everyone. My name is Yohan Raju. I'm a solutions architect here at Bright Data. I've been with the company for about six years, and I've been working in this space of data collection and web scraping. We initially started off as a proxy infrastructure company, uh, with the largest residential IP network in the world, and then we moved into the data collection platform. Now we're the world's largest web data collection platform, collecting petabytes of data per

month, only second to Google. So just a little information for you guys on, you know, where we started and, and how, how we moved over to today. So I'm gonna be covering two topics. First is I'm gonna be speaking about building AI agents with real-time web data. Some tools that people are generally using when it comes to, you know, AI agents versus depending on your RAG model with pre-indexed data and what are the advantages, what we see in the industry and, you know, what changes, uh, are to come and

what it takes to, uh, run these agents. And then finally, I'm gonna have my hands-on exercise that you guys can follow along with me, uh, where I have a product intelligence agent that will essentially use a product from Amazon, get sentiment on Reddit, and then provide you analysis with, you know, should I buy that product? Yes or no.

So

working in the scraping space for a long period of time, one thing I've noticed is, you know, when you think about building, scraping, or collecting data, what does it really take, right? Now, the early customers in the scraping space that I've worked with, you know, when, when you first start, it's always about, "Hey, I wanna-- these are my target sites, and this is what I wanna scrape," right? You know, people usually start off with low volume. You know, that's what comprises a general traditional scraping architec-architecture.

Now, let's say I'm an e-commerce, uh, user, and I wanna collect e-commerce data. You know, my biggest targets might be Amazon and Walmart because, you know, I can find a lot of products on there, and I can get price intelligence and data on reviews and, you know, all the general information for a vast majority of products. Now, when does scraping become challenging? And this is a change that I've actually seen with customers that I work with today, um, occurring in the industry when it's not just about

Amazon and Walmart anymore. Now you want Shopee, now you want Shopify stores, now you want, um, Etsy and, you know, a bunch of other sites. Now, if we take a step back, and I mentioned, you know, you know your target sites. In order to build scrapers around your target sites, it takes generally about, you know, three to five days to develop a scraper if you're a developer. And that's not only developing it, it's QAing it, running it in development, testing it out and, you know, the whole end-to-end pipeline. Now, if we

think about, you know, ten sites, okay, it might take a bit of time. Now you think, "Okay, I wanna add a hundred sites, a thousand sites." This is where things start getting complicated. Now, before AI even came into the picture, we would-- developers would have to manually understand the site, you know, understand where the selectors are located. Can I target these data points in the API and actually build crawlers around that knowledge? And I've worked with many people, you know, that's, that have been scraping for ten plus years that

have been managing and maintaining these crawlers. But when it comes to scaling up, then you're thinking, "Okay, like, now I'm adding more sites to what I already am scraping, but I also need to manage the scrapers that I al-already have." So let's say Amazon makes a change tomorrow, and two other sites make a change tomorrow, but you're managing a thousand sites. This requires a lot more engineering resources and a lot more time for you to actually figure out, okay, what change was made? How can I counter-counteract it? How can I fix it? And is this actually impacting

my customers, my SLA, and my data quality? Now, before we had this bottleneck of having to scale up your resources and when-- at those times when you would have to manage the thousand sites, you'd need, you know, two times more engineers, four times more engineers, et cetera. But now in this AI space, none of that is needed because we have things like agents, and agents can actually go out and discover and build sites, build scrapers for these sites a lot easier than, you know, a team of developers

and engineers. Now, this is one classic example of an e-commerce site of targeting selectors on the page. So you can see here we're just-- in this example, it's just targeting price and title. Now, what happens if the price has changed, the selector for the price has changed, you know, tomorrow? That will require some downtime on your end because, you know, if you're the one scraping the site because, you know, now you're not getting price and that can affect the rest of your pipeline that you have to maybe analyze products, you know, et

cetera. Now, usually, that require like a team to go in, fix it, then maybe backfill on the data that, you know, you've lost. But now with AI agents, that can be au-automated and done automatically. So when you've also built scrapers, ev-- one thing to always know is every website is different. Now, sites in the example I use like Amazon and Walmart, they have two different layouts, even though they both might have things like price, title,

description. So each site requires a unique scraper to built, build for. And the even more important than that is each site wants to protect their data. Now, sites usually like to ensure that no one else is, uh, you know, has it enough-- like the same product intelligence they have or, you know, depending on what type of data that they're, you know, trying to keep secure. They will always implement different types of anti-bot and different types of

CAPTCHA to throw, you know, scrapers and bots off from collecting that data. Now, I've seen all different types of blocks. I've seen, you know, the most basic sites that you can get through a data center, and then I've seen sites that, you know, are protected by Cloudflare, Akamai, you know, you name it, that you actually need to have a residential IP address, you need to have a real browser, and you need to seem like a real user. You know, if you click too fast on the page, if you scroll too fast, that could all trigger anti-bot. If you're blocking images, things like that.

Now, like I mentioned before, engineering has been a bottleneck until the AI space came here, when we, we had the AI agents ready, like, you know, able to build crawlers. And you'd have to actually have a deep understanding of the site. You know, um, a lot of sites, like I mentioned, because the layout is very different, it means that they'll try to make sure the data is h- as hard as possible to collect for you, and you'd have to have a deep level of domain knowledge expertise to get to that data.

Now, this is a diagram of how the traditional scraping architecture look like. So just to briefly go over this, we have, on the left-hand side, the, you know, the a- the actor, essentially what's triggering this crawl. So some people like using like schedulers, you know, um, cron jobs, you name it, where they want the data fed to them daily, weekly, et cetera, and that kicks off the job directly over to the next layer. Some people like doing, you know, specific config,

specific URLs. Same, uh, same idea right there. It goes r- and that goes right over to the proxy layer. Now, the proxy's essentially how you mask yourself from being detected. There's different types of proxies that you can use. You know, some static IP addresses like data center ISP, some rotating residential IP addresses that come from like laptops, tablets, TVs that are more authentic to sites because they can say, "Okay, this is registered to a actual device. We'll let them in." And then

some tools that have anti-bot capabilities built in, like what Bright Data offers, Web Unlocker or Browser API, which is in the browser automation side of th- of, of it, where the, the actual product itself manages, "Okay, is my request getting blocked? Am I gonna retry? Am I using the right IP?" That's all in a part of the layer when you think about the anti-bot side of things. Now, from there, then the next layer is, you know, parsing the data and actually validating if the data is, you know,

authentic. That's a big part of, you know, after you actually get to the site and collect the data, am I getting price back ninety percent of the time, and do I wanna validate that I'm getting it back ninety percent of the time? Do I need review data? Do I need ratings? You know, things like that. You have to think about, okay, like how do I ensure that I'm not gonna be setting up, when the data goes through my pipeline, that it's gonna be checked accurately by validation rules? Then we go into the post-processing

side of like the ingestion layer. So, you know, deduping records, enriching your data, pro- processing your batches. That is once you... You know, some people like delivering it to cloud storages. Some people like delivering it to their internal systems. Once it gets there, you know, what are you doing with the data? How do you ensure that you're not providing the same record twice or, you know, un- unclean data? So, you know, moving it from, in the medallion architecture, bronze, silver, and then finally moving it over to your database. And if anything

fails in that process, then you need to start the whole, uh, process over. And this is where I was mentioning, you know, if something breaks in the anti-bots layer or if it breaks and, you know, selector has changed, you'd need to start the whole process over, sometimes backfilling. You know, maybe you might miss SLAs and things like that.

And so here I'm gonna be talking about now specifically AI. Now, how many of you guys today are using things like, you know, Claude, GPT, any of the, uh, actual like AI brains? Okay, cool. How many of you guys have entered a query where it's, it's, you see it says, "Thinking," or, "Looking for an answer," or, "Searching the web"?

Okay, cool. Nice. So this is where you see, and I've split it up into two, where the left-hand side is where you see, okay, when the AI is actually thinking or searching the web. That's where it decides, "Okay, the query that you gave me, I actually need to go out and find the answer to this. I can't use my pre-trained data." So some prompts, like I'll give a basic example. "When's the best time to travel to San Francisco?" So if I say that, sometimes the, the, uh,

AI brain will usually go over to the RAG side, the Async, where it'll u- be used the pre-trained data that, you know, "Hey, maybe the best time to travel is during the summer." And it'll actually provide that back to you. Now, if I change my query and say, "When's the best time to travel this year to San Francisco?" Then the AI, the AI has to actually go out into the web and see, okay, like, do I need to look at what's going on? You know, is my user asking me for events that are going on? You know, how do I... Like, I need to make a

decision this year real-time to find out when I can recommend for them to travel. So these are two different paths that, um, each ag- the, each AI can actually make and provide you an output with. Some, some things I noticed is like responses sometimes can be cached. Like, you know, if someone's asking, "What's the weather today in N- in New York City?" like a few times a day, that might be a good response to cache because you only need one person to really ask that, and

you can, you know what the weather is going to be for the day, and you can just provide it. You know, so you don't have to burn through tokens or, you know, um, uh, like money to g- actually go out on the web and, and actually find that data.

Now, like I mentioned, there's, like, two paths to, uh, gather this intelligence. The sync path and then the async path. Both are advantageous for different cases. The sync path is, you know, if you really actually want to go out on the web and actually find what you're looking for, and async is if, you know, you-- it's okay to rely on pre-trained data. And both require different sort of setups. And we also do offer a tool, which I'm gonna cover in the next slide, um, where you can

actually use-- leverage Bright Data's tools to actually go out into the web and collect data. Now, one thing I actually noticed is if you go into any website, and if any of you guys are familiar with, like, robots.txt, a lot of sites usually block things like Claude bot and GPT bot. So if you ask these engines these questions, they're actually going out to the web and trying to find the data. They might not be able to get those data from those specific sites because they're actually blocking those, you know, GPT or Claude from actually

accessing the data. Now, that's where we sort of come in and 'cause what our advantageous is-- advantage is we're able to actually unlock the web. You know, whether it's if you have a scraper or if you have an agent or you're using, um, you know, something like Claude or GPT, we have tools that will allow it to actually go out i-into the web without actually getting blocked and scraping data, and we've been masters at this for, for a long time. Now, I'm gonna be

talking about the, the Web Model Context Protocol. That's what MCP means. So it usually integrates with something like Claude Code, LangChain, you know, name it. Like, you could think of it as like a brain that it needs to speak to, and that brain essentially connects to our MCP endpoint, which is essentially sixty plus APIs that we control in our library. And some examples you can see here are, you know, search engine. So, you know, Google results.

Scraping pages as markdown files and scraping pages as HTML. Now, if I want to target specific sites like, you know, Amazon, Reddit, LinkedIn, Instagram, that's all included as well. I actually-- As an example, I, I worked with someone two weeks ago that, you know, was trying to build a social media listener with the MCP, and one use case they said was, "Hey, um, this is my name. I wanna go out into the web and collect me all of my social

media profiles and r-return me how many followers I have." So what this did was he was using it on Claude Code. It made a call to our MCP using the search engine first, found the results, his pages on Instagram, TikTok, LinkedIn, brought that back, made the API request to our data pipeline, and then brought back the results on how many followers you have and gave him the context back. So that's the power of, you know, what this can do and how much of this-- the web the, the

MCP can unlock. And also this works for browser automations. Um, so, you know, creating things that you want to use a browser for, like clicking, scrolling. You know, maybe DoorDash, like automating an order or, um, automating, you know, on a travel site trying to find, you know, what's the best times to buy a fi- buy a flight, book a flight. Um, so things like that it's very advantageous for. And I'm gonna show you a little video on how the MCP works.

Let me bring... So on the left-hand side, you can see this is Claude with Bright Data's MCP set up, and on the right side, this is without Bright Data's MCP set up on Claude. So you can see it's the same prompt being asked, and it's use your browser to get the contents of this page, which is a Glassdoor URL. And on the left side, you can see with Bright Data's MCP, it actually went over to Glassdoor and got the results. But on

the right-hand side, it actually got blocked from going into Glassdoor. So this is one perfect example of, you know, sites like Glassdoor I've worked with on a, on a data collection level that's like a very protected site. And, you know, this is one great example on why the MCP is advantageous. You know, pairing it up with something like Claude or, you know, GPT or a-any sort of brains that, you know, you'd want to run it on.

Okay. So now I'm gonna be speaking about an example use case.

So working with c-- AI companies, these are examples of, you know, customers that I speak with that use Bright Data in any form of way. So, you know, training models, um, fine-tune enriching their, uh, enriching their models or fine-tuning their models, um, providing AI agents real-time access like, like we mentioned, or enriching RAG. So in this example, I'm gonna be speaking about specifically jobs, right? So let's say someone came to me and said, "Hey,

listen, I, I need job data, and I wanna train a model with jobs data." Like maybe resume matching, you know, as an example. So one thing we need to know is when you're thinking about jobs data, it's not like other-- you know, collecting other forms of data. You know, where it's I can pull in the data, and then I can just forget about it or train a model. You need continuous updates about jobs. You know which companies are posting what. You know how many jobs are available, what's the salary, you know, location, et cetera.

So this is why it's important that the data is actually structured and has a consistent output. The actual performance is very key as well because, you know, if you're building a model to analyze like job patterns or, you know, anything in the job market, you need that data flowing to you consistently. And you need-- also need this to be very reliable because if something breaks or, you know, if there's downtime, then that can actually affect the act-- the entire pipeline that you're-- you want to run. And

the timeline for this, you know, it's not only about getting data starting from today about jobs throughout. Some people need jobs data previously, you know, like the past few months or past few years. So that's why it's very important that, you know, you're covering all of these because if you wanna build a big picture on, you know, in this example for jobs on, you know, how do I find-- train my model on jobs data so I can find the best matches for this position? You need to understand how the trends

changed. You know, maybe salary has increased for software engineers over time, and you need to understand why, what the demand is, what's the current, you know, offering and, you know, you can tr-- you build your model around that. So this is, like, the example use case, like an AI agent for resume matching. So it's exactly what we discussed here. Um, so immediately you need real-time access to actually go through and, um, you know, scrape the data and be able to train this model along with it.

Okay. So I'm gonna be covering a hands-on exercise here. So I have a Google Notebook here, and if any of you guys are following along, there's two things that you require to actually run this script. One is a Bright Data API token, and one is a Gemini API key. And I have the list of instructions in the Google Drive on how to set up both. Um, and essentially I have everything set up here for you to

run. So I'll speak a bit about what we're doing here. So first we're calling Bright Data's API endpoint to collect Amazon data. We're collecting the response from Amazon, then calling the API for Reddit itself, capturing the raw Reddit data response, and then performing AI analysis over that. Now, before I run this, I wanna show you the exact data that we're getting from each of these APIs. So Amazon,

we're getting product-level data. So I picked a, a tumbler cup as an example, and essentially we have es-- every single data that-- about the product. So, you know, price, images, URLs, et cetera. And I, I'll show you this once it runs as well. My screen's a little small, so it might not show the entire thing, but... And we also have a Reddit API where what I'm going to do in this, uh, script that I

built is I'll actually run a keyword search for that specific tumbler cup, and I'm going to be outputting posts that, you know, talk about what, what, what's exactly, you know, is-- should I buy this or not? Is this worth it? Is this not worth it? What are people saying? So let's run this live.

All right, so we got our keys loaded here.

All right, so we got our Amazon data that came back. You can see it has price, rating, you know, other sellers that are selling it.

All of the actual data from the product itself, features, you know, you name it.

Now we'll go ahead and trigger a Reddit search

for the Stanley Quencher, um, tumbler, just having, you know, reviews about the actual product itself.

And while this is running, I will show you a bit about what I spoke about the model context protocol, the MCP servers. So we have different layers. I know I had talked about things like, you know, browser automation and search engine, but we also do cover things like, you know, e-commerce, social media, financial intelligence, research, business intelligence. So these are-- can all be plugged into an MCP endpoint, you know, app store travels,

and we are still adding to it. Currently, we have about, like, sixty-plus tools available to use for AI agents. Um, some people prefer actually using scrapers themselves, but we do-- this will provide-- allow your AI agent to choose which tools to use, so it'll be a lot more streamlined and easier.

While this is running, I can take any questions if anyone has.

You said that you can scrape through the resumes but not other, the other bots cannot do that. How can-- For most of the companies, you know, they get it by doing process drive forwards.

Yeah. So the question was, how come you guys can scrape resume-level data but, you know, bots can't do it, right? So- What, what gives Bright Data the advantage, and this is why we're able to do it where as, like, you know, things like bots aren't able to, is because we have the world's largest infrastructure, like proxy infrastructure. And where you're talking about, you know, a few requests, maybe like, you know, maybe a few days' worth of scraping pages, like, you know, Glassdoor as y- I use as an example, might work, right? But

let's say after a while, Glassdoor catches on. Like, "Hey, you know, he's using a data center IP. He's accessing the page." You know, nothing's actually being done on the page, and then they start adding all these layers of anti-bot into the picture. That's when companies like us, Bright Data, would come into the play because you need to actually not only own, like, a scraper or a bot, but you need to think about, "How do I bypass, you know, what the site is protected with, the captcha, and how do I do it consistently without triggering

the block?" So we've been doing it for years, and we have tools that bake into this anti-bot logic, where we're changing our approach to a site every time at scale. But sometimes, um, com- like, people that are starting off scraping might be doing it in-house and managing it. You have to also think that's also engineering time and resource time because trying to figure out what change did that site make also takes time, even probably more time than managing the scraper.

Okay, cool. Looks like it came back with data here, so I can show you. So it ha- it cau- captured a few Reddit posts about the actual tumbler itself, so some comments about, you know, what people were saying about the actual tumbler. It looks like people were saying it was embarrassing, and people were obsessed, but ... And then it gives the actual response here. So you could see, "Actually buy it.

It's on a decent discount." You know, "There's overwhelmingly positive reviews, but a few minor mixed opinions. There's high demand but low stock on Amazon," 'cause it returned that from the data. And, you know, a few other analysis points about the actual tumbler and the amount of reviews it has. It actually dissected all the data points that we returned from our API and gave you a decision on, "Yes, should I buy it? Should I not buy it?" Now, this is just one example of, you know, using data from ... starting from Amazon, running it on

Reddit, and then making a decision, but you can think of it at, think of this on, like, a bigger picture, you know? You can ... It doesn't have to be e-commerce. It can be any sort of use case, you know. As long as we ha- like, we have the tools that can allow you to go to the page and access the data. Now, it's up to you what logic and how you want to build that pipeline and build it at scale as well 'cause what one thing we do best is actually provide the data at scale.

Cool. I think we're, uh, able to wrap up, and if you guys, um, wanna c- if you have questions after, I'll be available or you could stop t- by at the Bright Data booth, but thank you for your time, guys.