AI Engineer World's Fair 2025

How to defend your sites from AI bots

Loading slides…

Read the talk

How to defend your site from AI bots

Effective bot defense starts with deciding which automation serves your users, then combines crawler verification, network signals, computational challenges and durable rate-limit keys.

From a talk by David Mytton

Before you start: Familiarity with HTTP requests, DNS and basic web application hosting will help you follow the implementation details.

What does an automated request cost your site?

What happens when bots visit your site at a scale your application was never designed to serve? David Mytton, who introduces himself as Arcjet’s founder, frames bot protection as something developers can implement themselves. He cites nearly 50% of web traffic overall and nearly 60% in gaming as automated, before widespread adoption of autonomous agents. Automation already includes both useful search crawlers and clients that consume resources without providing anything in return.

Slide titled “Up to 50% of internet traffic = bots” with stacked horizontal bars for industries and a legend for bad bot, good bot, and human traffic.Open full source frame
Bot and human traffic across industries.

A static page may be inexpensive to serve. A dynamic page can trigger database queries or other backend work on every request, with a serverless platform adding a per-request charge. Hundreds of thousands of automated requests turn that small unit cost into a substantial bill. Downloading large assets adds bandwidth costs and consumes capacity that legitimate users need. Eventually, the same pressure can make the service unavailable. Serverless shifts much of the capacity problem into billing; it does not make resources infinite.

0:160:29
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

0:16 · section reference included

Separate AI attribution from measurable traffic

A traffic spike does not establish that AI caused it. The more useful evidence identifies the clients involved or measures what changes when those clients are blocked. Mytton moves from general complaints to three concrete examples:

SiteReported observation
DiasporaMytton reports that GPTBot accounted for 24% of traffic.
Read the DocsTemporarily blocking Cloudflare-classified AI crawlers reduced downloaded-file bandwidth from about 800 GB/day to 200 GB/day.
WikipediaMytton cites roughly 35% bot traffic; the underlying measure is bot pageviews, not GPTBot’s share.

The Read the Docs incident report scopes its reduction to downloaded files, rather than total platform bandwidth. These examples expose different parts of the problem: a named crawler’s traffic share, the cost of bulk downloads, and the aggregate pressure from automation.

Slide titled “Is AI making it worse?” lists Diaspora: 24% of traffic = GPTBot; ReadTheDocs: 800GB/day to 200GB/day; Wikipedia: 35% of traffic = bots.Open full source frame
Traffic examples from Diaspora, ReadTheDocs, and Wikipedia.

The operational difference is how the content gets collected. A crawler that works gradually gives the site room to serve other users. A scraper that pulls content through hundreds of thousands of requests, without respecting the site’s rules, can overwhelm it regardless of why the scraper wants the data.

2:232:38
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

2:23 · section reference included

Decide which automation provides value

Search crawling has a familiar exchange: Googlebot visits and understands a site so that Google can index it, potentially sending visitors back. Bulk scrapers downloading images, content and files provide the contrasting case. AI crawlers sit between those categories because their value depends on what they do and what the site owner wants. Training collection does not necessarily produce referrals or any direct benefit for the source site.

Mytton distinguishes four kinds of OpenAI automation, beginning with search indexing. OAI-SearchBot collects content for ChatGPT search. Its role resembles a conventional search crawler: appearing in the index can produce citations, referrals and sign-ups. That gives a site owner a reason to allow it even when they do not want their content collected for training.

ChatGPT-User serves a different purpose: fetching content in response to a user’s request. Someone might paste a URL to summarize a page or ask for help implementing something from documentation. Mytton describes this retrieval as separate from training and notes that the response may not include a citation. Even so, blocking it can obstruct an existing user who has chosen an LLM as their interface.

GPTBot is the training crawler. Unlike search indexing, its collection does not promise a direct referral or citation. Keeping these purposes separate lets a site express a more useful policy than allowing or blocking every request associated with one AI provider.

Browser agents make the decision harder. An Operator-type agent can act for a person through a browser running in a virtual machine. Inbox triage in Gmail might be welcome; buying 500 concert tickets for resale probably is not. Both are automated actions on behalf of a user, but their effects on the service differ. Declared crawlers can identify themselves and undergo verification. In the demonstration’s framing, Operator instead arrives looking like Chrome. Whether automation is acceptable depends on the action, not merely the client’s name.

3:493:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

3:49 · section reference included

Start with robots.txt

robots.txt is a useful place to turn those decisions into crawler-specific instructions. For example, allowing search indexing while declining training collection can be expressed as:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

The file communicates policy; it does not enforce access. A crawler can still request a disallowed URL.

Mytton describes Googlebot and other cooperative search crawlers as following these instructions, and OpenAI as generally doing so. The distinction between crawling and user-directed retrieval matters here: OpenAI’s current crawler documentation says robots.txt may not apply to user-triggered actions. Hostile clients can ignore the file entirely—or use its disallowed paths to discover content worth fetching. Start here because it clarifies what you want each crawler to do, not because it protects private resources.

7:337:42
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

7:33 · section reference included

Verify the identity a crawler declares

The next signal is the User-Agent header. It commonly appears in request logs and provides a convenient starting point for classification. It is recommended, rather than mandatory, under HTTP Semantics. Many bots voluntarily identify themselves, so matching their declared names can still be productive.

Arcjet’s Well Known Bots catalog provides patterns and verification metadata for building those rules; Mytton describes a collection containing several thousand user agents. But the header is just a client-controlled string. A malicious client can claim to be Googlebot or Chrome as easily as it can give its own name. A declared identity is a classification hint, not authentication.

Verification checks the source behind that claim. Mytton names Apple, Bing, Google and OpenAI as providers whose crawlers can be verified. The procedure is provider-specific: Google supports reverse DNS followed by forward confirmation, while OpenAI’s current documentation publishes crawler IP lists. A reverse lookup alone is not a universal verification method.

The Googlebot slide demonstrates the round trip:

  1. Look up the request’s source IP address to obtain its hostname.
  2. Check that the hostname belongs to the provider’s documented crawler domain.
  3. Resolve that hostname and confirm that it includes the original source IP.

For the address shown on the slide, the shell lookups are:

bash

host 66.249.66.1
host crawl-66-249-66-1.googlebot.com

The displayed results map 66.249.66.1 to crawl-66-249-66-1.googlebot.com and back to the same address. Combining this source verification with the User-Agent lets an allow rule distinguish a desired crawler from a client merely borrowing its name.

“Defense 3/8: Verification” slide shows a terminal resolving 66.249.66.1 to crawl-66-249-66-1.googlebot.com, then resolving that hostname back to the same IP address.Open full source frame
Verifying a Googlebot address with reverse and forward DNS lookups.
8:408:55
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

8:40 · section reference included

Interpret network signals in application context

When declared identity and verification are insufficient, build a picture of normal behavior for individual IP addresses and address ranges. Bot detection is imperfect, so this is an accumulation of evidence rather than a single decisive test. Useful metadata includes network ownership, country, data-center origin and associations with particular operators. Mytton names MaxMind and IPinfo as sources, describing mostly paid databases alongside some free metadata APIs.

VPN, proxy, residential and mobile classifications add context. Mytton reports that AWS originated 12% of bot traffic reaching Cloudflare’s network in the year before the talk. That does not make every AWS request unwanted: a crawler visiting public pages may reasonably run in a data center. A data-center request submitting a form intended only for human sign-ups deserves different scrutiny. The same network attribute has different significance at different endpoints.

Country blocking is especially vulnerable to weak assumptions. Satellite and cellular connections can be geolocated to the IP owner rather than the person using the connection. Residential classification is also no guarantee of human activity: purchased proxy services route automated requests through home or mobile networks. Combine these attributes with observed behavior and retain local evidence about the traffic your application actually receives.

10:2710:39
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

10:27 · section reference included

Make abuse expensive without exhausting users

CAPTCHAs traditionally distinguish humans from automation by asking users to solve visual puzzles or manipulate objects. AI weakens that distinction: a model can interpret an image challenge, while an audio alternative can be downloaded and transcribed. Mytton describes these bypasses as cheap and fast, without presenting a task-specific benchmark.

Proof of work changes the question from whether the client can solve a human puzzle to whether it will spend computation before accessing a resource. The server requires a computational answer, shifting some cost onto the requester. Mytton illustrates the economics with a puzzle taking one or two seconds on an individual phone or laptop, repeated across tens of thousands or millions of sites. A small delay for one visitor can become a substantial aggregate CPU expense for a broad crawler.

That expense only deters an attacker when it changes the economics of the attack. In the concert-ticket example, paying a couple of dollars to bypass a challenge may still leave an attractive margin on a ticket resold for $200–$300. Signals such as an unverified IP address or suspicious request characteristics can justify harder puzzles, but increasing difficulty also penalizes legitimate users. Repeatedly refreshing a CAPTCHA that cannot be solved is an accessibility failure as well as a frustrating interaction.

For self-hosting, Mytton names Anubis, Go Away and Nepenthes as projects to consider in front of an application or in a Kubernetes deployment. He groups them under protective proxies and challenges; Anubis specifically documents configurable proof-of-work challenges. The deployment pattern places the protection layer before the application so that suspicious clients encounter a challenge before consuming the protected resource. The projects should not be assumed to implement identical mechanisms.

13:0113:13
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

13:01 · section reference included

Verify requests cryptographically

Cryptographic request attestation offers another way to establish who a client is. Mytton describes Cloudflare’s proposal to attach an HTTP message signature to an automated request so that the recipient can verify it quickly. At talk time, he characterizes the proposal as only weeks old and still developing, with an open question about how much it improves on source-IP verification. A verified identity still needs an associated policy deciding what that client may do.

Apple’s Private Access Tokens address a related problem: reducing CAPTCHA prompts through private attestation. Mytton introduces them through Privacy Pass and the Apple ecosystem. Apple’s actual requirements are supported software and a signed-in Apple ID, with checks on device legitimacy and account standing—not a paid iCloud subscription. Support includes WebKit and URLSession as well as Safari. Mytton describes limited website adoption at the time, making deployment and ecosystem support practical constraints alongside the cryptography.

15:3415:48
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

15:34 · section reference included

Connect requests across changing IP addresses

An IP address is an unstable basis for recognizing a client. Crawlers can rotate through large banks of addresses, particularly with IPv6, while retaining the same underlying client software. Fingerprinting extracts characteristics of that software’s network requests and turns them into a hash, allowing related traffic to be grouped even when its source address changes.

JA4 fingerprints TLS client characteristics. The HTTP counterpart identified in the companion material, JA4H, uses headers and other HTTP request characteristics. Their licensing differs: JA4 uses BSD 3-Clause, while JA4H’s published source uses a FoxIO license with monetization restrictions. The talk’s description of the HTTP version as proprietary should not be read as meaning its source is unavailable.

Once requests are grouped by fingerprint, a rule can act on their aggregate behavior. A client producing excessive traffic across many addresses can be blocked by its shared fingerprint instead of requiring a separate rule for every IP. A fingerprint describes client characteristics, however, not a permanent unique identity: similar stacks can share characteristics, and TLS-library updates can change the hash.

Rate limiting uses that grouping to impose a quota rather than necessarily blocking all access. The rate-limit key determines what the quota actually constrains. IP-only limits are vulnerable both to ordinary address changes and deliberate rotation.

KeyWhat the quota followsMain consideration
Source IPTraffic from one addressRotation splits activity across buckets.
Authenticated session IDActivity within a logged-in sessionRequires an authenticated session.
JA4 fingerprintTraffic sharing TLS client characteristicsGroups stacks, not unique people.

Mytton recommends a user session ID for logged-in traffic or a JA4 hash when fingerprinting supplies the grouping. The goal is to make changing addresses less effective as a way to escape the intended limit.

16:5617:17
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

16:56 · section reference included

Add defenses as the resource demands

The adoption order starts with a modest baseline: express policy through robots.txt, classify declared user agents, and verify the crawlers you intend to allow. Mytton argues that this combination is sufficient for most sites. More popular services, valuable content and scarce goods create stronger incentives to bypass that baseline, warranting additional network reputation checks, computational challenges and consideration of emerging request signatures. Fingerprinting combined with rate limits is his common destination for stronger protection.

These controls can live in application code rather than remaining an opaque infrastructure concern. Mytton’s implementation context is Arcjet, and he closes by pointing developers to a fuller writeup with examples. That companion now leads to the updated bot detection guide. The engineering task is to choose which clients may consume which resources, verify the signals supporting that decision, and enforce a cost or quota that matches the resource’s value.

18:4518:56
Suggest correction

This note stays in this page until you copy or download it. Nothing is submitted; reloading clears the draft.

18:45 · section reference included

Resources

From the talk

Updates since the talk

Transcript download
Full transcript

Loading transcript…