# Most of Your 'AI Crawler' Traffic Isn't AI We checked two days of Cloudflare logs for our own site: roughly six in ten requests wearing an AI crawler name were vulnerability scanners probing for .env files and keys. How to tell them apart, two free Cloudflare rules that block them, and the one mistake to avoid. Everyone wants to know whether AI systems are reading their site. So when we connected our own marketing site to the Cloudflare integration and asked the AI which pages the crawlers were fetching, we expected a list of blog posts. We got a list of secrets. Over two days, Cloudflare recorded just over a thousand requests to the site from user agents on our list of known AI crawlers: GPTBot, ClaudeBot, PerplexityBot, Applebot, Amazonbot, Bytespider and the rest. Roughly six in ten of those requests were for paths like these: ``` /.env /.env.production /.ssh/id_rsa /.aws/credentials /firebase-admin.json /terraform.tfstate /.docker/config.json /@fs/proc/self/environ ``` Nothing on that list exists on a static marketing site, and no AI system has any reason to ask for it. These are vulnerability scanners hunting for leaked credentials, and every one of them was announcing itself as GPTBot, or ClaudeBot, or Perplexity, or Apple. The same probe for `/.env` arrived under eight different crawler names in 48 hours. The actual AI traffic was a small minority: ClaudeBot and GPTBot reading `robots.txt` and the sitemap, ChatGPT and Perplexity fetching a handful of recent articles because someone had asked about them, and Applebot pulling a few stylesheets. That is the number we cared about, and it was buried under scanner noise. ## Why Scanners Pretend to Be Crawlers A user agent is just a string the client sends. Anyone can send any value, and scanners have always borrowed respectable names: Googlebot for years, and now the AI crawlers, for the same two reasons. First, site owners allow them. A firewall that blocks unknown bots but waves through anything called GPTBot is a firewall with a password written on the door. Second, they hide in the crowd. AI crawler traffic has grown enough that a few hundred extra requests under a crawler name look normal in a log, where the same requests from an anonymous Python client would stand out. The result is that any report counting "requests from AI crawlers" by user agent alone is inflated, sometimes by several times, and the inflation lands on exactly the question you were trying to answer. ## How to Tell Them Apart Three signals separate a real crawler from a scanner wearing its name. You do not need all three. **Verification.** The major AI operators publish the IP ranges their crawlers use, and Cloudflare checks every request claiming a known bot name against them. A request that claims to be GPTBot and does not come from OpenAI's ranges fails that check. Cloudflare exposes the result as a verified-bot flag you can use in firewall rules on every plan, including Free. **The response code.** A real crawler fetches pages that exist. A scanner asks for files that do not, so almost all of its requests return 404. On a static site, the scanner share of your "AI crawler" traffic is very close to the share that returned an error. **The path.** You know what your site contains. Requests for `.env`, `.ssh`, `.git`, `.aws`, key files and credential JSON are not reconnaissance by an AI, whatever the user agent says. The second and third signals are also the ones you can check without any tooling: open your server or CDN logs, filter to the crawler user agents, and look at the status codes and paths. ## Blocking Them in Cloudflare on the Free Plan If your site is behind Cloudflare, two custom firewall rules deal with this. Both work on the Free plan, which allows five custom rules. Both are written in the free-plan syntax: the `matches` regex operator needs a Business plan, so these use `contains` and the `ends_with()` function instead. **Rule 1 blocks spoofed crawlers.** It fires when a request claims one of the big crawler names but fails Cloudflare's verification. ``` (http.user_agent contains "GPTBot" or http.user_agent contains "ClaudeBot" or http.user_agent contains "OAI-SearchBot" or http.user_agent contains "PerplexityBot" or http.user_agent contains "Amazonbot" or http.user_agent contains "Applebot" or http.user_agent contains "Bytespider" or http.user_agent contains "CCBot" or http.user_agent contains "Meta-ExternalAgent" or http.user_agent contains "YouBot" or http.user_agent contains "cohere-ai") and not cf.client.bot and http.host eq "example.com" ``` Action: Block. Replace the host with your public site's hostname. **Rule 2 blocks credential probes from anyone**, since scanners also arrive with ordinary browser user agents. ``` (lower(http.request.uri.path) contains "/.env") or (lower(http.request.uri.path) contains "/.ssh/") or (lower(http.request.uri.path) contains "/.git/") or (lower(http.request.uri.path) contains "/.aws/") or (lower(http.request.uri.path) contains "/@fs/") or (lower(http.request.uri.path) contains "/proc/self/") or (lower(http.request.uri.path) contains "credentials") or (lower(http.request.uri.path) contains "service_account") or (lower(http.request.uri.path) contains "service-account") or (lower(http.request.uri.path) contains "firebase-admin") or (lower(http.request.uri.path) contains "terraform.tfstate") or ends_with(lower(http.request.uri.path), ".pem") or ends_with(lower(http.request.uri.path), ".key") or ends_with(lower(http.request.uri.path), ".s3cfg") or ends_with(lower(http.request.uri.path), ".htpasswd") or ends_with(lower(http.request.uri.path), ".bashrc") or ends_with(lower(http.request.uri.path), ".zshrc") ``` Action: Block. Paste both into the "Edit expression" view rather than the visual builder, which does not offer the functions. If your site legitimately serves a path that one of these lines matches, remove that line. ### The Mistake We Made Our first version of rule 1 did not have the host condition, and it had `Claude-User` and `ChatGPT-User` in the list. Within the hour, every customer who had connected Claude to our app through our MCP endpoint was getting a 403. Two lessons in that. The AI *assistant* user agents, the ones ending in `-User` plus `Meta-ExternalFetcher` and `DuckAssistBot`, are not the training crawlers. They are live fetches made because a person just asked an AI about you, and some of them are also what AI products send when they call an API on a user's behalf. That is traffic you want, and not all of it passes verification. Leave those names out of rule 1 and let rule 2 catch scanners who borrow them. And scope the rule to your public website's hostname. If the same Cloudflare zone also fronts an app, an API or an integration endpoint, a bot rule written for the marketing site will quietly break them. ## Checking Your Own Site If you want the numbers rather than a feeling, connect Cloudflare to AI Data Stream and ask in chat: ``` Which pages did AI crawlers fetch on the site in the last 7 days, and how many of those requests failed? ``` As of this week the answer separates the two on its own. The Cloudflare tools and the dashboard widget count successful fetches as crawler requests and report 4xx and 5xx responses as a separate failed total, with a note about what failed requests on a static site usually are. To see exactly what the scanners asked for: ``` Show me the AI crawler paths with status 400 and above. ``` Run both before and after adding the rules and the second list should go quiet, while the first one finally shows what AI systems are actually reading. ## The Fine Print - **Crawler identification is still by user agent** unless you pay for Cloudflare's Bot Management. Bot Fight Mode, the free toggle, is a different product: it challenges bots but does not label them in your analytics. The verified-bot flag is the one piece of Bot Management available to everyone, which is why rule 1 can lean on it. - **Blocked requests are still logged**, with a 403, so a tool that counts raw requests by user agent will keep counting them. Counting successful fetches is what makes the number honest. - **Not every AI operator can be verified.** Cloudflare only verifies bots whose operators publish IP ranges and behave, and it has removed some from the list. Check the current verified bots list in your dashboard before adding a name to rule 1. Cloudflare is available on **Pro and Business plans**. Add it from your property's **Connections** page, or read the [connection documentation](/docs/connecting-data-sources/#cloudflare).