How to verify real AI bots in your server logs

A line in your server log that says GPTBot or Googlebot tells you what the visitor called itself. It does not tell you who the visitor was. The name comes from the user-agent string, a piece of text the visitor's software sends with each request, and anyone can type any name into it.

To confirm a real AI bot, check the IP address the request came from. Vendors document two ways to do that. Some publish a list of the IP ranges their crawlers use. Some support a reverse DNS lookup, where you ask which hostname an IP address belongs to and then confirm the hostname points back to the same address. As of October 2026, Google, Bing, Apple and Common Crawl document both methods. OpenAI, Anthropic, Perplexity and Amazon publish IP lists only. Meta's crawler page publishes neither.

This guide gives each vendor's method as documented, a routine to follow, and what log data can and cannot tell you.

Why the user-agent string is not proof

Every request to your site carries a user-agent string. Browsers send one that names the browser. Crawlers send one that names the bot, built around a short token such as OAI-SearchBot or PerplexityBot. Our guide to AI crawlers and robots.txt lists every token and what each bot is for.

Because the string is plain text supplied by the visitor, scrapers borrow well-known names to get past filters. The operators say so themselves. Bing's crawler page warns that "user agent strings are easy to spoof, so not every request with these user agent strings may be coming from a real Bing crawler." Google says verification is useful "if you're concerned that spammers or other troublemakers are accessing your site while claiming to be from Google."

The IP address is harder to fake, because the response has to travel back to it. That is why every documented verification method starts from the IP address in your log, not the name.

The two verification methods

Matching against a published IP list

The vendor hosts a file, usually in JSON format, listing the address ranges its crawlers use. The ranges are written in CIDR notation, a compact way of writing a block of addresses, such as 192.0.2.0/24. If the IP in your log falls inside one of the ranges, the request came from that vendor's network. Your developer, host or firewall can do the comparison automatically.

Reverse and forward DNS

Google's verification page describes the manual method in four steps, which it says "is sufficient for most use cases."

1. "Run a reverse DNS lookup on the accessing IP address from your logs, using the host command."
2. "Verify that the domain name is either googlebot.com, google.com, or googleusercontent.com."
3. "Run a forward DNS lookup on the domain name retrieved in step 1 using the host command on the retrieved domain name."
4. "Verify that it's the same as the original accessing IP address from your logs."

In Google's example, host 66.249.66.1 returns crawl-66-249-66-1.googlebot.com, and host crawl-66-249-66-1.googlebot.com returns 66.249.66.1. The second lookup matters. Whoever controls an IP address can set its reverse name to anything, so a hostname alone proves nothing until it resolves back to the same address.

What each vendor publishes

Google

Google sorts its crawlers into three groups, each with its own hostnames and IP files. The files are linked from its verification page.

For anything else arriving from Google, such as Apps Script, Google says to match the address against its general list of Google IP addresses.

OpenAI, Anthropic and Perplexity

OpenAI. The only verification method in OpenAI's crawler documentation is a set of IP files, one per agent: openai.com/searchbot.json for OAI-SearchBot, openai.com/gptbot.json for GPTBot, openai.com/chatgpt-user.json for ChatGPT-User and openai.com/adsbot.json for OAI-AdsBot. Having one file per agent lets you check that a request labeled as the search crawler came from the search crawler's ranges. OpenAI also notes that when its bots fetch robots.txt they may add a robots.txt marker to the user-agent string, which helps "especially when their logs do not include paths."

Anthropic. Anthropic's crawler article links one list, at claude.com/crawling/bots.json, and says: "If a crawler has a source IP address on this list, it indicates that the crawler is coming from Anthropic." The list is not split between ClaudeBot, Claude-User and Claude-SearchBot, so an IP match tells you it was Anthropic, not which bot. Anthropic also says that blocking its IP addresses "may not work correctly or persistently guarantee an opt-out, as doing so impedes our ability to read your robots.txt file."

Perplexity. Perplexity's bot documentation gives www.perplexity.com/perplexitybot.json for PerplexityBot and www.perplexity.com/perplexity-user.json for Perplexity-User. Its advice is to "combine both User-Agent string matching and IP address verification." It adds: "Always use the most current IP ranges from the official JSON endpoints."

Bing, Apple, Amazon, Meta and Common Crawl

Bing. Microsoft offers three routes. The reverse DNS method is to confirm that the IP "resolves to a name that end with search.msn.com" and that the name resolves back to the same IP. There is a Verify Bingbot tool in Bing Webmaster Tools under Tools & Enhancements, also offered as a public page, where you paste an IP address. And there is a published list of Bingbot IP addresses, with the instruction: "Refresh this list daily as we can change this list any time." One caution: the file's own creation date reads January 2024, so use the tool or reverse DNS as a cross-check.

Apple. Apple's Applebot page says traffic "is generally identified by using reverse DNS in the *.applebot.apple.com domain," and that you can also match the IP against its published file of CIDR prefixes.

Amazon. Amazon publishes three IP address pages on its developer site, one each for Amazonbot, Amzn-SearchBot and Amzn-User. No reverse DNS method is described.

Meta. As of October 2026, Meta's web crawlers page publishes no IP range file and no reverse DNS method. Its only reference to IP addresses is advice, for its link-preview crawler, to allow "either the user agent strings or the IP addresses (more secure) used by the crawler," and the page does not say where those addresses are listed. Until Meta publishes them there, a request labeled meta-externalagent cannot be verified by a method that page documents.

Common Crawl. CCBot is "run on dedicated IP address ranges with reverse DNS (except over IPv6 where reverse DNS is not yet supported.)" Its example IP resolves to a name ending in crawl.commoncrawl.org, and the ranges are also published as a JSON file.

Tokens you will never see in a log

Two names from robots.txt discussions have no traffic of their own. Google says "Google-Extended doesn't have a separate HTTP request user agent string. Crawling is done with existing Google user agent strings." Apple says Applebot-Extended "does not crawl webpages." Both are switches that control how already-crawled content may be used. If a report claims to show visits from either, the report is wrong.

A routine for checking your logs

Log formats vary by server and host, so we will not show a sample line as if it were standard. What you need from each entry is the same everywhere: the IP address, the date and time, the path requested, the status code returned and the user-agent string.

1. Get the right log. If your site sits behind a CDN or firewall, a request blocked there never reaches your server, so your server log will not show it. Use the CDN's log or analytics too.
2. Filter by token. Pull the entries whose user-agent contains the tokens you care about.
3. List the distinct IP addresses for each token.
4. Verify each IP with the method in the table above. Download the vendor's list that day.
5. Sort the results into verified and unverified. Unverified requests using a famous bot's name are imitations or cannot be confirmed.
6. Read the status codes for the verified requests. A run of 403 or 429 responses to a search crawler you want means something is turning it away. Our guide to CDNs and firewalls blocking AI crawlers covers the usual causes.
7. Note which pages were fetched. Important pages that are never requested may be hard to discover.

Some platforms do this work for you. Cloudflare and Vercel both describe verifying bots by published IP list or reverse DNS. Relying on that is a reasonable shortcut, as long as you know which method the label rests on.

What log data tells you and what it doesn't

The JavaScript row is its own topic; see our guide to JavaScript rendering and AI crawlers. For clicks from assistants to your site, logs are the wrong tool. That is analytics work, covered in how to track AI traffic in GA4 and our explainer on the AI Assistant channel in GA4. Browser-based analytics has the opposite blind spot: Google notes that client-side analytics may not give a full or accurate picture of Googlebot activity, and points to the crawl stats report in Search Console.

Conclusion

Verifying AI bots comes down to one habit: trust the IP address, not the name. Use reverse and forward DNS where the operator supports it, which today means Google, Bing, Apple and Common Crawl. Use the published IP files for OpenAI, Anthropic, Perplexity and Amazon, fetched fresh each time. Accept that Meta's crawler page documents no verification method as of October 2026. Then read the verified entries for what they are good for, which is confirming access and spotting blocks, and do not read them as proof of citations. Log review is one item on a full technical audit checklist for GEO. If you would like help making sense of what your logs show, request a free GEO audit. More guides are in our GEO and AEO Guide.

Frequently asked questions