AI companies reach a website for several different reasons. One bot builds a search index. Another fetches a page because a person asked a question. A third collects material for model improvement. Ordinary search crawlers also supply pages that later appear in an AI answer. Those are four different jobs, and a single “AI bot traffic” number cannot tell you which one you are looking at.
The short answer
Most crawler activity falls into four useful groups:
- Search discovery: a crawler finds and indexes pages for a search product.
- User-triggered retrieval: an agent fetches a page while answering a specific person.
- Training or model improvement: a crawler gathers material under the provider's training controls.
- Ordinary search and utility crawling: conventional indexes and platform services fetch the page and may indirectly support AI experiences.
A successful, verified visit proves that the provider could access that URL at that time. It does not prove that the URL was indexed, that it was retrieved for a later prompt, that it was cited in an answer, or that it produced a business result.
Four events teams often mix up
| Event | What it means | Strong evidence | What it does not prove |
|---|---|---|---|
| Access | A crawler successfully fetched the URL. | Verified request, usable status, correct content. | Indexing, retrieval or citation. |
| Discovery / indexing | A system knows the page exists and may store it. | Provider console, index evidence or repeated verified discovery. | That the page will be used for a specific answer. |
| Retrieval | The page was selected or fetched for a query or agent action. | Provider trace, verified user-triggered fetch or answer-time evidence. | That the answer cited or relied on it. |
| Answer selection | The answer mentioned or cited the page or brand. | Saved answer, citation URL, prompt, engine, region and timestamp. | That a prior crawl alone caused the selection. |
Crawler and control reference
Provider names and policies change without notice. Use this as an operational map, then verify against the linked provider documentation. This table was last checked against those sources on .
| Provider | Agent or token | Operational role | What it does | Measurement note |
|---|---|---|---|---|
| OpenAI | OAI-SearchBot | Search discovery | Surfaces websites in ChatGPT's search results, summaries and links. | Verify against OpenAI's published ranges. |
| OpenAI | ChatGPT-User | User-triggered retrieval | Fetches a page because a person or a custom GPT asked for it. | OpenAI states robots.txt rules may not apply. A fetch is not a visible citation. |
| OpenAI | GPTBot | Training / improvement | Crawls content that may be used to train foundation models. | Keep entirely separate from search visibility. |
| Perplexity | PerplexityBot | Search discovery | Builds and refreshes Perplexity's search index. | Perplexity states it is not used for foundation-model training. Verify against published ranges. |
| Perplexity | Perplexity-User | User-triggered retrieval | Fetches pages in response to user actions. | Perplexity states these requests generally ignore robots.txt. Access is not citation. |
| Anthropic | Claude-SearchBot | Search discovery | Supports search quality and result discovery for Claude. | The role is distinct from training collection. |
| Anthropic | Claude-User | User-triggered retrieval | Fetches pages when a user asks Claude to access them. | Check the status and the exact path served. |
| Anthropic | ClaudeBot | Training / improvement | Collects material for model development under Anthropic's controls. | Apply a separate robots policy if the decision differs from search. |
| Googlebot | Search discovery | Builds Google Search's index, which can supply search and AI experiences. | Use Google's reverse-and-forward DNS method. | |
| Google-Extended | Control token | Controls specified Gemini training and grounding uses. | Not a separate HTTP user agent. It should not appear as a requester. | |
| Apple | Applebot | Search / platform discovery | Supports Apple search products and suggestions. | Verify using Apple's documented DNS method. |
| Apple | Applebot-Extended | Control token | Controls specified generative-AI uses of Applebot-crawled content. | It does not crawl as a separate bot. |
| Microsoft | Bingbot | Search discovery | Builds Bing's index, which can support Microsoft search experiences. | Verify using Bing's documented method. |
Primary documentation: OpenAI crawlers, Perplexity bots, Anthropic web crawlers, Google common crawlers, About Applebot.
Why a user-agent name is not proof
A user-agent header is text supplied by the requester. Anyone can copy a known crawler name into it. AirPulse has observed requests labelled PerplexityBot attempting paths such as environment and configuration files. Those requests are consistent with scanners spoofing a trusted name, not with a search crawler doing normal discovery.
So label unverified records as user-agent-labelled. Upgrade an event to verified only when the source passes the provider's documented verification method. Where a provider publishes no reliable method, the honest classification is not verifiable, rather than a guess at the likely vendor.
How to verify a crawler visit
- Keep the raw timestamp, timezone, source IP, full user-agent, host, path, status and response content type.
- Identify the provider's official crawler documentation and the role assigned to that agent.
- Verify the source against published IP ranges or the provider's reverse-and-forward DNS method. OpenAI publishes ranges for OAI-SearchBot, ChatGPT-User and GPTBot; Perplexity publishes them for PerplexityBot and Perplexity-User; Google and Apple document DNS verification instead.
- Confirm the exact URL returned a usable response. A redirect loop, block page, 403, empty shell or 5xx is not successful access.
- Classify the event as verified, failed verification, or not verifiable. Do not force every event into a provider bucket.
- Correlate verified access with sitemap discovery, index evidence, prompt runs, citations and outcomes — but do not claim causality without stronger evidence.
What happened on AirPulse's first cited research page
AirPulse examined the research page How do I prove that a GEO or AEO change improved AI visibility? and the exact non-branded query with the same wording. What follows is an observed sequence in AirPulse data over a frozen window. It is not proof that any single crawl caused the citation.
| Date | Observed event |
|---|---|
| Jul 27, 2026 | The page went live. The Research hub link, XML sitemap entry and llms.txt entry shipped in the same commit. |
| Jul 28, 2026 | The first exact-page user-agent-labelled crawler request appeared. GPTBot- and ClaudeBot-labelled requests fetched the sitemap. |
| Jul 29, 2026 | A PerplexityBot-labelled request fetched the Research hub. Applebot- and PetalBot-labelled requests fetched the exact page. |
| Jul 31, 2026 | A GoogleOther-labelled request fetched the exact page. A PerplexityBot-labelled request fetched the sitemap. |
| Aug 1, 2026 | Perplexity cited the page for the target query. The citation repeated across three consecutive daily jobs and appeared in five regional observations. |
The aggregate: during the observed window the page received nine successful main-page requests across seven user-agent-labelled crawler names — FacebookBot, Applebot, PetalBot, Bytespider, ChatGPT-User, GoogleOther and Meta-ExternalAgent. Some names appeared more than once.
The important gap: AirPulse did not find an aggregate record of PerplexityBot or Perplexity-User fetching that exact page URL before the citation. It did observe PerplexityBot-labelled requests to the Research hub and to the sitemap.
The safe conclusion: the page became discoverable and was later cited. The logs do not prove the route Perplexity used to find or retrieve it, and none of these user-agent-labelled events has passed provider verification. The citation is reportable because it repeated: three consecutive daily jobs and five regional observations rather than a single screenshot. That is the standard set out in the repeated-run study.
Can a page be cited without a visible crawler hit?
Yes. A missing exact-page log record does not mean the page was invisible. Possible explanations include:
- the provider already held the page in an index or cache;
- discovery happened through the Research hub, the sitemap, a link graph or a search partner;
- the answer system used a conventional search index rather than a dedicated AI crawler;
- the request came from infrastructure the classifier did not recognise;
- logs were sampled, expired, filtered or stored in a different layer;
- the cited source was selected from a search result without a new fetch reaching the origin.
A sitemap is a discovery aid, not a guarantee of crawling, indexing, retrieval or citation.
What the crawler log should retain
| Field | Why it matters |
|---|---|
| Timestamp and timezone | Reconstruct the sequence and match it to prompt runs. |
| Source IP and verification result | Separate verified providers from spoofed labels. |
| Full user-agent | Preserve the original signal so events can be reclassified later. |
| Host, path and query | Know exactly what was requested; redact sensitive values in reporting. |
| Status and content type | Distinguish successful HTML from redirects, errors and assets. |
| Bytes and latency | Spot empty responses, truncation and operational failures. |
| Referer and cache indicators | Understand routing and whether the origin served the request. |
| robots.txt and WAF decision | Explain allow, block, challenge and rate-limit outcomes. |
| Parser and verifier version | Make historical classifications reproducible. |
Collect this at the origin server, CDN or edge. Analytics platforms filter bot traffic out before you see it, and bot traffic is what this audit is made of.
Set robots policy by purpose
Do not use one vague “AI bots” switch. Decide separately whether you want to allow:
- search discovery crawlers that make pages eligible for search-backed answers;
- user-triggered agents that fetch a page on someone's request;
- training or model-improvement crawlers;
- ordinary search crawlers that support existing search visibility.
robots.txt is a crawler preference, not an access-control system. Two of the agents above may not follow it at all, because the fetch is user-initiated. Use authentication, authorisation, network controls and rate limits for anything sensitive. Remember also that Google-Extended and Applebot-Extended are control tokens, not separate HTTP user agents that should appear as requests in server logs.
Use a five-layer measurement model
| Layer | Question | Recommended output |
|---|---|---|
| 1. Access | Could the provider fetch the page? | Verified successful requests by URL and role. |
| 2. Discovery | Did the system learn that the page exists? | Sitemap and hub fetches, index evidence, discovery timestamps. |
| 3. Retrieval | Was the page selected for a prompt or agent action? | Answer-time fetches, traces or provider evidence. |
| 4. Answer | Was the page or brand mentioned or cited? | Saved answer, citation URL, prompt, engine, region and time. |
| 5. Outcome | Did visibility create value? | Qualified visits, conversions, pipeline or support deflection. |
Layers 1 and 2 come from your own logs. Layers 3 and 4 come from repeated prompt runs with stored answers and cited URLs — the calculation rules are in the measurement methodology, and the prompts they run against are chosen using the prompt-selection method.
What not to report
- “Perplexity crawled us 43 times” when the count comes only from a user-agent regex.
- “The page is indexed” because a crawler fetched the sitemap.
- “The model used our content” because the server returned 200.
- “This crawl caused the citation” because it happened earlier in the timeline.
- “Crawler traffic equals AI traffic” when bot requests and human referrals are blended.
- One combined crawler score that mixes search, user retrieval, training and ordinary search.
A practical audit workflow
- Choose the priority pages and the exact non-branded questions they answer.
- Confirm each page is public, indexable, canonical and present in the hub and sitemap.
- Collect raw request evidence at the server or edge, before analytics filters remove bots.
- Classify each agent by operational role using current provider documentation.
- Verify source identity with official IP or DNS methods wherever one is published.
- Report successful access separately from discovery, retrieval, answer and outcome evidence.
- Run the same prompts repeatedly across engines and regions; save the full answer and the cited URLs.
- Review deltas at fixed checkpoints and fix the weakest layer first.
If the goal is to test whether one published page moved AI visibility rather than to inventory crawler traffic, this workflow is the access-layer half of the first-citation trial protocol. Fix technical access before rewriting the article. A page that returns 403 to a search crawler is failing an access test, not a content test.
Crawler audit worksheet
The downloadable worksheet mirrors this page: the fields to retain per request, the operational role and verification method per agent, the verified / failed / not-verifiable classification, the five measurement layers, the four separate robots decisions and a provider-policy review date. It contains no customer data.