AI companies reach a website for four different jobs, and a single “AI bot traffic” number cannot tell you which one you are looking at.
- Search discovery. A crawler finds and indexes pages for a search product.
- User-triggered retrieval. An agent fetches a page while answering a specific person.
- Training or model improvement.A crawler gathers material under the provider's training controls.
- Ordinary search and utility crawling. Conventional indexes and platform services fetch the page and may indirectly support AI experiences.
A verified visit proves the provider could access that URL at that time. It does not prove the URL was indexed, retrieved for a later prompt, or cited in an answer.
Four events teams often mix up
| Event | What it means | Strong evidence | What it does not prove |
|---|---|---|---|
| Access | A crawler successfully fetched the URL. | Verified request, usable status, correct content. | Indexing, retrieval or citation. |
| Discovery / indexing | A system knows the page exists and may store it. | Provider console, index evidence or repeated verified discovery. | That the page will be used for a specific answer. |
| Retrieval | The page was selected or fetched for a query or agent action. | Provider trace, verified user-triggered fetch or answer-time evidence. | That the answer cited or relied on it. |
| Answer selection | The answer mentioned or cited the page or brand. | Saved answer, citation URL, prompt, engine, region and timestamp. | That a prior crawl alone caused the selection. |
Crawler and control reference
Provider names and policies change without notice. Verify against the linked documentation. Last checked .
| Provider | Agent or token | Operational role | What it does | Measurement note |
|---|---|---|---|---|
| OpenAI | OAI-SearchBot | Search discovery | Surfaces websites in ChatGPT's search results, summaries and links. | Verify against OpenAI's published ranges. |
| OpenAI | ChatGPT-User | User-triggered retrieval | Fetches a page because a person or a custom GPT asked for it. | OpenAI states robots.txt rules may not apply. A fetch is not a visible citation. |
| OpenAI | GPTBot | Training / improvement | Crawls content that may be used to train foundation models. | Keep entirely separate from search visibility. |
| Perplexity | PerplexityBot | Search discovery | Builds and refreshes Perplexity's search index. | Perplexity states it is not used for foundation-model training. Verify against published ranges. |
| Perplexity | Perplexity-User | User-triggered retrieval | Fetches pages in response to user actions. | Perplexity states these requests generally ignore robots.txt. Access is not citation. |
| Anthropic | Claude-SearchBot | Search discovery | Supports search quality and result discovery for Claude. | The role is distinct from training collection. |
| Anthropic | Claude-User | User-triggered retrieval | Fetches pages when a user asks Claude to access them. | Check the status and the exact path served. |
| Anthropic | ClaudeBot | Training / improvement | Collects material for model development under Anthropic's controls. | Apply a separate robots policy if the decision differs from search. |
| Googlebot | Search discovery | Builds Google Search's index, which can supply search and AI experiences. | Use Google's reverse-and-forward DNS method. | |
| Google-Extended | Control token | Controls specified Gemini training and grounding uses. | Not a separate HTTP user agent. It should not appear as a requester. | |
| Apple | Applebot | Search / platform discovery | Supports Apple search products and suggestions. | Verify using Apple's documented DNS method. |
| Apple | Applebot-Extended | Control token | Controls specified generative-AI uses of Applebot-crawled content. | It does not crawl as a separate bot. |
| Microsoft | Bingbot | Search discovery | Builds Bing's index, which can support Microsoft search experiences. | Verify using Bing's documented method. |
Primary documentation: OpenAI crawlers, Perplexity bots, Anthropic web crawlers, Google common crawlers, About Applebot.
Why a user-agent name is not proof
A user-agent header is text the requester supplies. Anyone can copy a crawler name into it. AirPulse has observed requests labelled PerplexityBot probing environment and configuration file paths, which is scanner behaviour, not search discovery.
Label unverified records user-agent-labelled. Upgrade to verifiedonly when the source passes the provider's documented method. Where no reliable method exists, classify as not verifiable rather than guessing the vendor.
How to verify a crawler visit
- Keep the raw timestamp, timezone, source IP, full user-agent, host, path, status and response content type.
- Identify the provider's official crawler documentation and the role assigned to that agent.
- Verify the source against published IP ranges or the provider's reverse-and-forward DNS method. OpenAI publishes ranges for OAI-SearchBot, ChatGPT-User and GPTBot; Perplexity publishes them for PerplexityBot and Perplexity-User; Google and Apple document DNS verification instead.
- Confirm the exact URL returned a usable response. A redirect loop, block page, 403, empty shell or 5xx is not successful access.
- Classify the event as verified, failed verification, or not verifiable.
- Correlate verified access with sitemap discovery, index evidence, prompt runs, citations and outcomes without claiming causality.
What happened on AirPulse's first cited research page
AirPulse examined the page How do I prove that a GEO or AEO change improved AI visibility? and the exact non-branded query with the same wording, over a frozen window. This is an observed sequence, not proof that a crawl caused the citation.
| Date | Observed event |
|---|---|
| Jul 27, 2026 | The page went live. The Research hub link, XML sitemap entry and llms.txt entry shipped in the same commit. |
| Jul 28, 2026 | The first exact-page user-agent-labelled crawler request appeared. GPTBot- and ClaudeBot-labelled requests fetched the sitemap. |
| Jul 29, 2026 | A PerplexityBot-labelled request fetched the Research hub. Applebot- and PetalBot-labelled requests fetched the exact page. |
| Jul 31, 2026 | A GoogleOther-labelled request fetched the exact page. A PerplexityBot-labelled request fetched the sitemap. |
| Aug 1, 2026 | Perplexity cited the page for the target query. The citation repeated across three consecutive daily jobs and appeared in five regional observations. |
In the window the page received nine successful main-page requests across seven user-agent-labelled crawler names: FacebookBot, Applebot, PetalBot, Bytespider, ChatGPT-User, GoogleOther and Meta-ExternalAgent. Some appeared more than once.
No record shows PerplexityBot or Perplexity-User fetching that exact page URL before the citation. PerplexityBot- labelled requests did reach the Research hub and the sitemap.
The page became discoverable and was later cited. The logs do not prove the route Perplexity used, and none of the user-agent-labelled events passed provider verification. The citation is reportable because it repeated across three consecutive daily jobs and five regional observations (repeated-run study). The full series, 16 frozen Perplexity observations with one later miss, is on How Stable Is a New AI Citation?
Can a page be cited without a visible crawler hit?
Yes. A missing exact-page log record does not mean the page was invisible. Possible explanations include:
- the provider already held the page in an index or cache;
- discovery happened through the Research hub, the sitemap, a link graph or a search partner;
- the answer system used a conventional search index rather than a dedicated AI crawler;
- the request came from infrastructure the classifier did not recognise;
- logs were sampled, expired, filtered or stored in a different layer;
- the cited source was selected from a search result without a new fetch reaching the origin.
A sitemap is a discovery aid, not a guarantee of crawling, indexing, retrieval or citation.
What the crawler log should retain
| Field | Why it matters |
|---|---|
| Timestamp and timezone | Reconstruct the sequence and match it to prompt runs. |
| Source IP and verification result | Separate verified providers from spoofed labels. |
| Full user-agent | Preserve the original signal so events can be reclassified later. |
| Host, path and query | Know exactly what was requested; redact sensitive values in reporting. |
| Status and content type | Distinguish successful HTML from redirects, errors and assets. |
| Bytes and latency | Spot empty responses, truncation and operational failures. |
| Referer and cache indicators | Understand routing and whether the origin served the request. |
| robots.txt and WAF decision | Explain allow, block, challenge and rate-limit outcomes. |
| Parser and verifier version | Make historical classifications reproducible. |
Collect this at the origin server, CDN or edge. Analytics platforms filter bot traffic out before you see it.
Set robots policy by purpose
Not one “AI bots” switch. Decide separately whether to allow:
- search discovery crawlers that make pages eligible for search-backed answers;
- user-triggered agents that fetch a page on someone's request;
- training or model-improvement crawlers;
- ordinary search crawlers that support existing search visibility.
robots.txt is a crawler preference, not access control. The user-triggered agents may not follow it. Use authentication, network controls and rate limits for anything sensitive. Google-Extended and Applebot-Extended are control tokens, not user agents that should appear in logs.
Use a five-layer measurement model
| Layer | Question | Recommended output |
|---|---|---|
| 1. Access | Could the provider fetch the page? | Verified successful requests by URL and role. |
| 2. Discovery | Did the system learn that the page exists? | Sitemap and hub fetches, index evidence, discovery timestamps. |
| 3. Retrieval | Was the page selected for a prompt or agent action? | Answer-time fetches, traces or provider evidence. |
| 4. Answer | Was the page or brand mentioned or cited? | Saved answer, citation URL, prompt, engine, region and time. |
| 5. Outcome | Did visibility create value? | Qualified visits, conversions, pipeline or support deflection. |
Layers 1 and 2 come from your logs. Layers 3 and 4 come from repeated prompt runs with stored answers and cited URLs; the rules are in the measurement methodology and the prompts are chosen with the prompt-selection method.
What not to report
- “Perplexity crawled us 43 times” when the count comes only from a user-agent regex.
- “The page is indexed” because a crawler fetched the sitemap.
- “The model used our content” because the server returned 200.
- “This crawl caused the citation” because it happened earlier in the timeline.
- “Crawler traffic equals AI traffic” when bot requests and human referrals are blended.
- One combined crawler score that mixes search, user retrieval, training and ordinary search.
A practical audit workflow
- Choose the priority pages and the exact non-branded questions they answer.
- Confirm each page is public, indexable, canonical and present in the hub and sitemap.
- Collect raw request evidence at the server or edge, before analytics filters remove bots.
- Classify each agent by operational role using current provider documentation.
- Verify source identity with official IP or DNS methods wherever one is published.
- Report successful access separately from discovery, retrieval, answer and outcome evidence.
- Run the same prompts repeatedly across engines and regions; save the full answer and the cited URLs.
- Review deltas at fixed checkpoints and fix the weakest layer first.
To test whether one published page moved AI visibility, this workflow is the access-layer half of the first-citation trial protocol. Fix access before rewriting the article. A page that returns 403 to a search crawler is failing an access test, not a content test.
Crawler audit worksheet
The worksheet mirrors this page: fields to retain per request, role and verification method per agent, the verified / failed / not-verifiable classification, the five layers, the four robots decisions and a policy review date. No customer data.