How AI answer engines find and cite your site — and the one text file that decides whether they can.
When a customer asks ChatGPT or Perplexity to recommend a business like yours, one of two things happens: your site shows up as a cited source, or it doesn’t exist. There’s no page two in an AI answer. For small businesses, nonprofits, and professional practices, that citation is quickly becoming as valuable as a first-page Google ranking — and a single misconfigured file on your server can quietly lock you out of it.
What actually controls whether AI engines can read your site?
Your robots.txt file — a plain text file at yourdomain.com/robots.txt — tells automated bots which parts of your site they may access. In 2026, AI companies run separate bots for training, search, and live user-requested fetches, so this file now decides whether ChatGPT, Gemini, Perplexity, and Claude can read and cite your pages.
It’s a request, not a lock. The standard (formalized as RFC 9309) works by good-faith compliance: the major crawlers from OpenAI, Anthropic, Google, and Perplexity all document their bots and generally honor robots.txt directives. But a directive only works if the bot reads and respects it, and the file is the first place these companies look.
The crawlers that matter in 2026
Each AI company now separates its bots by job. The distinction matters, because you can welcome the ones that drive citations while making a different choice about the ones that only collect training data.
- Training crawlers — GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended, CCBot. These collect content that may be used to train models.
- Search crawlers — OAI-SearchBot (OpenAI), Claude-SearchBot (Anthropic), PerplexityBot. These index your pages so you’re eligible to be cited in AI answers.
- User-action bots — ChatGPT-User, Claude-User, Perplexity-User. These fetch a page in real time when a person asks the assistant to read it.
Google-Extended is the odd one out: it’s a permission token, not a bot you’ll see in your logs. Disallowing it opts you out of Gemini training and grounding without affecting your Google Search ranking at all.
Can I allow AI search but block AI training?
Yes. Because training and search now run as separate bots, you can allow the search and user agents (OAI-SearchBot, Claude-SearchBot, PerplexityBot) so your content stays eligible for citation, while disallowing the training agents (GPTBot, ClaudeBot, Google-Extended) so it isn’t used to train models.
This is the nuanced middle ground most site owners want, and it’s the choice a large share of publishers are now making. There’s one important exception worth knowing: Google merges much of its search and AI crawling, so blocking Google’s AI use can also cost you search visibility. That’s a trade few businesses should make.
A welcome-mat robots.txt for AI visibility
If your priority is being found and cited (the right call for most local businesses and practices), the simplest safe configuration lists each crawler explicitly and allows it. Listing crawlers individually beats a single wildcard rule because it keeps full control over which engine sees what. Here’s a starting point:
# OpenAI
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
# Anthropic
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
# Perplexity
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
# Google AI (does not affect Google Search ranking)
User-agent: Google-Extended
Allow: /
Prefer to stay out of model training but keep citation eligibility? Swap Allow: / for Disallow: / under GPTBot, ClaudeBot, and Google-Extended only, and leave the search and user agents on Allow.
How to confirm your file is actually working
- Visit yourdomain.com/robots.txt in a browser. If nothing loads, you don’t have one — and the safe default is that compliant bots may crawl freely, which may or may not be what you want.
- Check your server or CDN logs for the bot tokens above. A user-agent string alone proves nothing — anyone can spoof it — so verify real crawlers against the vendor’s published IP ranges. Healthy requests to public pages return a 200 status; repeated 403s mean a firewall rule is blocking access you may have intended to allow.
- Watch for host-level surprises. Some hosts and CDNs now block AI crawlers on your behalf by default. That protects your bandwidth but can erase you from AI answers without you realizing — so confirm the decision is yours, not your host’s.
Why this belongs on your to-do list now
AI-driven crawling grew roughly 300% year over year heading into 2026, and automated requests now make up more than half of all web traffic, per Cloudflare’s network data. The businesses that get cited in AI answers over the next year will largely be the ones whose sites were readable when the assistants came looking. A ten-minute check of one text file is the lowest-effort, highest-leverage move you can make on AI visibility this quarter.
From invisible to citable: your next step
Pull up your robots.txt, match it against the configuration above, and decide deliberately which AI engines you want reading your content. If you’d rather not touch server files yourself — or you want to confirm your host isn’t quietly blocking crawlers — this is exactly the kind of quick, high-impact audit we handle for clients across the DC region. Reach out and we’ll check it for you.
