A growing share of the people who might hire you never scroll through a list of links anymore. They ask ChatGPT, Perplexity, or Google’s AI Overviews a question, and the assistant reads a handful of websites, then hands back a synthesized answer with a citation or two. If the AI systems can’t actually reach or parse your website, you’re not in the running for that answer — no matter how good your business is.
Getting found by AI search happens in two layers. The first is access: can the AI’s crawlers even get to your pages? The second is readability: once they arrive, can they make sense of what’s there? This post covers the access layer — the technical checklist that makes sure your site is reachable in the first place. A companion post covers the content side: what makes AI engines actually quote you once they can read you.
Meet the crawlers that are already knocking
Every major AI company runs its own bots, and most of them split their crawlers by purpose — one for training future models, a separate one for real-time search retrieval, and sometimes a third for on-demand page fetches when a user asks the assistant to look at a specific page. That split matters, because blocking the wrong one costs you differently than blocking the right one.
- OpenAI: GPTBot (model training), OAI-SearchBot (search indexing), ChatGPT-User (fetches a page when a user asks ChatGPT to look at it directly)
- Anthropic: ClaudeBot (training), Claude-SearchBot (search indexing), Claude-User (on-demand fetches)
- Perplexity: PerplexityBot (real-time search retrieval — Perplexity has no separate training crawler)
- Google: Google-Extended (controls whether your content trains Gemini and feeds AI Overviews, separate from classic Googlebot indexing)
- Apple: Applebot-Extended (Apple Intelligence)
The practical takeaway: blocking a training crawler keeps your content out of a future model’s training data without necessarily removing you from that company’s live search answers, because the search and on-demand bots are governed separately.
Step 1: Find out if you’re already blocking them by accident
A lot of small business sites block AI crawlers without meaning to. Security plugins, Cloudflare rules, and CDN configurations set up before 2023 often treat any unfamiliar bot as a scraping threat and block it outright — which catches legitimate AI search crawlers in the same net as malicious ones. If your site hasn’t had its robots.txt or firewall rules reviewed since AI search became mainstream, this is worth checking first, before anything else on this list.
Two quick checks: open yoursite.com/robots.txt in a browser and read what’s actually there, and ask your host or security plugin whether it maintains a bot-blocking list that includes AI user agents. If you have server log access, a quick search for “GPTBot,” “ClaudeBot,” or “PerplexityBot” in recent logs will tell you whether they’re visiting at all.
Step 2: Write a deliberate robots.txt policy
Once you know your current state, the fix is a robots.txt file that makes an intentional choice for each crawler rather than one blanket rule for all of them. A common, sensible default for a local service business: allow the search and on-demand bots so you stay eligible for citations, and decide separately — based on your comfort with AI training on your content — whether to allow the training-only bots.
- Allow (so you can appear in live AI answers): OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot
- Your call (training data only, no effect on live search visibility): GPTBot, ClaudeBot, Google-Extended
- Always include a Sitemap: line pointing to your sitemap.xml so any bot that does respect robots.txt can find your full page list
Connect4 Tip: robots.txt is a voluntary request, not a lock. It’s standardized under RFC 9309 and respected by every major AI company’s crawlers, but it carries no technical enforcement — a small number of bots (and some stealth scrapers) ignore it, and real access control for sensitive pages has to happen at the server or firewall level instead.
Step 3: Make sure the content itself can be read
A crawler that’s allowed in still has to be able to parse the page. Two common local-business setups quietly break that: content that only renders after JavaScript runs (so the raw HTML a bot fetches is mostly empty), and mandatory cookie-consent redirects that send every visitor — bots included — to a consent screen before they see any content. Neither of those is an AI-specific problem, but both have gotten more costly now that AI crawlers, unlike a human visitor, often read the raw HTML response and move on rather than waiting for scripts to finish.
- Check that your homepage’s core content — headline, key services, location, contact info — appears in the page’s raw HTML, not only after JavaScript executes
- Make sure cookie-consent or age-gate redirects exempt known bot user agents, or at least don’t block the underlying content from being fetched
- Use a clean heading hierarchy (one H1, logical H2/H3 nesting) so both search engines and AI systems can tell what a page is actually about
Step 4: Add structured data so machines understand context
Structured data (schema markup) is a translation layer that spells out, in a format machines parse reliably, what a page is: a business, an article, a set of questions and answers, a service. FAQPage, Organization, LocalBusiness, and Article schema are the highest-value types for a typical small business site. One nuance worth knowing: Google retired FAQ rich results from its visible search snippets in May 2026, so FAQPage schema no longer earns you the expandable Q&A box in a Google search result — but it still functions as a clearly labeled data source for AI systems doing answer extraction, so it’s still worth implementing.
Step 5: Is llms.txt worth your time?
llms.txt is a proposed standard — a plain-text file at your site’s root that summarizes your content for AI systems, similar in spirit to robots.txt. It gets a lot of buzz in marketing circles, so it’s worth setting expectations honestly.
Connect4 Tip: As of mid-2026, independent research covering roughly 300,000 domains found no measurable link between having an llms.txt file and getting cited more often by AI search engines, and major AI crawlers rarely fetch the file at all. It’s genuinely useful for developer-documentation sites read by coding assistants, but for a typical local business site it’s a low priority — nice to have once the basics above are done, not a substitute for them.
What you can do right now
- Open yoursite.com/robots.txt and confirm it isn’t silently blocking every bot
- Ask your host or security plugin whether AI user agents are on a default block list
- View your homepage’s raw HTML source and confirm your core content is actually there
- Confirm any cookie-consent flow doesn’t gate content behind a click
- Check whether FAQPage, LocalBusiness, and Article schema are already implemented on your key pages
Where Connect4 can help
- Auditing your current robots.txt and server/CDN rules for accidental AI-bot blocks
- Writing a deliberate, documented robots.txt policy for your site
- Implementing FAQPage, LocalBusiness, and Article schema through your WordPress site
- Reviewing whether your theme or page builder is hiding content from raw HTML
- Ongoing monitoring as your care plan retainer, so AI-visibility checks happen on a schedule instead of once
Frequently Asked Questions
Do I need to block AI crawlers to protect my content?
That’s a business decision, not a technical requirement. Blocking training-only crawlers (GPTBot, ClaudeBot, Google-Extended) keeps your content out of future model training without necessarily affecting whether you appear in that company’s live AI search answers, since search and on-demand crawlers are usually governed separately.
Will blocking AI crawlers hurt my Google rankings?
Blocking Google-Extended does not affect classic Googlebot indexing or your regular search rankings — it only controls whether your content trains Gemini and feeds AI Overviews. Your standard SEO visibility is governed by Googlebot, which is a separate user agent.
How do I know if AI crawlers are currently visiting my site?
Check your server access logs for user agents like GPTBot, ClaudeBot, PerplexityBot, and OAI-SearchBot. If you don’t have direct log access, your host or security plugin dashboard often has a bot-activity report you can pull.
Is llms.txt required for AI visibility?
No. It’s an optional file with real value for developer-documentation sites but no measured citation benefit for typical local business sites as of mid-2026. Prioritize robots.txt and structured data first.
How long does a robots.txt change take to affect AI visibility?
Policy changes typically propagate to the AI company’s systems within about 24 hours, but the crawler’s actual next visit depends on its own scheduling and your site’s crawl demand. Plan to monitor logs for at least two weeks before judging the impact.
Being Crawlable Isn’t Optional Anymore
None of the steps above guarantee that ChatGPT or Perplexity will start quoting your business tomorrow — that depends on the content itself, which is the subject of the companion post. But skipping this checklist guarantees the opposite: if the bots can’t reach you or can’t parse what they find, no amount of great content on the page will matter, because it never gets read in the first place. Access is the floor, not the ceiling.
