Bots are now the majority of web traffic. Here’s how to protect your server and your bill without erasing yourself from ChatGPT, Gemini, and Perplexity.

There’s a line item hiding in your hosting plan that most business owners never see. The bandwidth, CPU cycles, and visits you pay for are increasingly consumed not by customers, but by AI crawlers fetching your pages — sometimes the same unchanged page, over and over. The problem is real. But the common fix, blocking those bots, can quietly cost you something more valuable than bandwidth: your presence in AI answers.

How much of my traffic is actually bots?

By mid-2026, automated requests made up roughly 57% of all web traffic — machines now outnumber humans online, according to Cloudflare Radar. On a small site, AI crawlers alone routinely account for 30% to 50% of raw bandwidth, and by Cloudflare’s own numbers, over half of AI crawler traffic re-fetches pages that haven’t even changed.

That growth has been steep: AI-specific crawling grew roughly 300% year over year, and agentic AI traffic — assistants browsing on a person’s behalf — grew by an even larger multiple. For a sole proprietor or small practice on a modest plan, that’s real money and real server strain.

The trap: blocking bots can erase you from AI search

Here’s the tension every site owner now faces. Blocking AI crawlers with robots.txt protects your bandwidth — but it also removes your content from the pool that ChatGPT, Perplexity, and Google AI Overviews draw on to build answers. Block indiscriminately and you save a few dollars while vanishing from the fastest-growing discovery channel on the web.

The stranger risk: some hosts have started blocking AI crawlers on your behalf by default. That protects your bill and erases you from AI answers at the same time — a decision you should be making, not discovering.

As of September 15, 2026, for example, Cloudflare began blocking the Training and Agent crawler categories by default on ad-bearing pages for newly onboarded domains, while leaving Search crawlers allowed. Kinsta stopped counting bad-bot bandwidth against plan limits back in late 2025; other hosts rate-limit bots at the platform level. The point is that your host’s defaults may already be shaping your AI visibility — so the first step is finding out what they are.

The smart approach: separate training from search

The winning move for most small businesses is selective, not all-or-nothing: block the training-only crawlers that give nothing back (CCBot, and optionally GPTBot and ClaudeBot), while allowing the search and user-action bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) that keep you eligible for citations. You cut wasteful traffic without cutting off discovery.

One important caveat: Google merges much of its search and AI crawling, so blocking Google’s AI training also forfeits Google Search visibility — a trade almost no business should make. Leave Googlebot fully allowed.

Four ways to control crawler load

• Start with robots.txt. It’s free, it’s the first place compliant bots look, and it lets you make the training-vs-search distinction cleanly. Just remember it’s advisory — well-behaved bots honor it, bad actors ignore it.
• Block at the edge for the traffic that ignores robots.txt. A CDN or firewall (like Cloudflare’s WAF) stops unwanted bots before the request ever reaches your server — no PHP execution, no database query, no bandwidth consumed. This is where you stop spoofed or misbehaving scrapers that robots.txt can’t touch.
• Use your host’s bot controls. Many hosts now offer crawler dashboards or plugins that update robots.txt and apply rules automatically. On Cloudflare, the AI Crawl Control panel shows exactly which AI bots are hitting your site and lets you allow or block each one.
• Right-size your plan if bots are a genuine cost. If crawler bandwidth is pushing you over plan limits, a host that doesn’t count bot traffic against your allowance — or a higher-allowance plan — can be cheaper than fighting every bot.

Start by measuring, not blocking

Before you change anything, look at your logs or your CDN’s crawler report and answer three questions: which AI bots are visiting, how much traffic each one drives, and whether they’re returning 200s or getting blocked. Only then can you make a deliberate choice — because the goal isn’t the least bot traffic, it’s the right bot traffic: heavy on the crawlers that send you customers through AI citations, light on the ones that just run up your bill.

Control the crawlers, keep the customers

AI crawler traffic is a cost worth managing and an opportunity worth protecting — and those two goals don’t have to conflict. Measure first, block selectively, and confirm your host’s defaults reflect your priorities rather than theirs. If you’d like a clear read on what’s crawling your site and a configuration that trims the waste without hiding you from AI answers, that’s a quick audit we run for clients throughout the DMV. Get in touch and we’ll take a look.