Stop the AI Tool and scraper madness – Set up Cloudflare today

You’ve likely heard about the crazy amount of crawling going on by AI scrapers, AI search tools, bots, and others now that AI Search has arrived. But have you seen what it looks like? I mean have you REALLY seen the graphs and charts showing how bad this is? Well, today is your lucky day.

I’ve been running my blog for nearly two decades (literally) and I have never run into any problems with traffic overages with my hosting provider (which is now Kinsta). And by the way, Kinsta is incredible and I’ve been extremely happy with their services over time. But a few months ago something new happened. And it was disturbing as heck.

When traffic isn’t traffic, yet you’re billed for that traffic.
Several months ago I started receiving emails from Kinsta that I was approaching my ‘visits’ limit and that I would be charged for the overage. That never happened before, ever. Even when I had blog posts go viral and tons of traffic was arriving on my site I didn’t receive those emails.

Clearly I thought it was an error and that Kinsta should look into the “traffic”. And again, I have never come close to the limits before, so something changed – Hint, hint, the AI Search ecosystem arrived. :) 

The first month I received that email from Kinsta I actually shrugged it off, but I did quickly contact some people at Kinsta who looked into it. To me, it was pretty clear that AI tools, scrapers and bots were hammering my site and that was causing the overage. And for all we know, even official AI search platforms could be crawling my site that were masking who they really were. We know that’s happened before, cough Perplexity, so it was entirely possible.

And to be clear, I’m not saying you should block official AI platforms like ChatGPT, Perplexity, Claude, etc. Instead, there are many tools and scrapers out there hammering sites for content, data, etc., and that is what I’m referring to here. And they don’t use specific user-agents which could be blocked by robots.txt, flagged by hosting providers, etc. They can easily look like human visitors…

Here is what my “visits” graph looked like in Kinsta. There is no way I had 366K human visits to my site during that timeframe:

Kinsta: All clear on the scraping front. But was it?
Anyway, Kinsta dug in a bit and explained there wasn’t anything they could do since a number of the IP addresses weren’t tied to official bots and those other visits were getting mixed in with normal site visitors. And although Kinsta provides Cloudflare as a CDN, that feature doesn’t include more advanced bot management and blocking functionality. Therefore, the solution was to sign up for my own Cloudflare account on top of my hosting package that was already leveraging Cloudflare as a CDN.

I added that task to my to-do list but so many things got in the way. Client work, Google updates, and the insane pace of change in the industry as AI Search platforms hit the scene. And that last part was ironic considering AI tools, scrapers and bots were probably causing the massive amount of crawling which was yielding hosting overages!

A few months went by and I didn’t take action simply based on my crazy schedule. And as you can guess, more emails arrived from Kinsta about overages. It was insane.

Finally pulling the trigger with Cloudflare:
I finally found some time to set up Cloudflare separately from my Kinsta account. And then I waited to see what would happen. Would their bot detection systems mitigate the onslaught of crawling or would those tools, crawlers and scrapers still get by? And again, I wasn’t blocking official AI bots. I just wanted to block automated “traffic” which wasn’t human and was eating up my hosting plan.

I made sure that Super Bot Fight mode was set up in my account:

Yep, those AI bots and scrapers were hammering my site. And I know many other site owners are in this situation. So, if you are running into hosting overages based on exceeding your traffic limits with your provider, then don’t hesitate… set up Cloudflare soon. There are many benefits to using Cloudflare beyond just bot management, but this is a big one considering the insane amount of crawling and scraping going on as AI Search platforms and tools expand.

Of course, you’ll probably be a bit overwhelmed with the settings and options in Cloudflare, but once you get the right setup running, you’ll be very happy. Bye-bye bots and scrapers and hello lower hosting costs.

Bonus: Paying for bandwidth versus visits. Another option for site owners.
I also wanted to mention another option for site owners for those not wanting to set up Cloudflare for some reason. Literally one day after setting up Cloudflare, Kinsta sent out an email to their customers based on the crazy amount of AI crawling going on (and many receiving overage notices). Kinsta explained that you can switch your account to bandwidth pricing versus visits pricing, which can help keep overage charges down. I still recommend setting up Cloudflare based on the amount of features and functionality you receive, but I did want to mention this option. You can see part of the email below. So definitely speak with your hosting provider in case you want to go down that path.

Summary: Stop the AI madness and block the (bad) bots:
Again, this just underscores the AI insanity right now. There are so many AI bots, scrapers, crawlers, and other BS “visits” going on that it’s tough to manage. And that can cost you a lot of wasted money. So set up Cloudflare, get the essential features rocking and rolling in your settings, and block those BS “visits”. Your bank account will thank you.

GG