.png)
6 hours ago
3
My clarified stance on blocking AI crawlers is this: block all of them. I won't speak for everyone on the matter (I can't), but feel ok walling them all off.
We updated the readme on the ai.robots.txt repository to read:
This list contains AI-related crawlers of all types, regardless of purpose.
It clarifies the purpose of the project and the contents of the primary of the list. It's neither dictum nor directive, it's simply descriptive.
What we do offer is a table of data to guide your specific implementation, which I wholeheartedly encourage you to reference when guiding your implementation.
In the year plus that we've been working on this project I've only seen developments that reinforce my position.
We have chatbots spitting out text (that may or may not be correct — they don't hallucinate, people hallucinate). Said text is put to work as slop flooding social media platforms, dumping worthless books on retailers and making codebases harder to maintain.
We've got abundant and altogether mediocre imagery.
We've got spectacularly odd and altogether uninteresting video.
We've got documents and emails you couldn't be bothered to write and I couldn't be bothered to read.
We've got utilities being pitched as cure-alls. Utilities don't get you the valuation you want.
I saw a play recently, seating on the stage, theater in the round — real actors and all that. It was wonderful, heartfelt and something AI could never do. It was raw and distinctly human — we can't automate art. Art should challenge you, not accede to a prompt or demands. That's slop. I could make the same point about concerts — the performers and the audience are part of the experience. Extruding sound from a prompt is not the same. It never will be.
Anyways. Block some crawlers, I'd encourage it. Pick which ones, pick all of them, evaluate it against what you're trying to accomplish, but I'd encourage you to consider putting up your own small speed bumps as we're forcibly pushed ever faster towards the torment nexus.
