INSIGHT · CRAWLER GOVERNANCE
AI crawlers are not one thing:
search, agents and training need separate decisions.
A single “block AI” decision can mix different purposes. Start with the business objective and the documented crawler identity, then decide what access is appropriate.
DIRECT ANSWER · CURRENT SCOPE
Should a website block every AI bot at once?
Not without separating purpose. A site should distinguish crawlers that build search indexes, agents that fetch content for a user in real time, and bots used for model training or fine-tuning. It can then set policies around discoverability, copyright, data and service needs. robots.txt mainly communicates preferences; technical blocking requires the relevant enforcement controls.
Boundary: A crawler request only shows that access occurred. It does not prove that the content was cited, recommended or responsible for AI referral traffic.
Three purposes, three questions
Search discovery
Can a search product crawl and index public pages so they remain discoverable?
User-directed agents
Can an agent fetch a page to complete a user-requested task?
Model training
Can content be collected for model development? This is a distinct rights and policy question.
Build a crawler register
Record the documented user agent, provider, stated purpose, verification method, current policy, owner and review date. Do not infer crawler purpose from a name alone.
Use layered controls
robots.txt, provider controls, CDN or firewall rules and legal terms have different effects. Test the actual response and retain a rollback path before deployment.
SOURCE PRACTICE
Verify current provider documentation
Cloudflare and AI providers change crawler descriptions and controls. Recheck their official documentation before implementation.
This article is a governance framework, not legal advice and not a recommendation to allow or block every crawler.