OpenRobots
Free, in-browser generator for robots.txt and llms.txt files that control which AI crawlers can access your site.
🔗 Visit OpenRobots
Description
AI companies now run dozens of different bots that crawl the web to train models, power search, or feed chatbots — and deciding which ones you want touching your content usually means hand-writing robots.txt rules for crawlers you've never heard of. OpenRobots turns that into a checklist: toggle on or off 82+ named AI crawlers (or use a preset like 'block all AI' or 'allow search engines only') and it generates the correct robots.txt and the newer llms.txt file for you.
It runs entirely client-side in the browser — no account, no server, no data ever leaves your machine — and doubles as an auditor that can analyze an existing robots.txt for gaps. It's free and open source (MIT). The tool is upfront about its own limits: robots.txt is a voluntary standard, and it explicitly lists six known crawlers (including Bytespider and DeepSeekBot) that ignore it outright, so this controls compliant bots only, not a hard block.
💬 Our review
The short version: a genuinely useful, honest little utility — it does one specific job (generate AI-crawler rules) well, and it's refreshingly upfront that robots.txt can't stop bots that don't want to be stopped.
Compared to writing robots.txt by hand or using generic robots.txt generators that don't track the fast-moving list of AI-specific bots, OpenRobots' edge is staying current on which crawlers are which (training bots vs. search vs. assistants) and bundling the emerging llms.txt standard in the same flow. Its built-in disclaimer is the right call: for bots that actively ignore robots.txt, you need server- or firewall-level blocking instead. Worth using as a five-minute first step for any site owner thinking about AI crawler policy; not a substitute for real bot-blocking infrastructure if a persistent scraper ignores the rules.
💰 Pricing
📊 Global score
🤖 AI-enriched data
Gratuit, open source, sans compte.
Pros
100% client-side, zéro donnée envoyée
82+ crawlers IA couverts
Génère aussi llms.txt
Cons
robots.txt reste volontaire
Pas d'application forcée
Petit projet, un seul mainteneur