AI Crawlers and llms.txt: Let AI Assistants Read Your Site
AI assistants read the web using their own crawlers. You control whether they can read your site in your robots.txt file. Many websites block them accidentally, often due to default settings from their hosting provider.
The Key Crawlers
A few names you might see in your robots.txt:
- GPTBot and OAI-SearchBot from OpenAI (ChatGPT)
- ClaudeBot from Anthropic (Claude)
- PerplexityBot from Perplexity
- Google-Extended, which controls whether Google can use your content for its AI models
- Googlebot and Bingbot for standard search engines
Allowing or Blocking
If you want to be mentioned in AI answers, allow the search crawlers. You can choose which crawlers to permit individually. Also check your hosting firewall or bot protection settings, as they sometimes block anything that is not a standard human visitor.
What is llms.txt?
llms.txt is a proposed file in your website's root directory containing a clean, readable summary for AI models: who you are and which pages are most important. It is not (yet) an official standard and not all assistants use it. It does no harm to have one, but it does not replace having good, well-structured pages.
Frequently asked questions
- How do I know if my site is blocking AI crawlers?
- Open yoursite.com/robots.txt and look for the names listed above. Magnifi's website scan checks this as well.