Category · 2 articles
The llms.txt spec, format, hosting, and why every B2B site should ship it.
← All articles
llms.txt is the AI-crawler equivalent of robots.txt — a structured map of what your site is, what content matters, and how AI systems should describe it. Here is the spec, the format, and a copy-pasteable starter.
Should you let GPTBot, ClaudeBot and PerplexityBot crawl your site? Yes. The 2026 default is to allow the major AI crawlers — that is how you get cited. AI crawler robots.txt allow-list, exceptions, and why blocking is wrong for B2B.