1. Home›
  2. Guides›
  3. AI Crawler & llms.txt Checker
Free tool · No email

AI Crawler & llms.txt Checker

Enter a site and see which of 22 known AI crawlers from 10 vendors its robots.txt lets in, whether Googlebot and Bingbot are safe, and whether its llms.txt is live and well formed. Takes a few seconds, no signup.

We read only /robots.txt and /llms.txt from the address you enter, from our server. Free, no signup, up to 15 checks an hour.

How to read the result

  1. Googlebot and Bingbot — if either is blocked you are hidden from search results. That is the first thing to fix.
  2. AI crawler stance — the crawlers fall into groups: those that collect content to train models, those that power AI search and answers, and those that fetch a page because a person asked. Letting search in while keeping training out is a common choice.
  3. Assistants that may ignore robots.txt — fetchers acting for a person, such as ChatGPT-User, may not follow it. Their vendors say so in their own documentation.
  4. llms.txt — a proposed convention. We check that it is live and follows the format; its effect on AI citations is unproven, as our llms.txt guide explains.

Want both files built and checked for your site? See llms.txt Config. More free tools and guides in the TapasSEO guides library.