A focused AI-discoverability quick win by senior experts. You get a properly formatted llms.txt built from your live pages, and a robots.txt that sets deliberate allow or block rules for 22 AI crawlers from 10 vendors (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, Meta-ExternalAgent and more), checked before it reaches you. $150, 3 business days, transparent pricing, no surprises.
Delivered as a finished asset, ready to use. Self-serve from there. Want it scaled? Talk to Leverage →
Get My llms.txt → ✓ Transparent pricing, no surprises · 3 business days · $150 fixed priceTapasSEO's llms.txt Config is a focused AI-discoverability engagement by Leverage (20+ years, 50+ countries). You receive a properly formatted llms.txt and llms-full.txt, and a robots.txt with explicit allow or block rules for 22 AI crawlers, each checked against its vendor's documentation, plus an install guide for your platform. Every file is validated before delivery, and a free checker confirms them once you upload. $150, 3 business days, transparent pricing, no surprises.
A curated llms.txt built from your live pages, plus a fuller llms-full.txt. You upload both; the install guide for your platform (WordPress, Shopify, Wix, Webflow, Cloudflare/static or other) shows exactly where.
GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Meta-ExternalAgent, Applebot-Extended, Amazonbot, CCBot, DuckAssistBot and more. Googlebot and Bingbot are never blocked. Crawlers that fetch a page because a person asked may ignore robots.txt; your report says which.
The report lists the schema types found on the pages we read and what your homepage is missing. We do not change your markup; if you want it added or fixed, see Structured Data Implementation.
robots.txt points crawlers to your sitemap, and the pages you name as most important are listed first in llms.txt.
Both files are checked by code before delivery: the format, that Googlebot and Bingbot stay allowed, and that every listed page returns 200. After you upload, the free checker confirms they are live.
A monthly refresh checklist for when you ship new services or pages.
What we found, what we changed and why, a policy table for every crawler with its vendor source, the quality checks, and your install steps.
llms.txt + robots.txt AI-crawler rules + install guide
One-time. 3 business days. Transparent pricing, no surprises.
🔒 Fixed scope and price, documented up front — no surprises.
Want this scaled across multiple sites? Talk to Leverage →
Fixed scope and price, documented up front — no surprises.
Every set of files is reviewed by a senior specialist from the Leverage team before it reaches you.
You order, we deliver in 3 business days, with a human review of every set of files first.
We list what each crawler does (training, search, or fetching for a person), what the vendor's documentation says, and whether it honours robots.txt.
TapasSEO is the productized arm of Leverage, a boutique SEO consultancy with 20+ years of experience across 50+ countries and 20+ languages.
llms.txt is a proposal (llmstxt.org) for a short Markdown map of your site that AI systems can read. Its effect on citations is unproven: no major AI provider has said it ranks or cites on the strength of it. What does have a documented effect is robots.txt: the major AI vendors publish which crawlers they run and say they honour it. That is why this service starts with the access rules and ships llms.txt as cheap, tidy housekeeping. Our guide at /guides/llms-txt/ lays out the evidence.
No service can guarantee AI citation. AI engines decide which sources to cite based on training data, real-time signals, query relevance, and ongoing model changes. What we guarantee: your llms.txt and robots.txt are built correctly and checked before they reach you, so AI engines that do crawl your site have a clear map and clear rules. The work removes a barrier to citation; it does not force citation.
Twenty-two, from OpenAI, Anthropic, Google, Perplexity, Meta, Apple, Amazon, Common Crawl, DuckDuckGo and Mistral, each checked against that vendor's own documentation on the date shown in your report. Vendors add and rename crawlers, so the list is re-verified before each release rather than promised to be current forever. Googlebot and Bingbot are search engines and are never blocked.
Some sites want to disallow all AI crawlers (for example publishers blocking training data scraping). You choose one of three policies when you order: allow everything, allow AI search and answers while blocking AI training, or block all AI crawlers. Crawlers that fetch a page because a person asked, such as ChatGPT-User, may ignore robots.txt; your report tells you which ones.
Transparent pricing, no surprises. Fixed scope and price, documented before you order.
You receive ready-to-upload files (robots.txt, llms.txt and llms-full.txt), an install guide for your platform (WordPress, Shopify, Wix, Webflow, Cloudflare/static or other) and a PDF report that explains every choice, reviewed by a senior consultant before it reaches you. You upload the files; the free checker confirms they are live. If you want this scaled across multiple sites or properties, talk to Leverage.
GPTBot is OpenAI's crawler for content that may be used to train its generative models. Disallowing it tells OpenAI not to use your site for that training. It is separate from OAI-SearchBot, which decides whether you appear in ChatGPT search results, and from ChatGPT-User, which opens pages when a person asks ChatGPT about them and which OpenAI says may not follow robots.txt. Blocking GPTBot alone does not remove you from ChatGPT search.
Per Anthropic's help article, ClaudeBot collects web content that could contribute to training, Claude-SearchBot navigates the web to improve search result quality, and Claude-User fetches pages when a person asks Claude a question. Each can be controlled in robots.txt. Blocking Claude-SearchBot or Claude-User may reduce your visibility in Claude's search results and in answers to user questions.
Google-Extended is a robots.txt control token, not a separate crawler. It controls whether content Google has crawled can be used to train future Gemini models and for grounding in Gemini Apps. It does not affect Search inclusion or ranking.
llms.txt and llms-full.txt, robots.txt rules for 22 AI crawlers, a read-only structured data check, an install guide for your platform. 3 business days. $150 once.
Get My llms.txt — $150One-time payment · 3 business days · Transparent pricing, no surprises
If you need this across multiple subdomains, multi-region rule sets, or ongoing monthly llms.txt updates — drop your details and a free senior SEO consultant from the Leverage team will reach out within 24 hours.
You get the llms.txt config. If you’d rather a team handle the full AI-search setup — that’s Leverage: done-for-you SEO, 20+ years.
