What WordPress does with robots.txt by default
WordPress generates a robots.txt for you. If you have never created one, yoursite.com/robots.txt still returns a file — but you will not find it in your directory listing over FTP, because it does not exist on disk. WordPress builds it on request.
The default output is short:
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Sitemap: https://yoursite.com/wp-sitemap.xmlThree things are worth noticing about that default, because they explain most of the confusion people have with this file.
It blocks the admin area but deliberately allows admin-ajax.php, because themes and plugins route front-end functionality through it. Blocking it can stop a crawler rendering parts of your pages.
It does not block /wp-content/, and that is correct. Your images, CSS and JavaScript live there, and Google needs them to render the page as a visitor sees it.
It is virtual. The moment you upload a real robots.txt file to your web root, WordPress stops generating anything and your file is served instead — including whatever you left out of it.
When the default is already fine
Most WordPress sites do not need to touch this file at all.
robots.txt controls crawling — which URLs a bot may request. It does not control indexing, it does not influence rankings, and it does not secure anything. A URL blocked in robots.txt can still appear in search results if other pages link to it, because Google can list a URL it has never fetched.
So if your aim is to keep a page out of the index, robots.txt is the wrong instrument. Use a noindex meta robots tag, and leave the page crawlable so the tag can be read. Blocking a page you have also set to noindex is self-defeating: the crawler cannot fetch the page, so it never sees the instruction.
Edit robots.txt when you have a concrete crawling problem: a bot hammering a URL pattern that generates infinite variations, a staging area being fetched, or a search-results path producing thousands of low-value URLs. "Tidying it up" is not a reason.
How to edit it on WordPress
Three routes, in increasing order of permanence.
Through an SEO plugin. Yoast SEO and Rank Math both expose a robots.txt editor in their tools section. This is the usual route, and the file remains virtual — the plugin filters WordPress's generated output rather than writing to disk.
Through a filter in your theme or a small plugin. WordPress exposes a robots_txt filter, so you can append rules in code. This keeps the file virtual and version-controlled with your theme.
By uploading a real file. Put robots.txt in your web root and it wins. This is the most predictable option and the easiest one to forget about: once a physical file exists, plugin editors may appear to save changes that never take effect, because WordPress is no longer generating the file at all.
If you have ever been confused by a robots.txt that refuses to change, check for a physical file first. It is the most common cause.
Rules that are safe, and rules that cost you
The rules below are safe on a standard WordPress install.
| Rule | Why |
|---|---|
Disallow: /wp-admin/ | Administration, never useful in search |
Allow: /wp-admin/admin-ajax.php | Front-end functionality routes through it |
Disallow: /?s= | Internal search results, which can generate unlimited URLs |
Disallow: /*?replytocom= | Comment-reply links, one per comment per post |
Sitemap: … | Not a crawl rule, but this is where crawlers look for it |
The rules below are widely copied and frequently damaging.
Disallow: /wp-content/ blocks your images, CSS and JavaScript. Google renders pages before assessing them; block the assets and it sees a broken layout. This single line is the most expensive piece of robots.txt advice still circulating.
Disallow: /wp-includes/ blocks core JavaScript that some themes depend on at render time. It was standard advice years ago and has not been correct for a long while.
Disallow: /category/ or /tag/ is usually aimed at thin archive pages. It is the wrong tool: it prevents crawling, so Google never sees any noindex you set, and those archives often carry real internal-linking value. Handle archives with meta robots and internal linking, not a crawl block.
Crawl-delay: is ignored by Google entirely. Bing and some others honour it. If Googlebot is genuinely over-crawling you, the crawl-rate setting in Search Console is the mechanism, not this line.
The rules that matter more than the file
Two behaviours catch people out often enough to be worth stating plainly.
A blocked page can still rank. If other sites link to a URL you have disallowed, Google may index the URL with no description, using the anchor text it found elsewhere. People discover this when a blocked staging URL turns up in search with "No information is available for this page."
Disallow does not remove anything. If a URL is already indexed and you then block it, you have removed Google's ability to re-crawl it and discover a noindex or a 404. The page can sit in the index for a long time. Remove first, block afterwards — not the other way round.
Multisite, subdomains and staging
robots.txt is per host. blog.yoursite.com/robots.txt is a different file from yoursite.com/robots.txt, and a rule on one has no effect on the other. On WordPress multisite each site in the network generates its own.
Staging environments deserve particular care. Blocking a staging site in robots.txt does not reliably keep it out of search, for the reason above — the URL can still be indexed. HTTP authentication is the correct answer for staging: a crawler that cannot get past the password cannot index anything.
This is also the most common way a live site breaks. A Disallow: / from a staging environment gets copied to production during a launch, and the entire site becomes uncrawlable. If organic traffic falls off a cliff shortly after a launch or migration, robots.txt is the first file to read.
How to check what is actually live
Four checks, in order, and none takes more than a minute.
- Open
yoursite.com/robots.txtin a browser. This is what crawlers receive. If it does not match what your plugin shows, you have a physical file overriding it. - Use the robots.txt report in Google Search Console. It shows the version Google last fetched, when it fetched it, and any parse errors.
- Test a specific URL with Search Console's URL Inspection tool. It tells you whether that URL is blocked, which is more useful than reading rules and guessing.
- Check your sitemap line resolves. A
Sitemap:directive pointing at a 404 is common after a plugin change, and quietly costs you discovery.
Frequently asked questions
Where is the robots.txt file in WordPress?
Usually nowhere on disk. WordPress generates it on request, which is why you can load /robots.txt in a browser but not find it over FTP. It becomes a real file only if you or a plugin writes one to the web root.
Should I block /wp-content/ in robots.txt?
No. Your images, CSS and JavaScript live there, and Google needs to fetch them to render your pages. Blocking that directory makes your pages look broken to the crawler.
Does robots.txt stop a page being indexed?
No. It stops a page being crawled. A blocked URL can still be indexed if other pages link to it — usually with no description. To keep a page out of the index, allow crawling and use a noindex meta robots tag.
Why won't my robots.txt changes save?
Almost always because a physical robots.txt file exists in your web root. WordPress only generates a virtual file when no real one is present, so a real file silently overrides anything a plugin editor appears to save.
Get your crawl directives set correctly, once
robots.txt is a small file with a large blast radius. Most WordPress sites need the default plus a line or two; the damage comes from copied rules that block rendering assets, or from a staging Disallow: / reaching production.
If you would rather have this set correctly than take a guess, the Technical SEO Starter Setup covers crawl directives, sitemaps and indexation together — $497, one-off, no retainer.