--- title: robots.txt Tester slug: robots-txt-tester last_reviewed: 2026-08-28 author: editorial source: LemAudit Free Tools --- ## Intro A robots.txt tester answers one question: given the file you actually serve, will a specific crawler fetch a specific URL? It is a different job from writing the file. Most mistakes we see in audits are not syntax errors — the file parses fine — they are rules that match more or less than the person who wrote them believed. Paste your file, give it a URL and pick a crawler. The tester applies the same matching rules Google documents: group selection by user-agent, longest-pattern-wins precedence, `*` and `$` wildcards, and the empty-`Disallow` special case. It runs in your browser and logs nothing. ## How crawlers actually choose which rules apply Two steps, and both surprise people. **Step one: exactly one group applies.** A crawler scans every `User-agent` line and picks the group whose token is the longest match for its own name. Googlebot reading a file with a `User-agent: *` group and a `User-agent: Googlebot` group obeys the Googlebot group and ignores the wildcard group completely — including rules that only exist in the wildcard group. If your `Disallow: /admin/` lives under `*` and you added a `Googlebot` group later to allow one image folder, you have just opened `/admin/` to Google. **Step two: within that group, the longest matching path pattern wins.** Not the first rule, not the last. `Disallow: /admin/` and `Allow: /admin/public/` both match `/admin/public/page` — the Allow is longer, so the page is crawlable. If two matching patterns are the same length, Allow wins. Order in the file is irrelevant to precedence. It only matters for grouping consecutive `User-agent` lines. ## Wildcards, and where they bite `*` matches any run of characters. `$` anchors the match to the end of the URL. `Disallow: /*?` blocks every URL containing a query string — including the faceted URLs you wanted blocked and the paginated ones you did not. `Disallow: /*.pdf$` blocks PDFs but leaves `/report.pdf?download=1` crawlable, because the query string means the URL no longer ends in `.pdf`. `Disallow: /search` blocks `/search`, `/searching`, and `/search-results`, because there is no implicit boundary at the end of a pattern. Matching runs against the path plus query string, and it is case-sensitive. `/Admin/` and `/admin/` are two different things to a crawler. ## The empty Disallow `Disallow:` with no value means "nothing is disallowed" — it is the standard way to write an explicit allow-all group. `Disallow: /` with a slash blocks everything. One character apart, opposite meanings, and it is a genuinely common typo in hand-edited files. ## Blocking is not de-indexing This is the single most expensive misunderstanding in technical SEO. A `Disallow` rule stops a crawler fetching the page. It does not remove the page from the index, and it never will, because the crawler can no longer read the page to see any instruction you put inside it. If a blocked URL has inbound links, Google can and does list it — usually with no description, sometimes with an anchor-derived title. To keep a page out of results you must let it be crawled and serve `` or an `X-Robots-Tag` header. Blocking a page you have also noindexed cancels the noindex. The same logic applies to canonical tags, hreflang and structured data: all of them are inside pages a blocked crawler cannot read. ## What to test before you ship a change Run these five through the tester after any robots.txt edit: Your homepage, for the obvious reason. One URL from each section you intended to block. One URL from a section you intended to keep, that sits close to a blocked pattern — this is where over-broad wildcards show up. A CSS and a JavaScript file, because blocking render resources makes Google evaluate your mobile layout against a page it cannot paint. And one parameterised URL, if your site has them. Test with `Googlebot` selected, then again with `*`. If the answers differ and you did not intend them to, your groups are fighting each other. ## AI crawlers The same syntax now governs a second population of crawlers, and they are separate agents: `GPTBot`, `ClaudeBot`, `PerplexityBot`, `Google-Extended` for Gemini training, `CCBot` for Common Crawl. Google-Extended is worth understanding precisely — blocking it removes you from Gemini training data but has no effect on Googlebot or on your web search ranking. Decide deliberately. Blocking all of them protects content from training use and removes you from AI answer surfaces that increasingly sit above organic results. Our [robots.txt generator](/tools/robots-txt-generator) writes the AI-crawler block for you either way, and the [CMS detector](/tools/detect-cms) reports which AI bots a live site currently allows. ## Serving rules that break otherwise-valid files The file must be at the root: `https://example.com/robots.txt`. A file at `/blog/robots.txt` is ignored. Subdomains need their own file — `shop.example.com` does not inherit from the apex. It must return HTTP 200 with `text/plain`. A 404 is treated as allow-all. A 5xx for more than a short window makes Google stop crawling the whole site rather than risk fetching something you meant to block. HTML error pages served with a 200 status are parsed as robots.txt and produce nonsense. Google reads up to 500 KiB and truncates the rest, which only matters on generated files with thousands of rules — a sign the logic belongs in meta tags instead. ## When not to use This tester answers what your rules mean, not whether Google agrees your file is being fetched at all. If a URL is missing from the index and the tester says it is allowed, the problem is elsewhere: check the live response code, the canonical tag, the meta robots tag and the URL Inspection tool in Search Console before editing robots.txt again. Do not use it as a de-indexing tool. If the goal is removing a page from search results, robots.txt is the wrong lever entirely — the [CMS detector](/tools/detect-cms) will show you what a live page currently serves, and a `noindex` on a crawlable page is the fix. And if you are writing a file from scratch rather than checking one, start with the [robots.txt generator](/tools/robots-txt-generator) and test the result here afterwards. ## FAQ ### Why does Googlebot ignore my "User-agent: *" rules? Because a more specific group exists for it. A crawler obeys exactly one group — the longest user-agent match — and ignores every other group in the file, including the wildcard. Duplicate any shared rules into the named group. ### Which wins, Allow or Disallow? The longer pattern. Only when two matching patterns are exactly the same length does Allow win by default. ### Does robots.txt remove a page from Google? No. It blocks crawling, not indexing. A blocked URL with inbound links can appear in results with no snippet. Use a `noindex` meta tag on a crawlable page instead. ### Is robots.txt case-sensitive? The paths are. Directive names are not. `Disallow: /Admin/` will not block `/admin/`. ### Do I need a separate file for each subdomain? Yes. Each host serves its own robots.txt, and protocols count too — HTTP and HTTPS are treated as distinct origins for this purpose. ### What happens if my robots.txt returns a 500 error? Google pauses crawling of the site while the error persists, on the assumption that it might otherwise crawl something you intended to block. A 404, by contrast, is safely interpreted as no restrictions. ### Should I block AI crawlers? It is a business decision, not an SEO one. Blocking `GPTBot`, `ClaudeBot` and `PerplexityBot` keeps your content out of their answers as well as their training. Blocking `Google-Extended` affects Gemini training only and leaves web search untouched.