--- title: Free robots.txt Generator slug: robots-txt-generator last_reviewed: 2026-08-09 author: editorial source: LemAudit Free Tools --- ## Intro `robots.txt` is the first file most crawlers request from a domain, and one of the few places where a single stray character can quietly remove a site from search. It is a plain text file at the root of your host that tells automated clients which paths they may fetch. It is not a security control, it is not an indexing directive, and it is read before anything else on your site — which is exactly why it deserves more care than it usually gets. This generator builds the file from a short form and, more usefully, tells you when what you have asked for is not what you meant. It runs entirely in your browser; nothing you type is uploaded anywhere. ## What robots.txt actually controls The file controls fetching, not indexing. That distinction causes more damage than any other misunderstanding in technical SEO. If you disallow a URL, a compliant crawler will not request it. But if other pages link to that URL, a search engine can still list it in results — it simply shows the URL with no description, because it was never allowed to read the page. Blocking a page you want removed from search therefore has the opposite effect of the one intended: it locks in the listing and destroys the snippet. To remove a page from search results, allow the crawl and serve a `noindex` meta tag or `X-Robots-Tag` header. The crawler has to be able to read the page to see the instruction. Once the page has dropped out, you can block it if you want to save crawl budget. ## Rules of precedence Within a group, the most specific matching rule wins, measured by path length. `Disallow: /blog/` and `Allow: /blog/public/` coexist happily: the longer allow rule wins for that subtree. Groups are selected by user-agent, and a crawler obeys exactly one group — the most specific one that names it. If you write a `GPTBot` group, GPTBot stops reading the `*` group entirely. This trips people up constantly: they add a two-line AI block and accidentally exempt that crawler from every rule in the default group. The generator repeats the relevant rules inside each group for that reason. ## Wildcards and anchors Two special characters are widely supported. `*` matches any run of characters, and `$` anchors the end of the URL. `Disallow: /*?sessionid=` blocks every URL carrying that parameter regardless of where it appears. `Disallow: /*.pdf$` blocks PDF files but not a page whose URL merely contains `.pdf` in the middle. Both are far better tools than listing paths one by one, and both are easy to over-apply — a wildcard rule that matches more than you expected is the second most common cause of accidental deindexing after a stray `Disallow: /`. ## Crawl-delay and crawl budget `Crawl-delay` is honoured by Bing and Yandex and ignored by Google. If Googlebot is overwhelming your server, the fix is in Search Console's crawl rate settings or, better, in your server's response times. A crawl-delay line does no harm, but do not expect it to solve a Googlebot load problem. Crawl budget matters far less than most people assume. It becomes relevant when a site has hundreds of thousands of URLs, or when faceted navigation generates near-infinite parameter combinations. For a site with a few thousand pages, blocking a handful of directories will not change how fast your content gets indexed. ## AI and LLM crawlers The crawler landscape now has three distinct populations, and treating them as one is usually a mistake. Training crawlers such as `CCBot`, `Google-Extended` and `Applebot-Extended` collect text for model training and send you nothing back. Answer engines such as `OAI-SearchBot` and `PerplexityBot` fetch pages to answer live questions, and they cite sources — which sends real traffic. On-demand fetchers such as `ChatGPT-User` fetch a page because a person asked for it specifically. Blocking all three because you object to training also removes you from the answers people are asking for right now. The generator's third option splits them: training crawlers blocked, citing engines allowed. That is the setting most businesses actually want, and it is worth revisiting every few months as new agents appear. ## The sitemap directive One line, absolute URL, and it applies to the whole file regardless of which group it sits near. Point it at your sitemap index rather than listing every shard — the index is the thing you maintain, and crawlers follow it down. The directive is not a substitute for submitting your sitemap in Search Console and Bing Webmaster Tools, but it is how any other crawler finds it, and it costs nothing. ## Before you upload Check three things. The file must be at the root of the host, at `/robots.txt`, not in a subdirectory. Each subdomain needs its own file, because `shop.example.com` does not inherit anything from `example.com`. And the response must be a 200 with a plain-text content type — a 404 means everything is allowed, while a 5xx sustained over days causes Google to stop crawling the site altogether out of caution. After you upload, fetch the live URL in a browser and read it. That thirty-second check catches template-engine mangling, HTML wrappers and stale cached copies more reliably than any validator. ## FAQ ### Does robots.txt stop a page appearing in Google? No. It stops the page being fetched. A blocked URL can still be listed, without a description, if something links to it. Use a `noindex` meta tag on a crawlable page to remove it properly. ### Is robots.txt case sensitive? The paths are. `/Blog` and `/blog` are different rules. The directive names are not — `Disallow` and `disallow` both work, but conventional capitalisation makes the file easier to read. ### Do I need a robots.txt at all? Not strictly. A missing file returns 404 and crawlers assume everything is permitted. Having one is still worth it for the sitemap directive alone. ### Can I block a specific bot that ignores the rules? Not with this file. Compliance is voluntary. Bots that ignore robots.txt have to be handled at the server or firewall level, by user-agent or IP. ### Should I block my staging site here? Block it, but do not rely on it. Use HTTP authentication as well. A robots.txt on staging routinely gets deployed to production by mistake, which is one of the most common self-inflicted deindexing events there is. ### How large can the file be? Google reads the first 500 kibibytes and ignores the rest. If you are anywhere near that, the rules are too granular and should be replaced with wildcard patterns.