Robots.txt Generator – Build a Custom Robots.txt File
About the Robots.txt Generator
This generator builds a properly formatted robots.txt file through a point-and-click interface: set a default rule for all crawlers, override it for individual search engines, add crawl-delay values, declare your sitemap, and define path rules for the folders you want kept out of crawling.
A robots.txt file sits at the root of your domain and is the first thing most crawlers request. It is a set of instructions about what to crawl, and understanding what that does and does not mean is where most robots.txt mistakes originate.
The biggest one is confusing crawling with indexing. Blocking a path stops compliant crawlers from fetching it, but a URL blocked in robots.txt can still appear in search results if other pages link to it — Google simply shows it without a description, having never seen the content. Worse, the standard way to remove a page from search is a noindex tag on the page itself, and a crawler blocked by robots.txt can never fetch the page to discover that tag. Blocking a page you want deindexed is therefore the one move that guarantees it stays indexed. Allow crawling, add noindex, wait for recrawl, then block if you still want to.
The second is treating it as a security control. Your robots.txt is public at yourdomain.com/robots.txt, so listing /admin/ or /private-backups/ tells anyone who looks exactly where those directories are. Anything genuinely sensitive needs authentication, not a polite request that well-behaved bots stay away.
Compliance also varies. The file is a convention rather than an enforcement mechanism: Google, Bing, and other major crawlers respect it, scrapers and malicious bots frequently do not. Google specifically ignores the crawl-delay directive altogether, managing its own crawl rate instead, though Bing and Yandex do honor it.
One more worth knowing: blocking CSS and JavaScript files, which used to be common practice, now actively hurts you. Google renders pages to evaluate them, and a page whose stylesheets and scripts are blocked gets assessed as a broken layout.
The SEO side of the BigToolSite team checked the generated output against Google’s robots.txt specification, tested the syntax through Search Console’s own parser, and verified that directive order, wildcards, and user-agent grouping produce the behavior the settings describe.
Before uploading anything, running the file through the Robots.txt Tester shows you exactly which URLs each rule blocks, which is the fastest way to catch a pattern that matches more than you intended.
Declaring a sitemap in robots.txt only helps if the sitemap is current, and the XML Sitemap Generator produces one you can reference from the Sitemap line.
For pages that should be removed or sent elsewhere rather than merely left uncrawled, the htaccess Redirect Generator writes the server-level rules that actually accomplish that.
How to Use the Robots.txt Generator
- Set your default rule. “Allowed” is right for almost every site; a blanket refusal blocks your entire site from search and should only be used deliberately, such as on a staging server.
- Optionally set a Crawl-Delay. This asks crawlers to wait between requests, which helps on servers struggling under bot load. Google ignores it; Bing and Yandex respect it.
- Add your Sitemap URL as a full absolute address including https. This is one of the more useful lines in the file.
- Set individual rules for named search engines where you want them to differ from the default — Allow, Refuse, or Same as Default.
- Use Custom Robots to name any other user-agent, such as an SEO crawler or an AI training bot, and set its rule.
- Add Path Rules for the directories and URLs you want blocked or explicitly allowed. Paths are case-sensitive and are matched from the start of the URL.
- Click Generate robots.txt, review the output, and check it against a few real URLs before you deploy it.
- Download the file and upload it to your site root so it resolves at yourdomain.com/robots.txt. It must be at the root; a robots.txt in a subfolder is ignored entirely.
Example Usage
A typical WordPress configuration:
User-agent: * Allow: / Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /?s= Disallow: /search/ Sitemap: https://example.com/sitemap_index.xml
Every part of that earns its place. The admin directory is blocked but admin-ajax.php is allowed back in, because plugins and themes call it to render front-end content. Internal search result pages are blocked because they generate near-unlimited thin URLs that waste crawl budget without ever ranking.
To slow an aggressive crawler without blocking it entirely, give it its own group:
User-agent: AhrefsBot Crawl-delay: 10
Crawlers match the most specific user-agent group that applies to them and ignore the rest, so a bot with its own block will not also read the rules under User-agent: *. Anything that should apply to a named bot must be repeated inside its own group.
For a staging site that must stay out of search entirely:
User-agent: * Disallow: /
Two cautions with that one. It is the single most damaging line to accidentally deploy to production, so check before any launch that it did not travel across with the code. And because it only blocks crawling, a staging site already indexed will not disappear from it — password protection is the reliable way to keep a staging environment out of search.
Frequently Asked Questions
It tells crawlers which parts of your site they may request. It is used mainly to steer crawl budget away from low-value URLs and to keep bots out of areas that serve no purpose in search.
In the root directory, so it loads at yourdomain.com/robots.txt. Crawlers only look there. A file placed in a subfolder has no effect at all.
No. It stops crawling, not indexing. A blocked URL that other pages link to can still be listed, just without a description. To remove a page from search, allow crawling and add a noindex tag, because a blocked page can never be read to discover that tag.
No. The file is publicly readable, so listing a directory in it advertises that directory to anyone curious enough to look. Sensitive areas need authentication.
Major search engines do. Scrapers, content harvesters, and malicious bots often ignore it, since compliance is voluntary. Blocking those requires server-level rules or a firewall.
No. Google determines its own crawl rate and ignores the directive. Bing and Yandex do honor it, so it is still worth setting if those crawlers are straining your server.
Yes, by adding a custom user-agent rule for each one. Whether you should depends on whether you want your content used for training and whether you want to appear in AI-generated answers, since some crawlers serve both purposes.
No. Google renders pages to evaluate them, so blocking the assets it needs makes your page look broken to the crawler. This was once common advice and is now actively harmful.
Disallow controls crawling and lives in robots.txt. Noindex controls indexing and lives on the page as a meta tag or HTTP header. They solve different problems, and using the wrong one is the most common robots.txt mistake there is.
Google and Bing support * to match any sequence of characters and $ to match the end of a URL, so Disallow: /*.pdf$ blocks PDF files. Not every crawler supports them, so keep patterns simple.
Yes. Each subdomain, and each protocol, is treated as its own host, so blog.example.com needs its own file at its own root.
Test specific URLs against the rules before deploying, then confirm in Search Console after upload. Reading the file and assuming a pattern behaves as intended is how over-broad rules get shipped.
