Robots.txt Tester – Validate Your Robots.txt Rules
About the Robots.txt Tester and Validator
This tester reads a robots.txt file, checks it for syntax problems, and shows exactly which rules a given crawler will follow. Point it at a live URL or paste the file contents directly, choose a user agent such as Googlebot or Bingbot, and see the active Allow and Disallow rules alongside any errors or warnings.
Testing matters more with robots.txt than with almost any other file on a site, because the failure mode is silent and expensive. A malformed directive does not throw an error or break a page. It simply gets ignored, or worse, matches far more than you intended, and you find out weeks later when traffic to a section has quietly disappeared. A single stray character in a Disallow line can remove an entire category from search.
The most common cause of an unexpected block is not a typo but the group matching rules. A crawler reads only the most specific user-agent group that applies to it and ignores every other group entirely. If you add a block for User-agent: Googlebot with two rules in it, Googlebot stops reading the User-agent: * section completely — including any Disallow lines you assumed were global. This is exactly the kind of thing that is invisible when reading the file and obvious the moment you test a specific URL against a specific bot.
Path matching is the other frequent surprise. Rules match from the start of the URL path and are case-sensitive, so Disallow: /Blog does nothing on a site serving /blog/. And a rule without a trailing slash matches as a prefix: Disallow: /news blocks /news/, /newsletter/, and /news-archive/ alike, which is rarely what someone means when they write it.
Testing before deployment is the habit worth building. Pasting a draft file into this tool and checking your important URLs against it takes a minute and catches the over-broad pattern before it reaches production, rather than after.
Members of the BigToolSite SEO team ran this against files with deliberate syntax errors, conflicting Allow and Disallow rules, wildcard patterns, and multiple user-agent groups, verifying that the reported active rules match what Google’s own parser resolves for the same input.
If a test reveals rules you need to rewrite, the Robots.txt Generator builds a corrected file with the syntax already valid.
A Sitemap line only helps if the file it points at exists and is current, and the XML Sitemap Generator produces one you can verify against the URL in your robots.txt.
Crawl problems often turn out to be redirect problems rather than robots problems, and the Redirect Chain Checker traces where a URL actually ends up.
How to Use the Robots.txt Tester
- Enter your robots.txt URL, such as
https://example.com/robots.txt, or paste the file contents to test a draft before publishing. - Select a User Agent from the list, or type any user-agent string. Test the specific bot you care about rather than assuming rules apply universally.
- Click Test.
- Check the syntax status first. Errors mean directives that crawlers cannot parse; those lines are ignored entirely.
- Read the Warnings. These flag things that parse but probably do not do what you meant, such as a malformed sitemap URL or a rule outside any user-agent group.
- Review the Active Rules table. This is the important part: it shows only the rules your selected bot will actually obey, after group matching is resolved.
- Expand View Raw robots.txt Content to confirm the file being tested is the one you think it is, which catches caching and staging mix-ups.
- Run the test again for each crawler you care about. Different bots frequently resolve to different rule sets from the same file.
Example Usage
Testing this file as Googlebot:
User-agent: * Disallow: /private/ Disallow: /tmp/ User-agent: Googlebot Disallow: /no-google/
Active rules for Googlebot: only Disallow: /no-google/
The /private/ and /tmp/ blocks do not apply to Googlebot at all, because it matched its own group and stopped reading. Anyone scanning this file visually would assume all three rules were in force for Google. This single behavior accounts for a large share of robots.txt incidents.
A prefix-matching example worth testing on your own site:
User-agent: * Disallow: /news
Testing /newsletter-signup/ against that rule returns blocked, because the pattern matches any path beginning with /news. Adding the trailing slash as Disallow: /news/ limits it to the directory you meant.
When conflicting rules exist, Google applies the most specific match rather than the first one listed. Given Disallow: /folder/ followed by Allow: /folder/public/, the longer Allow path wins for URLs inside it. Rule order does not decide the outcome, path length does, which is another thing that is much easier to confirm by testing than by reading.
Frequently Asked Questions
Syntax validity, unknown or misplaced directives, malformed sitemap URLs, and which Allow and Disallow rules actually apply to the user agent you select once group matching is resolved.
Yes. Paste the draft contents instead of entering a URL. Testing a draft against your important paths is the reliable way to catch an over-broad rule before it goes live.
Most likely because Googlebot has its own user-agent group elsewhere in the file. A crawler follows only the most specific group that matches it and ignores all others, including the wildcard group.
Rules match as prefixes from the start of the path. Disallow: /news blocks /newsletter/ and /news-archive/ as well as /news/. Add a trailing slash to limit a rule to a directory.
The paths are. Disallow: /Blog will not block /blog/. Directive names such as Disallow and User-agent are not case-sensitive, but the paths after them must match your URLs exactly.
Google applies the most specific rule, meaning the one with the longest matching path, regardless of the order they appear in. Other crawlers may resolve conflicts differently, so test rather than assume.
Everything is crawlable by default. A missing file is a valid state, not an error, though the file is worth having so you can declare a sitemap and steer crawlers away from low-value URLs.
No. It only confirms crawlers are permitted to fetch them. Indexing depends on content quality, internal linking, canonical tags, and whether a noindex directive is present.
Yes. Enter any user-agent string, including SEO crawlers, AI training bots, and custom scrapers, to see which rules would apply to them.
Usually caching. Your browser may be serving an older copy of the file. Expand the raw content view to confirm which version was actually tested.
After every change, and before any site launch or migration. A staging file carrying Disallow: / into production is one of the most damaging and most common SEO accidents there is.
