robots.txt Checker & Tester
Validate a robots.txt file and test how crawlers interpret its rules.
- Home
- Web Dev
- robots.txt Checker & Tester
How to Use
- Paste the rules — Drop your robots.txt content into the input area, or click Valid Example or Invalid Example to load a sample file.
- Run the check — Click Check robots.txt to parse every line and validate the directives.
- Read the report — Review the directive, group, and sitemap counts, then work through the errors and warnings that were found.
- Fix and re-check — Correct the flagged lines and run the check again until the report shows no syntax problems.
Everything runs 100% client-side in your browser. The file you paste is never uploaded or stored on any server.
What is robots.txt?
robots.txt is a plain text file placed at the root of a website that tells search engine crawlers which pages they may or may not request. It is part of the Robots Exclusion Protocol and is read before a crawler fetches any other URL on the site. It uses simple name: value directives such as User-agent, Disallow, Allow, and Sitemap.
The file is grouped into sections. Each group starts with one or more User-agent lines and is followed by the rules that apply to those crawlers. A group for User-agent: * applies to every bot that does not have a more specific group of its own, which is why it is usually the safest place to put site-wide rules.
This checker parses a robots.txt file and produces a quick report so you can catch mistakes before they affect indexing. For each file it shows how many directives were recognised, how many groups were detected, and how many sitemap references were found. It also flags line-level errors (such as a missing colon) and warnings (such as an unknown directive, an empty user-agent, a rule that appears before any user-agent, or a value that contains extra whitespace).
Supported Directives
| Directive | What it does |
|---|---|
| User-agent | Names the crawler the following rules apply to (* means all crawlers). |
| Disallow | Blocks the crawler from a path. Disallow: / blocks the entire site. |
| Allow | Explicitly permits a path, useful to re-open a sub-path inside a disallowed directory. |
| Sitemap | Points crawlers to your XML sitemap so they can discover URLs. |
| Crawl-delay | Suggests a delay between requests. Note that Google ignores this directive. |
| Host / Clean-param / Request-rate | Non-standard directives understood by some crawlers; recognised but not required. |
robots.txt Examples
Allow everything (default)
User-agent: *
Disallow:
Sitemap: https://example.com/sitemap.xmlBlock a directory but keep one path
User-agent: *
Disallow: /admin/
Allow: /admin/public/Block everything
User-agent: *
Disallow: /Separate rules per crawler
User-agent: Googlebot
Disallow: /private/
User-agent: *
Disallow: /tmp/Common robots.txt Mistakes
- Missing the colon separator — every directive must be written as
Name: value. A line with no colon is an error. - Rules before any User-agent — a
DisalloworAllowthat appears before the firstUser-agentline has no group to attach to. - Whitespace inside values — a path such as
Disallow: /my folder/is ambiguous; URLs should not contain spaces. - Misspelled directives — typos like
Disaloware ignored by crawlers, so the rule silently does nothing. - Blocking the whole site —
Disallow: /underUser-agent: *removes your pages from search. Use it only for staging or private sites. - Blocking CSS and JS — crawlers need these files to render pages; blocking them can hurt your rankings.
- Forgetting the Sitemap line — adding a
Sitemapdirective helps crawlers discover your URLs faster.
Frequently Asked Questions
Can robots.txt prevent a page from appearing in search results?
It can prevent crawling, but it is not a reliable way to remove a page from search results. If a page is already indexed, blocking it in robots.txt may keep crawlers from seeing a noindex tag. Use the noindex meta tag or HTTP header for that, and combine it carefully with robots.txt.
Where must robots.txt be located?
It must be at the root of the domain, for example https://example.com/robots.txt. Crawlers do not check subdirectories for this file, so a copy placed deeper in the site will be ignored.
Does this tool fetch my live robots.txt?
No. You paste the content yourself and everything is processed in your browser; no request is made to your server. That makes it safe to test files before you publish them.
How does Allow interact with Disallow?
When both match, the more specific (longer) rule wins. For example, Disallow: /admin/ with Allow: /admin/public/ lets crawlers reach the public sub-path while blocking the rest of the admin directory.
Is robots.txt case-sensitive?
Directive names are case-insensitive, so User-agent and user-agent behave the same. Path values, however, are case-sensitive on most servers, so /Images/ and /images/ are treated as different URLs.
Does robots.txt stop all bots and scrapers?
No. robots.txt is a voluntary protocol. Reputable search engines honour it, but malicious scrapers can ignore it entirely. Use it to guide legitimate crawlers, not as a security control.