robots.txt Generator
Generate robots.txt files with custom rules for search engine crawlers.
- Home
- > Web Dev >
- robots.txt Generator
What is robots.txt?
robots.txt is a text file placed at the root of your website that instructs web crawlers (like Googlebot) which pages they can and cannot crawl. It follows the Robots Exclusion Standard.
This tool features:
- Multi-rule support — Create rules for different user agents (Googlebot, Bingbot, etc.).
- Dynamic path management — Add or remove Disallow and Allow paths for each rule.
- Crawl-delay — Set crawl delay for specific crawlers.
- Sitemap directive — Add your sitemap URL to help crawlers discover pages.
- Ready presets — Start from common configurations for WordPress, Laravel, and more.
robots.txt Directives Reference
| Directive | Purpose | Example |
|---|---|---|
User-agent |
Targets a specific crawler or * for all | User-agent: Googlebot |
Disallow |
Prevents crawling of specified paths | Disallow: /admin/ |
Allow |
Overrides a Disallow for a specific path | Allow: /public/ |
Crawl-delay |
Sets delay in seconds between requests | Crawl-delay: 10 |
Sitemap |
Points crawlers to your XML sitemap | Sitemap: https://site.com/sitemap.xml |
How to Use
- Choose a preset or start from scratch — Select a preset from the dropdown or add rules manually.
- Configure rules — Set user agent, add disallow/allow paths, and optionally set crawl delay. Add multiple rule blocks for different crawlers.
- Add sitemap — Enter your sitemap URL to help crawlers discover your content.
- Copy or download — Use the copy button or download the file. Place it at the root of your website.
Frequently Asked Questions
Where should I place my robots.txt file?
The robots.txt file must be placed in the root directory of your website, accessible at https://yoursite.com/robots.txt. It must be a plain text file named exactly robots.txt.
Can robots.txt prevent search engines from indexing my pages?
No. robots.txt only prevents crawling, not indexing. If a page is linked from elsewhere, it may still appear in search results without a description. Use noindex meta tags to prevent indexing.
Does Disallow: / block everything?
Yes. Disallow: / tells crawlers not to crawl any page on the site. However, they may still index pages found from external links.
