AES Encryption Explained: How It Works and Why It Matters
Jun 23, 2026
robots.txt Generator
Generate robots.txt files with custom rules for search engine crawlers.
- Home
- Web Dev
- robots.txt Generator
What is robots.txt?
robots.txt is a text file placed at the root of your website that instructs web crawlers (like Googlebot) which pages they can and cannot crawl. It follows the Robots Exclusion Standard.
This tool features:
- Multi-rule support — Create rules for different user agents (Googlebot, Bingbot, etc.).
- Dynamic path management — Add or remove Disallow and Allow paths for each rule.
- Crawl-delay — Set crawl delay for specific crawlers.
- Sitemap directive — Add your sitemap URL to help crawlers discover pages.
- Ready presets — Start from common configurations for WordPress, Laravel, and more.
robots.txt Directives Reference
| Directive | Purpose | Example |
|---|---|---|
User-agent |
Targets a specific crawler or * for all | User-agent: Googlebot |
Disallow |
Prevents crawling of specified paths | Disallow: /admin/ |
Allow |
Overrides a Disallow for a specific path | Allow: /public/ |
Crawl-delay |
Sets delay in seconds between requests | Crawl-delay: 10 |
Sitemap |
Points crawlers to your XML sitemap | Sitemap: https://site.com/sitemap.xml |
How to Use
- Choose a preset or start from scratch — Select a preset from the dropdown or add rules manually.
- Configure rules — Set user agent, add disallow/allow paths, and optionally set crawl delay. Add multiple rule blocks for different crawlers.
- Add sitemap — Enter your sitemap URL to help crawlers discover your content.
- Copy or download — Use the copy button or download the file. Place it at the root of your website.
Frequently Asked Questions
Where should I place my robots.txt file?
The robots.txt file must be placed in the root directory of your website, accessible at https://yoursite.com/robots.txt. It must be a plain text file named exactly robots.txt.
Can robots.txt prevent search engines from indexing my pages?
No. robots.txt only prevents crawling, not indexing. If a page is linked from elsewhere, it may still appear in search results without a description. Use noindex meta tags to prevent indexing.
Does Disallow: / block everything?
Yes. Disallow: / tells crawlers not to crawl any page on the site. However, they may still index pages found from external links.
Last updated: 24 Jun 2026
Blog
Latest from Our Blog
CSS Gradient Generator: Create Stunning Gradients for Your Website
Jun 23, 2026
Diff Checker: How to Compare Text and Files Effectively
Jun 23, 2026
JSON Validation: How to Validate and Fix JSON Data
Jun 23, 2026
Regex Testing: How to Test and Debug Regular Expressions
Jun 23, 2026
SSL Certificate Checker: How to Verify SSL/TLS Certificates Online
Jun 23, 2026