🤖 Robots.txt Generator
Create a valid robots.txt file with user-agent rules, sitemaps, and crawl-delay directives.
User-agent: * Allow: /
How to Use the Robots.txt Generator
Use the Quick Presets for common configurations, or build custom rules by adding user-agent blocks with allow/disallow paths. Add your sitemap URL(s) so search engines can discover all your pages. When finished, click Download to save the file and upload it to your website's root directory.
Robots.txt Syntax Reference
- User-agent: * — Applies the rule to all crawlers
- Disallow: /private/ — Blocks crawling of the /private/ directory
- Allow: /private/public-page.html — Overrides a Disallow for a specific page
- Crawl-delay: 10 — Wait 10 seconds between requests (not supported by Google)
- Sitemap: — Points crawlers to your XML sitemap for better discoverability
Common Mistakes to Avoid
- Blocking CSS/JS files that Google needs to render your pages
- Using robots.txt to hide sensitive content (use authentication or noindex instead)
- Forgetting to allow your sitemap URL
- Placing robots.txt in a subdirectory instead of the root domain
Frequently Asked Questions
A robots.txt file tells search engine crawlers which pages or sections of your site they can or cannot access. It sits in your site's root directory and helps manage crawl budget by preventing bots from indexing duplicate, private, or low-value pages.
Yes. The entire tool runs client-side in your browser. Your site structure and rules are never sent to any server. You can safely generate robots.txt files for client projects or private websites.
No. Google needs to access CSS and JS files to render your pages properly. Blocking them can prevent Google from understanding your page layout and content, which may negatively impact your rankings.
Crawl-delay tells bots to wait a specified number of seconds between requests. It can help reduce server load from aggressive crawlers. Note that Google ignores this directive — use Google Search Console's crawl rate settings instead for Googlebot.
Common mistakes include accidentally blocking your entire site with Disallow: /, blocking important CSS/JS resources, forgetting to add your sitemap URL, using robots.txt to hide pages instead of noindex tags, and not testing the file before deploying it.