🤖 robots.txt Generator

Build a robots.txt for 20+ major crawlers (Googlebot, Bingbot, GPTBot, ClaudeBot, PerplexityBot, Bytespider and more) with Allow / Disallow, Crawl-delay, Sitemap and Host directives.

✅ 100% free · no signup · runs entirely in your browser (nothing is uploaded)

🔒 About Privacy

・All processing happens in your browser
・Inputs are never sent to any server
・No signup, no login, no payment ever

⚡ Presets

🤖 Crawlers (User-agent)

Allow paths (one per line)

Disallow paths (one per line)

Crawl-delay (sec)

Sitemap URL

Host (for Yandex)

📄 robots.txt

📖 How to Use

1

Pick crawlers

Check the crawlers and toggle Allow / Disallow.
2

Add paths and options

Enter Allow / Disallow paths, Crawl-delay, Sitemap URL and Host.
3

Copy or download

Copy the generated robots.txt or save as file.

❓ FAQ

Does blocking GPTBot or ClaudeBot affect SEO ranking?

No. They are separate AI training crawlers, distinct from Googlebot or Bingbot. SEO ranking is unaffected. Allow them if you want AI citations (LLMO).

Why is my Disallow ignored?

robots.txt is advisory. Malicious bots ignore it. For hard blocking, combine with HTTP 401 / Basic Auth or IP-level rules.

Where should robots.txt be placed?

It must be at the domain root (https://example.com/robots.txt). Subdirectories are not recognized.

🔗 Related Tools

🐛 Found a bug or issue with this tool?

Free to use, no signup. Even just the steps to reproduce are helpful. Reports go directly to the operator and help us fix issues.