Skip to content
Zoyku

robots.txt Generator

Build a robots.txt with AI crawler blocklists or allowlists, per-bot rules, crawl-delay and sitemap directives — then copy or download.

Presets

Site-wide rules (User-agent: *)

AI crawlers

  • GPTBotOpenAI · model training
  • OAI-SearchBotOpenAI · search index
  • ChatGPT-UserOpenAI · user-triggered fetch
  • ClaudeBotAnthropic · model training
  • Claude-UserAnthropic · user-triggered fetch
  • PerplexityBotPerplexity · answer engine
  • Google-ExtendedGoogle · AI training (not Search)
  • CCBotCommon Crawl · open dataset
  • Applebot-ExtendedApple · AI training
  • BytespiderByteDance · aggressive crawler
  • AmazonbotAmazon · AI and Alexa
  • FacebookBotMeta · AI training
  • meta-externalagentMeta · AI training (new)

robots.txt

This tool runs entirely in your browser. Your data never leaves your device — it is not uploaded to any server.

How to use

  1. 1Pick a preset: allow all, block AI training bots or block everything.
  2. 2Fine-tune each AI crawler with Allow, Block or no rule, and add Disallow paths for User-agent: *.
  3. 3Add your sitemap URL, then copy or download the file to your site root.

robots.txt Generator

robots.txt controls crawler access before a page is even fetched. Beyond the classic User-agent: * rules, you can now decide per AI bot: allow search-adjacent crawlers such as OAI-SearchBot and PerplexityBot, block training crawlers like GPTBot, ClaudeBot, Google-Extended, CCBot and Bytespider, or mix your own policy.

The generator groups user-agents that share a rule, adds crawl-delay where it is honored, and appends your sitemap directive. Copy the file to your domain root at /robots.txt and verify it in Search Console's robots report.

Frequently asked questions

Is robots.txt a security control?

No. It is advisory: compliant crawlers obey it, but anyone can read it — and it advertises your private paths. Protect sensitive areas with authentication; never rely on Disallow for security.

Do AI crawlers actually respect robots.txt?

The major documented ones — GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot — state robots.txt support, but compliance is voluntary and varies by operator. For hard guarantees, block at the CDN or firewall layer.

Why is my page still indexed after Disallow?

Blocking crawling is not blocking indexing: if other pages link to the URL, Google may still index it without its content. To remove a page from results, allow crawling and use a noindex meta tag instead.

Related tools

All SEO tools →

XML Sitemap Generator

Paste a URL list and get a valid sitemap.xml with lastmod, per-section changefreq and priority — deduplicated and ready to submit.

All tools →

Meta Tag Generator

Generate title, description, keywords, robots and viewport meta tags with live HTML output you can paste straight into your <head>.

All tools →

Canonical Tag Generator

Generate a correct rel=canonical link tag and learn how to normalize URLs so search engines index the one version you want.

All tools →

Embed this tool

Put this tool on your own site with one line — it runs entirely in the visitor's browser.

<iframe src="https://zoyku.com/en/tools/robots-txt-generator?embed=1" width="100%" height="640" style="border:0;border-radius:16px" title="Zoyku — robots-txt-generator" loading="lazy"></iframe>
100% local processing