Robots.txt Generator — Free Online Tool with Presets

Robots.txt Generator — Build Your Crawler Control File

Create a complete robots.txt file in seconds. Choose from common presets (WordPress, e-commerce, staging), customize User-agent rules, add multiple sitemaps, and optionally block AI training crawlers or aggressive SEO bots. Output follows the official robots.txt specification with proper structure and comments.

Presets

• Allow all crawlers: empty Disallow — search engines can crawl everything
• Block all crawlers: Disallow: / — useful for unpublished or private sites
• WordPress: blocks /wp-admin/, /wp-includes/, plugin/cache directories, and search query URLs while allowing admin-ajax.php
• E-commerce: blocks /cart/, /checkout/, /my-account/, and add-to-cart URLs that don’t belong in search indices
• Staging / dev site: blocks everything — prevents accidental indexing

Block AI training crawlers

Toggle this option to add Disallow rules for 13 known AI bots:

• GPTBot: OpenAI training crawler
• ChatGPT-User: ChatGPT’s browse-on-demand bot
• ClaudeBot, anthropic-ai: Anthropic crawlers
• Google-Extended: Google AI training (does NOT affect Google Search)
• CCBot: Common Crawl (used by many AI companies for training)
• PerplexityBot: Perplexity AI
• Bytespider, FacebookBot, Diffbot, Omgilibot, ImagesiftBot, cohere-ai and others

Important: robots.txt is a polite request. Bots can ignore it. Reputable crawlers respect it; malicious ones don’t.

Block aggressive SEO bots

Big SEO crawlers (AhrefsBot, SemrushBot, MJ12bot, DotBot, etc.) can hammer your server. If you don’t use those tools and want to save bandwidth, enable this option. Caution: if you’re an SEO professional, you might want to keep them allowed.

Common rules

• One User-agent can have multiple Disallow/Allow lines
• Allow overrides Disallow when more specific
• Wildcards: use * for any sequence of chars, $ for end-of-URL
• Comments: start with #, ignored by crawlers
• Sitemap directive: always uses absolute URL, can appear multiple times

Deploy your robots.txt

Place the file at https://yourdomain.com/robots.txt — exactly at the root, accessible publicly. Test it with Google Search Console’s robots.txt Tester before relying on it.

Privacy

Everything runs in your browser. Your robots rules are never uploaded.