robots.txt Generator
Build rule groups per crawler, block AI bots, set crawl delays, and add your sitemap. The preview updates live and the syntax is always valid. Everything runs in your browser.
About this tool
Generates a robots.txt file with correct syntax: one User-agent group per crawler, Allow / Disallow / Crawl-delay directives inside it, blank lines between groups, and the Sitemap line at the end. Paths are normalized to start with /, and empty groups are skipped so the output is always valid. The AI-crawler preset covers the bots most commonly blocked today: GPTBot, CCBot, anthropic-ai, Google-Extended, PerplexityBot, and Bytespider.
Keep in mind robots.txt is advisory — well-behaved crawlers honor it, but it is not access control and does not hide pages from someone who requests them directly. Use authentication or noindex for anything that genuinely must stay out of reach. Nothing you type here leaves the page.