Generate robots.txt files
User Agents
Sitemaps
Enter sitemap URLs, one per line
Quick Templates
robots.txt Preview
User-agent: * Disallow: /private Disallow: /admin Sitemap: https://example.com/sitemap.xml
About robots.txt
The robots.txt file tells search engine crawlers which pages or files they can or cannot request from your site. Place the file in the root directory of your website (e.g., example.com/robots.txt).