robots.txt generator
Build a valid robots.txt file to control how search engines index your site.
How it works
The robots.txt file is placed at the root of your site (for example https://example.com/robots.txt) and tells search engine robots which parts of the site they're allowed to crawl. Each rule associates a User-agent (the targeted robot, * for all) with one or more Allow or Disallow directives on paths. A syntax error or malformed path can accidentally block indexing of the entire site, which is why it's worth generating this file with a dedicated tool rather than by hand.
Does robots.txt really prevent access to a page?
No. robots.txt is a convention that well-behaved robots (Googlebot, Bingbot, etc.) choose to follow, but it's not a technical protection. Anyone can view the content of a "Disallow" page if they know its URL. To actually protect a resource, use server-side authentication, not robots.txt.
Should I always include a Sitemap line?
It's not mandatory, but it's good practice: specifying the URL of your sitemap.xml in robots.txt helps search engines discover all your site's pages more quickly, in addition to a direct submission via Search Console.
100% local processing in your browser — nothing is sent to a server.