Robots.txt Generator
Build a robots.txt file that controls exactly which crawlers can access your site.
What is robots.txt? It's the first file search engines check on your site. It tells crawlers which paths they can and can't access, and where to find your sitemap.
Quick Presets
Disallow Rules
Paths crawlers should not access.
Allow Rules (optional)
Paths crawlers can access, overrides a Disallow rule above it.
Common Paths to Block
Your Robots.txt File
Save this as robots.txt in your site's root directory (e.g. example.com/robots.txt).
Copied to clipboard.
Need more than a free tool?
We build the strategy, the site, and the growth engine behind it.