🤖 Robots.txt Generator
Control search engine crawlers with a custom robots.txt file
📌 What is Robots.txt? The robots.txt file tells search engine crawlers which URLs they can access on your site. It's placed in the root directory of your website and is the first file search engines check.
Quick Presets
Basic Settings
Enter your sitemap URL to help search engines find your content
Time delay between successive crawler requests (optional)
User-Agent Rules
Disallow Rules
Specify paths that crawlers should NOT access
Allow Rules (Optional)
Specify paths that crawlers CAN access (overrides Disallow)
Common Paths to Block
Your Robots.txt File
Instructions: Copy the content below and save it as robots.txt in the root directory of your website (e.g., https://example.com/robots.txt)