Micro Tools
Advertisement

Robots.txt Generator

Generate a valid robots.txt file in seconds. Add allow and disallow rules per user-agent, plus your sitemap URL.

Rules

Add one rule per crawler group.

Advertisement

What robots.txt does

Robots.txt sits at your domain root and tells well-behaved crawlers which paths they may fetch. Each block names a user-agent (or * for all bots) followed by Allow and Disallow rules. It is a request, not a lock: major search engines honor it, but malicious bots ignore it.

Common patterns

Blocking /admin/, /cart/ and search-result pages keeps crawlers on content that matters. Allowing everything with Disallow: (empty) is the default open posture. Adding your sitemap URL at the bottom points crawlers at the full page list in one step.

Related tools

JSON formatterFormat and validate JSON, privately. Regex testerTest regex live with highlighting. Base64 encoderBase64 encode and decode, plus URL encoder and JWT decoder.

Robots.txt Generator FAQs

Where do I put the robots.txt file?
At the root of your domain, so it is reachable at example.com/robots.txt. Crawlers only check that exact location.
Does robots.txt hide pages from Google?
It stops crawling but not necessarily indexing. A blocked URL can still appear in results if other sites link to it. For truly private content, use passwords or noindex instead.
What does Disallow with an empty value mean?
It allows everything. Disallow: / blocks the whole site, while Disallow: (blank) blocks nothing.
Can I block a specific bot?
Yes. Put the bot's user-agent name, like GPTBot, in the user-agent field and add Disallow: / to block it while leaving others untouched.
How do I test my robots.txt?
After uploading, use Google Search Console's robots.txt tester to confirm Google parses it the way you intended.
Advertisement