SEO Tools

Robots.txt
Generator

Q: Where do I put robots.txt?

It must live at the root of your domain — https://example.com/robots.txt. Crawlers always look there; it will not work in a subfolder.

Q: Should I add my sitemap to robots.txt?

Yes. A Sitemap: line pointing to your sitemap.xml helps search engines discover all your pages faster. It is independent of the allow/disallow rules.

Build a valid robots.txt to tell search engines what to crawl. Start from a preset, add disallow rules and your sitemap, then copy it to your site root.

Crawler rules

Sitemap line

Presets

User-agent: *
Disallow: /admin/
Allow: /

Sitemap: /sitemap.xml

Frequently Asked Questions

Where do I put robots.txt?

It must live at the root of your domain — https://example.com/robots.txt. Crawlers always look there; it will not work in a subfolder.

Does Disallow hide a page from Google?

No. Disallow stops crawling, but a blocked URL can still be indexed if other sites link to it. To keep a page out of search results, use a noindex meta tag and allow crawling.

What does User-agent: * mean?

The asterisk applies the rules to all crawlers. You can target a specific bot instead, e.g. User-agent: Googlebot, to give it different rules.

Should I add my sitemap to robots.txt?

Yes. A Sitemap: line pointing to your sitemap.xml helps search engines discover all your pages faster. It is independent of the allow/disallow rules.

Related Tools

Redirect Generator Nginx & Apache Config Word & Character Counter

Robots.txtGenerator

Frequently Asked Questions

Related Tools

Put Your Site on a Fast, SEO-Ready VPS

Robots.txt
Generator