What is robots.txt used for?

robots.txt tells crawlers which paths they can access and which paths should be blocked from crawling.

Can I add a sitemap URL in robots.txt?

Yes, you can include your sitemap.xml URL so search engines can discover your important pages faster.

No, it is completely free and runs directly in your browser.

Only block your entire website if it is a temporary staging or private environment. Public production websites usually allow crawling for key pages.

robots.txt controls crawling, not guaranteed indexing. For stronger indexing control, use meta robots tags or x-robots-tag headers where needed.

Common disallow paths include admin areas, private dashboards, internal APIs, and duplicate utility paths that should not be crawled.

Free SEO Tool

Generate a valid robots.txt file for your website with custom allow/disallow rules, crawl delay, and sitemap URL support.

Custom disallow paths

Sitemap URL support

Copy or download instantly

Configure crawl directives and generate a clean robots.txt file.

Website URLDisallow Paths (one per line)

Crawl Delay

Allow all crawlers by defaultInclude sitemap line

Copy the file or download it directly.

User-agent: * Allow: / Disallow: /admin/ Disallow: /private/ Disallow: /api/ Crawl-delay: 1 Sitemap: https://example.com/sitemap.xml

Enter your domain so the sitemap line is generated correctly.

Choose allow/disallow behavior and list restricted paths.

Copy the final output or download it directly as robots.txt.

Keep robots.txt simple and intentional. Block only low-value or private sections, while allowing pages you want indexed.

Always include a sitemap reference when possible. This improves crawl discovery and helps search engines find important URLs faster.

Recheck directives after site migrations or URL structure changes to avoid accidental crawl blocking.