⚡ Free Tool · No sign-up

Free Robots.txt
Generator

Create a valid robots.txt file in seconds. Control which pages search engines crawl, set a crawl-delay and point bots to your sitemap.

✓ No sign-up✓ Instant results✓ Free forever

What is robots.txt?

robots.txt is a plain-text file at the root of your domain that tells search-engine crawlers which paths they may or may not request. It's the first file most bots fetch. Used correctly it protects crawl budget; used carelessly it can accidentally hide your whole site from Google.

How to use it

  • Choose a default rule, then add specific disallow/allow paths.
  • Add your sitemap URL so crawlers discover your pages faster.
  • Save the output as robots.txt and upload it to your domain root.

Found unindexed backlinks?

Don't wait weeks for Google. Submit them to BacklinkIndex and get them crawled in minutes.

Start with 5 free credits →

Frequently asked questions

Where do I put the robots.txt file?

At the root of your domain, e.g. https://example.com/robots.txt. It won't work in a subfolder.

Does Disallow remove a page from Google?

No. Disallow stops crawling but a blocked URL can still be indexed without content. Use a noindex meta tag to remove pages.

Should I block low-value pages?

Yes, blocking faceted/search URLs preserves crawl budget so Google spends it on pages and backlinks that matter.

My backlinks still aren't indexed — why?

robots.txt only controls your own site. Backlinks live on other domains; if they're uncrawled, force-index them with BacklinkIndex.

More free SEO tools

Backlink Index CheckerDomain Authority CheckerFree Backlink CheckerKeyword Rank CheckerRedirect CheckerHTTP Header CheckerUTM BuilderSchema GeneratorKeyword Difficulty CheckerKeyword Density CheckerSitemap ValidatorSERP Snippet PreviewOpen Graph Generator