Question 1

What is robots.txt?

Accepted Answer

robots.txt is a text file at the root of your site (e.g. example.com/robots.txt) that tells search engine crawlers which parts of your site they may or may not crawl. It uses the Robots Exclusion Protocol.

Question 2

What's the difference between Disallow and noindex?

Accepted Answer

Disallow tells crawlers not to crawl a URL — but if a URL is linked from elsewhere, it may still appear in search results without a description. To prevent indexing entirely, use the robots meta tag with noindex instead.

Question 3

What are wildcard patterns?

Accepted Answer

The * wildcard matches any sequence of characters and $ marks the end of a URL. For example, Disallow: /*? blocks all URLs with a query string, and Disallow: /*.pdf$ blocks all PDF files.

Question 4

What extra features does this tool have compared to others?

Accepted Answer

(1) User-agent presets — Googlebot, Bingbot, Slurp, DuckDuckBot, Baiduspider, YandexBot, facebookexternalhit, AhrefsBot, SemrushBot, GPTBot. (2) Allow/disallow toggle per rule. (3) Path pattern builder with wildcard support. (4) Sitemap auto-include. (5) Crawl-delay directive. (6) Comments. (7) Live preview. (8) Validation warnings (e.g., path without leading slash). (9) Stats — rule count per user-agent. (10) History (localStorage, last 20).

Question 5

Is my data sent anywhere?

Accepted Answer

No. robots.txt generation is pure string templating in your browser. History is stored in localStorage on this device only.

Robots.txt Generator

User-agent groups

Sitemaps

Comments

How to use

User-agent groups

Sitemaps

Comments