1. Start from a preset

2. Rules for all crawlers (User-agent: *)

Default rule

Allow all Disallow all

Allow paths One per line, for example /wp-admin/admin-ajax.php

Disallow paths One per line, for example /cart or /*?sort=

Crawl-delay in seconds (optional) Google ignores this. Bing and some other crawlers respect it.

Extra user-agent groups

Give a specific crawler its own rules. A crawler follows only the most specific group that names it, and ignores the * group.

3. AI crawlers

Tick a crawler to block it from the whole site. Unticked crawlers follow your rules above.

4. Sitemaps

Sitemap URLs One per line. Use full absolute URLs.

robots.txt

robots.txt is a request, not enforcement. Well behaved crawlers follow it, but it does not stop anyone from fetching your pages, and a blocked URL can still be indexed if other sites link to it. Use authentication or noindex for anything private.