Robots.txt Generator for AI Crawlers: GPTBot and ClaudeBot

Quick setup

"Block AI training only" keeps you visible in Google, Bing, ChatGPT search, Claude and Perplexity answers, while asking AI companies not to train on your content.

Rules for every crawler

WordPress siteAdds the WordPress defaults: block /wp-admin/ but allow admin-ajax.php. Also block these paths (optional), one per linePaths start with /. * matches anything, $ marks the end of the URL. Sitemap URL (optional)

robots.txt

Upload it to the root of your site so it opens at yoursite.com/robots.txt. On WordPress, a real robots.txt file replaces the one WordPress makes for you. Most crawlers see changes within 24 hours. DuckDuckGo says 72 hours for DuckAssistBot.

Runs in your browser. Nothing you type or upload is sent to our server.

What does this robots.txt generator do?

It builds a robots.txt file that lets you decide, bot by bot, which AI crawlers may read your site. Each crawler sits in one of four groups: search engines, AI search, AI training and user-triggered fetchers. You switch each one on or off, add your sitemap URL, and copy the result. The WordPress site option adds the rules WordPress itself uses.

How do you use it?

Choose a preset: allow all, block AI training only, or block all AI.
Adjust individual bots with the toggles. The note under each bot explains what blocking it does.
Keep “WordPress site” ticked if your site runs WordPress.
Enter your sitemap URL, for example https://yourdomain.com/wp-sitemap.xml or the sitemap index from your SEO plugin.
Copy the file and replace the contents of /robots.txt at the root of your domain.

Which AI crawlers should you block?

Blocking is a trade-off. Training bots take content to build models and send no visitors. Search bots decide whether your pages can be cited and linked in AI answers. For most businesses that want leads, the sensible default is allow search, decide on training.

Group	Bots	If you block them
Search engines	Googlebot, Bingbot, Applebot	You disappear from normal search. Almost never block.
AI search	OAI-SearchBot, Claude-SearchBot, PerplexityBot, DuckAssistBot, Amzn-SearchBot	Your pages are less likely to be cited in ChatGPT, Claude, Perplexity or DuckDuckGo answers.
AI training	GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, Amazonbot	You ask companies not to use your content for model training. Search visibility is not affected.
User fetchers	ChatGPT-User, Claude-User, Perplexity-User, meta-externalfetcher, Amzn-User	The assistant may refuse to open your page when a user pastes your link. Several of these state they may ignore robots.txt.

Two control tokens are easy to misread. Google-Extended does not crawl and does not affect Google Search; Google says it controls use of content for Gemini training and grounding only. AI Overviews and AI Mode are part of Search, and Google states on its AI features page that Googlebot rules and snippet controls are the way to manage them. Applebot-Extended works the same way for Apple.

How do robots.txt rules work?

The format is defined in RFC 9309. The rules that matter most:

One group per bot. A crawler follows the group whose User-agent matches its name. It ignores the * group if a named group exists. So a bot with its own group does not inherit your WordPress rules from *.
Longest match wins. When an Allow and a Disallow both match a URL, the rule with the longer path applies. If they are the same length, Allow wins.
Paths are case sensitive. Bot names are not.
Wildcards. * matches any characters and $ marks the end of a URL.
It is a request, not a lock. The RFC says the rules are not a form of access authorization. Polite crawlers follow them; others may not.
Caching. Crawlers should not cache the file for more than 24 hours, so changes take up to a day. DuckDuckGo states 72 hours for DuckAssistBot.

What does WordPress put in robots.txt by default?

If there is no physical robots.txt file, WordPress serves a virtual one from do_robots(). Since WordPress 5.5, core also adds a sitemap line when the site is public (WP_Sitemaps::add_robots).

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

Sitemap: https://yourdomain.com/wp-sitemap.xml

Keep the admin-ajax.php line. Some themes and plugins load front-end content through it, and blocking it can stop crawlers from rendering pages correctly.

Worked example: block training, keep AI search

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: CCBot
Disallow: /

Sitemap: https://yourdomain.com/wp-sitemap.xml

OAI-SearchBot, Claude-SearchBot and PerplexityBot have no group of their own, so they follow the * group and can still read your public pages.

Common mistakes

Blocking /wp-content/ or /wp-includes/. This hides CSS, JavaScript and images that Google needs to render pages.
Using robots.txt to hide a page from Google. Google can still index a blocked URL without a snippet. Use a noindex tag instead.
Two robots.txt sources. A physical file in the web root replaces the WordPress virtual file, and SEO plugins may also edit it. Check what yourdomain.com/robots.txt actually shows.
Leaving Disallow: / from staging after a site goes live.
Relying on Crawl-delay. Google and Amazon do not support it.

For a full plan on getting cited by AI tools, read how to rank in ChatGPT or see our SEO and AI search optimization service. WordPress owners can also manage these settings with WP Toolkit Studio.

Last updated: October 19, 2026

Questions, answered

Questions about the robots.txt Generator (AI Crawlers).

Still unsure? Ask the developer directly on a free 30-minute call.

Book a free consultation

How do I block AI crawlers with robots.txt?

Add a group for each bot you want to block, such as User-agent: GPTBot followed by Disallow: /. Leave search bots like Googlebot and OAI-SearchBot without a blocking group so they can still read your pages.

Does blocking GPTBot remove my site from ChatGPT search?

No. OpenAI uses OAI-SearchBot for ChatGPT search and GPTBot for training. Blocking GPTBot only signals that your content should not be used for training.

Does Google-Extended affect AI Overviews?

No. Google says Google-Extended does not affect Google Search inclusion or ranking. AI Overviews and AI Mode are part of Search and follow Googlebot rules and snippet controls.

Do all AI bots respect robots.txt?

No. OpenAI, Meta, Amazon and Perplexity all say their user-triggered fetchers may not follow robots.txt. Robots.txt is a request, not access control.

What is the default WordPress robots.txt?

WordPress serves User-agent: *, Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php. Since version 5.5 it adds a Sitemap line pointing to wp-sitemap.xml when the site is public.

How long do robots.txt changes take to work?

The standard says crawlers should not cache the file for more than 24 hours. OpenAI, Meta and Perplexity mention up to about 24 hours and DuckDuckGo says 72 hours for DuckAssistBot.

Should a business block ClaudeBot or GPTBot?

It depends on whether you mind your content being used for model training. Blocking training bots does not reduce AI search visibility, but blocking search bots does.

Where do I upload robots.txt in WordPress?

Upload it to the web root, usually the same folder as wp-config.php. A physical file there is served instead of the WordPress virtual file.

See all free tools

Your developer Jamshaid Ali Founder, Handicraft Marketing

Need a tool like this built for your business?

We build fast WordPress sites, custom calculators and software that run your daily work. Tell us what you need and get a clear plan within 24 hours.

Book Free Consultation Chat on WhatsApp (+92 306 9626307)

Reply within 24 hours
Free plan, no obligation
Talk directly to the developer

Robots.txt generator for AI crawlers and WordPress