On this page
  1. What did we find?
  2. Which AI crawlers do Pakistani websites block?
  3. How do the blocking sites set their rules?
  4. Which sectors are most and least prepared?
  5. How many Pakistani websites have an llms.txt file?
  6. What does blocking each type of AI crawler do?
  7. Should your business block AI crawlers?
  8. How we did this
  9. What this data cannot show
  10. Summary
  11. References

Very few do. Of the 200 most visited .pk websites in the Tranco ranking, only 11 (5.5%) block any AI crawler in robots.txt, and 9 (4.5%) block GPTBot, OpenAI’s training crawler. AI search crawlers such as OAI-SearchBot and PerplexityBot were blocked by just one site, and only because it blocks every crawler, Googlebot included. Six of the 11 blocking sites use rules that Cloudflare adds for them rather than rules of their own.

We build websites and do SEO and AI search work for businesses in Pakistan, and we made a free robots.txt generator for AI crawlers. For this study we downloaded the current Tranco top sites list, took the 200 highest ranked .pk domains and read each one’s robots.txt and llms.txt on 24 October 2026.

What did we find?

MeasureSitesShare of 200
Readable robots.txt file13869%
No robots.txt (404 Not Found)3417%
A web page instead of a robots.txt file84%
Error response (403 or 502)63%
No response from our connection147%
Name at least one AI crawler in their rules147%
Block at least one AI crawler115.5%
Block GPTBot94.5%
Serve a real llms.txt file4723.5%

123 of the 138 readable robots.txt files (89%) have no rule for any AI crawler and no blanket block, so every AI bot simply follows the general rules written for all crawlers.

Which AI crawlers do Pakistani websites block?

Training crawlers are blocked far more often than search crawlers. For 14 AI user agents we checked whether the home page (/) is blocked, reading each file the way a compliant crawler does.

Bar chart: of the top 200 .pk websites, 10 block CCBot, 9 block GPTBot, 7 or 8 block each other training crawler, and 1 blocks each AI search crawler
Training crawlers are blocked by 7 to 10 of the 200 sites; AI search and user crawlers by only 1, a site that blocks every crawler.
CrawlerCompanyWhat it is forSites blocking it
CCBotCommon CrawlOpen web archive, widely used as AI training data10
GPTBotOpenAITraining9
ClaudeBotAnthropicTraining8
Google-ExtendedGoogleControl token for Gemini training and grounding8
BytespiderByteDanceReported as training (no official page)8
meta-externalagentMetaTraining and indexing for Meta products8
AmazonbotAmazonProduct improvement, may train Amazon AI8
anthropic-aiAnthropicOlder token, not on Anthropic’s current page7
Applebot-ExtendedAppleControl token for Apple model training7
OAI-SearchBotOpenAIChatGPT search1
ChatGPT-UserOpenAIOpens pages a user asks for1
Claude-SearchBotAnthropicClaude search1
Claude-UserAnthropicOpens pages a user asks for1
PerplexityBotPerplexityPerplexity search1

The one block on OAI-SearchBot, Claude-SearchBot, PerplexityBot, ChatGPT-User and Claude-User comes from a government site whose whole robots.txt is User-agent: * and Disallow: /, which shuts out Googlebot and Bingbot as well. No site in the sample blocked an AI search crawler while letting Google in.

How do the blocking sites set their rules?

Most of the blocking is not hand written:

  • 6 use Cloudflare’s managed robots.txt. Their files contain a “BEGIN Cloudflare Managed content” block, the line Content-Signal: search=yes,ai-train=no,use=reference and Disallow rules for a list of AI training crawlers. Cloudflare’s managed robots.txt documentation says it adds these rules in front of a site’s own file when the owner turns the feature on and sets an AI crawler policy to block. jazz.com.pk is one of the six.
  • 4 wrote their own rules. dailytimes.com.pk blocks seven of the training crawlers we checked. foodpanda.pk blocks GPTBot but gives OAI-SearchBot and ChatGPT-User their own groups with Allow: /, the “no to training, yes to search” setup in its simplest form. Two smaller sites block only CCBot.
  • 1 blocks every crawler with the blanket rule described above.

The intent is clear in the numbers: 8 of the 9 sites that block GPTBot leave OAI-SearchBot free to read the site. OpenAI’s crawler documentation says GPTBot collects content that may train its models, while sites that opt out of OAI-SearchBot “will not be shown in ChatGPT search answers”.

Going the other way, 7 sites give at least one AI crawler its own group that allows it. Three news sites, nation.com.pk, dailypakistan.com.pk and nawaiwaqt.com.pk, use the same list, giving GPTBot, ChatGPT-User, ClaudeBot, Google-Extended, CCBot, PerplexityBot and Amazonbot each a group with Allow: /. One side effect: under RFC 9309 a crawler with its own group ignores the * group, so these bots may also enter folders the * group closes.

Which sectors are most and least prepared?

SectorSitesReadable robots.txtNo file or a web pageBlock an AI crawlerReal llms.txt
Government34141420
Education44242022
News1414013
Shopping50480139
Telecom and internet116310
Banks and finance41000
Other4331543

Government and education sites were the least prepared: 34 of their 78 sites had no robots.txt at all or served a web page at that address, leaving crawlers without instructions. All 14 news sites and 48 of the 50 shopping sites had a readable file, mostly generated by their platform. Blocking was rare in every sector, never more than 4 sites.

How many Pakistani websites have an llms.txt file?

47 of the 200 sites (23.5%) returned a real text file at /llms.txt. That looks like fast adoption until you see who made them. 37 are the same “Agent Instructions” file that Shopify serves for its stores: Shopify’s theme documentation says it manages an agents.md file for every store and that /llms.txt mirrors it by default. All 37 Shopify stores in the sample had one.

Among the other 163 sites, only 10 (6.1%) had an llms.txt. Two say they were generated by Yoast SEO and one by Rank Math. The other seven are a mix of hand-written files, on sites such as rozee.pk, ucp.edu.pk and arabnews.pk, and two product catalog files on stores. Arab News Pakistan’s file states that it “is not a license to train on our content”. Another 30 sites answered /llms.txt with a 200 status but sent something that was not an llms.txt file, such as a web page.

What llms.txt can and cannot do is covered in our guide what is llms.txt. If you decide to publish one, our llms.txt generator builds it from your sitemap in the browser.

What does blocking each type of AI crawler do?

TypeTokensIf you block them
TrainingGPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot, Bytespider, meta-externalagent, AmazonbotYou ask the company not to use your pages to train models. Google’s crawler overview says Google-Extended does not affect inclusion or ranking in Google Search.
AI searchOAI-SearchBot, Claude-SearchBot, PerplexityBotYour pages are less likely to be shown or cited in ChatGPT search, Claude or Perplexity answers.
User fetchersChatGPT-User, Claude-UserThe assistant may not open your page when a user asks it to. OpenAI says robots.txt rules may not apply to ChatGPT-User because a user starts the action.
Older tokenanthropic-aiAnthropic’s help article on its crawlers lists ClaudeBot, Claude-SearchBot and Claude-User and does not mention it. Use ClaudeBot instead.

Should your business block AI crawlers?

The top Pakistani sites that block anything mostly land on the same answer: 10 of the 11 block training crawlers and leave AI search crawlers open. It is a sensible default for any business that wants customers.

A checklist:

  1. Decide on training. If you do not want your text used to train models, give GPTBot, ClaudeBot, Google-Extended, CCBot and the other training tokens a group with Disallow: /. This does not remove you from Google Search or ChatGPT search.
  2. Leave the AI search crawlers alone unless you want to stay out of AI answers. No site in the sample blocked them except through a blanket block on every crawler.
  3. Never use a blanket Disallow: / on a live site. It also blocks Googlebot.
  4. Repeat your general rules in named groups. A bot with its own group no longer follows the * group, so a path such as /wp-admin/ has to be listed again if you care about it.
  5. Check Cloudflare first. 110 of the 200 sites answered our robots.txt request through Cloudflare, which can add its own rules to the top of your file.
  6. Make sure /robots.txt is a plain text file, not a web page. Eight sites in the sample returned a web page instead.

Our robots.txt generator has a “block AI training only” preset and a tester that shows which rule decides for each crawler, and the robots.txt guide for WordPress explains every rule.

How we did this

  • Sample: Tranco list 262N9, created on 23 October 2026. Tranco combines five rankings (CrUX, Farsight, Majestic, Radar and Umbrella) over 30 days and keeps registrable domains only. We took the first 200 domains ending in .pk, ranked 2,352 to 127,548. Names were reduced to the registrable domain with the Public Suffix List rules for .pk (com.pk, gov.pk, edu.pk and the rest), so www and other subdomains would merge into one site. There were no duplicates in this range, and no site was removed.
  • Fetching: on 24 October 2026 a Python script requested https://domain/robots.txt and /llms.txt once each from one connection in Pakistan, with the user agent “Mozilla/5.0 (compatible; handicraftm-research; +https://handicraftm.com)”, a 15 second timeout and no more than 2 requests per second. Redirects were followed up to five times. If HTTPS could not connect we tried plain HTTP once (14 sites), if a certificate did not verify we still read the file and noted it (10 sites), and the 15 sites that never answered were tried once more later (1 answered).
  • Parsing: RFC 9309 rules. A crawler uses every group that names its token, otherwise the * group; the longest matching rule wins and Allow wins a tie. We tested the home page and an ordinary inner page for 14 AI tokens, plus Googlebot and Bingbot for comparison, and both paths gave the same answer for every site. A file counted as robots.txt only if it came back with a 2xx status and was not an HTML page.
  • llms.txt: counted only if the final URL ended in /llms.txt, the status was 2xx and the body was non-empty text, not HTML.
  • Sectors: government and education by domain suffix (gov.pk, gop.pk, gos.pk, edu.pk) plus a few checked by hand; news, shopping, telecom and finance by hand from known brands or a Shopify storefront note in the site’s own files. Anything unclear stayed in Other.

What this data cannot show

  • Rules are requests. RFC 9309 says robots.txt rules “are not a form of access authorization”. Some crawlers may ignore the rules we saw, and sites can block bots at the firewall or CDN, which robots.txt does not show. Cloudflare’s own page points to its AI Crawl Control feature for enforcement.
  • One snapshot, one connection. A site can send different files to different user agents or countries, and 14 sites did not answer us at all.
  • Popular is not typical. Tranco ranks domains by traffic signals worldwide, not by Pakistani ownership, and the 200 busiest .pk sites do not represent small business websites.
  • Sector labels are partly manual and the groups are small, so treat sector differences as indicative.

Summary

AI crawler blocking is rare on Pakistan’s most visited websites. In October 2026 only 11 of the top 200 .pk sites (5.5%) blocked any AI crawler and 9 blocked GPTBot; six of those relied on Cloudflare’s managed rules, and the sites that block training almost always leave AI search crawlers open. Government and education sites were the most likely to have no usable robots.txt at all. llms.txt looks common at 47 sites, but 37 of those are Shopify’s automatic file. Set your own rules with our robots.txt generator, or see our SEO and AI search optimization service for help being found in Google and AI answers.

References