Most explicitly allowed-all bots

User-Agent Mentions
(%)
(%)
Googlebot 21,057,332 1.6% 656,834 3% 1,719,048 78,202 13,513,400 1,206,538 3,880,383 2,927
bingbot 17,777,028 1.35% 1,148,491 6% 671,384 7,057,256 5,666,925 574,943 2,657,869 160
facebookexternalhit 1,489,051 0.11% 189,191 12% 671,169 25,101 63,454 336,217 203,177 742
Googlebot-Image 2,212,319 0.17% 405,495 18% 528,134 33,065 246,254 240,091 759,187 93
Twitterbot 1,171,241 0.09% 91,287 7% 493,780 37,390 69,299 207,839 271,623 23
Baiduspider 15,378,609 1.16% 8,952,075 58% 491,337 25,303 5,269,715 169,892 470,224 63
Mediapartners-Google 20,294,091 1.54% 82,373 0% 483,906 22,979 12,069,953 216,404 7,418,343 133
AdsBot-Google 16,409,635 1.24% 95,532 0% 406,597 23,420 11,292,500 4,313,583 277,950 53
Yandex 9,966,109 0.75% 1,050,799 10% 219,570 7,029,574 598,984 687,899 379,089 194
AhrefsBot 16,489,721 1.25% 4,026,035 24% 217,591 20,878 4,037,518 57,597 8,130,013 89
Slurp 4,226,870 0.32% 489,736 11% 197,780 23,678 194,689 164,434 3,156,523 30
googlebot-mobile 836,566 0.06% 208,361 24% 196,278 1,399 30,377 106,313 293,768 70
MSNBot 5,482,633 0.42% 501,856 9% 191,805 20,521 153,857 170,747 4,443,809 38
ia_archiver 17,043,078 1.29% 1,635,243 9% 176,572 2,944 4,820,442 124,358 10,283,449 70
Mediapartners (Googlebot) 169,565 0.01% 894 0% 166,100 1 573 1,283 714 0
GPTBot 11,803,297 0.89% 6,893,684 58% 164,016 85,887 70,164 4,100,538 488,800 208
ChatGPT-User 4,364,787 0.33% 1,986,215 45% 144,557 13,757 48,106 2,056,672 115,404 76
PerplexityBot 3,616,851 0.27% 1,253,074 34% 135,549 5,261 16,047 2,020,930 185,935 55
360Spider 879,276 0.07% 459,640 52% 132,205 3,865 8,840 34,044 240,679 3
Googlebot-News 299,277 0.02% 29,887 9% 126,052 21,431 12,961 89,674 19,258 14
Sogou web spider 2,413,357 0.18% 1,298,644 53% 113,619 429 14,093 2,659 983,908 5
Applebot 1,321,343 0.1% 785,105 59% 113,392 3,361 73,160 99,909 246,346 70
DuckDuckBot 579,203 0.04% 67,623 11% 110,384 20,907 57,288 287,575 35,358 68
ClaudeBot 10,484,835 0.79% 6,002,169 57% 109,622 59,631 35,018 4,062,656 215,680 59
OAI-SearchBot 2,115,230 0.16% 1,711,863 80% 109,311 1,685 42,118 56,749 193,453 51
Google-Extended 9,047,722 0.69% 4,592,643 50% 108,460 70,266 35,627 4,152,043 88,606 77
LinkedInBot 263,626 0.02% 23,872 9% 106,263 23,003 18,804 57,919 33,756 9
YisouSpider 584,507 0.04% 299,545 51% 104,115 3,826 17,767 4,210 155,024 20
YandexImages 240,497 0.02% 74,475 30% 89,590 1,730 14,442 55,547 4,711 2
anthropic-ai 6,353,708 0.48% 2,138,602 33% 88,752 4,209 10,075 4,055,294 56,739 37
TermlyBot 84,066 0.01% 59 0% 83,678 1 18 299 11 0
AdsBot-Google-Mobile 11,424,274 0.87% 24,553 0% 83,162 23,226 7,111,098 4,155,670 26,556 9
YandexBot 6,337,001 0.48% 1,020,123 16% 81,292 20,806 5,140,557 25,393 48,765 65
sogou spider 6,762,147 0.51% 490,116 7% 78,543 3,822 5,109,681 3,182 1,076,798 5
SemrushBot 5,784,790 0.44% 4,266,512 73% 75,986 2,962 65,542 51,484 1,322,189 115
Bytespider 8,363,234 0.63% 6,012,119 71% 70,951 83,900 16,422 2,054,970 124,823 49
Claude-Web 3,032,158 0.23% 869,608 28% 70,375 3,948 21,330 2,011,619 55,237 41
CCBot 10,423,395 0.79% 6,159,046 59% 67,726 27,674 41,091 4,033,461 94,280 117
Yeti 1,252,724 0.09% 1,016,544 81% 66,589 1,286 43,227 63,962 61,086 30
Google 217,201 0.02% 9,117 4% 65,208 4,778 37,654 56,357 44,080 7
Sosospider 6,689,462 0.51% 1,281,703 19% 58,970 127 5,113,674 1,824 233,162 2
Yahoo! Slurp 12,418,621 0.94% 176,590 1% 58,416 7,014,334 5,110,736 42,173 16,367 5
FacebookBot 5,600,998 0.42% 1,309,472 23% 56,406 68,558 6,244 4,036,405 123,878 35
houzzbot 109,042 0.01% 4,759 4% 52,136 0 7 52,138 2 0
AhrefsSiteAudit 4,210,122 0.32% 86,340 2% 48,928 123 3,953,563 74,109 47,046 13
Baiduspider-image 616,223 0.05% 366,514 59% 48,100 267 17,705 4,838 178,796 3
Googlebot-Video 213,246 0.02% 20,430 9% 47,342 21,377 5,975 103,908 14,202 12
cohere-ai 5,203,077 0.39% 1,066,643 20% 46,119 2,732 7,915 4,025,525 54,107 36
Claude-User 247,578 0.02% 147,386 59% 43,996 193 25,024 28,746 2,204 29
Facebot 167,668 0.01% 29,277 17% 43,680 20,698 44,284 11,523 18,199 7
Diffbot 1,034,332 0.08% 931,185 90% 41,134 2,584 2,499 2,019 54,875 36
Bing 74,286 0.01% 953 1% 40,676 124 26,536 1,416 4,581 0
Screaming Frog SEO Spider 181,867 0.01% 128,113 70% 40,410 1,105 2,317 5,319 4,599 4
YoudaoBot 5,513,427 0.42% 359,398 6% 37,922 61 5,111,361 1,633 3,034 18
Scrapy 1,805,947 0.14% 1,711,988 94% 37,094 508 1,798 879 53,642 38
MJ12bot 10,457,817 0.79% 5,210,090 49% 36,820 1,762 119,974 14,978 5,074,069 124
Sogou inst spider 373,058 0.03% 154,703 41% 36,059 221 9,980 764 171,330 1
AdX-Bot 35,892 0.0% 0 0% 35,892 0 0 0 0 0
Google-Ad-Manager 35,892 0.0% 0 0% 35,892 0 0 0 0 0
meta-externalagent 9,492,431 0.72% 5,221,943 55% 35,412 10,857 12,498 4,069,293 141,302 1,126
Baiduspider-news 56,695 0.0% 15,815 27% 34,723 132 1,844 1,920 2,261 0
rogerbot 2,196,279 0.17% 1,908,848 86% 34,270 20,554 6,821 97,028 128,512 246
PetalBot 9,736,146 0.74% 9,474,164 97% 33,565 1,627 14,406 27,164 185,149 71
dotbot 13,194,319 1.0% 5,448,074 41% 32,930 20,951 36,193 11,150 7,644,960 61
Sogou News Spider 266,155 0.02% 53,117 19% 32,115 38 9,702 405 170,777 1
PowerMapper 36,224 0.0% 188 0% 31,230 0 1,020 3,673 112 1
Sogou spider2 265,205 0.02% 53,713 20% 30,540 81 9,727 676 170,467 1
Sogou Orion spider 267,321 0.02% 56,185 21% 30,495 29 9,565 403 170,644 0
Sogou blog 230,411 0.02% 48,013 20% 30,237 6 9,390 367 142,398 0
AdsBot-Google-Mobile-Apps 4,205,847 0.32% 16,042 0% 28,158 19,941 9,208 4,121,416 11,073 9
Perplexity-User 263,365 0.02% 192,419 73% 28,156 279 2,262 38,165 2,070 14
YouBot 2,575,154 0.2% 413,783 16% 27,510 1,406 2,982 2,048,075 81,354 44
Claude-SearchBot 184,173 0.01% 129,526 70% 23,833 220 24,612 4,061 1,895 26
Baiduspider-video 381,906 0.03% 347,929 91% 23,429 196 1,865 4,339 4,147 1
googlebot-Smartphone 23,547 0.0% 15 0% 23,288 0 61 73 110 0
AdIdxBot 290,608 0.02% 16,514 5% 22,961 1,340 37,720 36,812 175,254 7
msnbot-media 160,470 0.01% 66,884 41% 22,712 23,467 11,378 12,905 23,113 11
SiteAuditBot 566,118 0.04% 409,029 72% 22,189 1,306 10,569 27,907 95,112 6
Amazonbot 9,657,156 0.73% 5,357,087 55% 21,900 30,296 52,455 4,074,322 120,983 113
Meta-ExternalFetcher 377,462 0.03% 257,668 68% 20,453 2,106 6,988 9,778 80,431 38
HaosouSpider 354,552 0.03% 192,041 54% 17,515 64 1,502 484 142,944 2
BingPreview 310,629 0.02% 7,926 2% 16,693 20,157 5,450 90,603 169,790 10
DataForSeoBot 1,361,803 0.1% 1,313,925 96% 16,357 447 2,274 5,611 23,173 16
NaverBot 1,105,116 0.08% 968,201 87% 15,636 536 35,120 32,780 52,829 14
Pinterestbot 164,825 0.01% 41,595 25% 15,618 4,497 34,630 34,826 33,653 6
serpstatbot 1,035,920 0.08% 987,019 95% 15,561 32 17,185 199 15,918 6
Applebot-Extended 8,278,025 0.63% 4,067,732 49% 15,483 57,474 4,745 4,051,706 80,846 39
SEOkicks 355,491 0.03% 332,095 93% 15,426 21 384 177 7,385 3
YandexDirect 93,493 0.01% 61,369 65% 15,304 8 1,197 673 14,942 0
Baiduspider-render 58,033 0.0% 2,065 3% 14,777 8 7,600 247 33,336 0
WhatsApp 26,683 0.0% 4,442 16% 14,468 211 1,585 3,161 2,813 3
Google-InspectionTool 75,222 0.01% 12,495 16% 13,701 1,005 2,587 42,510 2,922 2
Teoma 1,506,479 0.11% 1,317,637 87% 13,201 1,493 10,753 87,596 75,791 8
archive.org_bot 1,011,013 0.08% 917,695 90% 12,786 507 11,689 62,308 5,981 47
GoogleOther 2,530,744 0.19% 304,849 12% 12,418 45,536 3,980 2,079,092 84,847 22
Nutch 5,477,719 0.41% 5,392,703 98% 12,251 1,003 4,889 569 66,254 50
Sogou Pic Spider 17,925 0.0% 5,494 30% 12,141 3 188 43 56 0
Storebot-Google 69,066 0.01% 11,932 17% 12,027 1,191 2,115 35,790 6,001 10
Sogou 7,225,182 0.55% 7,154,469 99% 11,673 68 7,461 29,369 22,118 24
Exabot 1,688,786 0.13% 1,626,574 96% 11,342 776 8,875 33,105 8,101 13

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.