Most explicitly allowed-all bots

User-Agent Mentions
(%)
(%)
Googlebot 23,052,427 1.63% 745,174 3% 2,668,216 84,456 13,955,500 1,337,491 4,258,556 3,034
bingbot 19,003,156 1.34% 1,191,656 6% 1,160,680 7,286,673 5,924,819 645,533 2,793,574 221
facebookexternalhit 1,945,896 0.14% 211,556 10% 1,054,540 26,939 101,447 380,506 170,801 107
Googlebot-Image 2,819,700 0.2% 412,877 14% 1,043,657 33,729 261,994 265,491 801,870 82
AdsBot-Google 15,184,119 1.07% 115,307 0% 844,035 24,217 10,234,128 3,679,421 286,962 49
Twitterbot 1,489,412 0.11% 109,136 7% 758,196 38,946 71,993 227,073 284,043 25
Baiduspider 16,318,490 1.15% 9,331,559 57% 736,241 26,139 5,521,902 199,639 502,901 109
Googlebot-News 747,426 0.05% 25,999 3% 567,877 23,861 14,317 95,056 20,300 16
Googlebot-Video 674,347 0.05% 22,246 3% 491,717 21,829 7,426 115,811 15,304 14
Mediapartners-Google 20,905,100 1.48% 90,923 0% 489,032 22,843 12,527,116 227,176 7,547,885 125
Google-InspectionTool 511,041 0.04% 12,879 2% 446,633 941 3,911 43,533 3,140 4
GPTBot 14,641,235 1.03% 10,017,954 68% 419,756 89,541 141,630 3,516,353 455,729 272
Feedfetcher-Google 493,144 0.03% 48,404 9% 412,930 162 1,016 27,878 2,753 1
APIs-Google 476,818 0.03% 6,597 1% 412,164 19,241 1,065 34,421 3,323 7
AdSense 409,242 0.03% 40 0% 408,487 0 8 3 704 0
Google-NotebookLM 504,731 0.04% 92,458 18% 408,169 104 236 3,733 21 10
AdsBot 646,242 0.05% 188,068 29% 408,005 284 999 35,595 13,287 4
PerplexityBot 2,722,299 0.19% 1,677,457 61% 367,820 7,348 78,353 430,774 160,473 74
ChatGPT-User 3,511,725 0.25% 2,485,705 70% 349,286 15,504 102,160 473,586 85,396 88
ClaudeBot 13,214,791 0.93% 9,080,274 68% 319,448 63,326 89,247 3,468,038 194,386 72
AhrefsBot 16,295,347 1.15% 4,721,582 28% 307,613 21,734 4,048,590 82,917 7,112,794 117
Google-Extended 11,646,877 0.82% 7,592,662 65% 301,190 71,195 76,671 3,544,377 60,691 91
Yandex 10,411,631 0.74% 1,122,072 10% 263,913 7,259,824 602,116 741,416 422,059 231
anthropic-ai 6,559,385 0.46% 2,763,369 42% 260,681 6,241 49,872 3,450,929 28,232 61
MSNBot 5,943,756 0.42% 541,615 9% 246,260 21,919 169,579 177,827 4,786,513 43
Slurp 4,585,410 0.32% 523,692 11% 243,161 25,097 173,914 168,670 3,450,842 34
OAI-SearchBot 2,355,510 0.17% 1,784,085 75% 233,815 2,598 96,071 75,963 162,925 53
YandexBot 6,853,466 0.48% 1,124,459 16% 219,490 21,349 5,389,043 47,685 51,364 76
Applebot 1,673,630 0.12% 944,235 56% 216,509 4,387 115,175 165,815 227,412 97
CCBot 13,147,260 0.93% 9,314,840 70% 212,954 29,429 102,306 3,422,401 65,202 128
meta-externalagent 11,719,537 0.83% 7,859,426 67% 212,786 14,602 51,345 3,459,440 119,918 2,020
360Spider 1,062,315 0.08% 526,393 49% 206,985 4,073 12,735 36,324 275,766 39
Googlebot-Mobile 927,315 0.07% 219,700 23% 195,401 1,565 29,517 143,794 337,271 67
Bytespider 12,717,640 0.9% 9,227,090 72% 186,792 84,796 54,926 3,068,309 95,623 104
DuckDuckBot 680,544 0.05% 82,606 12% 185,195 22,333 55,423 299,329 35,584 74
ia_archiver 17,967,404 1.27% 1,778,945 9% 182,955 3,197 5,045,581 124,217 10,832,405 104
Sogou web spider 2,528,026 0.18% 1,322,530 52% 178,694 482 18,645 11,994 995,675 6
Mediapartners (Googlebot) 170,001 0.01% 880 0% 166,550 1 604 1,265 701 0
Facebot 302,036 0.02% 28,752 9% 159,000 25,495 44,268 26,025 18,489 7
YisouSpider 714,203 0.05% 344,962 48% 158,838 4,011 22,735 9,708 173,920 29
Claude-Web 2,115,746 0.15% 1,485,866 70% 133,605 5,810 61,336 402,856 26,212 61
SemrushBot 6,273,855 0.44% 4,626,325 73% 128,494 3,258 81,896 77,043 1,356,707 132
LinkedInBot 294,445 0.02% 26,976 9% 124,769 25,164 19,880 63,724 33,921 11
Perplexity-User 411,760 0.03% 228,538 55% 107,656 696 25,588 46,566 2,698 18
sogou spider 7,089,597 0.5% 539,748 7% 106,945 3,984 5,343,927 5,406 1,089,582 5
Facebook 135,028 0.01% 28,330 20% 106,246 7 72 55 317 1
Claude-SearchBot 362,602 0.03% 190,248 52% 105,953 576 47,629 14,316 3,844 36
Meta 108,928 0.01% 3,221 2% 105,592 6 26 6 77 0
AdsBot-Google-Mobile 9,733,403 0.69% 25,575 0% 105,105 24,013 6,036,867 3,511,934 29,893 16
YandexImages 262,078 0.02% 80,781 30% 104,338 1,757 14,112 56,249 4,839 2
Sogou inst spider 458,087 0.03% 162,581 35% 98,279 271 14,106 1,129 181,720 1
Baiduspider-image 695,481 0.05% 374,130 53% 97,586 305 25,992 10,025 187,441 2
Sosospider 6,995,523 0.49% 1,300,621 18% 95,736 150 5,349,902 2,291 246,821 2
MJ12bot 11,340,020 0.8% 5,938,950 52% 95,621 1,884 119,450 28,550 5,155,415 150
Applebot-Extended 10,725,699 0.76% 7,054,329 65% 91,748 60,618 40,604 3,427,433 50,921 46
dotbot 12,676,992 0.9% 5,836,835 46% 88,527 21,549 119,498 28,651 6,581,830 102
rogerbot 2,355,057 0.17% 2,008,225 85% 88,504 21,135 7,425 101,338 128,264 166
PetalBot 8,993,934 0.64% 8,622,889 95% 88,066 1,818 64,789 46,558 169,726 88
FacebookBot 5,328,325 0.38% 1,610,268 30% 86,830 70,562 45,310 3,421,701 93,610 44
Baiduspider-news 108,314 0.01% 16,429 15% 85,576 150 1,903 1,939 2,317 0
TermlyBot 85,618 0.01% 104 0% 85,144 1 18 340 11 0
cohere-ai 5,272,538 0.37% 1,718,825 32% 82,878 3,717 43,645 3,399,741 23,695 37
Yahoo! Slurp 12,923,415 0.91% 202,091 1% 76,819 7,242,190 5,343,615 42,113 16,583 4
DataForSeoBot 1,902,943 0.13% 1,780,177 93% 72,800 610 2,536 22,192 24,610 18
YoudaoBot 5,835,639 0.41% 410,784 7% 72,551 95 5,347,695 2,038 2,441 35
Yeti 1,285,182 0.09% 1,034,428 80% 71,923 1,325 42,018 71,421 64,039 28
serpstatbot 1,156,027 0.08% 1,050,281 90% 71,461 35 17,889 310 16,044 7
SEOkicks 455,481 0.03% 375,949 82% 71,235 18 486 214 7,576 3
YouBot 4,128,182 0.29% 922,813 22% 68,560 2,519 38,031 3,044,442 51,769 48
Sogou spider2 317,157 0.02% 54,537 17% 66,667 121 13,808 1,026 180,997 1
Sogou News Spider 315,923 0.02% 53,754 17% 66,325 67 13,805 764 181,207 1
Google 221,686 0.02% 7,650 3% 65,704 6,270 37,656 56,937 47,463 6
Sogou Orion spider 318,057 0.02% 57,035 17% 65,525 59 13,646 719 181,073 0
Sogou blog 288,032 0.02% 50,785 17% 64,672 35 13,465 681 158,394 0
Claude-User 328,293 0.02% 180,234 54% 58,284 564 46,725 39,434 3,020 32
Bing 89,073 0.01% 987 1% 56,224 121 25,149 1,468 5,124 0
houzzbot 109,146 0.01% 4,979 4% 52,078 0 6 52,081 2 0
Amazonbot 12,240,672 0.86% 8,512,222 69% 51,865 33,124 89,196 3,463,239 90,901 125
Meta-ExternalFetcher 513,716 0.04% 352,533 68% 51,818 2,353 42,521 14,461 49,982 48
Baiduspider-video 418,166 0.03% 351,750 84% 51,314 221 1,919 8,628 4,333 1
Diffbot 1,324,969 0.09% 1,202,161 90% 49,276 3,839 37,532 7,073 25,043 45
AhrefsSiteAudit 4,249,294 0.3% 118,078 2% 49,124 137 3,947,446 79,625 54,871 13
Seekr 104,193 0.01% 56,169 53% 43,889 104 398 3,587 33 13
HaosouSpider 401,194 0.03% 197,071 49% 42,309 106 1,851 943 158,912 2
Screaming Frog SEO Spider 236,338 0.02% 178,374 75% 40,020 1,006 2,410 9,052 5,473 3
Baiduspider-render 88,531 0.01% 8,599 9% 39,702 9 12,621 1,638 25,962 0
Scrapy 1,821,088 0.13% 1,711,597 93% 35,611 669 36,518 13,628 23,023 42
Google-Ad-Manager 35,284 0.0% 0 0% 35,284 0 0 0 0 0
AdX-Bot 35,284 0.0% 0 0% 35,284 0 0 0 0 0
PowerMapper 40,543 0.0% 214 0% 34,775 0 1,657 3,783 113 1
Storebot-Google 94,523 0.01% 12,127 12% 34,035 1,209 3,477 37,758 5,908 9
Bingbot-Image 33,299 0.0% 163 0% 31,363 17 77 1,424 255 0
AdsBot-Google-Mobile-Apps 3,560,965 0.25% 15,951 0% 29,009 20,457 9,767 3,474,581 11,191 9
ToutiaoSpider 200,250 0.01% 16,079 8% 25,269 31 587 469 157,814 1
AdIdxBot 321,397 0.02% 31,838 9% 24,092 1,355 44,407 43,555 176,144 6
msnbot-media 173,354 0.01% 73,615 42% 23,598 23,749 14,825 14,409 23,147 11
NaverBot 1,119,348 0.08% 972,799 86% 23,242 561 33,555 37,323 51,853 15
WhatsApp 41,528 0.0% 5,548 13% 22,971 271 1,888 7,919 2,927 4
Sogou 7,517,090 0.53% 7,405,057 98% 22,418 91 7,728 38,793 42,947 56
Pinterestbot 207,478 0.01% 59,178 28% 22,414 4,229 36,702 49,323 35,625 7

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.