Most explicitly allowed-all bots

User-Agent Mentions
(%)
(%)
Googlebot 19,780,460 1.66% 560,392 2% 1,455,020 75,715 12,849,741 1,142,716 3,694,103 2,773
facebookexternalhit 1,320,410 0.11% 165,126 12% 552,994 24,664 59,619 325,958 192,001 48
bingbot 16,680,976 1.4% 1,117,325 6% 504,161 6,787,523 5,099,597 549,537 2,622,691 142
Mediapartners-Google 19,391,963 1.63% 60,643 0% 466,013 22,250 11,255,260 212,299 7,375,369 129
Googlebot-Image 2,081,696 0.17% 359,027 17% 451,054 32,457 233,501 228,641 776,934 82
AdsBot-Google 14,632,708 1.23% 68,375 0% 395,095 22,941 10,479,672 3,395,750 270,827 48
Twitterbot 1,020,687 0.09% 69,435 6% 389,295 36,962 65,963 201,216 257,798 18
Baiduspider 14,290,085 1.2% 8,630,187 60% 343,144 24,878 4,713,090 155,533 423,198 55
Yandex 9,557,560 0.8% 1,017,635 10% 184,294 6,760,237 583,430 665,443 346,335 186
Slurp 3,986,029 0.34% 430,046 10% 175,711 23,201 183,066 159,059 3,014,918 28
AhrefsBot 15,357,687 1.29% 3,721,801 24% 172,661 20,486 3,852,037 24,378 7,566,250 74
googlebot-mobile 777,119 0.07% 180,626 23% 169,211 1,392 27,699 99,860 298,277 54
ia_archiver 15,829,198 1.33% 1,449,945 9% 165,899 2,491 4,271,671 123,607 9,815,522 63
Mediapartners (Googlebot) 168,670 0.01% 822 0% 165,225 1 568 1,262 792 0
MSNBot 5,203,898 0.44% 432,415 8% 154,996 20,289 145,749 167,280 4,283,135 34
360Spider 755,377 0.06% 408,457 54% 109,292 3,872 8,943 22,153 202,656 4
Googlebot-News 267,783 0.02% 21,850 8% 108,027 20,819 11,682 87,838 17,554 13
GPTBot 9,339,306 0.79% 5,417,189 58% 104,275 82,282 60,667 3,195,017 479,707 169
ChatGPT-User 5,361,963 0.45% 1,924,441 35% 98,813 11,736 37,574 3,199,166 90,173 60
Sogou web spider 2,349,933 0.2% 1,269,049 54% 96,642 401 12,189 2,352 969,295 5
Yisouspider 507,016 0.04% 249,834 49% 91,505 3,832 16,625 3,984 141,221 15
LinkedInBot 232,969 0.02% 19,129 8% 90,843 22,557 17,937 55,471 27,025 7
YandexImages 231,680 0.02% 71,378 30% 85,922 1,721 13,767 54,379 4,511 2
DuckDuckBot 515,807 0.04% 50,342 9% 83,633 20,638 56,855 277,412 26,875 52
PerplexityBot 4,608,577 0.39% 1,166,021 25% 82,644 2,955 14,691 3,162,284 179,941 41
Applebot 1,238,726 0.1% 759,264 61% 82,588 3,281 68,151 93,831 231,549 62
AdsBot-Google-Mobile 9,940,302 0.84% 20,124 0% 81,051 22,529 6,579,673 3,210,994 25,922 9
TermlyBot 81,338 0.01% 54 0% 80,958 0 16 296 13 1
OAI-SearchBot 1,996,106 0.17% 1,645,137 82% 78,081 815 32,241 51,475 188,320 37
Google-Extended 6,774,743 0.57% 3,267,152 48% 73,583 67,398 33,570 3,248,485 84,494 61
ClaudeBot 8,130,650 0.68% 4,601,585 56% 70,238 57,227 32,780 3,159,733 209,046 41
Google 205,333 0.02% 7,279 3% 64,528 5,328 31,416 56,175 40,600 7
Yeti 1,233,036 0.1% 1,008,948 81% 62,420 1,278 41,383 62,984 55,998 25
YandexBot 5,694,334 0.48% 949,684 16% 59,894 20,885 4,597,123 21,903 44,782 63
sogou spider 6,181,755 0.52% 495,897 8% 59,704 3,850 4,567,307 3,127 1,051,865 5
anthropic-ai 5,294,503 0.45% 2,014,767 38% 58,345 3,692 8,552 3,155,446 53,671 30
Claude-Web 4,079,304 0.34% 806,477 19% 53,592 3,235 7,930 3,155,945 52,093 32
SemrushBot 5,464,823 0.46% 4,013,769 73% 53,175 2,717 53,941 49,479 1,291,636 106
houzzbot 108,963 0.01% 4,750 4% 52,102 0 5 52,104 2 0
Sosospider 6,116,968 0.51% 1,280,726 20% 50,648 141 4,569,446 1,731 214,274 2
FacebookBot 4,584,692 0.39% 1,208,219 26% 46,385 66,786 5,603 3,137,162 120,508 29
AhrefsSiteAudit 3,975,008 0.33% 76,432 1% 46,030 119 3,775,597 42,681 34,139 10
Yahoo! Slurp 11,560,960 0.97% 145,314 1% 44,335 6,745,383 4,567,997 41,743 16,184 4
Claude-User 149,685 0.01% 59,720 39% 40,706 99 21,741 26,346 1,054 19
cohere-ai 4,182,342 0.35% 953,256 22% 39,384 2,447 6,944 3,129,152 51,133 26
Facebot 152,305 0.01% 23,287 15% 38,921 20,279 41,559 11,520 16,734 5
Baiduspider-image 568,479 0.05% 350,936 61% 38,391 253 12,614 4,636 161,646 3
Scrapy 1,747,961 0.15% 1,656,922 94% 37,284 465 1,443 791 51,025 31
Diffbot 967,581 0.08% 872,681 90% 37,133 2,090 2,331 797 52,521 28
Googlebot-Video 193,926 0.02% 17,969 9% 35,424 21,053 5,510 101,110 12,849 11
Google-Ad-Manager 35,265 0.0% 0 0% 35,265 0 0 0 0 0
AdX-Bot 35,265 0.0% 0 0% 35,265 0 0 0 0 0
YoudaoBot 4,933,114 0.41% 327,506 6% 34,766 61 4,567,128 1,566 2,072 15
Bytespider 4,844,914 0.41% 4,590,578 94% 34,465 86,970 15,181 8,304 109,384 32
Sogou inst spider 335,890 0.03% 138,674 41% 33,050 198 8,825 731 154,410 2
Bing 61,263 0.01% 888 1% 30,670 128 24,099 1,369 4,109 0
PowerMapper 35,339 0.0% 177 0% 30,541 0 939 3,576 105 1
Screaming Frog SEO Spider 164,387 0.01% 122,408 74% 30,281 1,142 1,995 4,607 3,948 6
Baiduspider-news 49,934 0.0% 15,274 30% 28,666 126 1,623 1,927 2,318 0
Sogou News Spider 239,494 0.02% 49,764 20% 28,228 42 8,366 363 152,730 1
Sogou spider2 240,142 0.02% 50,264 20% 28,115 62 8,494 616 152,590 1
Sogou Orion spider 242,308 0.02% 52,686 21% 28,105 28 8,352 364 152,773 0
AdsBot-Google-Mobile-Apps 3,257,870 0.27% 12,224 0% 27,845 19,545 9,015 3,178,540 10,692 9
Sogou blog 209,544 0.02% 43,857 20% 27,831 5 8,168 334 129,349 0
CCBot 8,061,990 0.68% 4,752,918 58% 24,999 25,423 36,004 3,133,885 88,666 95
msnbot-media 151,446 0.01% 62,241 41% 21,696 22,860 10,243 12,699 21,696 11
SiteAuditBot 535,501 0.05% 392,285 73% 18,095 1,290 5,149 27,479 91,197 6
YouBot 419,839 0.04% 317,813 75% 17,765 885 2,652 2,370 78,316 38
Baiduspider-video 364,505 0.03% 336,673 92% 17,558 181 1,822 4,198 4,072 1
AdIdxBot 279,130 0.02% 14,804 5% 16,281 1,084 35,416 37,244 174,295 6
HaosouSpider 340,090 0.03% 192,433 56% 16,257 43 1,255 432 129,668 2
meta-externalagent 7,215,592 0.61% 3,907,687 54% 16,006 7,979 11,556 3,168,751 102,992 621
YandexDirect 93,624 0.01% 62,329 66% 15,007 10 1,157 651 14,470 0
Baiduspider-render 49,641 0.0% 1,784 3% 14,247 6 4,101 217 29,286 0
NaverBot 1,094,357 0.09% 962,126 87% 14,102 527 33,673 32,199 51,716 14
PetalBot 9,036,447 0.76% 8,800,096 97% 13,813 1,472 14,721 25,989 180,297 59
Teoma 1,491,510 0.13% 1,302,505 87% 12,809 1,567 11,035 87,617 75,970 7
Amazonbot 7,267,810 0.61% 3,888,138 53% 12,418 27,846 47,888 3,175,468 115,955 97
Google-InspectionTool 70,456 0.01% 11,287 16% 12,140 1,086 1,963 41,373 2,605 2
MJ12bot 9,723,694 0.82% 4,700,403 48% 10,848 1,658 114,781 12,819 4,883,090 95
Perplexity-User 150,309 0.01% 100,885 67% 10,781 219 1,648 35,841 923 12
rogerbot 1,965,096 0.17% 1,703,279 86% 10,626 20,131 8,294 96,310 126,235 221
BingPreview 300,307 0.03% 7,225 2% 10,308 19,595 3,912 89,946 169,312 9
Exabot 1,673,668 0.14% 1,614,153 96% 10,297 789 8,780 32,811 6,823 15
WhatsApp 20,109 0.0% 3,954 19% 10,179 187 1,499 1,808 2,481 1
daumoa 73,176 0.01% 18,663 25% 9,731 196 12,216 29,578 2,788 4
Pinterestbot 148,343 0.01% 36,754 24% 9,595 4,336 33,050 32,852 31,751 5
Sogou 6,916,222 0.58% 6,873,193 99% 9,170 69 6,770 17,522 9,476 22
YandexNews 16,318 0.0% 5,967 36% 9,111 51 332 368 489 0
dotbot 12,095,589 1.02% 4,923,198 40% 8,748 20,310 37,124 9,561 7,096,597 51
Seznambot 5,002,226 0.42% 376,535 7% 8,739 1,225 4,572,851 31,971 10,893 12
Yahoo-MMCrawler 1,271,703 0.11% 1,087,525 85% 8,706 54 9,350 9,986 156,075 7
Qwantify 134,245 0.01% 86,667 64% 8,579 54 19,800 4,157 14,982 6
facebookcatalog 60,083 0.01% 2,527 4% 8,506 291 2,292 32,086 14,381 0
Claude-SearchBot 67,649 0.01% 35,497 52% 8,446 119 21,090 1,825 654 18
Sogou Pic Spider 13,278 0.0% 5,042 37% 7,949 4 190 34 59 0
TelegramBot 27,865 0.0% 13,990 50% 7,650 1,067 1,408 2,005 1,742 3
googlebot-Smartphone 7,686 0.0% 11 0% 7,476 0 47 70 82 0
Yahoo-slurp 23,018 0.0% 1,902 8% 7,165 15 591 8,067 5,278 0
Nutch 5,266,292 0.44% 5,188,353 98% 7,011 952 4,803 509 64,618 46

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.