Most explicitly allowed-all bots

User-Agent Mentions
(%)
(%)
Googlebot 20,960,376 1.61% 631,733 3% 1,649,017 78,609 13,597,449 1,190,613 3,810,070 2,885
facebookexternalhit 1,443,357 0.11% 186,898 12% 634,625 25,892 62,282 333,931 199,230 499
bingbot 17,719,210 1.36% 1,135,760 6% 623,354 7,099,368 5,708,778 565,286 2,586,507 157
Googlebot-Image 2,181,626 0.17% 401,389 18% 508,743 33,811 244,442 237,032 756,111 98
Mediapartners-Google 20,296,996 1.56% 77,529 0% 475,743 23,009 12,154,283 214,563 7,351,734 135
Twitterbot 1,135,666 0.09% 91,138 8% 459,250 38,181 73,898 206,052 267,123 24
Baiduspider 15,383,481 1.18% 8,969,788 58% 457,465 25,297 5,302,490 171,265 457,116 60
AdsBot-Google 16,373,104 1.25% 90,348 0% 402,887 23,437 11,257,906 4,321,282 277,184 60
Yandex 9,976,754 0.76% 1,038,455 10% 207,374 7,070,817 599,979 682,241 377,696 192
AhrefsBot 16,357,614 1.25% 3,969,368 24% 204,745 20,922 4,005,485 25,507 8,131,486 101
Slurp 4,151,003 0.32% 472,335 11% 192,155 24,379 198,526 162,552 3,101,024 32
googlebot-mobile 817,464 0.06% 203,299 24% 190,644 1,404 29,077 104,095 288,880 65
MSNBot 5,383,127 0.41% 484,728 9% 180,005 21,325 155,943 169,341 4,371,743 42
ia_archiver 17,022,157 1.3% 1,589,041 9% 173,068 2,614 4,864,520 124,446 10,268,399 69
Mediapartners (Googlebot) 169,435 0.01% 873 0% 165,975 1 569 1,289 728 0
GPTBot 11,381,476 0.87% 6,486,130 56% 144,609 84,952 69,126 4,107,444 489,017 198
360Spider 852,882 0.07% 445,553 52% 129,502 3,861 8,779 35,847 229,337 3
ChatGPT-User 4,605,147 0.35% 1,978,117 42% 128,036 13,371 48,701 2,327,087 109,764 71
Googlebot-News 294,642 0.02% 29,916 10% 122,030 22,190 12,769 88,806 18,917 14
PerplexityBot 3,849,538 0.3% 1,236,795 32% 116,919 4,517 15,661 2,290,621 184,975 50
Sogou web spider 2,398,539 0.18% 1,293,186 53% 110,206 431 12,453 2,553 979,705 5
Applebot 1,299,173 0.1% 770,070 59% 110,056 4,134 73,405 97,385 244,056 67
YisouSpider 567,551 0.04% 291,333 51% 102,454 3,816 16,194 4,252 149,475 27
LinkedInBot 254,905 0.02% 21,416 8% 101,428 23,778 18,467 57,880 31,926 10
DuckDuckBot 570,305 0.04% 67,741 11% 101,115 21,709 59,402 284,950 35,320 68
OAI-SearchBot 2,090,258 0.16% 1,699,595 81% 99,444 1,307 43,131 55,099 191,636 46
Google-Extended 8,667,589 0.66% 4,218,306 48% 96,539 69,557 35,187 4,160,204 87,722 74
ClaudeBot 10,075,590 0.77% 5,600,018 55% 96,101 59,021 34,509 4,070,500 215,387 54
YandexImages 236,157 0.02% 72,099 30% 88,884 1,742 14,290 54,559 4,581 2
TermlyBot 83,608 0.01% 60 0% 83,202 2 18 314 11 1
AdsBot-Google-Mobile 11,439,504 0.88% 23,529 0% 82,550 23,252 7,118,251 4,165,566 26,347 9
anthropic-ai 6,331,308 0.49% 2,120,212 33% 77,690 4,082 9,699 4,063,764 55,825 36
YandexBot 6,346,445 0.49% 997,854 15% 77,336 20,843 5,177,948 23,788 48,607 69
sogou spider 6,783,460 0.52% 485,724 7% 74,378 3,816 5,148,083 3,158 1,068,296 5
SemrushBot 5,727,090 0.44% 4,222,838 73% 71,332 2,888 66,196 50,558 1,313,145 133
Yeti 1,255,941 0.1% 1,015,739 80% 65,951 1,284 48,312 63,618 61,008 29
Google 221,317 0.02% 9,265 4% 64,902 4,719 42,519 56,302 43,603 7
Claude-Web 3,286,759 0.25% 864,193 26% 64,386 3,543 18,219 2,282,307 54,072 39
Bytespider 7,683,377 0.59% 5,606,454 72% 63,031 84,085 16,275 1,791,632 121,856 44
CCBot 10,023,038 0.77% 5,758,611 57% 58,601 27,226 41,829 4,042,829 93,835 107
Sosospider 6,714,086 0.51% 1,276,608 19% 57,381 125 5,150,610 1,836 227,524 2
FacebookBot 5,587,725 0.43% 1,291,698 23% 54,168 67,982 6,142 4,045,141 122,560 34
Yahoo! Slurp 12,486,358 0.96% 169,294 1% 53,481 7,055,744 5,149,138 42,577 16,118 6
houzzbot 108,973 0.01% 4,756 4% 52,103 0 7 52,105 2 0
AhrefsSiteAudit 4,138,058 0.32% 83,900 2% 46,977 124 3,919,178 43,309 44,551 19
Baiduspider-image 597,314 0.05% 355,492 59% 46,973 268 15,096 4,699 174,783 3
Googlebot-Video 207,044 0.02% 19,297 9% 43,389 22,198 5,905 102,144 14,099 12
cohere-ai 5,196,537 0.4% 1,053,530 20% 43,312 2,657 7,794 4,035,886 53,328 30
Claude-User 243,342 0.02% 145,000 59% 43,035 181 26,125 27,159 1,816 26
Facebot 171,735 0.01% 30,566 17% 42,767 20,675 48,612 11,403 17,707 5
Diffbot 1,023,105 0.08% 922,877 90% 39,654 2,236 2,488 1,705 54,112 33
Scrapy 1,796,747 0.14% 1,703,567 94% 37,127 494 1,754 885 52,886 34
YoudaoBot 5,541,987 0.42% 351,736 6% 37,071 61 5,148,336 1,604 3,161 18
bing 73,804 0.01% 941 1% 36,069 122 30,825 1,409 4,438 0
Google-Ad-Manager 35,866 0.0% 0 0% 35,866 0 0 0 0 0
AdX-Bot 35,866 0.0% 0 0% 35,866 0 0 0 0 0
Screaming Frog SEO Spider 175,972 0.01% 127,258 72% 35,470 1,850 2,066 5,114 4,209 5
Sogou inst spider 361,684 0.03% 150,146 41% 34,879 217 8,461 764 167,216 1
MJ12bot 10,279,902 0.79% 5,073,658 49% 34,534 1,756 120,271 13,789 5,035,759 135
Baiduspider-news 54,619 0.0% 15,645 28% 32,814 131 1,828 1,902 2,299 0
PetalBot 9,724,015 0.75% 9,465,383 97% 32,448 1,611 14,164 27,012 183,315 82
Sogou News Spider 258,797 0.02% 52,482 20% 31,291 41 8,195 404 166,383 1
PowerMapper 35,865 0.0% 191 0% 30,954 0 994 3,613 112 1
Sogou spider2 257,934 0.02% 53,026 20% 29,702 77 8,318 665 166,145 1
Sogou Orion spider 260,066 0.02% 55,475 21% 29,678 28 8,160 400 166,325 0
rogerbot 2,095,881 0.16% 1,813,545 86% 29,555 20,583 6,805 97,287 127,862 244
Sogou blog 222,784 0.02% 46,924 21% 29,407 5 7,983 368 138,097 0
dotbot 13,048,167 1.0% 5,305,058 40% 28,183 20,969 36,254 10,133 7,647,491 79
meta-externalagent 9,092,476 0.7% 4,826,226 53% 28,147 9,820 12,223 4,077,811 137,048 1,201
AdsBot-Google-Mobile-Apps 4,215,694 0.32% 15,160 0% 28,096 19,970 9,161 4,132,208 11,090 9
YouBot 2,296,851 0.18% 403,826 17% 22,818 1,033 2,885 1,785,654 80,594 41
msnbot-media 157,825 0.01% 65,897 41% 22,735 23,270 10,319 12,827 22,766 11
Perplexity-User 254,936 0.02% 190,534 74% 21,947 272 2,161 38,351 1,658 13
Baiduspider-video 369,433 0.03% 337,458 91% 21,636 195 1,830 4,214 4,099 1
AdIdxBot 285,771 0.02% 15,488 5% 20,916 1,335 36,355 36,635 175,035 7
googlebot-Smartphone 19,884 0.0% 17 0% 19,638 0 58 74 97 0
SiteAuditBot 559,954 0.04% 406,115 72% 19,343 1,305 10,431 28,029 94,724 7
Amazonbot 9,264,960 0.71% 4,961,396 53% 18,986 29,496 51,835 4,083,506 119,632 109
Claude-SearchBot 173,832 0.01% 125,812 72% 17,924 211 25,641 2,655 1,566 23
HaosouSpider 348,011 0.03% 189,975 54% 17,435 61 1,398 477 138,663 2
NaverBot 1,109,085 0.09% 967,310 87% 15,456 535 40,315 32,456 52,999 14
YandexDirect 92,849 0.01% 61,111 65% 15,157 9 1,157 665 14,750 0
Meta-ExternalFetcher 367,493 0.03% 255,321 69% 14,771 2,077 6,867 8,711 79,710 36
Baiduspider-render 57,032 0.0% 1,857 3% 14,742 7 6,505 157 33,764 0
DataForSeoBot 1,346,391 0.1% 1,301,101 96% 14,677 426 2,255 4,927 22,974 31
serpstatbot 1,026,359 0.08% 978,627 95% 14,096 37 17,308 190 16,095 6
SEOkicks 350,208 0.03% 328,359 93% 13,940 17 390 175 7,324 3
Google-InspectionTool 74,250 0.01% 11,609 15% 13,461 1,785 2,475 41,987 2,931 2
Pinterestbot 161,153 0.01% 41,854 25% 13,396 5,291 34,022 33,687 32,897 6
BingPreview 305,843 0.02% 7,517 2% 13,189 20,193 4,858 90,329 169,747 10
Teoma 1,495,001 0.11% 1,305,261 87% 13,172 2,274 10,753 87,350 76,183 8
WhatsApp 24,217 0.0% 4,256 17% 13,067 201 1,610 2,196 2,884 3
Sogou Pic Spider 18,444 0.0% 5,278 28% 12,879 3 188 41 55 0
Applebot-Extended 7,905,900 0.61% 3,690,658 46% 12,037 56,930 4,691 4,061,754 79,793 37
Nutch 5,437,354 0.42% 5,352,592 98% 11,947 992 4,871 576 66,327 49
GoogleOther 2,259,874 0.17% 299,089 13% 11,298 45,015 3,940 1,816,528 83,983 21
archive.org_bot 971,749 0.07% 879,895 90% 11,153 192 11,609 62,042 6,811 47
Sogou 7,261,191 0.56% 7,193,715 99% 11,066 68 7,542 31,161 17,615 24
Exabot 1,680,440 0.13% 1,619,242 96% 10,946 779 8,897 32,900 7,665 11
Daumoa 74,015 0.01% 19,070 25% 9,912 201 12,186 29,636 3,006 4

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.