Most explicitly allowed-all bots

User-Agent Mentions
(%)
(%)
Googlebot 23,453,586 2.04% 481,182 2% 1,314,851 73,760 17,979,279 1,104,135 2,497,668 2,711
facebookexternalhit 1,242,498 0.11% 153,975 12% 501,692 23,564 58,169 314,283 190,771 44
Mediapartners-Google 17,923,708 1.56% 59,818 0% 440,003 22,020 10,080,444 209,530 7,111,765 128
Googlebot-Image 1,932,140 0.17% 315,972 16% 424,091 31,302 230,486 223,543 706,671 75
bingbot 15,123,307 1.32% 1,051,819 6% 419,530 6,334,614 4,339,620 534,507 2,443,083 134
AdsBot-Google 15,278,780 1.33% 64,408 0% 382,919 22,580 11,329,507 3,318,767 160,551 48
Twitterbot 948,081 0.08% 54,198 5% 346,474 35,822 67,216 191,591 252,761 19
Baiduspider 12,846,023 1.12% 8,064,026 62% 259,524 24,623 3,992,501 142,428 362,868 53
ia_archiver 13,338,751 1.16% 1,256,284 9% 166,095 2,435 3,551,903 122,604 8,239,369 61
Mediapartners (Googlebot) 165,204 0.01% 807 0% 161,779 4 561 1,229 824 0
AhrefsBot 15,802,856 1.37% 3,334,118 21% 155,056 20,173 3,898,993 22,998 8,371,440 78
Slurp 2,722,171 0.24% 362,820 13% 154,817 22,099 180,710 155,717 1,845,982 26
googlebot-mobile 652,100 0.06% 162,955 24% 147,240 1,396 27,161 96,872 216,421 55
MSNBot 3,831,159 0.33% 363,598 9% 145,907 19,313 148,909 165,161 2,988,237 34
Yandex 9,013,017 0.78% 1,001,110 11% 137,506 6,307,702 572,294 649,912 344,304 189
Googlebot-News 250,849 0.02% 18,935 7% 98,932 19,770 11,009 86,232 15,958 13
360Spider 659,730 0.06% 385,609 58% 89,096 3,866 8,238 10,112 162,806 3
Sogou web spider 2,324,600 0.2% 1,259,130 54% 89,070 406 12,521 2,247 961,222 4
YandexImages 226,293 0.02% 66,501 29% 86,464 1,690 13,461 54,248 3,927 2
LinkedInBot 212,512 0.02% 17,455 8% 85,110 21,472 17,397 51,451 19,620 7
TermlyBot 79,862 0.01% 36 0% 79,502 0 16 294 13 1
AdsBot-Google-Mobile 10,678,996 0.93% 19,143 0% 78,424 22,188 7,388,518 3,140,311 30,403 9
Yisouspider 457,280 0.04% 220,927 48% 77,074 3,829 16,484 4,113 134,838 15
GPTBot 8,006,036 0.7% 4,248,952 53% 72,015 69,820 62,094 3,121,984 431,022 149
DuckDuckBot 476,437 0.04% 34,990 7% 68,838 19,646 55,936 271,532 25,446 49
ChatGPT-User 5,158,651 0.45% 1,831,148 35% 68,119 1,499 40,392 3,128,009 89,427 57
Google 204,369 0.02% 7,319 3% 64,509 4,914 30,895 56,004 40,721 7
Applebot 1,190,270 0.1% 738,624 62% 63,620 2,453 67,028 88,658 229,824 63
OAI-SearchBot 1,894,072 0.16% 1,558,442 82% 63,360 680 33,855 49,141 188,568 26
Yeti 1,221,794 0.11% 1,002,076 82% 62,417 1,305 41,135 62,408 52,429 24
PerplexityBot 4,442,666 0.39% 1,088,748 24% 58,757 2,866 22,157 3,090,700 179,411 27
Google-Extended 5,635,738 0.49% 2,226,367 39% 52,990 56,525 41,093 3,175,641 83,071 51
YandexBot 4,892,747 0.43% 877,751 17% 52,393 19,798 3,879,857 20,058 42,828 62
houzzbot 108,782 0.01% 4,733 4% 52,019 0 6 52,022 2 0
ClaudeBot 6,990,653 0.61% 3,562,361 50% 48,534 46,403 40,240 3,085,797 207,289 29
SemrushBot 5,194,382 0.45% 3,767,076 72% 47,699 2,536 52,196 46,500 1,278,245 130
Sosospider 5,300,175 0.46% 1,209,062 22% 45,196 124 3,854,333 1,726 189,732 2
anthropic-ai 5,122,756 0.45% 1,928,541 37% 44,268 3,518 8,136 3,086,120 52,145 28
FacebookBot 4,423,351 0.38% 1,123,101 25% 42,304 65,519 5,273 3,068,655 118,477 22
sogou spider 5,318,413 0.46% 422,219 7% 38,805 3,813 3,850,760 3,121 999,691 4
Claude-Web 3,905,063 0.34% 719,272 18% 38,372 3,092 7,545 3,085,979 50,773 30
Scrapy 1,664,157 0.14% 1,573,591 94% 37,741 411 1,356 719 50,315 24
Yahoo! Slurp 10,372,752 0.9% 131,918 1% 37,308 6,294,293 3,852,161 41,818 15,252 2
Facebot 141,083 0.01% 15,608 11% 37,268 20,039 41,130 11,326 15,707 5
AhrefsSiteAudit 4,003,882 0.35% 69,729 1% 36,521 116 3,823,843 40,998 32,664 11
AdX-Bot 34,203 0.0% 0 0% 34,203 0 0 0 0 0
Google-Ad-Manager 34,203 0.0% 0 0% 34,203 0 0 0 0 0
cohere-ai 4,026,387 0.35% 872,193 21% 33,792 2,302 6,599 3,061,163 50,320 18
Claude-User 139,887 0.01% 50,848 36% 33,351 34 30,438 24,587 617 12
YoudaoBot 4,206,574 0.37% 318,539 7% 32,228 62 3,851,999 1,578 2,151 17
Diffbot 880,269 0.08% 791,383 89% 32,215 1,987 2,237 630 51,797 20
Sogou inst spider 318,274 0.03% 130,440 40% 31,281 195 9,416 703 146,237 2
Googlebot-Video 182,702 0.02% 16,820 9% 30,431 20,035 5,393 98,746 11,267 10
PowerMapper 34,697 0.0% 173 0% 29,986 0 887 3,545 105 1
Baiduspider-image 547,346 0.05% 352,006 64% 29,980 264 12,375 4,694 148,024 3
Screaming Frog SEO Spider 150,370 0.01% 110,438 73% 29,675 370 2,006 3,942 3,936 3
AdsBot-Google-Mobile-Apps 3,186,394 0.28% 11,624 0% 27,206 19,290 8,482 3,109,244 10,539 9
Baiduspider-news 48,207 0.0% 15,527 32% 26,693 135 1,616 2,027 2,209 0
Sogou News Spider 224,897 0.02% 44,517 19% 26,486 58 8,960 362 144,513 1
Sogou spider2 225,590 0.02% 45,009 19% 26,374 64 9,096 590 144,456 1
Sogou Orion spider 227,737 0.02% 47,392 20% 26,348 28 8,966 369 144,634 0
Sogou blog 201,368 0.02% 43,165 21% 26,093 5 8,773 338 122,994 0
bing 55,177 0.0% 848 1% 24,893 129 24,088 1,353 3,866 0
msnbot-media 148,926 0.01% 60,132 40% 22,025 22,572 10,251 12,519 21,416 11
SiteAuditBot 404,930 0.04% 263,515 65% 19,143 1,277 5,090 27,533 88,367 5
Bytespider 3,617,587 0.31% 3,396,110 93% 18,638 76,315 15,082 7,928 103,487 27
Baiduspider-video 365,240 0.03% 338,927 92% 16,902 191 1,820 4,274 3,125 1
AdIdxBot 271,285 0.02% 11,433 4% 15,173 1,063 35,744 34,691 173,175 6
HaosouSpider 334,821 0.03% 194,930 58% 14,789 46 1,316 394 123,344 2
YandexDirect 92,325 0.01% 61,634 66% 14,538 10 1,123 634 14,386 0
NaverBot 1,087,984 0.09% 958,933 88% 14,345 541 33,590 31,319 49,243 13
CCBot 6,900,870 0.6% 3,675,061 53% 14,159 24,139 35,359 3,064,371 87,700 81
Teoma 1,370,790 0.12% 1,184,960 86% 13,139 791 11,076 86,790 74,027 7
Baiduspider-render 41,179 0.0% 1,720 4% 12,933 6 3,381 188 22,951 0
Google-InspectionTool 66,438 0.01% 10,835 16% 11,733 315 1,935 39,278 2,340 2
MJ12bot 9,491,237 0.83% 4,424,904 46% 11,464 1,523 114,848 12,117 4,926,292 89
rogerbot 1,932,623 0.17% 1,667,060 86% 10,478 19,840 10,948 96,067 128,014 216
meta-externalagent 6,102,760 0.53% 2,870,671 47% 10,442 4,694 13,324 3,098,225 105,084 320
Sogou 6,445,016 0.56% 6,414,315 99% 9,764 69 6,389 5,459 9,000 20
YouBot 401,518 0.03% 309,197 77% 9,666 746 2,317 1,537 78,029 26
daumoa 72,127 0.01% 18,552 25% 9,590 212 12,068 28,960 2,742 3
YandexNews 14,550 0.0% 3,806 26% 9,522 48 310 371 493 0
Exabot 1,645,659 0.14% 1,591,315 96% 9,501 768 8,528 30,041 5,493 13
Yahoo-MMCrawler 1,174,930 0.1% 1,054,031 89% 8,884 52 9,164 9,901 92,891 7
Seznambot 4,260,283 0.37% 353,901 8% 8,781 447 3,856,951 30,692 9,499 12
BingPreview 292,715 0.03% 6,945 2% 8,692 19,354 3,850 85,459 168,406 9
Qwantify 128,230 0.01% 81,112 63% 8,539 59 19,484 4,040 14,989 7
Pinterestbot 137,164 0.01% 34,580 25% 8,377 3,297 31,028 29,434 30,444 4
facebookcatalog 58,315 0.01% 2,335 4% 8,332 264 2,172 31,193 14,019 0
dotbot 12,628,329 1.1% 4,653,083 36% 8,298 19,982 39,008 8,982 7,898,927 49
WhatsApp 14,603 0.0% 2,084 14% 7,977 186 1,314 1,519 1,523 0
Yahoo-slurp 22,862 0.0% 1,887 8% 7,267 16 588 7,921 5,183 0
Nutch 5,307,506 0.46% 5,231,623 98% 6,905 950 4,773 449 62,757 49
TelegramBot 23,476 0.0% 12,826 54% 6,868 293 1,299 1,371 817 2
Amazonbot 6,125,894 0.53% 2,827,716 46% 6,636 24,010 47,012 3,105,646 114,789 85
Pinterest 4,054,323 0.35% 17,551 0% 6,630 42 37,198 59,229 3,933,671 2
google-xrawler 6,895 0.0% 380 5% 6,461 6 20 20 8 0
Perplexity-User 137,471 0.01% 93,914 68% 6,449 153 1,104 35,267 573 11
archive.org_bot 769,121 0.07% 682,873 88% 6,330 161 12,769 61,290 5,656 42
PhindBot 21,107 0.0% 13,924 65% 5,918 14 799 256 181 15

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.