Most explicitly allowed-all bots

User-Agent Mentions
(%)
(%)
Googlebot 19,562,605 1.83% 395,868 2% 1,130,054 71,080 15,217,264 1,016,529 1,729,701 2,109
facebookexternalhit 1,075,721 0.1% 86,716 8% 438,371 23,106 62,566 283,550 181,382 30
Mediapartners-Google 16,464,640 1.54% 57,102 0% 401,177 21,397 9,056,251 189,071 6,739,507 135
Googlebot-Image 1,796,460 0.17% 290,704 16% 382,413 30,708 225,825 200,739 665,994 77
AdsBot-Google 12,347,826 1.15% 58,648 0% 366,446 20,255 8,844,934 2,911,255 146,248 40
bingbot 13,577,887 1.27% 949,890 6% 345,756 5,840,930 3,768,250 482,663 2,190,309 89
Twitterbot 854,553 0.08% 45,619 5% 296,553 35,460 71,691 165,563 239,651 16
Baiduspider 11,359,040 1.06% 7,302,378 64% 167,036 24,499 3,431,983 112,582 320,517 45
ia_archiver 11,331,152 1.06% 1,061,460 9% 161,813 2,281 3,022,855 101,629 6,981,090 24
Mediapartners (Googlebot) 160,542 0.02% 795 0% 157,224 3 544 1,183 793 0
AhrefsBot 13,243,550 1.24% 3,295,250 24% 143,222 19,718 3,521,295 21,651 6,242,346 68
msnbot 2,991,112 0.28% 325,867 10% 141,102 19,201 148,006 142,447 2,214,455 34
googlebot-mobile 585,171 0.05% 146,588 25% 139,201 1,312 25,387 92,817 179,851 15
Slurp 1,893,697 0.18% 317,882 16% 136,655 20,008 181,723 132,249 1,105,153 27
Yandex 8,228,101 0.77% 785,211 9% 115,200 5,815,563 559,230 593,536 359,247 114
Googlebot-News 242,360 0.02% 16,912 6% 97,731 19,509 9,636 83,483 15,077 12
YandexImages 221,315 0.02% 66,380 29% 84,162 1,625 12,103 53,460 3,583 2
LinkedInBot 189,401 0.02% 14,541 7% 76,199 21,146 24,541 34,187 18,780 7
TermlyBot 75,306 0.01% 29 0% 74,981 0 17 267 11 1
AdsBot-Google-Mobile 8,154,765 0.76% 17,025 0% 74,416 19,986 5,276,931 2,746,377 20,021 9
Sogou web spider 2,276,910 0.21% 1,227,014 53% 66,495 364 10,734 1,846 970,454 3
360Spider 572,874 0.05% 349,699 61% 64,957 4,071 7,330 5,295 141,520 2
Google 199,160 0.02% 7,196 3% 63,672 4,372 30,125 55,065 38,722 8
Yeti 1,198,212 0.11% 987,418 82% 60,241 1,243 40,430 61,474 47,383 23
DuckDuckBot 438,402 0.04% 27,052 6% 59,790 19,434 53,448 256,813 21,851 14
Applebot 1,094,985 0.1% 663,751 60% 58,464 2,375 62,984 83,135 224,211 65
Yisouspider 391,879 0.04% 188,066 47% 52,062 4,035 13,965 3,499 130,238 14
houzzbot 107,345 0.01% 4,590 4% 51,373 0 4 51,376 2 0
YandexBot 4,084,818 0.38% 645,507 15% 44,853 19,707 3,321,461 17,780 35,451 59
SemrushBot 5,091,144 0.48% 3,735,647 73% 41,692 2,453 55,362 31,783 1,224,125 82
Sosospider 4,682,696 0.44% 1,163,322 24% 41,603 114 3,296,915 1,658 179,082 2
FacebookBot 3,803,123 0.36% 903,880 23% 39,874 61,856 3,882 2,682,630 110,979 22
Scrapy 1,598,435 0.15% 1,513,280 94% 36,612 331 845 657 46,690 20
Facebot 133,548 0.01% 13,684 10% 35,898 19,823 39,452 10,653 14,033 5
AhrefsSiteAudit 3,617,985 0.34% 66,198 1% 34,626 102 3,449,042 39,624 28,387 6
Google-Ad-Manager 33,794 0.0% 0 0% 33,794 0 0 0 0 0
AdX-Bot 33,794 0.0% 0 0% 33,794 0 0 0 0 0
sogou spider 4,686,779 0.44% 354,704 7% 31,960 4,010 3,294,047 2,858 999,196 4
Yahoo! Slurp 9,290,955 0.87% 111,839 1% 31,067 5,802,564 3,303,870 27,595 14,018 2
Sogou inst spider 290,977 0.03% 116,687 40% 29,262 174 7,791 591 136,471 1
YoudaoBot 3,625,149 0.34% 298,333 8% 29,065 61 3,294,181 1,413 2,078 18
PowerMapper 32,367 0.0% 156 0% 28,081 0 746 3,292 91 1
Baiduspider-image 376,728 0.04% 196,437 52% 27,854 260 9,534 4,477 138,163 3
Screaming Frog SEO Spider 138,040 0.01% 101,076 73% 27,320 471 1,991 3,629 3,549 4
Googlebot-Video 168,881 0.02% 15,481 9% 27,297 19,859 5,052 90,445 10,737 10
AdsBot-Google-Mobile-Apps 2,788,509 0.26% 9,694 0% 25,559 18,985 7,445 2,716,728 10,089 9
Baiduspider-news 45,179 0.0% 14,456 31% 24,718 136 1,538 2,136 2,195 0
Sogou News Spider 211,282 0.02% 43,988 20% 24,577 61 7,548 288 134,819 1
Sogou spider2 211,802 0.02% 44,427 20% 24,481 54 7,586 492 134,761 1
Sogou Orion spider 213,877 0.02% 46,618 21% 24,445 28 7,559 296 134,931 0
GPTBot 6,440,012 0.6% 3,166,729 49% 24,395 65,907 41,922 2,709,272 431,655 132
Sogou blog 193,019 0.02% 42,804 22% 24,206 5 7,374 271 118,359 0
ChatGPT-User 4,630,544 0.43% 1,772,298 38% 23,939 1,701 21,750 2,727,399 83,407 50
msnbot-media 138,900 0.01% 52,626 37% 22,714 22,291 9,639 11,864 19,754 12
OAI-SearchBot 1,783,081 0.17% 1,530,151 85% 22,310 563 18,651 26,156 185,226 24
bing 49,029 0.0% 794 1% 19,608 122 23,715 1,248 3,542 0
SiteAuditBot 334,837 0.03% 199,749 59% 16,887 1,271 4,882 28,489 83,555 4
PerplexityBot 3,777,022 0.35% 871,939 23% 16,029 2,974 4,438 2,705,775 175,843 24
Baiduspider-video 208,417 0.02% 183,761 88% 15,517 193 1,734 4,083 3,128 1
NaverBot 1,065,332 0.1% 946,685 88% 14,731 565 33,248 24,918 45,172 13
YandexDirect 76,253 0.01% 48,093 63% 13,572 10 972 463 13,143 0
Teoma 1,313,728 0.12% 1,140,290 86% 13,564 891 10,703 79,623 68,650 7
AdIdxBot 262,215 0.02% 10,643 4% 13,544 828 34,915 33,308 168,971 6
Google-Extended 4,317,901 0.4% 1,401,007 32% 13,284 57,038 21,833 2,748,093 76,604 42
HaosouSpider 322,689 0.03% 188,663 58% 13,055 34 1,853 346 118,736 2
Baiduspider-render 34,037 0.0% 1,732 5% 12,182 6 2,194 191 17,732 0
Bytespider 2,790,723 0.26% 2,594,839 92% 11,551 72,511 10,251 7,271 94,274 26
Google-InspectionTool 57,007 0.01% 9,841 17% 10,764 418 1,683 32,269 2,030 2
rogerbot 1,752,862 0.16% 1,497,289 85% 10,105 19,533 19,111 80,854 125,956 14
ClaudeBot 5,482,189 0.51% 2,504,912 45% 9,536 42,650 22,587 2,700,761 201,716 27
Seznambot 3,674,555 0.34% 331,069 9% 9,263 554 3,300,399 24,262 8,996 12
MJ12bot 8,892,002 0.83% 4,214,295 47% 9,221 1,409 116,196 11,269 4,539,537 75
Daumoa 64,431 0.01% 17,719 27% 9,173 216 11,837 22,827 2,657 2
YandexNews 13,747 0.0% 3,456 25% 9,100 41 303 365 482 0
meta-externalagent 4,749,758 0.44% 1,921,400 40% 8,548 2,423 9,849 2,703,358 104,134 46
Yahoo-MMCrawler 1,126,827 0.11% 1,032,130 91% 8,528 51 8,738 9,619 67,754 7
anthropic-ai 4,645,283 0.43% 1,879,103 40% 8,195 3,397 4,094 2,702,427 48,040 27
Qwantify 105,613 0.01% 60,968 57% 8,008 53 18,469 3,856 14,253 6
Sogou 5,923,759 0.55% 5,900,160 99% 7,926 63 5,788 1,078 8,725 19
BingPreview 284,471 0.03% 6,170 2% 7,850 18,911 3,741 83,472 164,318 9
Pinterestbot 128,493 0.01% 31,576 24% 7,756 3,063 28,765 28,378 28,951 4
facebookcatalog 53,410 0.0% 1,936 3% 7,696 262 1,115 29,362 13,039 0
Exabot 1,580,150 0.15% 1,535,374 97% 7,685 710 7,736 23,680 4,954 11
dotbot 10,372,179 0.97% 4,536,274 43% 7,261 19,597 37,292 8,758 5,762,957 40
Yahoo-slurp 22,255 0.0% 1,847 8% 7,152 15 547 7,679 5,015 0
Nutch 4,867,345 0.46% 4,798,339 98% 6,391 857 4,435 399 56,883 41
CCBot 5,518,772 0.52% 2,699,663 48% 6,327 23,175 31,013 2,676,992 81,526 76
Pinterest 3,648,568 0.34% 14,980 0% 6,300 35 33,843 58,554 3,534,853 3
google-xrawler 6,646 0.0% 326 4% 6,268 5 18 21 8 0
YouBot 369,338 0.03% 286,960 77% 5,975 800 1,274 944 73,364 21
archive.org_bot 645,555 0.06% 561,804 87% 5,951 121 12,245 60,207 5,221 6
NinjaBot 10,085 0.0% 3,257 32% 5,594 3 71 391 768 1
FeedFetcher-Google 43,640 0.0% 11,116 25% 5,434 12 929 23,944 2,204 1
WhatsApp 10,612 0.0% 1,797 16% 5,395 178 548 1,282 1,412 0
TelegramBot 20,208 0.0% 11,301 55% 5,367 400 1,195 1,205 739 1
vitals.vercel-insights.com 4,973 0.0% 0 0% 4,971 0 0 2 0 0
Perplexity-User 91,523 0.01% 66,325 72% 4,877 84 251 19,555 422 9
YandexMedia 9,489 0.0% 3,635 38% 4,789 34 387 186 458 0
yahoo-blogs/v3.9 250,738 0.02% 182,668 72% 4,684 23 5,942 752 56,669 0
psbot 2,268,383 0.21% 2,196,030 96% 4,633 466 3,831 1,542 61,875 6

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.