Most explicitly allowed-all bots

User-Agent Mentions
(%)
(%)
Googlebot 22,918,409 1.63% 739,258 3% 2,592,440 83,849 13,937,294 1,328,165 4,234,411 2,992
bingbot 18,932,933 1.34% 1,190,305 6% 1,126,377 7,278,284 5,916,547 624,624 2,796,611 185
Googlebot-Image 2,777,665 0.2% 412,818 14% 1,001,614 33,809 262,693 264,336 802,313 82
facebookexternalhit 1,886,991 0.13% 205,785 10% 1,000,084 26,736 96,052 378,291 179,933 110
AdsBot-Google 15,238,204 1.08% 113,463 0% 808,320 24,052 10,291,474 3,713,338 287,505 52
Twitterbot 1,465,800 0.1% 108,440 7% 738,178 39,008 71,650 224,769 283,730 25
Baiduspider 16,259,347 1.15% 9,301,180 57% 720,777 26,189 5,513,254 196,097 501,779 71
Googlebot-News 711,192 0.05% 26,161 3% 531,489 24,312 14,416 94,513 20,285 16
Mediapartners-Google 20,887,703 1.48% 89,139 0% 488,854 22,923 12,510,412 227,044 7,549,206 125
Googlebot-Video 634,500 0.05% 22,055 3% 452,571 21,906 7,368 115,307 15,279 14
Google-InspectionTool 473,836 0.03% 12,861 2% 410,165 946 3,862 42,891 3,107 4
GPTBot 14,447,915 1.02% 9,813,151 67% 400,610 89,053 135,093 3,545,346 464,397 265
Feedfetcher-Google 458,127 0.03% 48,042 10% 378,640 159 1,015 27,552 2,718 1
APIs-Google 442,113 0.03% 6,598 1% 377,783 19,304 1,053 34,073 3,295 7
AdSense 374,886 0.03% 40 0% 374,123 0 8 3 712 0
Google-NotebookLM 467,298 0.03% 89,452 19% 373,785 112 221 3,698 21 9
AdsBot 611,065 0.04% 187,208 30% 373,708 271 1,007 35,040 13,827 4
PerplexityBot 2,727,068 0.19% 1,642,952 60% 348,175 6,986 71,701 487,809 169,374 71
ChatGPT-User 3,517,676 0.25% 2,446,785 69% 335,299 15,154 95,989 529,734 94,629 86
ClaudeBot 13,034,954 0.92% 8,884,113 68% 303,573 62,795 83,981 3,496,998 203,426 68
AhrefsBot 16,298,832 1.16% 4,689,516 28% 298,094 21,770 4,046,019 79,833 7,163,488 112
Google-Extended 11,470,212 0.81% 7,398,183 64% 284,375 70,656 71,537 3,576,276 69,097 88
Yandex 10,380,615 0.74% 1,119,977 10% 262,291 7,251,108 601,920 722,396 422,728 195
anthropic-ai 6,537,902 0.46% 2,717,538 41% 250,629 5,812 44,859 3,482,354 36,661 49
MSNBot 5,911,986 0.42% 537,409 9% 242,834 22,013 169,852 176,606 4,763,228 44
Slurp 4,561,288 0.32% 523,237 11% 241,210 25,156 174,849 168,274 3,428,527 35
OAI-SearchBot 2,343,771 0.17% 1,780,054 75% 226,291 2,632 89,831 72,702 172,210 51
YandexBot 6,827,265 0.48% 1,113,885 16% 214,189 21,443 5,381,012 45,558 51,100 78
Applebot 1,652,161 0.12% 935,874 56% 208,937 4,406 104,995 161,996 235,860 93
360Spider 1,052,156 0.07% 520,587 49% 204,612 4,081 13,203 35,305 274,365 3
CCBot 12,964,960 0.92% 9,111,018 70% 201,792 28,957 95,924 3,453,543 73,603 123
Googlebot-Mobile 924,519 0.07% 218,879 23% 195,883 1,568 29,613 140,795 337,712 69
ia_archiver 17,920,963 1.27% 1,774,508 9% 183,504 3,194 5,038,058 124,247 10,797,350 102
DuckDuckBot 676,660 0.05% 83,012 12% 181,972 22,401 55,568 298,213 35,420 74
Bytespider 12,468,739 0.88% 9,016,272 72% 176,504 84,039 49,868 3,037,978 104,021 57
Sogou web spider 2,521,850 0.18% 1,321,828 52% 175,679 471 18,969 10,604 994,293 6
meta-externalagent 11,552,893 0.82% 7,696,781 66% 175,227 13,987 46,289 3,490,444 128,182 1,983
Mediapartners (Googlebot) 169,989 0.01% 893 0% 166,519 1 603 1,271 702 0
YisouSpider 703,821 0.05% 339,371 48% 156,087 4,025 23,062 8,925 172,322 29
Claude-Web 2,132,376 0.15% 1,444,837 67% 127,151 5,421 56,488 463,578 34,850 51
Facebot 262,151 0.02% 24,056 9% 125,828 25,567 44,045 24,181 18,467 7
SemrushBot 6,230,667 0.44% 4,588,281 73% 124,668 3,222 82,002 74,722 1,357,648 124
LinkedInBot 291,468 0.02% 26,931 9% 122,778 25,217 19,187 63,197 34,146 12
sogou spider 7,077,644 0.5% 537,848 7% 105,466 3,996 5,336,026 4,745 1,089,558 5
Perplexity-User 408,459 0.03% 230,362 56% 104,998 693 23,914 45,825 2,652 15
AdsBot-Google-Mobile 9,825,038 0.7% 25,563 0% 104,567 23,842 6,095,134 3,545,715 30,201 16
Claude-SearchBot 357,518 0.03% 190,541 53% 103,524 568 46,070 13,047 3,740 28
YandexImages 260,995 0.02% 80,973 31% 102,961 1,751 14,194 56,285 4,829 2
Sogou inst spider 455,374 0.03% 162,523 35% 96,552 269 14,409 1,031 180,589 1
Baiduspider-image 693,420 0.05% 374,871 54% 95,981 307 25,937 10,051 186,271 2
Sosospider 6,986,471 0.5% 1,300,367 18% 95,411 149 5,341,967 2,102 246,473 2
MJ12bot 11,305,639 0.8% 5,910,460 52% 91,761 1,862 119,468 25,666 5,156,280 142
TermlyBot 85,408 0.01% 93 0% 84,940 1 18 345 11 0
rogerbot 2,342,002 0.17% 1,998,804 85% 84,932 21,195 7,433 101,159 128,314 165
dotbot 12,685,835 0.9% 5,792,041 45% 84,763 21,610 119,507 25,719 6,642,094 101
PetalBot 9,034,035 0.64% 8,664,515 95% 84,680 1,818 60,019 44,187 178,736 80
Baiduspider-news 107,112 0.01% 16,437 15% 84,282 152 1,907 1,977 2,357 0
Applebot-Extended 10,566,464 0.75% 6,866,716 64% 83,824 59,878 35,740 3,460,676 59,586 44
FacebookBot 5,346,930 0.38% 1,597,729 29% 83,602 69,815 40,047 3,453,187 102,512 38
cohere-ai 5,260,737 0.37% 1,675,206 31% 78,181 3,349 38,911 3,432,657 32,399 34
Yahoo! Slurp 12,905,285 0.92% 201,467 1% 75,705 7,233,649 5,335,646 42,232 16,582 4
Facebook 99,622 0.01% 25,549 25% 73,635 7 74 55 301 1
Meta 76,292 0.01% 3,194 4% 72,985 6 26 5 76 0
YoudaoBot 5,823,084 0.41% 407,029 6% 71,952 94 5,339,545 1,958 2,471 35
Yeti 1,282,320 0.09% 1,033,603 80% 70,712 1,321 42,091 71,183 63,383 27
DataForSeoBot 1,853,683 0.13% 1,736,861 93% 69,594 574 2,510 19,950 24,175 19
serpstatbot 1,135,041 0.08% 1,032,502 90% 68,218 34 17,953 301 16,026 7
SEOkicks 437,639 0.03% 361,252 82% 68,117 17 489 212 7,549 3
Sogou News Spider 314,550 0.02% 53,625 17% 66,004 67 14,101 669 180,083 1
Sogou spider2 315,226 0.02% 54,495 17% 65,804 117 14,113 934 179,762 1
Google 220,914 0.02% 7,541 3% 65,681 6,069 37,682 56,949 46,985 7
Sogou Orion spider 316,705 0.02% 56,914 17% 65,205 58 13,953 630 179,945 0
YouBot 4,065,383 0.29% 887,648 21% 65,131 1,916 33,378 3,016,770 60,493 47
Sogou blog 286,583 0.02% 50,631 17% 64,328 34 13,770 591 157,229 0
Claude-User 327,023 0.02% 182,383 55% 57,123 542 45,213 38,638 3,093 31
Bing 88,421 0.01% 1,017 1% 55,452 121 25,284 1,464 5,083 0
houzzbot 109,211 0.01% 4,997 4% 52,101 0 7 52,104 2 0
Meta-ExternalFetcher 516,091 0.04% 352,714 68% 50,156 2,358 37,812 14,376 58,636 39
Baiduspider-video 417,607 0.03% 352,482 84% 49,957 222 1,917 8,655 4,373 1
AhrefsSiteAudit 4,230,648 0.3% 102,328 2% 49,170 137 3,945,446 78,859 54,695 13
Diffbot 1,317,661 0.09% 1,193,140 90% 47,986 3,224 32,823 6,700 33,748 40
Amazonbot 12,071,979 0.86% 8,313,076 68% 47,824 32,518 84,505 3,493,794 100,150 112
Seekr 103,782 0.01% 55,793 53% 43,873 105 385 3,578 35 13
HaosouSpider 399,675 0.03% 197,111 49% 42,044 101 1,924 837 157,656 2
Screaming Frog SEO Spider 218,044 0.02% 160,269 73% 39,976 1,015 2,436 8,947 5,398 3
Baiduspider-render 87,144 0.01% 7,755 8% 39,317 10 12,694 1,389 25,979 0
Scrapy 1,825,535 0.13% 1,713,514 93% 35,716 665 31,840 11,986 31,775 39
AdX-Bot 35,301 0.0% 0 0% 35,301 0 0 0 0 0
Google-Ad-Manager 35,301 0.0% 0 0% 35,301 0 0 0 0 0
PowerMapper 40,365 0.0% 212 0% 34,761 0 1,589 3,688 114 1
Storebot-Google 93,790 0.01% 12,126 12% 33,819 1,213 3,417 37,311 5,895 9
Bingbot-Image 32,606 0.0% 158 0% 30,624 17 69 1,479 259 0
AdsBot-Google-Mobile-Apps 3,595,792 0.26% 15,941 0% 28,935 20,525 9,747 3,509,430 11,205 9
ToutiaoSpider 198,766 0.01% 16,068 8% 25,007 30 646 463 156,551 1
NaverBot 1,119,764 0.08% 972,185 86% 24,527 560 33,630 37,276 51,571 15
AdIdxBot 319,840 0.02% 31,889 9% 24,092 1,352 43,632 42,753 176,116 6
msnbot-media 170,936 0.01% 71,826 42% 23,274 23,832 14,822 13,903 23,268 11
Sogou 7,503,485 0.53% 7,392,985 98% 22,497 85 7,805 37,974 42,116 23
WhatsApp 39,686 0.0% 5,121 12% 22,108 269 1,650 7,624 2,910 4
Pinterestbot 204,776 0.01% 58,974 28% 21,460 4,253 36,628 47,768 35,686 7

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.