Most explicitly allowed-all bots

User-Agent Mentions
(%)
(%)
Googlebot 21,546,503 1.58% 715,460 3% 2,174,792 79,855 13,327,100 1,258,884 3,987,464 2,948
bingbot 18,031,560 1.33% 1,189,333 6% 909,830 6,892,997 5,661,706 603,769 2,773,759 166
facebookexternalhit 1,652,317 0.12% 195,375 11% 799,735 26,251 64,527 363,473 202,581 375
Googlebot-Image 2,529,313 0.19% 417,259 16% 763,273 33,399 274,363 251,831 789,097 91
Baiduspider 15,431,586 1.13% 8,843,801 57% 625,957 25,583 5,263,895 186,333 485,948 69
AdsBot-Google 16,606,382 1.22% 106,927 0% 615,653 23,633 11,295,567 4,282,195 282,354 53
Twitterbot 1,321,962 0.1% 95,160 7% 614,728 38,511 72,915 223,994 276,631 23
Mediapartners-Google 20,227,078 1.49% 86,649 0% 483,668 22,540 11,887,596 221,628 7,524,867 130
Googlebot-News 508,330 0.04% 30,426 5% 330,329 21,592 13,343 92,777 19,847 16
AhrefsBot 16,772,861 1.23% 4,252,143 25% 262,247 21,259 4,082,880 73,959 8,080,258 115
Googlebot-Video 423,623 0.03% 21,000 4% 247,451 21,548 6,200 112,387 15,023 14
Yandex 9,918,294 0.73% 1,087,479 10% 241,255 6,865,343 604,270 704,085 415,673 189
Slurp 4,347,073 0.32% 519,960 11% 226,216 24,764 188,405 166,160 3,221,531 37
GPTBot 13,358,151 0.98% 8,385,764 62% 225,872 86,658 87,713 4,085,862 486,061 221
MSNBot 5,654,529 0.42% 523,253 9% 221,623 21,590 161,106 175,034 4,551,885 38
Google-InspectionTool 283,161 0.02% 12,703 4% 220,988 942 2,708 42,665 3,151 4
googlebot-mobile 864,122 0.06% 217,069 25% 200,667 1,388 30,619 114,003 300,307 69
Feedfetcher-Google 273,826 0.02% 45,269 16% 197,285 156 1,009 27,329 2,777 1
APIs-Google 260,056 0.02% 6,505 2% 196,424 18,921 1,854 32,989 3,356 7
AdSense 193,210 0.01% 30 0% 192,540 0 11 3 626 0
AdsBot 416,033 0.03% 173,124 41% 192,379 280 1,054 35,286 13,907 3
Google-NotebookLM 201,451 0.01% 9,285 4% 191,910 41 153 43 10 9
PerplexityBot 3,821,742 0.28% 1,470,566 38% 186,631 6,177 32,297 1,936,093 189,910 68
ChatGPT-User 4,616,930 0.34% 2,254,991 48% 183,406 14,574 64,744 1,980,967 118,162 86
360Spider 975,192 0.07% 492,067 50% 180,124 3,909 12,725 33,755 252,609 3
ia_archiver 17,043,321 1.25% 1,736,629 10% 179,006 3,065 4,799,822 124,466 10,200,245 88
Mediapartners (Googlebot) 169,621 0.01% 897 0% 166,152 1 583 1,269 719 0
Google-Extended 10,492,641 0.77% 6,014,817 57% 154,539 70,544 37,320 4,125,042 90,292 87
Sogou web spider 2,478,679 0.18% 1,312,807 52% 152,153 428 17,953 3,117 992,216 5
ClaudeBot 11,974,685 0.88% 7,463,087 62% 149,804 60,762 37,089 4,042,912 220,968 63
Applebot 1,482,056 0.11% 856,591 57% 149,230 4,321 74,558 147,611 249,671 74
YandexBot 6,436,828 0.47% 1,057,598 16% 141,687 20,920 5,124,607 40,955 50,992 69
DuckDuckBot 626,950 0.05% 77,924 12% 141,288 22,000 56,791 293,051 35,823 73
YisouSpider 651,483 0.05% 313,784 48% 140,925 3,859 22,291 4,465 166,133 26
anthropic-ai 6,620,199 0.49% 2,382,552 35% 127,844 4,951 11,799 4,035,140 57,862 51
OAI-SearchBot 2,183,747 0.16% 1,739,684 79% 126,567 2,103 58,294 60,850 196,198 51
CCBot 11,952,000 0.88% 7,636,060 63% 124,651 27,790 57,188 4,010,933 95,256 122
Bytespider 9,975,140 0.73% 7,505,681 75% 119,556 81,731 17,618 2,124,898 125,599 57
LinkedInBot 279,688 0.02% 26,103 9% 113,605 24,542 18,979 62,555 33,894 10
Claude-Web 3,236,573 0.24% 1,130,178 34% 99,649 4,649 23,227 1,922,876 55,941 53
SemrushBot 6,051,004 0.44% 4,464,846 73% 99,618 3,060 67,734 71,711 1,343,917 118
sogou spider 6,796,118 0.5% 508,842 7% 98,887 3,872 5,094,610 3,285 1,086,617 5
AdsBot-Google-Mobile 11,347,986 0.83% 25,149 0% 97,562 23,427 7,055,036 4,119,988 26,814 10
YandexImages 247,667 0.02% 76,418 30% 93,750 1,737 14,951 55,942 4,867 2
TermlyBot 84,543 0.01% 64 0% 84,138 1 18 311 11 0
Sosospider 6,724,067 0.49% 1,297,156 19% 82,786 140 5,100,022 1,881 242,080 2
Baiduspider-image 663,168 0.05% 367,299 55% 76,735 273 24,166 8,959 185,734 2
Yeti 1,269,392 0.09% 1,022,685 80% 70,468 1,297 43,520 69,224 62,168 30
FacebookBot 5,664,701 0.42% 1,380,542 24% 69,989 68,631 7,563 4,012,231 125,708 37
Sogou inst spider 424,456 0.03% 160,541 37% 69,467 226 13,698 795 179,728 1
Yahoo! Slurp 12,264,117 0.9% 193,273 1% 69,236 6,848,986 5,093,941 42,134 16,543 4
MJ12bot 10,811,676 0.79% 5,464,115 50% 67,505 1,757 114,583 24,785 5,138,799 132
Google 220,152 0.02% 8,037 3% 65,521 5,706 38,089 56,405 46,387 7
Baiduspider-news 87,631 0.01% 16,016 18% 65,195 135 1,864 1,926 2,495 0
rogerbot 2,326,174 0.17% 2,007,168 86% 60,860 20,770 7,511 100,834 128,842 189
PetalBot 9,829,348 0.72% 9,525,649 96% 60,176 1,671 15,067 36,928 189,783 74
cohere-ai 5,438,670 0.4% 1,319,940 24% 59,939 2,992 8,420 3,992,850 54,492 37
dotbot 13,491,255 0.99% 5,690,575 42% 59,817 21,179 105,309 24,534 7,589,772 69
YoudaoBot 5,544,062 0.41% 382,193 6% 59,734 64 5,097,206 1,725 3,116 24
Sogou News Spider 300,876 0.02% 52,870 17% 54,868 38 13,410 439 179,250 1
Sogou spider2 300,315 0.02% 53,568 17% 53,626 85 13,430 700 178,905 1
Sogou Orion spider 301,883 0.02% 55,930 18% 53,165 29 13,265 416 179,078 0
Sogou blog 268,146 0.02% 49,483 18% 52,778 6 13,087 379 152,413 0
houzzbot 109,957 0.01% 5,668 5% 52,139 0 7 52,141 2 0
Bing 83,669 0.01% 965 1% 49,964 123 26,382 1,426 4,809 0
AhrefsSiteAudit 4,268,540 0.31% 95,230 2% 49,831 126 3,996,916 76,466 49,958 13
Claude-User 277,604 0.02% 155,883 56% 48,248 250 39,603 31,163 2,428 29
meta-externalagent 10,703,486 0.79% 6,435,616 60% 46,236 11,860 13,657 4,046,426 148,035 1,656
Diffbot 1,078,931 0.08% 969,089 89% 46,011 2,899 2,625 2,477 55,792 38
Facebot 179,801 0.01% 24,547 13% 45,943 24,967 44,389 21,273 18,675 7
DataForSeoBot 1,635,153 0.12% 1,548,209 94% 44,985 485 2,417 15,285 23,752 20
SEOkicks 397,961 0.03% 346,100 86% 43,762 17 479 188 7,412 3
serpstatbot 1,103,939 0.08% 1,026,477 92% 43,756 35 17,622 257 15,786 6
Screaming Frog SEO Spider 194,749 0.01% 138,592 71% 38,481 1,061 2,612 9,169 4,831 3
Scrapy 1,842,780 0.14% 1,739,426 94% 36,404 530 1,908 10,566 53,906 40
HaosouSpider 389,611 0.03% 197,169 50% 36,404 69 2,565 496 152,906 2
Google-Ad-Manager 35,745 0.0% 0 0% 35,745 0 0 0 0 0
AdX-Bot 35,745 0.0% 0 0% 35,745 0 0 0 0 0
Baiduspider-video 396,935 0.03% 346,633 87% 35,392 197 1,877 8,393 4,442 1
Applebot-Extended 9,654,704 0.71% 5,456,363 56% 35,058 58,096 5,099 4,018,784 81,257 47
Perplexity-User 298,416 0.02% 204,522 68% 35,038 410 17,138 38,995 2,299 14
PowerMapper 39,855 0.0% 201 0% 34,478 0 1,107 3,953 115 1
YouBot 2,862,791 0.21% 633,457 22% 34,278 1,625 3,283 2,107,786 82,314 48
Baiduspider-render 78,645 0.01% 3,173 4% 33,605 8 10,811 888 30,160 0
Claude-SearchBot 231,362 0.02% 151,119 65% 32,165 282 39,211 6,068 2,489 28
Amazonbot 11,097,274 0.82% 6,809,926 61% 30,703 30,598 53,491 4,050,175 122,267 114
AdsBot-Google-Mobile-Apps 4,171,742 0.31% 16,236 0% 29,110 20,145 9,506 4,085,442 11,294 9
Meta-ExternalFetcher 398,236 0.03% 272,290 68% 25,064 2,187 7,615 10,409 80,633 38
AdIdxBot 301,323 0.02% 18,062 5% 23,071 1,341 42,213 40,701 175,929 6
msnbot-media 166,043 0.01% 67,601 40% 22,913 23,661 15,031 12,822 24,004 11
Storebot-Google 79,874 0.01% 12,005 15% 22,581 1,189 2,217 35,953 5,919 10
googlebot-Smartphone 21,912 0.0% 15 0% 21,649 0 65 80 103 0
SiteAuditBot 592,802 0.04% 431,736 72% 20,915 1,308 10,619 31,231 96,987 6
WhatsApp 35,082 0.0% 4,557 12% 18,874 228 1,533 6,870 3,017 3
Pinterestbot 183,665 0.01% 44,508 24% 18,400 4,339 36,314 45,030 35,068 6
BingPreview 317,600 0.02% 8,525 2% 17,616 20,366 10,077 90,737 170,269 10
Sogou 7,089,428 0.52% 7,001,254 98% 16,714 73 7,711 38,901 24,750 25
NaverBot 1,111,373 0.08% 969,066 87% 16,453 527 35,023 36,929 53,361 14
YandexDirect 95,457 0.01% 62,767 65% 15,591 9 1,226 706 15,158 0
GoogleOther 2,605,039 0.19% 315,088 12% 15,031 45,915 5,490 2,139,387 84,103 25

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.