Most explicitly allowed-all bots

User-Agent Mentions
(%)
(%)
Googlebot 23,612,583 1.63% 807,384 3% 2,931,365 87,552 14,070,011 1,379,116 4,334,303 2,852
bingbot 19,247,230 1.33% 1,206,207 6% 1,264,883 7,325,985 5,993,248 670,053 2,786,639 215
facebookexternalhit 2,182,188 0.15% 265,687 12% 1,224,441 27,260 105,483 389,682 169,483 152
Googlebot-Image 2,997,320 0.21% 411,547 13% 1,215,812 34,034 262,465 268,395 804,982 85
AdsBot-Google 15,347,857 1.06% 115,842 0% 989,063 24,656 10,276,121 3,652,644 289,483 48
Twitterbot 1,559,649 0.11% 111,801 7% 809,298 39,292 73,468 234,790 290,972 28
Baiduspider 16,488,264 1.14% 9,403,178 57% 784,196 27,077 5,561,104 207,242 505,374 93
Googlebot-News 901,564 0.06% 25,949 2% 717,557 23,662 14,570 96,154 23,656 16
Googlebot-Video 830,741 0.06% 24,983 3% 644,427 22,047 7,485 116,999 14,786 14
Google-InspectionTool 655,333 0.05% 12,966 1% 588,194 911 3,948 45,987 3,323 4
APIs-Google 620,715 0.04% 6,620 1% 555,274 19,496 1,107 34,767 3,444 7
Feedfetcher-Google 635,156 0.04% 47,771 7% 555,162 164 1,017 28,202 2,838 2
AdSense 551,831 0.04% 42 0% 551,081 0 7 4 697 0
Google-NotebookLM 652,912 0.05% 97,578 14% 550,863 129 336 3,936 59 11
AdsBot 793,067 0.05% 192,756 24% 550,323 287 983 35,755 12,959 4
GPTBot 15,524,330 1.07% 10,817,752 69% 509,469 90,464 146,078 3,503,362 456,854 351
Mediapartners-Google 21,006,990 1.45% 95,685 0% 492,267 23,171 12,602,708 226,979 7,566,053 127
PerplexityBot 2,923,308 0.2% 1,833,124 62% 460,582 8,086 86,979 374,728 159,720 89
ChatGPT-User 3,666,370 0.25% 2,620,911 71% 421,792 16,429 110,005 413,992 83,147 94
ClaudeBot 14,076,037 0.97% 9,863,522 70% 404,832 64,462 95,984 3,454,447 192,643 147
Google-Extended 12,474,831 0.86% 8,350,031 66% 385,238 72,036 82,439 3,527,072 57,913 102
meta-externalagent 12,516,810 0.87% 8,537,071 68% 346,220 16,326 57,126 3,433,087 124,747 2,233
AhrefsBot 16,517,691 1.14% 4,885,498 29% 320,445 22,709 4,157,625 83,864 7,047,376 174
anthropic-ai 6,753,095 0.47% 2,929,622 43% 307,356 7,014 55,335 3,427,917 25,723 128
OAI-SearchBot 2,445,882 0.17% 1,789,014 73% 294,657 2,935 103,604 94,423 161,188 61
Yandex 10,483,754 0.73% 1,129,960 10% 270,818 7,299,359 607,585 752,213 423,602 217
Facebot 462,248 0.03% 79,520 17% 268,027 25,780 44,232 27,058 17,624 7
CCBot 13,976,200 0.97% 10,122,781 72% 255,789 30,274 104,666 3,400,523 62,031 136
MSNBot 6,074,243 0.42% 581,156 9% 253,188 22,109 170,280 180,847 4,866,619 44
Slurp 4,710,311 0.33% 563,455 11% 250,943 25,307 173,209 170,917 3,526,446 34
Applebot 1,727,387 0.12% 959,565 55% 245,383 4,398 118,022 172,309 227,546 164
YandexBot 6,992,360 0.48% 1,213,890 17% 233,361 21,648 5,422,622 49,152 51,608 79
360Spider 1,114,143 0.08% 559,923 50% 220,766 4,077 11,905 38,994 278,453 25
Bytespider 13,598,507 0.94% 10,028,623 73% 215,390 85,689 57,900 3,120,772 89,972 161
Facebook 240,452 0.02% 31,713 13% 208,269 8 72 53 336 1
Meta 210,973 0.01% 3,228 1% 207,625 5 27 7 81 0
DuckDuckBot 710,981 0.05% 85,150 11% 206,265 22,566 56,181 304,414 36,330 75
Googlebot-Mobile 945,603 0.07% 219,046 23% 198,673 1,550 29,906 163,806 332,554 68
Sogou web spider 2,565,698 0.18% 1,337,129 52% 191,811 482 19,303 16,545 1,000,423 5
ia_archiver 18,168,396 1.26% 1,822,761 10% 184,386 3,175 5,078,298 124,329 10,955,346 101
Claude-Web 2,236,135 0.15% 1,630,527 72% 169,695 6,571 71,185 334,600 23,436 121
YisouSpider 739,930 0.05% 352,252 47% 167,445 4,024 23,462 14,132 178,588 27
Mediapartners (Googlebot) 170,048 0.01% 879 0% 166,611 1 601 1,260 696 0
Applebot-Extended 11,516,255 0.8% 7,812,341 67% 143,717 61,661 45,816 3,404,650 48,017 53
LinkedInBot 316,836 0.02% 30,021 9% 138,737 25,492 21,851 66,835 33,889 11
SemrushBot 6,427,734 0.44% 4,767,431 74% 135,915 3,836 81,276 77,453 1,361,588 235
Perplexity-User 452,068 0.03% 224,008 49% 133,935 1,360 31,137 58,657 2,947 24
Claude-SearchBot 387,638 0.03% 194,419 50% 118,506 1,236 50,540 18,591 4,243 103
YandexImages 264,932 0.02% 81,338 30% 107,301 1,758 13,974 56,356 4,203 2
cohere-ai 5,426,889 0.38% 1,874,845 34% 107,229 4,564 46,106 3,373,589 20,512 44
sogou spider 7,149,121 0.49% 564,853 7% 106,217 3,983 5,376,677 5,496 1,091,890 5
AdsBot-Google-Mobile 9,636,486 0.67% 25,906 0% 104,891 24,520 5,969,936 3,481,452 29,765 16
Baiduspider-image 713,701 0.05% 380,683 53% 102,893 312 26,877 11,554 191,380 2
FacebookBot 5,360,911 0.37% 1,656,392 30% 100,583 71,574 50,395 3,391,894 89,960 113
MJ12bot 11,658,079 0.81% 6,159,943 52% 99,808 1,853 121,024 28,408 5,246,830 213
Sogou inst spider 467,226 0.03% 167,067 35% 97,677 270 14,680 1,451 186,080 1
Sosospider 7,041,651 0.49% 1,308,896 18% 97,108 196 5,384,107 2,271 249,072 1
dotbot 12,849,274 0.89% 6,067,293 47% 93,124 21,796 119,182 28,979 6,518,795 105
rogerbot 2,464,692 0.17% 2,113,781 85% 92,421 21,401 7,600 101,165 128,158 166
PetalBot 9,046,089 0.63% 8,627,846 95% 92,306 2,426 67,237 46,579 209,543 152
Baiduspider-news 112,615 0.01% 16,545 14% 90,182 153 1,907 1,548 2,280 0
YouBot 4,351,228 0.3% 1,080,705 24% 85,512 3,092 40,189 3,093,058 48,619 53
TermlyBot 86,588 0.01% 110 0% 85,046 1 19 340 1,072 0
Yahoo! Slurp 12,996,432 0.9% 199,780 1% 81,396 7,281,279 5,376,377 41,856 15,740 4
DataForSeoBot 2,080,392 0.14% 1,952,471 93% 76,932 598 2,745 21,727 25,893 26
serpstatbot 1,185,095 0.08% 1,075,235 90% 75,251 39 17,869 360 16,334 7
SEOkicks 469,665 0.03% 386,309 82% 75,096 19 492 215 7,531 3
Yeti 1,286,528 0.09% 1,030,904 80% 74,761 1,378 41,903 72,443 65,112 27
YoudaoBot 5,886,628 0.41% 426,420 7% 73,627 95 5,382,004 2,028 2,419 35
Amazonbot 13,036,362 0.9% 9,302,930 71% 73,335 34,071 99,345 3,436,714 89,773 194
Sogou spider2 324,102 0.02% 55,497 17% 67,743 122 14,367 1,012 185,360 1
Claude-User 344,451 0.02% 177,486 51% 67,481 1,284 49,610 45,418 3,091 81
Sogou News Spider 324,712 0.02% 56,785 17% 67,171 66 14,370 754 185,566 0
Sogou Orion spider 324,368 0.02% 57,622 17% 66,351 57 14,206 697 185,435 0
Google 220,720 0.02% 7,844 3% 65,726 4,084 37,711 56,987 48,361 7
Sogou blog 294,484 0.02% 51,243 17% 65,531 33 14,028 657 162,992 0
Meta-ExternalFetcher 521,095 0.04% 347,815 66% 59,958 2,977 47,157 16,429 46,646 113
Bing 91,836 0.01% 1,012 1% 59,407 124 24,736 1,464 5,093 0
Diffbot 1,383,595 0.1% 1,246,231 90% 58,667 4,466 42,237 10,096 21,847 51
Baiduspider-video 422,495 0.03% 353,555 83% 54,930 222 1,910 8,254 3,623 1
houzzbot 109,014 0.01% 4,950 4% 52,027 0 7 52,028 2 0
AhrefsSiteAudit 4,360,714 0.3% 119,268 2% 50,248 150 4,056,296 80,811 53,928 13
Seekr 104,776 0.01% 56,654 54% 44,004 108 371 3,591 33 15
HaosouSpider 415,421 0.03% 205,239 49% 43,916 155 1,763 936 163,410 2
Baiduspider-render 92,605 0.01% 10,597 11% 41,793 11 12,106 2,009 26,089 0
Screaming Frog SEO Spider 253,851 0.02% 194,116 76% 41,367 977 2,461 9,321 5,605 4
PowerMapper 41,394 0.0% 209 0% 35,791 0 1,570 3,706 117 1
Scrapy 1,826,231 0.13% 1,719,299 94% 35,158 1,237 38,346 12,614 19,533 44
Google-Ad-Manager 35,084 0.0% 0 0% 35,084 0 0 0 0 0
AdX-Bot 35,084 0.0% 0 0% 35,084 0 0 0 0 0
WhatsApp 55,387 0.0% 5,863 10% 33,147 287 3,662 10,094 2,329 5
Storebot-Google 96,809 0.01% 12,212 12% 32,361 1,202 3,778 41,018 6,229 9
Bingbot-Image 33,807 0.0% 164 0% 31,825 20 83 1,462 253 0
AdsBot-Google-Mobile-Apps 3,529,060 0.24% 16,125 0% 30,258 20,707 9,940 3,441,517 10,504 9
GoogleOther 3,780,815 0.26% 486,645 12% 29,376 47,333 43,990 3,123,967 49,478 26
BingPreview 336,267 0.02% 9,274 2% 27,358 21,076 12,949 96,333 169,268 9
TelegramBot 67,846 0.0% 18,695 27% 25,649 939 3,233 17,339 1,985 6
AdIdxBot 325,270 0.02% 31,426 9% 25,267 1,515 45,353 45,694 176,010 5
ToutiaoSpider 208,144 0.01% 19,529 9% 25,264 28 512 472 162,339 0
DuckAssistBot 3,672,558 0.25% 252,464 6% 24,471 1,294 26,591 3,365,499 2,201 38

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.