Most directed user-agents

User-Agent Mentions
(%)
(%)
* 255,761,331 17.73% 41,860,580 16% 70,277,410 433,120 44,809,241 64,123,231 33,948,352 309,397
Googlebot 23,559,496 1.63% 802,892 3% 2,913,675 87,630 14,053,390 1,374,450 4,324,561 2,898
bingbot 19,218,165 1.33% 1,202,875 6% 1,256,136 7,318,480 5,984,505 667,492 2,788,457 220
Baiduspider 16,469,977 1.14% 9,391,743 57% 781,922 27,177 5,558,338 205,478 505,214 105
AdsBot-Google 15,356,746 1.06% 115,542 0% 978,418 24,712 10,293,582 3,654,915 289,530 47
GPTBot 15,425,476 1.07% 10,729,013 69% 502,204 90,443 145,092 3,501,911 456,468 345
CCBot 13,885,970 0.96% 10,037,671 72% 251,687 30,251 104,017 3,400,119 62,089 136
ClaudeBot 13,977,334 0.97% 9,775,467 69% 397,608 64,429 94,715 3,452,298 192,676 141
Mediapartners-Google 20,993,811 1.46% 95,487 0% 491,005 23,237 12,592,232 227,128 7,564,596 126
Bytespider 13,505,150 0.94% 9,943,044 73% 212,998 85,645 57,089 3,115,866 90,341 167
Yahoo! Slurp 12,986,540 0.9% 200,014 1% 80,955 7,273,684 5,374,361 41,750 15,772 4
Amazonbot 12,941,376 0.9% 9,216,677 71% 71,028 34,019 93,581 3,436,010 89,870 191
Google-Extended 12,384,504 0.86% 8,268,574 66% 378,357 72,026 81,331 3,526,224 57,890 102
meta-externalagent 12,419,660 0.86% 8,454,787 68% 333,680 16,256 56,090 3,432,342 124,346 2,159
Applebot-Extended 11,428,013 0.79% 7,729,643 67% 139,422 61,629 44,740 3,404,333 48,194 52
BLEXBot 10,271,092 0.71% 10,043,329 97% 2,279 3,602 41,133 13,947 166,707 95
Yandex 10,478,198 0.73% 1,129,750 10% 270,329 7,291,807 609,387 753,063 423,634 228
AdsBot-Google-Mobile 9,639,379 0.67% 25,863 0% 104,504 24,579 5,972,486 3,482,127 29,804 16
AhrefsBot 16,517,492 1.14% 4,871,475 29% 319,174 22,611 4,170,423 82,889 7,050,744 176
PetalBot 9,023,282 0.63% 8,606,304 95% 91,555 2,419 67,139 46,588 209,126 151
Sogou 7,556,876 0.52% 7,445,298 98% 23,339 687 7,693 36,268 43,546 45
IPS-Agent 7,465,944 0.52% 191,053 2% 57 16 7,273,927 207 683 1
Pandalytics 7,444,905 0.52% 169,835 2% 59 453 7,273,509 264 783 2
IonCrawl 7,335,994 0.51% 62,428 0% 16 6 7,273,475 3 66 0
ia_archiver 18,147,359 1.26% 1,822,851 10% 184,513 3,182 5,075,900 124,335 10,936,476 102
YandexBot 6,980,027 0.48% 1,205,694 17% 231,480 21,730 5,420,576 48,944 51,525 78
Sosospider 7,038,885 0.49% 1,308,613 18% 96,881 200 5,382,012 2,282 248,896 1
anthropic-ai 6,732,865 0.47% 2,914,615 43% 303,555 7,022 54,285 3,427,522 25,741 125
MJ12bot 11,658,001 0.81% 6,148,053 52% 99,057 1,846 120,981 27,690 5,260,164 210
ichiro 6,328,874 0.44% 959,002 15% 409 13 5,366,766 594 2,090 0
dotbot 12,835,837 0.89% 6,053,325 47% 92,345 21,862 119,220 28,244 6,520,737 104
sogou spider 7,144,459 0.5% 562,440 7% 106,097 4,024 5,374,623 5,482 1,091,788 5
Seznambot 5,974,114 0.41% 529,726 8% 12,058 1,194 5,380,353 37,649 13,119 15
YoudaoBot 5,883,681 0.41% 425,755 7% 73,427 94 5,379,908 2,041 2,421 35
Nutch 5,632,054 0.39% 5,546,340 98% 13,329 1,534 5,092 559 65,161 39
cohere-ai 5,412,861 0.38% 1,863,021 34% 104,811 4,570 46,052 3,373,645 20,718 44
YetiBot 5,373,021 0.37% 5,439 0% 310 1 5,366,044 127 1,100 0
rdfbot 5,366,036 0.37% 22 0% 16 0 5,365,972 2 24 0
FacebookBot 5,338,133 0.37% 1,635,732 30% 99,666 71,538 49,330 3,391,587 90,170 110
omgili 5,106,959 0.35% 1,938,930 37% 5,123 23,920 40,479 3,078,857 19,611 39
SemrushBot 6,409,284 0.44% 4,750,818 74% 134,483 3,810 81,350 77,427 1,361,161 235
omgilibot 4,635,796 0.32% 1,459,167 31% 12,756 24,060 40,198 3,079,311 20,260 44
CloudflareBrowserRenderingCrawler 4,582,350 0.32% 4,581,728 99% 430 13 23 149 7 0
AhrefsSiteAudit 4,373,623 0.3% 118,986 2% 50,246 150 4,069,020 79,833 55,375 13
YouBot 4,331,310 0.3% 1,066,360 24% 83,657 3,099 40,139 3,089,312 48,689 54
GoogleOther 3,776,470 0.26% 486,617 12% 28,540 47,317 43,978 3,120,285 49,706 27
Quora-Bot 3,704,301 0.26% 351,456 9% 657 39 508 3,351,534 107 0
DuckAssistBot 3,666,552 0.25% 252,395 6% 23,412 1,292 21,775 3,365,544 2,095 39
aiHitBot 3,582,282 0.25% 500,915 13% 566 1,244 824 3,077,594 1,112 27
ChatGPT-User 3,649,030 0.25% 2,612,955 71% 415,656 16,434 104,432 416,238 83,223 92
cohere-training-data-crawler 3,566,323 0.25% 480,902 13% 2,747 1,599 1,479 3,078,438 1,129 29
Ai2Bot-Dolma 3,576,220 0.25% 440,747 12% 1,142 1,736 37,205 3,077,863 17,492 35
AI2Bot 3,539,315 0.25% 394,442 11% 6,302 1,426 38,487 3,079,643 18,979 36
AdsBot-Google-Mobile-Apps 3,529,324 0.24% 16,101 0% 29,563 20,772 9,932 3,442,384 10,563 9
img2dataset 3,456,771 0.24% 321,034 9% 922 1,191 37,737 3,078,335 17,517 35
GoogleOther-Image 3,351,237 0.23% 179,995 5% 3,761 2,354 38,713 3,082,229 44,166 19
GoogleOther-Video 3,348,870 0.23% 178,406 5% 3,490 2,339 38,557 3,082,039 44,020 19
TikTokSpider 3,248,538 0.23% 155,489 4% 910 932 1,013 3,089,292 875 27
MyCentralAIScraperBot 3,204,041 0.22% 125,523 3% 203 95 537 3,077,393 279 11
PerplexityBot 2,898,222 0.2% 1,821,583 62% 452,670 8,094 81,374 376,642 157,770 89
ImagesiftBot 2,897,095 0.2% 2,562,432 88% 3,832 2,969 41,314 6,261 280,228 59
rogerbot 2,458,351 0.17% 2,108,077 85% 91,713 21,462 7,579 101,204 128,150 166
OAI-SearchBot 2,432,260 0.17% 1,789,041 73% 288,489 2,959 97,999 92,559 161,150 63
Claude-Web 2,220,660 0.15% 1,621,357 73% 165,860 6,588 65,422 337,733 23,580 120
psbot 2,265,494 0.16% 2,183,566 96% 5,391 523 4,259 1,782 69,968 5
Googlebot-Image 2,984,552 0.21% 411,419 13% 1,203,081 34,100 262,545 268,227 805,094 86
magpie-crawler 2,113,429 0.15% 2,104,118 99% 707 997 802 3,910 2,893 2
TurnitinBot 2,048,816 0.14% 2,040,521 99% 1,171 811 1,086 548 4,659 20
DataForSeoBot 2,066,487 0.14% 1,940,268 93% 76,164 597 2,705 20,995 25,731 27
facebookexternalhit 2,158,213 0.15% 256,062 11% 1,208,775 27,316 105,460 390,785 169,663 152
MegaIndex.ru 1,939,355 0.13% 1,920,727 99% 496 784 1,305 315 15,721 7
Exabot 1,857,786 0.13% 1,777,610 95% 13,489 867 8,763 48,891 8,142 24
Scrapy 1,828,160 0.13% 1,720,935 94% 35,193 1,242 38,334 12,687 19,725 44
Gigabot 1,843,869 0.13% 1,772,654 96% 4,716 505 3,579 677 61,733 5
spbot 2,129,356 0.15% 1,621,786 76% 348 133 1,519 363 505,206 1
WebCopier 1,606,085 0.11% 1,599,413 99% 311 700 1,000 635 4,020 6
Teleport 1,589,190 0.11% 1,584,737 99% 262 616 747 624 2,197 7
WebZIP 1,587,808 0.11% 1,582,391 99% 284 619 1,461 622 2,425 6
Sogou web spider 2,560,874 0.18% 1,336,212 52% 189,142 481 19,307 15,934 999,793 5
WebStripper 1,552,318 0.11% 1,547,728 99% 268 610 748 618 2,342 4
TeleportPro 1,537,156 0.11% 1,533,237 99% 240 383 777 488 2,025 6
SiteSnagger 1,520,626 0.11% 1,516,138 99% 269 610 725 618 2,262 4
Offline Explorer 1,505,533 0.1% 1,500,608 99% 286 609 768 629 2,630 3
Xenu 1,497,253 0.1% 1,492,308 99% 343 1,010 757 506 2,326 3
Applebot 1,719,596 0.12% 958,644 55% 242,299 4,395 118,057 170,896 225,143 162
Teoma 1,567,640 0.11% 1,366,291 87% 14,431 1,360 10,519 86,603 88,428 8
Twiceler 1,454,202 0.1% 1,423,948 97% 1,861 16 2,133 232 26,012 0
EmailSiphon 1,426,599 0.1% 1,423,423 99% 190 598 595 151 1,639 3
larbin 1,423,336 0.1% 1,419,543 99% 223 478 636 500 1,952 4
NerdyBot 1,416,248 0.1% 1,415,140 99% 132 41 81 223 631 0
Zeus 1,408,180 0.1% 1,405,463 99% 197 605 551 152 1,209 3
Wget 1,404,915 0.1% 1,397,817 99% 521 714 2,216 740 2,901 6
MSIECrawler 1,393,487 0.1% 1,388,443 99% 312 406 756 496 3,070 4
EmailWolf 1,389,676 0.1% 1,387,092 99% 191 596 505 148 1,141 3
HTTrack 1,387,552 0.1% 1,379,838 99% 587 427 1,576 854 4,262 8
ExtractorPro 1,382,496 0.1% 1,379,447 99% 190 595 580 148 1,532 4
CherryPicker 1,378,601 0.1% 1,376,315 99% 168 502 502 142 970 2
WebBandit 1,377,281 0.1% 1,374,901 99% 179 494 532 144 1,029 2
CopyRightCheck 1,377,038 0.1% 1,374,710 99% 174 503 500 141 1,008 2
WebSauger 1,373,695 0.1% 1,371,110 99% 191 590 515 148 1,138 3

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.