Most directed user-agents

User-Agent Mentions
(%)
(%)
* 158,520,570 14.48% 22,912,713 14% 21,269,732 503,536 36,034,346 49,814,532 27,974,217 11,494
Googlebot 20,266,320 1.85% 416,894 2% 1,197,512 71,410 15,618,513 1,047,238 1,912,660 2,093
AdsBot-Google 13,127,774 1.2% 60,059 0% 370,030 20,404 9,416,909 3,112,398 147,928 46
bingbot 14,021,344 1.28% 997,528 7% 364,093 5,986,190 3,920,077 498,338 2,255,030 88
Baiduspider 11,769,610 1.07% 7,482,992 63% 212,645 24,690 3,585,991 131,984 331,260 48
Mediapartners-Google 16,896,305 1.54% 58,204 0% 414,179 21,442 9,333,555 204,345 6,864,444 136
Yahoo! Slurp 9,605,363 0.88% 119,224 1% 32,851 5,947,729 3,453,559 37,515 14,483 2
AdsBot-Google-Mobile 8,454,576 0.77% 17,777 0% 75,842 20,070 5,378,986 2,941,424 20,468 9
Yandex 8,408,628 0.77% 795,091 9% 117,602 5,960,803 556,716 616,717 361,588 111
BLEXBot 7,932,673 0.72% 7,765,025 97% 787 2,157 31,914 2,048 130,727 15
AhrefsBot 13,833,636 1.26% 3,295,002 23% 145,265 19,947 3,997,195 22,376 6,353,783 68
PetalBot 7,453,877 0.68% 7,243,229 97% 3,939 1,314 15,365 25,181 164,803 46
GPTBot 7,310,556 0.67% 3,391,743 46% 28,261 68,420 492,318 2,897,922 431,755 137
ClaudeBot 6,342,222 0.58% 2,721,725 42% 11,673 45,566 472,573 2,889,217 201,445 23
IPS-Agent 6,132,222 0.56% 183,642 2% 56 17 5,947,804 93 609 1
Pandalytics 6,110,541 0.56% 162,274 2% 43 8 5,947,600 7 608 1
Sogou 6,071,150 0.55% 6,046,559 99% 8,451 65 5,768 1,334 8,954 19
IonCrawl 5,971,187 0.55% 23,550 0% 14 5 5,947,558 3 57 0
CCBot 5,968,718 0.55% 2,955,849 49% 8,359 23,435 31,642 2,868,225 81,133 75
Nutch 5,350,765 0.49% 5,280,428 98% 6,530 904 4,472 403 57,982 46
ChatGPT-User 5,303,517 0.48% 1,793,228 33% 27,275 1,758 471,640 2,926,165 83,404 47
Google-Extended 5,173,776 0.47% 1,605,761 31% 15,159 56,188 472,130 2,947,980 76,517 41
meta-externalagent 5,137,527 0.47% 2,111,033 41% 8,918 2,479 10,680 2,900,561 103,804 52
Amazonbot 5,058,816 0.46% 1,970,292 38% 4,848 21,509 47,256 2,905,601 109,231 79
anthropic-ai 4,840,666 0.44% 1,884,190 38% 9,633 3,478 4,775 2,890,228 48,339 23
Sosospider 4,852,104 0.44% 1,170,845 24% 42,737 113 3,453,201 1,687 183,519 2
dotbot 10,476,512 0.96% 4,536,306 43% 7,774 19,766 37,598 9,110 5,865,918 40
ia_archiver 11,892,947 1.09% 1,119,413 9% 163,957 2,310 3,154,046 117,793 7,335,404 24
MJ12bot 9,406,670 0.86% 4,262,436 45% 10,219 1,479 111,729 12,166 5,008,561 80
ichiro 4,391,744 0.4% 944,366 21% 384 18 3,444,817 179 1,980 0
PerplexityBot 4,432,887 0.4% 887,433 20% 19,078 3,036 454,098 2,893,880 175,342 20
YandexBot 4,270,711 0.39% 672,281 15% 45,923 19,709 3,477,932 18,721 36,085 60
AhrefsSiteAudit 4,095,695 0.37% 65,983 1% 34,902 108 3,924,492 41,241 28,962 7
Applebot-Extended 4,119,277 0.38% 1,105,858 26% 1,909 43,955 5,741 2,888,989 72,809 16
FacebookBot 4,018,848 0.37% 921,609 22% 43,332 64,323 4,117 2,873,686 111,763 18
SemrushBot 5,098,500 0.47% 3,726,513 73% 43,100 2,465 52,060 41,830 1,232,449 83
sogou spider 4,857,783 0.44% 374,523 7% 33,966 4,051 3,450,246 2,959 992,034 4
Seznambot 3,844,973 0.35% 339,610 8% 8,974 476 3,456,226 30,635 9,040 12
YoudaoBot 3,790,222 0.35% 306,407 8% 29,974 62 3,450,230 1,437 2,095 17
cohere-ai 3,745,650 0.34% 825,335 22% 1,997 2,341 3,475 2,865,578 46,909 15
Claude-Web 3,610,739 0.33% 661,657 18% 4,591 2,911 4,277 2,890,008 47,270 25
YetiBot 3,448,784 0.31% 2,891 0% 177 1 3,444,631 103 981 0
rdfbot 3,444,023 0.31% 20 0% 15 0 3,443,963 2 23 0
Google-CloudVertexBot 3,097,159 0.28% 230,660 7% 1,304 941 395 2,863,715 136 8
Quora-Bot 3,079,020 0.28% 215,557 7% 53 7 35 2,863,356 12 0
DuckAssistBot 3,024,568 0.28% 141,348 4% 1,699 189 16,567 2,864,206 545 14
AdsBot-Google-Mobile-Apps 2,984,534 0.27% 10,443 0% 26,381 19,119 7,811 2,910,546 10,225 9
Bytespider 2,902,332 0.27% 2,700,588 93% 12,460 75,623 11,223 7,865 94,550 23
psbot 2,287,545 0.21% 2,213,904 96% 4,681 468 3,867 1,545 63,074 6
OAI-SearchBot 2,253,048 0.21% 1,531,789 67% 24,500 603 467,913 43,208 185,014 21
TurnitinBot 1,780,709 0.16% 1,775,027 99% 762 184 768 361 3,603 4
ImagesiftBot 1,998,725 0.18% 1,695,098 84% 562 901 3,861 1,526 296,747 30
rogerbot 1,796,883 0.16% 1,536,227 85% 10,119 19,671 12,954 90,987 126,755 170
MegaIndex.ru 1,654,769 0.15% 1,638,229 99% 261 98 1,104 159 14,914 4
Exabot 1,597,461 0.15% 1,545,448 96% 8,268 726 7,834 30,055 5,120 10
Scrapy 1,613,464 0.15% 1,528,102 94% 36,371 369 1,036 657 46,910 19
Gigabot 1,581,486 0.14% 1,514,443 95% 3,863 456 3,416 466 58,838 4
WebCopier 1,498,907 0.14% 1,493,083 99% 231 633 908 576 3,471 5
Teleport 1,486,304 0.14% 1,482,591 99% 211 554 681 563 1,700 4
spbot 1,935,896 0.18% 1,478,596 76% 201 113 1,462 135 455,388 1
WebZIP 1,459,735 0.13% 1,455,175 99% 224 559 1,337 563 1,872 5
WebStripper 1,453,691 0.13% 1,449,906 99% 213 551 683 560 1,774 4
TeleportPro 1,446,363 0.13% 1,443,105 99% 187 358 712 455 1,542 4
Offline Explorer 1,434,631 0.13% 1,430,501 99% 219 554 692 559 2,103 3
Xenu 1,432,684 0.13% 1,429,044 99% 227 389 688 472 1,861 3
SiteSnagger 1,424,009 0.13% 1,420,285 99% 211 549 656 560 1,744 4
magpie-crawler 1,375,520 0.13% 1,372,174 99% 266 606 490 63 1,918 3
EmailSiphon 1,360,627 0.12% 1,357,826 99% 177 535 580 124 1,382 3
larbin 1,350,624 0.12% 1,347,372 99% 189 450 600 472 1,537 4
MSIECrawler 1,342,733 0.12% 1,338,310 99% 266 383 697 462 2,611 4
Sogou web spider 2,291,953 0.21% 1,242,884 54% 72,298 381 11,262 1,881 963,244 3
Zeus 1,327,882 0.12% 1,325,551 99% 178 547 522 125 955 4
EmailWolf 1,327,622 0.12% 1,325,426 99% 177 534 490 122 870 3
WebBandit 1,324,232 0.12% 1,322,236 99% 167 441 512 112 762 2
ExtractorPro 1,322,380 0.12% 1,319,697 99% 176 535 562 123 1,283 4
WebSauger 1,316,768 0.12% 1,314,559 99% 171 531 496 122 886 3
NerdyBot 1,315,054 0.12% 1,314,069 99% 128 39 90 110 618 0
NetAnts 1,314,185 0.12% 1,311,989 99% 172 531 509 122 859 3
WebAuto 1,314,083 0.12% 1,311,912 99% 208 528 491 122 819 3
SurveyBot 1,310,336 0.12% 1,308,470 99% 218 75 727 55 787 4
LinkWalker 1,308,716 0.12% 1,305,820 99% 178 497 673 116 1,430 2
asterias 1,317,617 0.12% 1,301,712 98% 1,099 449 2,115 243 11,997 2
CherryPicker 1,303,647 0.12% 1,301,751 99% 156 449 478 110 701 2
CopyRightCheck 1,302,587 0.12% 1,300,649 99% 161 449 477 109 740 2
BuiltBotTough 1,299,148 0.12% 1,297,332 99% 160 445 471 101 637 2
Harvest/1.5 1,298,009 0.12% 1,296,007 99% 164 445 514 114 763 2
LinkextractorPro 1,297,900 0.12% 1,295,999 99% 162 449 477 109 702 2
lwp-trivial 1,297,234 0.12% 1,295,415 99% 162 445 480 110 620 2
True_Robot 1,296,397 0.12% 1,294,618 99% 163 445 447 109 613 2
DittoSpyder 1,296,473 0.12% 1,294,499 99% 162 448 556 110 696 2
BackDoorBot/1.0 1,296,326 0.12% 1,294,467 99% 159 449 497 110 642 2
Szukacz/1.4 1,297,881 0.12% 1,294,156 99% 183 445 583 115 2,397 2
Crescent 1,296,101 0.12% 1,294,212 99% 156 445 471 110 705 2
NICErsPRO 1,296,045 0.12% 1,294,174 99% 164 445 474 109 677 2
ProWebWalker 1,295,931 0.12% 1,294,056 99% 154 445 472 109 693 2
CheeseBot 1,295,679 0.12% 1,293,805 99% 155 445 474 109 689 2
hloader 1,295,557 0.12% 1,293,690 99% 158 444 468 110 682 5
cosmos 1,295,500 0.12% 1,293,579 99% 157 445 502 113 702 2
Telesoft 1,295,463 0.12% 1,293,591 99% 161 445 469 109 686 2
LexiBot 1,295,330 0.12% 1,293,466 99% 165 445 471 108 673 2

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.