Most directed user-agents

User-Agent Mentions
(%)
(%)
* 170,665,433 14.79% 25,598,282 14% 23,125,939 514,684 38,062,795 53,966,815 29,384,235 12,683
Googlebot 23,022,395 2.0% 488,371 2% 1,320,058 73,824 17,331,553 1,108,755 2,697,123 2,711
AdsBot-Google 15,356,542 1.33% 64,625 0% 383,940 22,694 11,397,260 3,327,143 160,832 48
bingbot 15,250,366 1.32% 1,055,636 6% 423,337 6,351,621 4,403,894 535,172 2,480,572 134
Baiduspider 12,955,131 1.12% 8,100,851 62% 259,741 24,731 4,052,707 144,782 372,266 53
Mediapartners-Google 18,019,759 1.56% 59,756 0% 441,282 22,102 10,158,477 209,621 7,128,393 128
AdsBot-Google-Mobile 10,756,022 0.93% 19,226 0% 78,576 22,303 7,456,725 3,148,536 30,647 9
Yahoo! Slurp 10,450,218 0.91% 132,165 1% 37,733 6,311,162 3,912,098 41,781 15,277 2
PetalBot 9,620,087 0.83% 9,397,568 97% 4,672 1,379 16,750 25,060 174,605 53
Yandex 9,048,696 0.78% 1,009,085 11% 142,467 6,325,536 573,355 650,552 347,514 187
BLEXBot 8,411,212 0.73% 8,234,586 97% 1,030 2,231 33,130 2,201 138,019 15
GPTBot 8,055,367 0.7% 4,291,156 53% 75,093 70,101 59,010 3,130,324 429,534 149
AhrefsBot 15,885,135 1.38% 3,350,580 21% 160,048 20,272 3,897,974 23,077 8,433,106 78
CCBot 6,950,907 0.6% 3,716,364 53% 14,777 24,386 35,405 3,072,437 87,457 81
ClaudeBot 7,037,698 0.61% 3,603,574 51% 49,702 46,449 37,134 3,093,892 206,918 29
IPS-Agent 6,497,723 0.56% 185,643 2% 61 15 6,311,246 127 630 1
Pandalytics 6,477,244 0.56% 165,497 2% 46 8 6,311,022 43 627 1
Sogou 6,464,676 0.56% 6,431,782 99% 9,678 69 6,419 7,724 8,984 20
IonCrawl 6,335,092 0.55% 24,033 0% 15 6 6,310,975 3 60 0
Amazonbot 6,177,991 0.54% 2,870,469 46% 6,799 25,387 47,132 3,113,662 114,455 87
meta-externalagent 6,152,998 0.53% 2,912,538 47% 10,628 6,033 13,447 3,106,380 103,646 326
Google-Extended 5,682,380 0.49% 2,267,045 39% 54,083 56,549 38,002 3,183,910 82,740 51
Nutch 5,302,312 0.46% 5,226,379 98% 6,888 949 4,777 455 62,817 47
Sosospider 5,371,536 0.47% 1,219,438 22% 45,069 123 3,914,072 1,722 191,110 2
ia_archiver 13,621,249 1.18% 1,273,632 9% 166,241 2,437 3,611,802 122,606 8,444,470 61
anthropic-ai 5,133,322 0.44% 1,930,118 37% 45,147 3,529 8,178 3,094,163 52,159 28
ChatGPT-User 5,169,200 0.45% 1,834,449 35% 70,742 1,502 37,261 3,136,114 89,075 57
YandexBot 4,959,995 0.43% 882,978 17% 53,359 20,822 3,939,857 20,193 42,724 62
Applebot-Extended 4,973,804 0.43% 1,749,062 35% 3,130 44,638 7,179 3,092,598 77,178 19
ichiro 4,860,600 0.42% 953,068 19% 393 16 3,904,914 216 1,993 0
dotbot 12,693,705 1.1% 4,656,684 36% 8,355 20,081 39,120 9,006 7,960,410 49
MJ12bot 9,497,426 0.82% 4,429,777 46% 11,538 1,527 121,401 12,151 4,920,944 88
sogou spider 5,399,578 0.47% 433,774 8% 39,200 3,822 3,910,669 3,123 1,008,986 4
FacebookBot 4,432,855 0.38% 1,124,583 25% 42,342 65,601 5,305 3,076,798 118,204 22
Seznambot 4,322,268 0.37% 355,864 8% 8,778 446 3,916,851 30,772 9,545 12
PerplexityBot 4,449,418 0.39% 1,089,516 24% 60,035 2,881 19,035 3,098,886 179,038 27
YoudaoBot 4,265,958 0.37% 318,154 7% 32,269 62 3,911,753 1,553 2,150 17
cohere-ai 4,037,787 0.35% 874,933 21% 34,385 2,311 6,631 3,069,169 50,340 18
AhrefsSiteAudit 4,007,629 0.35% 70,009 1% 40,670 116 3,822,890 41,032 32,901 11
SemrushBot 5,200,940 0.45% 3,776,640 72% 48,559 2,542 52,245 46,506 1,274,310 138
YetiBot 3,909,907 0.34% 3,868 0% 182 1 3,904,723 114 1,019 0
rdfbot 3,904,109 0.34% 21 0% 15 0 3,904,049 2 22 0
Claude-Web 3,915,628 0.34% 720,869 18% 39,198 3,104 7,590 3,094,038 50,799 30
Bytespider 3,663,967 0.32% 3,442,175 93% 18,293 76,328 15,112 7,941 104,092 26
Google-CloudVertexBot 3,329,112 0.29% 256,180 7% 1,729 2,964 689 3,067,373 164 13
Quora-Bot 3,300,067 0.29% 232,698 7% 224 8 132 3,066,978 27 0
DuckAssistBot 3,244,059 0.28% 155,004 4% 2,557 230 17,490 3,068,209 552 17
AdsBot-Google-Mobile-Apps 3,195,009 0.28% 11,663 0% 27,275 19,381 8,688 3,117,450 10,543 9
psbot 2,378,574 0.21% 2,299,975 96% 4,835 492 4,063 1,637 67,566 6
ImagesiftBot 2,221,301 0.19% 1,907,787 85% 688 822 4,270 1,567 306,132 35
TurnitinBot 1,859,615 0.16% 1,853,459 99% 783 212 910 421 3,824 6
rogerbot 1,929,994 0.17% 1,664,185 86% 10,514 19,937 11,018 96,076 128,048 216
OAI-SearchBot 1,895,631 0.16% 1,562,023 82% 64,412 689 31,100 49,241 188,140 26
MegaIndex.ru 1,699,448 0.15% 1,682,549 99% 336 112 1,200 200 15,046 5
Exabot 1,647,738 0.14% 1,593,248 96% 9,543 767 8,550 30,121 5,496 13
Scrapy 1,666,020 0.14% 1,575,445 94% 37,725 411 1,362 724 50,329 24
Gigabot 1,665,139 0.14% 1,593,670 95% 3,954 470 3,592 517 62,932 4
WebCopier 1,542,352 0.13% 1,536,142 99% 244 665 999 600 3,697 5
Teleport 1,529,434 0.13% 1,525,459 99% 224 581 744 585 1,837 4
spbot 1,996,771 0.17% 1,520,771 76% 248 123 1,558 172 473,898 1
WebZIP 1,503,023 0.13% 1,498,112 99% 236 586 1,434 584 2,065 6
WebStripper 1,493,352 0.13% 1,489,242 99% 226 578 746 581 1,975 4
TeleportPro 1,486,202 0.13% 1,482,742 99% 197 368 764 472 1,655 4
SiteSnagger 1,463,725 0.13% 1,459,738 99% 221 576 719 581 1,886 4
Offline Explorer 1,458,055 0.13% 1,453,562 99% 239 611 759 583 2,298 3
Xenu 1,455,786 0.13% 1,451,899 99% 241 407 744 489 2,003 3
magpie-crawler 1,417,477 0.12% 1,413,889 99% 324 460 674 102 2,025 3
EmailSiphon 1,377,062 0.12% 1,374,125 99% 187 563 609 130 1,445 3
larbin 1,368,575 0.12% 1,365,123 99% 196 464 643 487 1,658 4
Sogou web spider 2,324,404 0.2% 1,259,814 54% 88,181 408 12,371 2,256 961,370 4
MSIECrawler 1,363,462 0.12% 1,358,802 99% 275 394 763 479 2,745 4
Zeus 1,360,420 0.12% 1,357,953 99% 187 574 550 132 1,020 4
NerdyBot 1,358,350 0.12% 1,357,283 99% 128 47 94 147 651 0
EmailWolf 1,343,966 0.12% 1,341,647 99% 185 561 519 128 923 3
WebBandit 1,338,915 0.12% 1,336,805 99% 178 464 539 119 808 2
ExtractorPro 1,338,259 0.12% 1,335,453 99% 182 562 588 129 1,341 4
CherryPicker 1,335,104 0.12% 1,333,089 99% 164 472 510 117 750 2
CopyRightCheck 1,334,049 0.12% 1,331,995 99% 169 472 508 116 787 2
Wget 1,334,818 0.12% 1,328,164 99% 410 655 2,121 715 2,749 4
WebSauger 1,332,061 0.12% 1,329,723 99% 180 558 524 128 945 3
NetAnts 1,329,351 0.12% 1,327,028 99% 181 558 539 128 914 3
WebAuto 1,329,306 0.12% 1,327,006 99% 217 555 523 128 874 3
LinkWalker 1,326,502 0.11% 1,323,463 99% 186 516 713 123 1,499 2
asterias 1,333,664 0.12% 1,316,231 98% 1,242 472 2,158 256 13,303 2
SurveyBot 1,319,158 0.11% 1,317,179 99% 243 82 734 91 825 4
HTTrack 1,322,100 0.11% 1,314,847 99% 492 449 1,515 734 4,055 8
BuiltBotTough 1,313,861 0.11% 1,311,929 99% 168 468 500 107 687 2
LinkextractorPro 1,312,869 0.11% 1,310,841 99% 171 472 507 116 760 2
Harvest/1.5 1,312,453 0.11% 1,310,323 99% 173 468 546 121 820 2
lwp-trivial 1,312,118 0.11% 1,310,183 99% 171 468 505 117 672 2
Teoma 1,385,411 0.12% 1,199,472 86% 13,096 791 11,078 86,877 74,090 7
True_Robot 1,311,348 0.11% 1,309,456 99% 170 468 472 116 664 2
DittoSpyder 1,310,927 0.11% 1,308,827 99% 170 471 592 117 748 2
Szukacz/1.4 1,312,446 0.11% 1,308,592 99% 192 468 608 123 2,461 2
NICErsPRO 1,310,661 0.11% 1,308,668 99% 173 468 505 116 729 2
Crescent 1,310,566 0.11% 1,308,558 99% 164 468 502 117 755 2
ProWebWalker 1,310,524 0.11% 1,308,530 99% 163 468 502 116 743 2
BackDoorBot/1.0 1,310,449 0.11% 1,308,463 99% 167 472 535 117 693 2
Telesoft 1,310,165 0.11% 1,308,172 99% 170 468 501 116 736 2
CheeseBot 1,310,130 0.11% 1,308,137 99% 163 468 507 116 737 2

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.