Most directed user-agents

User-Agent Mentions
(%)
(%)
* 155,343,232 14.46% 22,121,289 14% 20,848,690 593,092 35,207,051 48,822,472 27,739,254 11,384
Googlebot 19,739,088 1.84% 400,415 2% 1,142,316 71,213 15,355,832 1,021,027 1,746,176 2,109
AdsBot-Google 12,437,364 1.16% 58,824 0% 367,210 20,344 8,896,372 2,947,788 146,785 41
bingbot 13,676,698 1.27% 953,984 6% 350,642 5,878,900 3,797,076 483,783 2,212,224 89
Baiduspider 11,449,309 1.07% 7,350,462 64% 174,663 24,614 3,460,665 115,036 323,822 47
Mediapartners-Google 16,582,891 1.54% 57,769 0% 406,553 21,477 9,136,884 190,095 6,769,977 136
Yahoo! Slurp 9,360,133 0.87% 114,153 1% 31,707 5,840,458 3,332,027 27,681 14,105 2
AdsBot-Google-Mobile 8,238,912 0.77% 17,139 0% 74,800 20,054 5,323,639 2,783,142 20,129 9
Yandex 8,272,908 0.77% 789,111 9% 116,004 5,853,484 559,390 595,117 359,689 113
BLEXBot 7,797,438 0.73% 7,630,543 97% 717 2,139 31,706 2,164 130,156 13
PetalBot 7,335,797 0.68% 7,127,191 97% 3,795 1,292 15,073 24,320 164,079 47
AhrefsBot 13,313,116 1.24% 3,313,146 24% 143,370 19,808 3,525,666 21,885 6,289,173 68
GPTBot 6,540,373 0.61% 3,228,346 49% 25,655 67,493 42,175 2,745,064 431,508 132
IPS-Agent 6,022,674 0.56% 181,373 3% 53 20 5,840,524 93 610 1
Pandalytics 6,000,495 0.56% 159,499 2% 40 8 5,840,331 7 609 1
Sogou 5,962,200 0.56% 5,938,302 99% 7,962 63 5,856 1,076 8,922 19
IonCrawl 5,863,813 0.55% 23,444 0% 14 6 5,840,289 3 57 0
CCBot 5,622,244 0.52% 2,767,293 49% 6,895 23,214 31,008 2,712,561 81,198 75
ClaudeBot 5,556,790 0.52% 2,540,910 45% 10,180 44,812 22,741 2,736,776 201,347 24
Nutch 4,873,841 0.45% 4,804,591 98% 6,400 869 4,432 403 57,104 42
meta-externalagent 4,820,810 0.45% 1,953,785 40% 8,637 2,434 10,127 2,741,973 103,803 51
dotbot 10,464,800 0.97% 4,581,026 43% 7,443 19,690 37,475 8,920 5,810,208 38
anthropic-ai 4,683,232 0.44% 1,880,763 40% 8,585 3,426 4,165 2,738,164 48,104 25
Amazonbot 4,710,595 0.44% 1,783,906 37% 4,131 21,307 52,311 2,739,872 108,987 81
ChatGPT-User 4,669,121 0.43% 1,774,094 37% 24,790 1,731 21,974 2,763,433 83,052 47
Sosospider 4,715,604 0.44% 1,164,840 24% 41,745 113 3,325,055 1,668 182,181 2
MJ12bot 8,950,405 0.83% 4,267,914 47% 9,442 1,428 116,478 11,481 4,543,586 76
ia_archiver 11,428,409 1.06% 1,073,700 9% 161,937 2,301 3,040,231 101,694 7,048,522 24
Google-Extended 4,387,802 0.41% 1,434,771 32% 13,791 56,470 21,896 2,784,543 76,291 40
ichiro 4,259,014 0.4% 939,503 22% 371 21 3,316,963 175 1,981 0
YandexBot 4,121,832 0.38% 653,912 15% 45,020 19,785 3,349,534 17,937 35,585 59
SemrushBot 5,111,578 0.48% 3,753,203 73% 41,922 2,460 55,503 31,877 1,226,528 85
Applebot-Extended 3,811,640 0.35% 951,639 24% 1,629 43,235 5,286 2,736,976 72,856 19
FacebookBot 3,845,359 0.36% 908,655 23% 40,258 63,420 3,888 2,718,242 110,876 20
sogou spider 4,719,817 0.44% 358,269 7% 32,319 4,024 3,322,210 2,852 1,000,139 4
Seznambot 3,704,718 0.34% 332,901 8% 9,543 536 3,328,451 24,279 8,996 12
YoudaoBot 3,655,212 0.34% 300,120 8% 29,208 61 3,322,311 1,414 2,080 18
PerplexityBot 3,814,876 0.36% 873,568 22% 16,747 2,999 4,630 2,741,524 175,386 22
AhrefsSiteAudit 3,623,355 0.34% 66,301 1% 34,683 103 3,453,514 40,173 28,575 6
cohere-ai 3,582,882 0.33% 818,875 22% 1,816 2,307 2,974 2,710,178 46,715 17
Claude-Web 3,441,007 0.32% 645,708 18% 3,881 2,875 3,726 2,737,853 46,940 24
YetiBot 3,320,874 0.31% 2,821 0% 176 1 3,316,791 103 982 0
rdfbot 3,316,178 0.31% 20 0% 15 0 3,316,118 2 23 0
Google-CloudVertexBot 2,932,864 0.27% 222,333 7% 1,240 921 196 2,708,029 135 10
Quora-Bot 2,917,048 0.27% 209,278 7% 34 7 33 2,707,684 12 0
DuckAssistBot 2,867,432 0.27% 140,598 4% 1,470 158 16,385 2,708,395 409 17
AdsBot-Google-Mobile-Apps 2,825,275 0.26% 9,810 0% 25,847 19,072 7,608 2,752,773 10,156 9
Bytespider 2,827,467 0.26% 2,629,216 92% 11,778 74,443 10,410 7,436 94,159 25
psbot 2,273,775 0.21% 2,201,122 96% 4,635 469 3,837 1,547 62,159 6
TurnitinBot 1,766,417 0.16% 1,760,820 99% 752 182 761 362 3,535 5
ImagesiftBot 1,973,505 0.18% 1,674,185 84% 538 900 3,532 1,517 292,802 31
rogerbot 1,786,959 0.17% 1,530,539 85% 10,126 19,622 19,215 80,874 126,569 14
MegaIndex.ru 1,640,875 0.15% 1,624,407 98% 239 96 1,032 156 14,941 4
OAI-SearchBot 1,783,461 0.17% 1,530,041 85% 22,710 596 18,721 26,504 184,866 23
Exabot 1,584,282 0.15% 1,539,257 97% 7,755 715 7,770 23,736 5,039 10
Scrapy 1,603,778 0.15% 1,518,648 94% 36,542 341 843 657 46,727 20
Gigabot 1,572,942 0.15% 1,506,773 95% 3,817 454 3,364 469 58,061 4
WebCopier 1,494,994 0.14% 1,489,261 99% 228 629 887 574 3,410 5
Teleport 1,480,412 0.14% 1,476,783 99% 208 554 665 561 1,637 4
spbot 1,922,971 0.18% 1,470,512 76% 184 111 1,441 135 450,587 1
WebZIP 1,456,019 0.14% 1,451,542 99% 221 560 1,320 562 1,810 4
WebStripper 1,449,938 0.14% 1,446,240 99% 210 551 664 558 1,711 4
TeleportPro 1,440,987 0.13% 1,437,801 99% 184 360 702 455 1,481 4
Offline Explorer 1,431,505 0.13% 1,427,481 99% 216 527 679 557 2,042 3
Xenu 1,427,436 0.13% 1,423,879 99% 224 391 670 471 1,798 3
SiteSnagger 1,420,463 0.13% 1,416,823 99% 208 549 642 558 1,679 4
magpie-crawler 1,370,900 0.13% 1,367,651 99% 261 606 469 63 1,847 3
EmailSiphon 1,357,444 0.13% 1,354,657 99% 175 535 572 124 1,378 3
larbin 1,346,733 0.13% 1,343,567 99% 186 450 583 471 1,472 4
MSIECrawler 1,339,628 0.12% 1,335,278 99% 263 385 689 461 2,548 4
Zeus 1,324,852 0.12% 1,322,531 99% 176 548 516 124 953 4
EmailWolf 1,324,567 0.12% 1,322,382 99% 175 535 483 121 868 3
WebBandit 1,321,629 0.12% 1,319,642 99% 165 443 503 111 763 2
ExtractorPro 1,319,486 0.12% 1,316,809 99% 174 535 561 122 1,281 4
WebSauger 1,313,798 0.12% 1,311,600 99% 169 531 489 121 885 3
NetAnts 1,311,288 0.12% 1,309,104 99% 170 531 502 121 857 3
WebAuto 1,311,155 0.12% 1,308,993 99% 206 528 486 121 818 3
Sogou web spider 2,280,352 0.21% 1,228,733 53% 67,502 367 10,874 1,865 971,008 3
SurveyBot 1,308,566 0.12% 1,306,720 99% 210 78 725 55 774 4
NerdyBot 1,307,583 0.12% 1,306,608 99% 125 39 86 110 615 0
LinkWalker 1,305,931 0.12% 1,303,047 99% 176 498 665 115 1,428 2
asterias 1,315,126 0.12% 1,299,293 98% 1,083 451 2,112 242 11,943 2
CherryPicker 1,301,050 0.12% 1,299,161 99% 154 451 473 109 700 2
CopyRightCheck 1,300,049 0.12% 1,298,119 99% 159 451 470 108 740 2
BuiltBotTough 1,296,620 0.12% 1,294,811 99% 158 447 467 100 635 2
Harvest/1.5 1,295,601 0.12% 1,293,605 99% 162 447 509 113 763 2
LinkextractorPro 1,295,465 0.12% 1,293,574 99% 160 451 470 108 700 2
lwp-trivial 1,294,690 0.12% 1,292,879 99% 160 447 473 109 620 2
DittoSpyder 1,294,066 0.12% 1,292,099 99% 160 450 550 109 696 2
True_Robot 1,293,955 0.12% 1,292,174 99% 161 447 449 108 614 2
BackDoorBot/1.0 1,293,957 0.12% 1,292,104 99% 157 451 494 109 640 2
Crescent 1,293,701 0.12% 1,291,819 99% 154 447 466 109 704 2
Szukacz/1.4 1,295,389 0.12% 1,291,672 99% 181 447 576 114 2,397 2
NICErsPRO 1,293,608 0.12% 1,291,743 99% 162 447 469 108 677 2
ProWebWalker 1,293,456 0.12% 1,291,587 99% 152 447 467 108 693 2
CheeseBot 1,293,247 0.12% 1,291,380 99% 153 447 469 108 688 2
hloader 1,293,137 0.12% 1,291,276 99% 156 446 463 109 682 5
cosmos 1,293,097 0.12% 1,291,185 99% 155 447 495 112 701 2
Telesoft 1,292,964 0.12% 1,291,097 99% 159 447 464 108 687 2
MIIxpc 1,292,842 0.12% 1,291,024 99% 162 447 463 108 636 2

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.