Most directed user-agents

User-Agent Mentions
(%)
(%)
* 179,191,113 15.02% 28,407,657 15% 19,415,332 535,799 40,088,007 60,605,664 30,125,616 13,038
Googlebot 19,895,114 1.67% 564,767 2% 1,461,343 76,089 12,933,700 1,144,768 3,711,675 2,772
bingbot 16,814,177 1.41% 1,120,559 6% 507,287 6,812,995 5,195,756 549,995 2,627,444 141
AdsBot-Google 14,454,104 1.21% 68,745 0% 395,551 23,310 10,295,657 3,399,010 271,783 48
Baiduspider 14,423,078 1.21% 8,657,247 60% 351,043 25,232 4,806,948 156,534 426,020 54
Mediapartners-Google 19,516,032 1.64% 60,901 0% 466,563 22,610 11,374,237 212,409 7,379,183 129
Yahoo! Slurp 11,679,192 0.98% 144,310 1% 44,748 6,770,501 4,661,665 41,738 16,226 4
AdsBot-Google-Mobile 9,752,920 0.82% 20,160 0% 81,061 22,898 6,388,868 3,214,246 25,678 9
Yandex 9,584,639 0.8% 1,017,320 10% 185,978 6,785,377 583,237 665,912 346,628 187
BLEXBot 9,168,531 0.77% 8,977,215 97% 1,265 2,362 30,482 2,364 154,829 14
GPTBot 9,459,482 0.79% 5,531,743 58% 105,372 82,330 61,575 3,198,452 479,841 169
PetalBot 8,855,998 0.74% 8,618,562 97% 14,438 1,477 14,498 26,018 180,946 59
CCBot 8,182,973 0.69% 4,869,919 59% 25,480 25,467 36,051 3,137,109 88,850 97
ClaudeBot 8,251,558 0.69% 4,717,872 57% 71,044 57,236 32,919 3,163,083 209,362 42
AhrefsBot 15,181,211 1.27% 3,728,522 24% 173,181 20,851 3,858,254 24,435 7,375,895 73
Amazonbot 7,388,400 0.62% 4,005,181 54% 12,692 27,900 47,921 3,178,577 116,031 98
meta-externalagent 7,338,996 0.62% 4,022,304 54% 16,219 8,011 11,349 3,172,067 108,417 629
IPS-Agent 6,954,034 0.58% 182,573 2% 67 13 6,770,595 175 610 1
Sogou 6,942,620 0.58% 6,898,537 99% 9,147 69 6,800 18,525 9,519 23
Pandalytics 6,933,468 0.58% 162,361 2% 48 6 6,770,362 86 604 1
Google-Extended 6,894,811 0.58% 3,382,991 49% 74,226 67,429 33,609 3,251,887 84,607 62
IonCrawl 6,794,425 0.57% 24,033 0% 15 5 6,770,309 3 60 0
ia_archiver 15,974,663 1.34% 1,458,914 9% 166,040 2,491 4,365,419 123,594 9,858,142 63
Applebot-Extended 6,156,559 0.52% 2,855,116 46% 6,523 55,404 4,681 3,157,027 77,779 29
Sosospider 6,214,608 0.52% 1,282,243 20% 50,997 138 4,663,113 1,731 216,384 2
YandexBot 5,792,080 0.49% 953,115 16% 59,928 21,222 4,690,802 22,002 44,948 63
ichiro 5,612,942 0.47% 956,002 17% 398 13 4,654,276 267 1,986 0
ChatGPT-User 5,375,433 0.45% 1,933,071 35% 99,310 11,741 38,429 3,202,463 90,357 62
anthropic-ai 5,307,590 0.44% 2,024,134 38% 58,693 3,720 8,573 3,158,663 53,777 30
sogou spider 6,279,716 0.53% 496,978 7% 61,067 3,844 4,660,976 3,125 1,053,722 4
Nutch 5,273,535 0.44% 5,195,717 98% 7,006 946 4,806 509 64,505 46
Seznambot 5,096,128 0.43% 376,531 7% 8,952 1,224 4,666,492 31,969 10,948 12
YoudaoBot 5,026,681 0.42% 327,460 6% 34,724 61 4,660,783 1,563 2,075 15
dotbot 11,912,093 1.0% 4,929,800 41% 8,794 20,671 36,890 9,560 6,906,328 50
Bytespider 4,963,692 0.42% 4,709,245 94% 34,328 86,891 15,165 8,326 109,705 32
MJ12bot 9,735,436 0.82% 4,705,207 48% 11,350 1,663 114,482 12,823 4,889,814 97
YetiBot 4,659,304 0.39% 3,994 0% 186 1 4,653,903 115 1,105 0
rdfbot 4,653,483 0.39% 21 0% 15 0 4,653,423 2 22 0
FacebookBot 4,598,917 0.39% 1,218,879 26% 46,589 66,831 5,623 3,140,332 120,634 29
PerplexityBot 4,622,564 0.39% 1,175,565 25% 83,491 2,983 14,722 3,165,626 180,135 42
SemrushBot 5,474,975 0.46% 4,023,038 73% 52,966 2,725 53,998 49,492 1,292,650 106
cohere-ai 4,195,220 0.35% 962,819 22% 39,440 2,461 6,958 3,132,292 51,224 26
Claude-Web 4,092,131 0.34% 815,530 19% 53,971 3,257 7,944 3,159,204 52,193 32
AhrefsSiteAudit 3,981,782 0.33% 76,779 1% 46,190 118 3,781,773 42,700 34,212 10
Google-CloudVertexBot 3,501,348 0.29% 362,744 10% 4,200 2,612 904 3,130,431 438 19
Quora-Bot 3,441,157 0.29% 310,644 9% 337 12 250 3,129,868 46 0
DuckAssistBot 3,316,143 0.28% 159,598 4% 5,709 301 17,816 3,131,802 893 24
AdsBot-Google-Mobile-Apps 3,261,429 0.27% 12,251 0% 27,828 19,902 9,022 3,181,722 10,695 9
psbot 2,480,396 0.21% 2,399,839 96% 4,770 506 4,105 1,697 69,473 6
ImagesiftBot 2,293,876 0.19% 1,978,915 86% 881 935 4,310 1,573 307,220 42
TurnitinBot 1,935,308 0.16% 1,928,988 99% 819 220 945 489 3,835 12
rogerbot 1,964,826 0.16% 1,702,761 86% 10,669 20,489 8,021 96,300 126,365 221
OAI-SearchBot 2,006,994 0.17% 1,654,298 82% 78,499 814 33,206 51,650 188,489 38
MegaIndex.ru 1,733,518 0.15% 1,716,266 99% 391 142 1,219 252 15,241 7
Scrapy 1,758,069 0.15% 1,666,980 94% 37,244 454 1,445 791 51,124 31
Gigabot 1,760,914 0.15% 1,687,831 95% 3,802 475 3,618 570 64,614 4
Exabot 1,675,121 0.14% 1,615,543 96% 10,303 787 8,795 32,808 6,870 15
WebCopier 1,552,310 0.13% 1,545,984 99% 257 675 1,030 605 3,754 5
Teleport 1,536,849 0.13% 1,532,775 99% 233 591 777 593 1,875 5
spbot 2,005,539 0.17% 1,526,547 76% 277 121 1,616 222 476,755 1
WebZIP 1,519,893 0.13% 1,514,890 99% 246 596 1,456 591 2,108 6
WebStripper 1,499,914 0.13% 1,495,705 99% 235 587 778 588 2,017 4
magpie-crawler 1,495,868 0.13% 1,491,893 99% 423 494 685 177 2,193 3
TeleportPro 1,493,016 0.13% 1,489,470 99% 202 375 797 477 1,691 4
SiteSnagger 1,469,847 0.12% 1,465,755 99% 232 586 752 588 1,930 4
Offline Explorer 1,462,133 0.12% 1,457,557 99% 242 601 790 589 2,351 3
Xenu 1,460,439 0.12% 1,456,474 99% 250 413 781 495 2,023 3
Teoma 1,496,851 0.13% 1,307,874 87% 13,004 1,567 10,962 87,571 75,866 7
Sogou web spider 2,351,014 0.2% 1,270,076 54% 95,991 404 12,181 2,354 970,003 5
NerdyBot 1,380,221 0.12% 1,379,114 99% 129 44 95 200 639 0
EmailSiphon 1,380,495 0.12% 1,377,492 99% 194 575 618 134 1,479 3
larbin 1,374,441 0.12% 1,370,923 99% 207 469 675 490 1,673 4
MSIECrawler 1,368,780 0.11% 1,364,048 99% 283 401 797 484 2,763 4
Zeus 1,365,026 0.11% 1,362,489 99% 197 585 564 134 1,053 4
Twiceler 1,375,687 0.12% 1,350,926 98% 1,718 16 2,086 208 20,733 0
Wget 1,354,580 0.11% 1,347,794 99% 432 683 2,143 724 2,800 4
EmailWolf 1,346,638 0.11% 1,344,252 99% 193 573 529 131 957 3
WebBandit 1,341,304 0.11% 1,339,123 99% 186 477 550 123 843 2
ExtractorPro 1,340,647 0.11% 1,337,776 99% 188 574 602 131 1,372 4
CherryPicker 1,339,750 0.11% 1,337,663 99% 173 486 522 122 782 2
CopyRightCheck 1,338,509 0.11% 1,336,383 99% 178 486 521 120 819 2
WebSauger 1,334,375 0.11% 1,331,967 99% 190 569 537 130 979 3
WebAuto 1,331,568 0.11% 1,329,197 99% 228 566 536 130 908 3
NetAnts 1,331,589 0.11% 1,329,195 99% 191 570 552 130 948 3
LinkWalker 1,328,554 0.11% 1,325,441 99% 197 528 728 127 1,531 2
HTTrack 1,330,371 0.11% 1,323,027 99% 504 425 1,546 786 4,075 8
asterias 1,336,449 0.11% 1,318,712 98% 1,351 486 2,135 265 13,498 2
SurveyBot 1,317,467 0.11% 1,315,439 99% 257 82 739 136 810 4
BuiltBotTough 1,316,582 0.11% 1,314,578 99% 177 482 522 111 710 2
LinkextractorPro 1,314,924 0.11% 1,312,821 99% 180 486 520 120 795 2
Harvest/1.5 1,314,713 0.11% 1,312,512 99% 182 482 558 125 852 2
lwp-trivial 1,314,234 0.11% 1,312,228 99% 180 481 517 122 704 2
True_Robot 1,313,323 0.11% 1,311,361 99% 176 481 486 120 697 2
DittoSpyder 1,312,983 0.11% 1,310,808 99% 179 485 608 121 780 2
Szukacz/1.4 1,314,591 0.11% 1,310,655 99% 200 482 617 128 2,507 2
NICErsPRO 1,312,691 0.11% 1,310,626 99% 182 482 518 120 761 2
Crescent 1,312,665 0.11% 1,310,585 99% 173 482 515 121 787 2
ProWebWalker 1,312,546 0.11% 1,310,480 99% 172 482 515 120 775 2
Telesoft 1,312,366 0.11% 1,310,299 99% 179 482 513 121 770 2
BackDoorBot/1.0 1,312,299 0.11% 1,310,260 99% 176 486 547 121 707 2

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.