Most directed user-agents

User-Agent Mentions
(%)
(%)
* 239,749,549 17.62% 35,581,402 14% 62,709,620 475,638 43,093,149 64,866,760 32,788,661 234,319
Googlebot 21,549,503 1.58% 715,510 3% 2,175,302 79,853 13,328,959 1,258,956 3,987,975 2,948
AdsBot-Google 16,605,754 1.22% 106,931 0% 615,994 23,629 11,294,725 4,282,076 282,346 53
bingbot 18,030,689 1.33% 1,189,346 6% 909,987 6,894,194 5,659,299 603,796 2,773,901 166
Baiduspider 15,433,843 1.13% 8,845,121 57% 626,085 25,581 5,264,615 186,329 486,043 69
GPTBot 13,360,070 0.98% 8,387,688 62% 226,017 86,641 87,713 4,085,772 486,018 221
Mediapartners-Google 20,229,040 1.49% 86,648 0% 483,680 22,535 11,889,489 221,629 7,524,929 130
Yahoo! Slurp 12,266,048 0.9% 193,286 1% 69,248 6,850,188 5,094,644 42,134 16,544 4
CCBot 11,953,709 0.88% 7,637,933 63% 124,595 27,776 57,199 4,010,831 95,253 122
ClaudeBot 11,976,621 0.88% 7,464,999 62% 149,911 60,761 37,097 4,042,820 220,970 63
AdsBot-Google-Mobile 11,347,638 0.83% 25,151 0% 97,577 23,423 7,054,773 4,119,887 26,817 10
Amazonbot 11,099,142 0.82% 6,811,886 61% 30,718 30,587 53,494 4,050,071 122,272 114
meta-externalagent 10,705,041 0.79% 6,437,153 60% 46,323 11,859 13,660 4,046,324 148,048 1,674
Google-Extended 10,494,462 0.77% 6,016,674 57% 154,593 70,529 37,329 4,124,955 90,295 87
Bytespider 9,979,213 0.73% 7,507,605 75% 119,483 81,719 17,622 2,127,130 125,597 57
PetalBot 9,829,127 0.72% 9,525,467 96% 60,085 1,670 15,070 36,937 189,824 74
Applebot-Extended 9,656,442 0.71% 5,458,167 56% 35,095 58,096 5,102 4,018,676 81,258 48
Yandex 9,919,580 0.73% 1,087,492 10% 241,300 6,866,544 604,243 704,076 415,736 189
BLEXBot 9,599,824 0.71% 9,394,649 97% 1,937 2,498 30,784 13,407 156,525 24
AhrefsBot 16,772,136 1.23% 4,252,247 25% 262,186 21,255 4,082,302 73,971 8,080,061 114
Sogou 7,090,726 0.52% 7,002,483 98% 16,732 73 7,713 38,895 24,805 25
IPS-Agent 7,035,201 0.52% 183,659 2% 58 14 6,850,428 225 816 1
Pandalytics 7,014,389 0.52% 162,867 2% 42 396 6,850,012 144 926 2
IonCrawl 6,882,698 0.51% 32,637 0% 15 5 6,849,975 2 64 0
ia_archiver 17,045,617 1.25% 1,736,618 10% 179,015 3,065 4,800,502 124,465 10,201,864 88
anthropic-ai 6,620,540 0.49% 2,382,878 35% 127,956 4,950 11,806 4,035,039 57,860 51
Sosospider 6,724,859 0.49% 1,297,216 19% 82,815 140 5,100,720 1,881 242,085 2
YandexBot 6,437,532 0.47% 1,057,644 16% 141,632 20,916 5,125,315 40,961 50,995 69
ichiro 6,047,169 0.44% 957,333 15% 410 12 5,086,941 318 2,155 0
dotbot 13,490,775 0.99% 5,690,389 42% 59,733 21,175 105,309 24,540 7,589,560 69
sogou spider 6,796,913 0.5% 508,909 7% 98,901 3,872 5,095,304 3,285 1,086,637 5
MJ12bot 10,810,997 0.79% 5,464,086 50% 67,419 1,757 114,560 24,790 5,138,252 133
Seznambot 5,606,470 0.41% 444,788 7% 10,577 1,148 5,100,745 36,830 12,369 13
YoudaoBot 5,544,856 0.41% 382,262 6% 59,759 64 5,097,906 1,725 3,116 24
FacebookBot 5,664,691 0.42% 1,380,623 24% 70,003 68,620 7,570 4,012,126 125,712 37
Nutch 5,544,389 0.41% 5,457,990 98% 12,933 906 4,911 546 67,053 50
cohere-ai 5,438,937 0.4% 1,320,300 24% 59,952 2,991 8,421 3,992,741 54,495 37
YetiBot 5,092,620 0.37% 4,922 0% 272 1 5,086,224 118 1,083 0
rdfbot 5,086,198 0.37% 22 0% 15 0 5,086,135 2 24 0
SemrushBot 6,051,161 0.44% 4,465,006 73% 99,545 3,061 67,734 71,722 1,343,975 118
ChatGPT-User 4,615,076 0.34% 2,255,337 48% 183,524 14,566 64,745 1,978,650 118,168 86
Quora-Bot 4,344,592 0.32% 357,767 8% 473 17 405 3,985,870 60 0
DuckAssistBot 4,221,083 0.31% 198,057 4% 10,209 498 21,552 3,989,027 1,710 30
AhrefsSiteAudit 4,267,988 0.31% 95,239 2% 49,834 126 3,996,336 76,470 49,970 13
AdsBot-Google-Mobile-Apps 4,171,646 0.31% 16,233 0% 29,124 20,141 9,508 4,085,333 11,298 9
PerplexityBot 3,819,842 0.28% 1,470,842 38% 186,763 6,169 32,311 1,933,777 189,912 68
omgili 3,556,107 0.26% 1,366,927 38% 3,432 23,139 2,462 2,106,224 53,888 35
omgilibot 3,298,706 0.24% 1,103,915 33% 9,196 22,697 2,027 2,106,327 54,503 41
Claude-Web 3,234,544 0.24% 1,130,447 34% 99,675 4,648 23,234 1,920,547 55,940 53
YouBot 2,865,335 0.21% 633,767 22% 34,288 1,624 3,284 2,110,008 82,316 48
aiHitBot 2,598,088 0.19% 489,724 18% 457 512 748 2,105,462 1,161 24
GoogleOther 2,607,332 0.19% 315,151 12% 15,033 45,915 5,492 2,141,611 84,105 25
AI2Bot 2,552,747 0.19% 386,103 15% 3,945 698 1,919 2,106,887 53,162 33
psbot 2,508,923 0.18% 2,425,084 96% 5,450 504 4,205 1,800 71,875 5
img2dataset 2,482,810 0.18% 322,419 12% 632 533 1,304 2,106,102 51,785 35
cohere-training-data-crawler 2,411,851 0.18% 302,553 12% 760 777 1,177 2,105,647 911 26
Ai2Bot-Dolma 2,430,809 0.18% 271,008 11% 702 911 760 2,105,618 51,778 32
GoogleOther-Image 2,426,872 0.18% 233,253 9% 2,356 2,314 1,107 2,109,247 78,576 19
GoogleOther-Video 2,424,997 0.18% 231,832 9% 2,222 2,312 1,092 2,109,075 78,445 19
Google-CloudVertexBot 2,344,478 0.17% 450,441 19% 6,888 1,519 1,319 1,883,115 1,173 23
TikTokSpider 2,305,583 0.17% 188,693 8% 450 246 694 2,114,935 541 24
MyCentralAIScraperBot 2,276,642 0.17% 170,552 7% 96 53 489 2,105,231 211 10
ImagesiftBot 2,589,519 0.19% 2,265,345 87% 1,902 1,542 4,604 2,209 313,863 54
rogerbot 2,325,804 0.17% 2,006,885 86% 60,770 20,766 7,511 100,839 128,844 189
TurnitinBot 1,996,979 0.15% 1,989,482 99% 958 638 1,040 572 4,273 16
OAI-SearchBot 2,183,879 0.16% 1,739,719 79% 126,654 2,096 58,295 60,861 196,203 51
MegaIndex.ru 1,808,387 0.13% 1,790,556 99% 533 177 1,221 315 15,579 6
Scrapy 1,842,849 0.14% 1,739,488 94% 36,405 529 1,909 10,569 53,909 40
magpie-crawler 1,765,450 0.13% 1,759,926 99% 530 929 810 280 2,973 2
Googlebot-Image 2,529,767 0.19% 417,246 16% 763,740 33,394 274,327 251,840 789,129 91
Exabot 1,742,268 0.13% 1,669,123 95% 12,151 794 8,876 42,897 8,414 13
Gigabot 1,779,693 0.13% 1,704,751 95% 4,632 488 3,604 685 65,528 5
DataForSeoBot 1,635,432 0.12% 1,548,574 94% 44,898 485 2,414 15,289 23,752 20
WebCopier 1,591,390 0.12% 1,584,581 99% 292 688 990 640 4,193 6
Teleport 1,571,718 0.12% 1,567,134 99% 247 596 737 630 2,369 5
WebZIP 1,565,702 0.12% 1,560,153 99% 265 600 1,456 626 2,596 6
spbot 2,066,794 0.15% 1,559,952 75% 312 137 1,541 269 504,582 1
WebStripper 1,535,804 0.11% 1,531,097 99% 249 590 737 623 2,504 4
TeleportPro 1,522,322 0.11% 1,518,287 99% 225 375 753 493 2,183 6
SiteSnagger 1,505,437 0.11% 1,500,831 99% 253 589 713 623 2,424 4
Offline Explorer 1,492,312 0.11% 1,487,261 99% 272 601 752 627 2,796 3
Sogou web spider 2,478,832 0.18% 1,312,868 52% 152,234 428 17,952 3,116 992,229 5
Xenu 1,485,186 0.11% 1,480,796 99% 256 411 738 513 2,469 3
facebookexternalhit 1,652,476 0.12% 195,365 11% 799,892 26,247 64,530 363,479 202,588 375
Teoma 1,517,036 0.11% 1,327,987 87% 13,739 1,428 10,740 87,430 75,704 8
NerdyBot 1,410,787 0.1% 1,409,646 99% 131 43 91 245 631 0
EmailSiphon 1,411,705 0.1% 1,408,422 99% 187 579 592 148 1,774 3
larbin 1,408,769 0.1% 1,404,836 99% 208 472 626 506 2,117 4
Zeus 1,393,801 0.1% 1,390,998 99% 191 586 544 151 1,327 4
Wget 1,395,012 0.1% 1,387,925 99% 470 700 2,209 749 2,953 6
Twiceler 1,409,931 0.1% 1,382,024 98% 1,818 15 2,128 223 23,723 0
MSIECrawler 1,383,679 0.1% 1,378,509 99% 295 401 756 501 3,213 4
EmailWolf 1,375,596 0.1% 1,372,923 99% 186 578 505 147 1,254 3
ExtractorPro 1,368,982 0.1% 1,365,819 99% 185 578 582 148 1,666 4
CherryPicker 1,366,465 0.1% 1,364,102 99% 164 484 495 141 1,077 2
WebBandit 1,365,084 0.1% 1,362,621 99% 176 477 526 143 1,139 2
CopyRightCheck 1,364,957 0.1% 1,362,554 99% 169 485 495 140 1,112 2
WebSauger 1,360,642 0.1% 1,357,972 99% 183 571 512 147 1,254 3
HTTrack 1,362,031 0.1% 1,354,193 99% 540 433 1,551 872 4,433 9
WebAuto 1,357,272 0.1% 1,354,621 99% 216 568 510 147 1,207 3

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.