Most directed user-agents

User-Agent Mentions
(%)
(%)
* 248,626,811 17.64% 39,110,679 15% 67,502,646 432,699 43,798,801 63,965,229 33,514,720 302,037
Googlebot 22,918,409 1.63% 739,258 3% 2,592,440 83,849 13,937,294 1,328,165 4,234,411 2,992
bingbot 18,932,933 1.34% 1,190,305 6% 1,126,377 7,278,284 5,916,547 624,624 2,796,611 185
Baiduspider 16,259,347 1.15% 9,301,180 57% 720,777 26,189 5,513,254 196,097 501,779 71
AdsBot-Google 15,238,204 1.08% 113,463 0% 808,320 24,052 10,291,474 3,713,338 287,505 52
GPTBot 14,447,915 1.02% 9,813,151 67% 400,610 89,053 135,093 3,545,346 464,397 265
Mediapartners-Google 20,887,703 1.48% 89,139 0% 488,854 22,923 12,510,412 227,044 7,549,206 125
CCBot 12,964,960 0.92% 9,111,018 70% 201,792 28,957 95,924 3,453,543 73,603 123
Yahoo! Slurp 12,905,285 0.92% 201,467 1% 75,705 7,233,649 5,335,646 42,232 16,582 4
ClaudeBot 13,034,954 0.92% 8,884,113 68% 303,573 62,795 83,981 3,496,998 203,426 68
Bytespider 12,468,739 0.88% 9,016,272 72% 176,504 84,039 49,868 3,037,978 104,021 57
Amazonbot 12,071,979 0.86% 8,313,076 68% 47,824 32,518 84,505 3,493,794 100,150 112
meta-externalagent 11,552,893 0.82% 7,696,781 66% 175,227 13,987 46,289 3,490,444 128,182 1,983
Google-Extended 11,470,212 0.81% 7,398,183 64% 284,375 70,656 71,537 3,576,276 69,097 88
Applebot-Extended 10,566,464 0.75% 6,866,716 64% 83,824 59,878 35,740 3,460,676 59,586 44
Yandex 10,380,615 0.74% 1,119,977 10% 262,291 7,251,108 601,920 722,396 422,728 195
BLEXBot 10,103,992 0.72% 9,882,729 97% 2,207 2,589 41,751 13,415 161,276 25
AdsBot-Google-Mobile 9,825,038 0.7% 25,563 0% 104,567 23,842 6,095,134 3,545,715 30,201 16
AhrefsBot 16,298,832 1.16% 4,689,516 28% 298,094 21,770 4,046,019 79,833 7,163,488 112
PetalBot 9,034,035 0.64% 8,664,515 95% 84,680 1,818 60,019 44,187 178,736 80
Sogou 7,503,485 0.53% 7,392,985 98% 22,497 85 7,805 37,974 42,116 23
IPS-Agent 7,418,044 0.53% 183,122 2% 48 14 7,233,903 211 745 1
Pandalytics 7,396,876 0.52% 161,899 2% 41 463 7,233,475 145 851 2
IonCrawl 7,285,829 0.52% 52,304 0% 13 6 7,233,439 3 64 0
ia_archiver 17,920,963 1.27% 1,774,508 9% 183,504 3,194 5,038,058 124,247 10,797,350 102
YandexBot 6,827,265 0.48% 1,113,885 16% 214,189 21,443 5,381,012 45,558 51,100 78
Sosospider 6,986,471 0.5% 1,300,367 18% 95,411 149 5,341,967 2,102 246,473 2
anthropic-ai 6,537,902 0.46% 2,717,538 41% 250,629 5,812 44,859 3,482,354 36,661 49
ichiro 6,289,321 0.45% 958,255 15% 398 12 5,328,084 514 2,058 0
MJ12bot 11,305,639 0.8% 5,910,460 52% 91,761 1,862 119,468 25,666 5,156,280 142
dotbot 12,685,835 0.9% 5,792,041 45% 84,763 21,610 119,507 25,719 6,642,094 101
sogou spider 7,077,644 0.5% 537,848 7% 105,466 3,996 5,336,026 4,745 1,089,558 5
Seznambot 5,879,575 0.42% 474,488 8% 11,466 1,202 5,341,824 37,368 13,212 15
YoudaoBot 5,823,084 0.41% 407,029 6% 71,952 94 5,339,545 1,958 2,471 35
Nutch 5,498,015 0.39% 5,413,150 98% 13,200 912 4,996 543 65,171 43
YetiBot 5,334,089 0.38% 5,255 0% 296 1 5,327,344 113 1,080 0
rdfbot 5,327,333 0.38% 22 0% 15 0 5,327,270 2 24 0
FacebookBot 5,346,930 0.38% 1,597,729 29% 83,602 69,815 40,047 3,453,187 102,512 38
cohere-ai 5,260,737 0.37% 1,675,206 31% 78,181 3,349 38,911 3,432,657 32,399 34
SemrushBot 6,230,667 0.44% 4,588,281 73% 124,668 3,222 82,002 74,722 1,357,648 124
omgili 4,849,157 0.34% 1,750,121 36% 4,323 23,258 32,452 3,007,393 31,576 34
omgilibot 4,393,472 0.31% 1,288,007 29% 10,734 22,774 32,079 3,007,629 32,209 40
AhrefsSiteAudit 4,230,648 0.3% 102,328 2% 49,170 137 3,945,446 78,859 54,695 13
YouBot 4,065,383 0.29% 887,648 21% 65,131 1,916 33,378 3,016,770 60,493 47
Quora-Bot 3,768,562 0.27% 349,184 9% 612 39 467 3,418,187 73 0
DuckAssistBot 3,713,368 0.26% 249,530 6% 14,358 680 21,149 3,425,668 1,953 30
GoogleOther 3,692,559 0.26% 477,942 12% 20,686 46,956 37,191 3,048,122 61,639 23
CloudflareBrowserRenderingCrawler 3,589,639 0.25% 3,589,542 99% 77 8 6 6 0 0
AdsBot-Google-Mobile-Apps 3,595,792 0.26% 15,941 0% 28,935 20,525 9,747 3,509,430 11,205 9
cohere-training-data-crawler 3,486,154 0.25% 471,329 13% 1,134 1,013 1,334 3,010,325 994 25
Ai2Bot-Dolma 3,504,723 0.25% 432,264 12% 963 1,147 30,683 3,010,126 29,509 31
aiHitBot 3,472,549 0.25% 459,459 13% 549 678 775 3,009,936 1,129 23
AI2Bot 3,479,960 0.25% 399,459 11% 5,268 874 31,871 3,011,549 30,907 32
ChatGPT-User 3,517,676 0.25% 2,446,785 69% 335,299 15,154 95,989 529,734 94,629 86
img2dataset 3,404,624 0.24% 331,656 9% 876 634 31,264 3,010,625 29,537 32
GoogleOther-Image 3,299,610 0.23% 191,996 5% 2,908 2,380 32,245 3,013,813 56,250 18
GoogleOther-Video 3,297,333 0.23% 190,431 5% 2,687 2,366 32,087 3,013,638 56,106 18
TikTokSpider 3,182,753 0.23% 159,057 4% 780 350 923 3,020,876 744 23
MyCentralAIScraperBot 3,146,155 0.22% 135,383 4% 191 111 500 3,009,720 241 9
PerplexityBot 2,727,068 0.19% 1,642,952 60% 348,175 6,986 71,701 487,809 169,374 71
ImagesiftBot 2,736,989 0.19% 2,399,916 87% 2,713 1,778 34,792 5,858 291,879 53
psbot 2,305,985 0.16% 2,223,672 96% 5,437 518 4,241 1,787 70,325 5
rogerbot 2,342,002 0.17% 1,998,804 85% 84,932 21,195 7,433 101,159 128,314 165
OAI-SearchBot 2,343,771 0.17% 1,780,054 75% 226,291 2,632 89,831 72,702 172,210 51
Claude-Web 2,132,376 0.15% 1,444,837 67% 127,151 5,421 56,488 463,578 34,850 51
TurnitinBot 2,032,790 0.14% 2,024,525 99% 1,019 801 1,081 573 4,775 16
Googlebot-Image 2,777,665 0.2% 412,818 14% 1,001,614 33,809 262,693 264,336 802,313 82
magpie-crawler 1,933,559 0.14% 1,924,358 99% 615 1,011 822 3,744 3,007 2
MegaIndex.ru 1,859,569 0.13% 1,841,529 99% 551 180 1,293 301 15,709 6
DataForSeoBot 1,853,683 0.13% 1,736,861 93% 69,594 574 2,510 19,950 24,175 19
Exabot 1,811,584 0.13% 1,731,992 95% 13,103 826 8,900 48,030 8,714 19
Scrapy 1,825,535 0.13% 1,713,514 93% 35,716 665 31,840 11,986 31,775 39
Gigabot 1,797,254 0.13% 1,724,769 95% 4,727 498 3,596 675 62,984 5
facebookexternalhit 1,886,991 0.13% 205,785 10% 1,000,084 26,736 96,052 378,291 179,933 110
WebCopier 1,595,375 0.11% 1,588,600 99% 302 694 1,006 636 4,131 6
WebZIP 1,579,384 0.11% 1,573,865 99% 272 616 1,471 622 2,532 6
Teleport 1,576,699 0.11% 1,572,154 99% 253 611 749 625 2,301 6
spbot 2,077,841 0.15% 1,570,537 75% 320 135 1,545 249 505,054 1
WebStripper 1,542,085 0.11% 1,537,404 99% 257 607 750 618 2,445 4
Sogou web spider 2,521,850 0.18% 1,321,828 52% 175,679 471 18,969 10,604 994,293 6
TeleportPro 1,525,655 0.11% 1,521,649 99% 232 379 780 489 2,120 6
SiteSnagger 1,511,431 0.11% 1,506,856 99% 259 607 725 618 2,362 4
Offline Explorer 1,496,467 0.11% 1,491,458 99% 279 602 767 623 2,735 3
Xenu 1,488,704 0.11% 1,484,248 99% 354 412 758 507 2,422 3
Teoma 1,527,822 0.11% 1,325,222 86% 14,331 1,395 10,555 86,661 89,650 8
Applebot 1,652,161 0.12% 935,874 56% 208,937 4,406 104,995 161,996 235,860 93
NerdyBot 1,416,227 0.1% 1,415,122 99% 128 41 82 228 626 0
EmailSiphon 1,416,921 0.1% 1,413,691 99% 186 596 594 148 1,703 3
larbin 1,413,137 0.1% 1,409,248 99% 212 476 636 502 2,059 4
Wget 1,403,589 0.1% 1,396,440 99% 508 719 2,223 746 2,948 5
Zeus 1,399,537 0.1% 1,396,782 99% 191 602 552 150 1,257 3
Twiceler 1,408,768 0.1% 1,380,274 97% 1,885 16 2,127 225 24,241 0
MSIECrawler 1,385,659 0.1% 1,380,537 99% 302 402 757 498 3,159 4
EmailWolf 1,380,886 0.1% 1,378,263 99% 185 594 505 146 1,190 3
ExtractorPro 1,374,012 0.1% 1,370,908 99% 185 594 580 147 1,594 4
CherryPicker 1,371,268 0.1% 1,368,952 99% 163 497 501 140 1,013 2
CopyRightCheck 1,369,701 0.1% 1,367,345 99% 168 498 500 139 1,049 2
WebBandit 1,368,805 0.1% 1,366,389 99% 175 490 532 142 1,075 2
HTTrack 1,371,087 0.1% 1,363,282 99% 575 431 1,555 854 4,381 9
WebSauger 1,365,302 0.1% 1,362,682 99% 184 587 515 146 1,185 3

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.