Most directed user-agents

User-Agent Mentions
(%)
(%)
* 248,644,077 17.64% 39,114,637 15% 67,509,635 432,672 43,804,576 63,963,487 33,517,000 302,070
Googlebot 22,919,587 1.63% 739,314 3% 2,592,887 83,802 13,937,521 1,328,477 4,234,593 2,993
bingbot 18,933,289 1.34% 1,190,317 6% 1,126,397 7,278,474 5,916,588 624,636 2,796,692 185
Baiduspider 16,259,746 1.15% 9,301,472 57% 720,881 26,146 5,513,282 196,105 501,789 71
AdsBot-Google 15,237,334 1.08% 113,472 0% 808,677 24,010 10,290,705 3,712,911 287,507 52
GPTBot 14,449,726 1.02% 9,815,475 67% 400,710 89,060 135,408 3,544,947 463,859 267
Mediapartners-Google 20,888,069 1.48% 89,144 0% 488,880 22,878 12,510,770 227,078 7,549,194 125
CCBot 12,966,333 0.92% 9,113,303 70% 201,535 28,958 96,230 3,453,120 73,063 124
Yahoo! Slurp 12,905,554 0.92% 201,475 1% 75,717 7,233,880 5,335,667 42,229 16,582 4
ClaudeBot 13,036,641 0.92% 8,886,336 68% 303,652 62,798 84,311 3,496,600 202,875 69
Bytespider 12,470,907 0.88% 9,018,590 72% 176,250 84,040 50,195 3,038,310 103,465 57
Amazonbot 12,073,681 0.86% 8,315,390 68% 47,863 32,519 84,833 3,493,379 99,585 112
meta-externalagent 11,554,568 0.82% 7,698,694 66% 175,640 13,997 46,620 3,490,020 127,634 1,963
Google-Extended 11,471,884 0.81% 7,400,405 64% 284,457 70,661 71,863 3,575,863 68,547 88
Applebot-Extended 10,568,063 0.75% 6,868,910 64% 83,878 59,880 36,067 3,460,258 59,026 44
Yandex 10,381,168 0.74% 1,119,992 10% 262,311 7,251,328 601,920 722,706 422,716 195
BLEXBot 10,104,382 0.72% 9,883,090 97% 2,211 2,590 41,756 13,414 161,296 25
AdsBot-Google-Mobile 9,823,799 0.7% 25,566 0% 104,546 23,800 6,094,391 3,545,283 30,197 16
AhrefsBot 16,298,132 1.16% 4,689,825 28% 297,852 21,729 4,046,069 79,828 7,162,717 112
PetalBot 9,032,881 0.64% 8,663,885 95% 84,379 1,818 60,339 44,187 178,193 80
Sogou 7,503,735 0.53% 7,393,219 98% 22,506 86 7,805 37,965 42,131 23
IPS-Agent 7,418,258 0.53% 183,104 2% 48 15 7,234,134 211 745 1
Pandalytics 7,397,084 0.52% 161,875 2% 41 464 7,233,706 145 851 2
IonCrawl 7,286,079 0.52% 52,323 0% 13 6 7,233,670 3 64 0
ia_archiver 17,921,304 1.27% 1,774,460 9% 183,491 3,194 5,038,062 124,245 10,797,750 102
YandexBot 6,827,106 0.48% 1,114,037 16% 213,895 21,399 5,381,033 45,554 51,110 78
Sosospider 6,986,472 0.5% 1,300,334 18% 95,407 150 5,341,989 2,102 246,488 2
anthropic-ai 6,537,718 0.46% 2,717,936 41% 250,682 5,815 45,180 3,481,943 36,112 50
ichiro 6,289,341 0.45% 958,253 15% 398 12 5,328,106 514 2,058 0
MJ12bot 11,305,384 0.8% 5,910,668 52% 91,465 1,863 119,460 25,662 5,156,124 142
dotbot 12,684,924 0.9% 5,792,169 45% 84,466 21,568 119,505 25,719 6,641,396 101
sogou spider 7,077,728 0.5% 537,892 7% 105,469 3,997 5,336,047 4,746 1,089,572 5
Seznambot 5,879,659 0.42% 474,551 8% 11,467 1,201 5,341,844 37,367 13,214 15
YoudaoBot 5,823,107 0.41% 407,030 6% 71,948 95 5,339,569 1,958 2,472 35
Nutch 5,498,004 0.39% 5,413,151 98% 13,199 913 4,997 543 65,158 43
YetiBot 5,334,109 0.38% 5,253 0% 296 1 5,327,366 113 1,080 0
rdfbot 5,327,355 0.38% 22 0% 15 0 5,327,292 2 24 0
FacebookBot 5,346,396 0.38% 1,597,842 29% 83,610 69,816 40,366 3,452,764 101,960 38
cohere-ai 5,260,510 0.37% 1,675,623 31% 78,193 3,348 39,234 3,432,238 31,840 34
SemrushBot 6,230,638 0.44% 4,588,504 73% 124,379 3,222 82,011 74,715 1,357,683 124
omgili 4,849,612 0.34% 1,750,502 36% 4,318 23,257 32,775 3,007,710 31,016 34
omgilibot 4,393,854 0.31% 1,288,311 29% 10,736 22,773 32,402 3,007,943 31,649 40
AhrefsSiteAudit 4,230,670 0.3% 102,327 2% 49,166 137 3,945,483 78,853 54,691 13
YouBot 4,065,783 0.29% 887,957 21% 65,134 1,915 33,702 3,017,096 59,932 47
Quora-Bot 3,768,131 0.27% 349,184 9% 612 39 468 3,417,755 73 0
DuckAssistBot 3,712,944 0.26% 249,537 6% 14,362 680 21,143 3,425,239 1,953 30
GoogleOther 3,692,693 0.26% 477,981 12% 20,698 46,956 37,516 3,048,440 61,079 23
CloudflareBrowserRenderingCrawler 3,592,540 0.25% 3,592,443 99% 77 8 6 6 0 0
AdsBot-Google-Mobile-Apps 3,595,313 0.26% 15,941 0% 28,931 20,483 9,750 3,508,994 11,205 9
cohere-training-data-crawler 3,486,512 0.25% 471,370 13% 1,133 1,013 1,333 3,010,643 995 25
Ai2Bot-Dolma 3,504,849 0.25% 432,314 12% 960 1,147 31,006 3,010,443 28,948 31
aiHitBot 3,472,799 0.25% 459,396 13% 545 679 774 3,010,253 1,129 23
AI2Bot 3,480,015 0.25% 399,437 11% 5,265 874 32,194 3,011,866 30,347 32
ChatGPT-User 3,517,083 0.25% 2,447,109 69% 335,355 15,159 96,285 529,018 94,070 87
img2dataset 3,404,689 0.24% 331,644 9% 874 634 31,587 3,010,942 28,976 32
GoogleOther-Image 3,299,652 0.23% 191,956 5% 2,909 2,380 32,570 3,014,130 55,689 18
GoogleOther-Video 3,297,376 0.23% 190,391 5% 2,689 2,366 32,412 3,013,955 55,545 18
TikTokSpider 3,183,041 0.23% 159,029 4% 778 350 922 3,021,195 744 23
MyCentralAIScraperBot 3,146,443 0.22% 135,355 4% 191 111 499 3,010,037 241 9
PerplexityBot 2,726,488 0.19% 1,643,217 60% 348,295 6,989 72,010 487,091 168,813 73
ImagesiftBot 2,737,076 0.19% 2,400,247 87% 2,711 1,777 35,114 5,858 291,316 53
psbot 2,305,707 0.16% 2,223,396 96% 5,437 519 4,240 1,786 70,324 5
rogerbot 2,341,629 0.17% 1,998,768 85% 84,636 21,154 7,435 101,154 128,317 165
OAI-SearchBot 2,343,549 0.17% 1,780,042 75% 226,327 2,635 90,136 72,711 171,646 52
Claude-Web 2,131,762 0.15% 1,445,163 67% 127,179 5,424 56,808 462,836 34,300 52
TurnitinBot 2,032,792 0.14% 2,024,527 99% 1,019 802 1,079 573 4,776 16
Googlebot-Image 2,778,143 0.2% 412,787 14% 1,002,044 33,766 262,806 264,329 802,329 82
magpie-crawler 1,933,875 0.14% 1,924,669 99% 617 1,011 822 3,745 3,009 2
MegaIndex.ru 1,859,591 0.13% 1,841,547 99% 553 181 1,294 301 15,709 6
DataForSeoBot 1,853,777 0.13% 1,737,210 93% 69,300 575 2,512 19,953 24,208 19
Exabot 1,811,631 0.13% 1,732,031 95% 13,108 827 8,899 48,033 8,714 19
Scrapy 1,825,244 0.13% 1,713,462 93% 35,709 666 32,163 11,988 31,216 40
Gigabot 1,797,201 0.13% 1,724,730 95% 4,726 499 3,595 675 62,971 5
facebookexternalhit 1,887,318 0.13% 205,782 10% 1,000,669 26,695 96,385 378,297 179,380 110
WebCopier 1,595,361 0.11% 1,588,589 99% 302 695 1,005 636 4,128 6
WebZIP 1,579,370 0.11% 1,573,853 99% 272 617 1,471 622 2,529 6
Teleport 1,576,686 0.11% 1,572,144 99% 253 612 748 625 2,298 6
spbot 2,077,879 0.15% 1,570,574 75% 320 136 1,545 249 505,054 1
WebStripper 1,542,074 0.11% 1,537,396 99% 257 608 749 618 2,442 4
Sogou web spider 2,521,906 0.18% 1,321,824 52% 175,697 472 18,966 10,615 994,326 6
TeleportPro 1,525,638 0.11% 1,521,632 99% 232 380 779 489 2,120 6
SiteSnagger 1,511,427 0.11% 1,506,855 99% 259 608 724 618 2,359 4
Offline Explorer 1,496,463 0.11% 1,491,454 99% 279 603 766 623 2,735 3
Xenu 1,488,698 0.11% 1,484,242 99% 354 413 757 507 2,422 3
Teoma 1,527,793 0.11% 1,325,192 86% 14,334 1,395 10,557 86,660 89,647 8
Applebot 1,651,738 0.12% 935,946 56% 208,658 4,408 105,318 162,004 235,310 94
NerdyBot 1,416,215 0.1% 1,415,110 99% 128 41 82 228 626 0
EmailSiphon 1,416,907 0.1% 1,413,680 99% 186 597 593 148 1,700 3
larbin 1,413,112 0.1% 1,409,226 99% 212 477 635 502 2,056 4
Wget 1,403,606 0.1% 1,396,458 99% 508 719 2,224 746 2,945 6
Zeus 1,399,528 0.1% 1,396,776 99% 191 603 551 150 1,254 3
Twiceler 1,408,753 0.1% 1,380,253 97% 1,885 16 2,127 225 24,247 0
MSIECrawler 1,385,638 0.1% 1,380,515 99% 302 403 756 498 3,160 4
EmailWolf 1,380,874 0.1% 1,378,254 99% 185 595 504 146 1,187 3
ExtractorPro 1,373,999 0.1% 1,370,898 99% 185 595 579 147 1,591 4
CherryPicker 1,371,262 0.1% 1,368,946 99% 163 498 500 140 1,013 2
CopyRightCheck 1,369,694 0.1% 1,367,338 99% 168 499 499 139 1,049 2
WebBandit 1,368,789 0.1% 1,366,373 99% 175 491 531 142 1,075 2
HTTrack 1,371,075 0.1% 1,363,271 99% 577 432 1,554 854 4,378 9
WebSauger 1,365,285 0.1% 1,362,668 99% 184 588 514 146 1,182 3

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.