Most directed user-agents

User-Agent Mentions
(%)
(%)
* 220,821,677 16.93% 30,993,561 14% 52,744,242 531,157 42,017,485 63,457,883 31,009,377 67,972
Googlebot 20,960,376 1.61% 631,733 3% 1,649,017 78,609 13,597,449 1,190,613 3,810,070 2,885
AdsBot-Google 16,373,104 1.25% 90,348 0% 402,887 23,437 11,257,906 4,321,282 277,184 60
bingbot 17,719,210 1.36% 1,135,760 6% 623,354 7,099,368 5,708,778 565,286 2,586,507 157
Baiduspider 15,383,481 1.18% 8,969,788 58% 457,465 25,297 5,302,490 171,265 457,116 60
Mediapartners-Google 20,296,996 1.56% 77,529 0% 475,743 23,009 12,154,283 214,563 7,351,734 135
Yahoo! Slurp 12,486,358 0.96% 169,294 1% 53,481 7,055,744 5,149,138 42,577 16,118 6
AdsBot-Google-Mobile 11,439,504 0.88% 23,529 0% 82,550 23,252 7,118,251 4,165,566 26,347 9
GPTBot 11,381,476 0.87% 6,486,130 56% 144,609 84,952 69,126 4,107,444 489,017 198
CCBot 10,023,038 0.77% 5,758,611 57% 58,601 27,226 41,829 4,042,829 93,835 107
ClaudeBot 10,075,590 0.77% 5,600,018 55% 96,101 59,021 34,509 4,070,500 215,387 54
Yandex 9,976,754 0.76% 1,038,455 10% 207,374 7,070,817 599,979 682,241 377,696 192
PetalBot 9,724,015 0.75% 9,465,383 97% 32,448 1,611 14,164 27,012 183,315 82
BLEXBot 9,667,228 0.74% 9,468,417 97% 1,488 2,440 30,226 2,612 162,010 35
Amazonbot 9,264,960 0.71% 4,961,396 53% 18,986 29,496 51,835 4,083,506 119,632 109
meta-externalagent 9,092,476 0.7% 4,826,226 53% 28,147 9,820 12,223 4,077,811 137,048 1,201
Google-Extended 8,667,589 0.66% 4,218,306 48% 96,539 69,557 35,187 4,160,204 87,722 74
AhrefsBot 16,357,614 1.25% 3,969,368 24% 204,745 20,922 4,005,485 25,507 8,131,486 101
Applebot-Extended 7,905,900 0.61% 3,690,658 46% 12,037 56,930 4,691 4,061,754 79,793 37
Bytespider 7,683,377 0.59% 5,606,454 72% 63,031 84,085 16,275 1,791,632 121,856 44
Sogou 7,261,191 0.56% 7,193,715 99% 11,066 68 7,542 31,161 17,615 24
IPS-Agent 7,236,208 0.55% 179,041 2% 63 13 7,055,988 241 861 1
Pandalytics 7,215,758 0.55% 159,115 2% 45 7 7,055,583 148 859 1
IonCrawl 7,082,203 0.54% 26,568 0% 16 6 7,055,549 2 62 0
ia_archiver 17,022,157 1.3% 1,589,041 9% 173,068 2,614 4,864,520 124,446 10,268,399 69
Sosospider 6,714,086 0.51% 1,276,608 19% 57,381 125 5,150,610 1,836 227,524 2
YandexBot 6,346,445 0.49% 997,854 15% 77,336 20,843 5,177,948 23,788 48,607 69
anthropic-ai 6,331,308 0.49% 2,120,212 33% 77,690 4,082 9,699 4,063,764 55,825 36
ichiro 6,102,066 0.47% 957,496 15% 407 13 5,141,722 336 2,091 1
sogou spider 6,783,460 0.52% 485,724 7% 74,378 3,816 5,148,083 3,158 1,068,296 5
Seznambot 5,597,580 0.43% 387,777 6% 9,531 1,935 5,154,448 32,425 11,450 14
YoudaoBot 5,541,987 0.42% 351,736 6% 37,071 61 5,148,336 1,604 3,161 18
FacebookBot 5,587,725 0.43% 1,291,698 23% 54,168 67,982 6,142 4,045,141 122,560 34
dotbot 13,048,167 1.0% 5,305,058 40% 28,183 20,969 36,254 10,133 7,647,491 79
Nutch 5,437,354 0.42% 5,352,592 98% 11,947 992 4,871 576 66,327 49
MJ12bot 10,279,902 0.79% 5,073,658 49% 34,534 1,756 120,271 13,789 5,035,759 135
YetiBot 5,146,631 0.39% 4,202 0% 258 1 5,140,968 115 1,087 0
cohere-ai 5,196,537 0.4% 1,053,530 20% 43,312 2,657 7,794 4,035,886 53,328 30
rdfbot 5,140,938 0.39% 23 0% 15 0 5,140,876 2 22 0
ChatGPT-User 4,605,147 0.35% 1,978,117 42% 128,036 13,371 48,701 2,327,087 109,764 71
SemrushBot 5,727,090 0.44% 4,222,838 73% 71,332 2,888 66,196 50,558 1,313,145 133
Quora-Bot 4,384,300 0.34% 351,168 8% 381 14 292 4,032,393 52 0
DuckAssistBot 4,249,631 0.33% 183,062 4% 7,749 374 22,481 4,034,850 1,088 27
AdsBot-Google-Mobile-Apps 4,215,694 0.32% 15,160 0% 28,096 19,970 9,161 4,132,208 11,090 9
AhrefsSiteAudit 4,138,058 0.32% 83,900 2% 46,977 124 3,919,178 43,309 44,551 19
PerplexityBot 3,849,538 0.3% 1,236,795 32% 116,919 4,517 15,661 2,290,621 184,975 50
Claude-Web 3,286,759 0.25% 864,193 26% 64,386 3,543 18,219 2,282,307 54,072 39
omgili 2,995,673 0.23% 1,131,591 37% 2,336 23,108 2,373 1,783,167 53,068 30
Google-CloudVertexBot 2,696,790 0.21% 436,595 16% 5,560 1,369 1,042 2,251,054 1,148 22
omgilibot 2,737,058 0.21% 868,163 31% 7,282 23,057 1,873 1,783,081 53,568 34
psbot 2,490,641 0.19% 2,407,659 96% 5,279 507 4,219 1,799 71,172 6
aiHitBot 2,247,719 0.17% 462,857 20% 329 99 684 1,782,545 1,183 22
YouBot 2,296,851 0.18% 403,826 17% 22,818 1,033 2,885 1,785,654 80,594 41
GoogleOther 2,259,874 0.17% 299,089 13% 11,298 45,015 3,940 1,816,528 83,983 21
AI2Bot 2,198,096 0.17% 357,208 16% 2,834 542 1,845 1,783,386 52,251 30
img2dataset 2,132,120 0.16% 295,967 13% 463 478 1,205 1,783,082 50,896 29
cohere-training-data-crawler 2,067,234 0.16% 281,969 13% 396 271 1,074 1,782,677 823 24
ImagesiftBot 2,360,733 0.18% 2,041,619 86% 1,383 1,020 4,424 2,033 310,207 47
Ai2Bot-Dolma 2,083,463 0.16% 248,346 11% 423 437 665 1,782,647 50,916 29
GoogleOther-Image 2,090,547 0.16% 222,019 10% 1,779 2,278 966 1,785,957 77,531 17
GoogleOther-Video 2,089,332 0.16% 221,183 10% 1,666 2,276 955 1,785,825 77,410 17
rogerbot 2,095,881 0.16% 1,813,545 86% 29,555 20,583 6,805 97,287 127,862 244
TurnitinBot 1,962,662 0.15% 1,955,872 99% 859 239 929 574 4,175 14
TikTokSpider 1,951,002 0.15% 167,371 8% 288 217 502 1,782,329 274 21
MyCentralAIScraperBot 1,930,754 0.15% 147,758 7% 41 36 416 1,782,299 194 10
OAI-SearchBot 2,090,258 0.16% 1,699,595 81% 99,444 1,307 43,131 55,099 191,636 46
Scrapy 1,796,747 0.14% 1,703,567 94% 37,127 494 1,754 885 52,886 34
MegaIndex.ru 1,747,719 0.13% 1,729,921 98% 513 163 1,237 332 15,530 23
Gigabot 1,747,469 0.13% 1,673,048 95% 4,341 486 3,629 651 65,310 4
Exabot 1,680,440 0.13% 1,619,242 96% 10,946 779 8,897 32,900 7,665 11
WebCopier 1,573,667 0.12% 1,566,873 99% 299 679 997 625 4,189 5
Teleport 1,555,992 0.12% 1,551,426 99% 252 596 736 614 2,363 5
magpie-crawler 1,553,728 0.12% 1,548,941 99% 444 505 776 266 2,793 3
WebZIP 1,550,838 0.12% 1,545,311 99% 272 599 1,447 611 2,592 6
spbot 2,018,708 0.15% 1,537,563 76% 305 129 1,580 289 478,841 1
WebStripper 1,519,233 0.12% 1,514,545 99% 254 591 736 608 2,495 4
TeleportPro 1,510,772 0.12% 1,506,753 99% 229 378 754 481 2,173 4
SiteSnagger 1,491,021 0.11% 1,486,431 99% 259 591 713 608 2,415 4
Offline Explorer 1,479,832 0.11% 1,474,784 99% 278 597 752 611 2,807 3
Xenu 1,477,840 0.11% 1,473,420 99% 253 415 742 499 2,508 3
Googlebot-Image 2,181,626 0.17% 401,389 18% 508,743 33,811 244,442 237,032 756,111 98
Sogou web spider 2,398,539 0.18% 1,293,186 53% 110,206 431 12,453 2,553 979,705 5
Teoma 1,495,001 0.11% 1,305,261 87% 13,172 2,274 10,753 87,350 76,183 8
EmailSiphon 1,398,393 0.11% 1,395,067 99% 193 580 602 148 1,800 3
larbin 1,394,352 0.11% 1,390,430 99% 213 474 625 495 2,111 4
NerdyBot 1,392,824 0.11% 1,391,673 99% 134 45 97 266 609 0
Wget 1,386,733 0.11% 1,379,768 99% 455 688 2,175 744 2,898 5
Zeus 1,382,121 0.11% 1,379,277 99% 196 588 551 150 1,355 4
MSIECrawler 1,373,751 0.11% 1,368,592 99% 302 403 754 488 3,208 4
EmailWolf 1,364,582 0.1% 1,361,865 99% 192 579 515 147 1,281 3
Twiceler 1,382,262 0.11% 1,356,051 98% 1,731 14 2,137 217 22,112 0
CherryPicker 1,357,827 0.1% 1,355,407 99% 171 488 506 139 1,114 2
ExtractorPro 1,358,169 0.1% 1,354,961 99% 190 579 589 147 1,699 4
CopyRightCheck 1,356,475 0.1% 1,354,016 99% 176 488 506 137 1,150 2
WebBandit 1,355,954 0.1% 1,353,439 99% 183 480 536 140 1,174 2
WebSauger 1,349,960 0.1% 1,347,245 99% 189 573 522 146 1,282 3
WebAuto 1,346,963 0.1% 1,344,260 99% 229 570 520 146 1,235 3
LinkWalker 1,345,432 0.1% 1,341,994 99% 202 530 705 144 1,855 2
NetAnts 1,344,344 0.1% 1,341,613 99% 190 575 539 146 1,278 3
asterias 1,355,457 0.1% 1,335,278 98% 1,349 488 2,078 286 15,976 2

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.