Most directed user-agents

User-Agent Mentions
(%)
(%)
* 227,322,456 17.22% 32,458,787 14% 56,352,596 548,212 42,322,477 63,800,792 31,742,422 97,170
Googlebot 21,057,332 1.6% 656,834 3% 1,719,048 78,202 13,513,400 1,206,538 3,880,383 2,927
AdsBot-Google 16,409,635 1.24% 95,532 0% 406,597 23,420 11,292,500 4,313,583 277,950 53
bingbot 17,777,028 1.35% 1,148,491 6% 671,384 7,057,256 5,666,925 574,943 2,657,869 160
Baiduspider 15,378,609 1.16% 8,952,075 58% 491,337 25,303 5,269,715 169,892 470,224 63
Mediapartners-Google 20,294,091 1.54% 82,373 0% 483,906 22,979 12,069,953 216,404 7,418,343 133
Yahoo! Slurp 12,418,621 0.94% 176,590 1% 58,416 7,014,334 5,110,736 42,173 16,367 5
AdsBot-Google-Mobile 11,424,274 0.87% 24,553 0% 83,162 23,226 7,111,098 4,155,670 26,556 9
GPTBot 11,803,297 0.89% 6,893,684 58% 164,016 85,887 70,164 4,100,538 488,800 208
CCBot 10,423,395 0.79% 6,159,046 59% 67,726 27,674 41,091 4,033,461 94,280 117
ClaudeBot 10,484,835 0.79% 6,002,169 57% 109,622 59,631 35,018 4,062,656 215,680 59
Yandex 9,966,109 0.75% 1,050,799 10% 219,570 7,029,574 598,984 687,899 379,089 194
PetalBot 9,736,146 0.74% 9,474,164 97% 33,565 1,627 14,406 27,164 185,149 71
Amazonbot 9,657,156 0.73% 5,357,087 55% 21,900 30,296 52,455 4,074,322 120,983 113
BLEXBot 9,661,548 0.73% 9,467,773 97% 1,653 2,448 30,304 3,679 155,673 18
meta-externalagent 9,492,431 0.72% 5,221,943 55% 35,412 10,857 12,498 4,069,293 141,302 1,126
Google-Extended 9,047,722 0.69% 4,592,643 50% 108,460 70,266 35,627 4,152,043 88,606 77
AhrefsBot 16,489,721 1.25% 4,026,035 24% 217,591 20,878 4,037,518 57,597 8,130,013 89
Bytespider 8,363,234 0.63% 6,012,119 71% 70,951 83,900 16,422 2,054,970 124,823 49
Applebot-Extended 8,278,025 0.63% 4,067,732 49% 15,483 57,474 4,745 4,051,706 80,846 39
Sogou 7,225,182 0.55% 7,154,469 99% 11,673 68 7,461 29,369 22,118 24
IPS-Agent 7,195,380 0.55% 179,640 2% 60 13 7,014,580 236 850 1
Pandalytics 7,175,446 0.54% 159,801 2% 44 327 7,014,171 144 957 2
IonCrawl 7,041,473 0.53% 27,250 0% 17 5 7,014,135 2 64 0
ia_archiver 17,043,078 1.29% 1,635,243 9% 176,572 2,944 4,820,442 124,358 10,283,449 70
Sosospider 6,689,462 0.51% 1,281,703 19% 58,970 127 5,113,674 1,824 233,162 2
anthropic-ai 6,353,708 0.48% 2,138,602 33% 88,752 4,209 10,075 4,055,294 56,739 37
YandexBot 6,337,001 0.48% 1,020,123 16% 81,292 20,806 5,140,557 25,393 48,765 65
ichiro 6,063,558 0.46% 957,613 15% 410 13 5,103,195 334 1,991 2
sogou spider 6,762,147 0.51% 490,116 7% 78,543 3,822 5,109,681 3,182 1,076,798 5
Seznambot 5,570,353 0.42% 398,949 7% 9,703 1,162 5,115,765 32,737 12,024 13
dotbot 13,194,319 1.0% 5,448,074 41% 32,930 20,951 36,193 11,150 7,644,960 61
YoudaoBot 5,513,427 0.42% 359,398 6% 37,922 61 5,111,361 1,633 3,034 18
FacebookBot 5,600,998 0.42% 1,309,472 23% 56,406 68,558 6,244 4,036,405 123,878 35
Nutch 5,477,719 0.41% 5,392,703 98% 12,251 1,003 4,889 569 66,254 50
MJ12bot 10,457,817 0.79% 5,210,090 49% 36,820 1,762 119,974 14,978 5,074,069 124
cohere-ai 5,203,077 0.39% 1,066,643 20% 46,119 2,732 7,915 4,025,525 54,107 36
YetiBot 5,108,234 0.39% 4,303 0% 268 1 5,102,455 118 1,089 0
rdfbot 5,102,427 0.39% 23 0% 15 0 5,102,364 2 23 0
SemrushBot 5,784,790 0.44% 4,266,512 73% 75,986 2,962 65,542 51,484 1,322,189 115
Quora-Bot 4,375,313 0.33% 352,920 8% 403 16 325 4,021,595 54 0
ChatGPT-User 4,364,787 0.33% 1,986,215 45% 144,557 13,757 48,106 2,056,672 115,404 76
DuckAssistBot 4,241,416 0.32% 185,813 4% 8,501 407 21,356 4,024,249 1,061 29
AdsBot-Google-Mobile-Apps 4,205,847 0.32% 16,042 0% 28,158 19,941 9,208 4,121,416 11,073 9
AhrefsSiteAudit 4,210,122 0.32% 86,340 2% 48,928 123 3,953,563 74,109 47,046 13
PerplexityBot 3,616,851 0.27% 1,253,074 34% 135,549 5,261 16,047 2,020,930 185,935 55
omgili 3,270,898 0.25% 1,143,316 34% 2,577 23,541 2,367 2,045,302 53,762 33
Claude-Web 3,032,158 0.23% 869,608 28% 70,375 3,948 21,330 2,011,619 55,237 41
omgilibot 3,011,453 0.23% 879,102 29% 7,637 23,175 1,919 2,045,305 54,278 37
aiHitBot 2,517,150 0.19% 469,805 18% 362 426 691 2,044,661 1,181 24
YouBot 2,575,154 0.2% 413,783 16% 27,510 1,406 2,982 2,048,075 81,354 44
GoogleOther 2,530,744 0.19% 304,849 12% 12,418 45,536 3,980 2,079,092 84,847 22
Google-CloudVertexBot 2,427,197 0.18% 438,346 18% 5,888 1,400 1,150 1,979,212 1,178 23
psbot 2,496,545 0.19% 2,413,642 96% 5,246 507 4,228 1,794 71,122 6
AI2Bot 2,468,592 0.19% 364,427 14% 3,183 557 1,840 2,045,618 52,934 33
img2dataset 2,401,139 0.18% 302,043 12% 516 492 1,233 2,045,242 51,578 35
cohere-training-data-crawler 2,336,841 0.18% 288,822 12% 629 615 1,108 2,044,806 835 26
Ai2Bot-Dolma 2,352,368 0.18% 254,050 10% 466 775 689 2,044,771 51,585 32
GoogleOther-Image 2,358,623 0.18% 226,839 9% 1,922 2,290 990 2,048,368 78,197 17
GoogleOther-Video 2,357,363 0.18% 225,976 9% 1,801 2,289 977 2,048,234 78,069 17
TikTokSpider 2,219,026 0.17% 173,133 7% 340 226 532 2,044,451 321 23
MyCentralAIScraperBot 2,202,043 0.17% 156,871 7% 71 38 434 2,044,421 198 10
rogerbot 2,196,279 0.17% 1,908,848 86% 34,270 20,554 6,821 97,028 128,512 246
ImagesiftBot 2,376,536 0.18% 2,055,770 86% 1,436 1,372 4,490 2,081 311,336 51
TurnitinBot 1,968,706 0.15% 1,961,497 99% 881 561 1,001 576 4,174 16
OAI-SearchBot 2,115,230 0.16% 1,711,863 80% 109,311 1,685 42,118 56,749 193,453 51
Scrapy 1,805,947 0.14% 1,711,988 94% 37,094 508 1,798 879 53,642 38
MegaIndex.ru 1,753,801 0.13% 1,736,058 98% 515 170 1,204 319 15,529 6
Gigabot 1,756,997 0.13% 1,682,754 95% 4,346 483 3,634 647 65,129 4
Exabot 1,688,786 0.13% 1,626,574 96% 11,342 776 8,875 33,105 8,101 13
WebCopier 1,579,111 0.12% 1,572,309 99% 294 679 995 630 4,199 5
Teleport 1,561,209 0.12% 1,556,630 99% 248 598 735 619 2,374 5
magpie-crawler 1,560,139 0.12% 1,554,893 99% 489 842 788 263 2,861 3
WebZIP 1,553,832 0.12% 1,548,289 99% 268 599 1,448 616 2,606 6
spbot 2,045,451 0.15% 1,540,567 75% 302 137 1,562 277 502,605 1
WebStripper 1,524,299 0.12% 1,519,600 99% 250 591 735 613 2,506 4
TeleportPro 1,514,999 0.11% 1,510,968 99% 226 377 752 482 2,190 4
SiteSnagger 1,495,293 0.11% 1,490,690 99% 255 591 711 613 2,429 4
Offline Explorer 1,483,891 0.11% 1,478,831 99% 276 597 751 615 2,818 3
Xenu 1,480,126 0.11% 1,475,711 99% 250 413 737 500 2,512 3
Googlebot-Image 2,212,319 0.17% 405,495 18% 528,134 33,065 246,254 240,091 759,187 93
Teoma 1,506,479 0.11% 1,317,637 87% 13,201 1,493 10,753 87,596 75,791 8
Sogou web spider 2,413,357 0.18% 1,298,644 53% 113,619 429 14,093 2,659 983,908 5
EmailSiphon 1,401,544 0.11% 1,398,219 99% 189 581 601 151 1,800 3
NerdyBot 1,399,822 0.11% 1,398,684 99% 132 45 99 255 607 0
larbin 1,397,478 0.11% 1,393,555 99% 209 473 623 496 2,118 4
Wget 1,389,965 0.11% 1,382,953 99% 452 689 2,190 750 2,926 5
Zeus 1,384,811 0.1% 1,381,971 99% 193 588 552 153 1,350 4
Twiceler 1,395,865 0.11% 1,369,560 98% 1,712 15 2,148 222 22,208 0
MSIECrawler 1,376,308 0.1% 1,371,150 99% 296 403 756 489 3,210 4
EmailWolf 1,366,759 0.1% 1,364,046 99% 188 580 513 150 1,279 3
CherryPicker 1,359,997 0.1% 1,357,586 99% 168 488 503 142 1,108 2
ExtractorPro 1,360,401 0.1% 1,357,193 99% 188 579 589 149 1,699 4
CopyRightCheck 1,358,522 0.1% 1,356,073 99% 173 488 503 140 1,143 2
WebBandit 1,358,300 0.1% 1,355,792 99% 180 480 534 143 1,169 2
WebSauger 1,352,371 0.1% 1,349,662 99% 186 573 520 149 1,278 3
WebAuto 1,349,079 0.1% 1,346,381 99% 226 570 518 149 1,232 3
LinkWalker 1,347,231 0.1% 1,343,487 99% 199 851 701 147 1,844 2
NetAnts 1,346,155 0.1% 1,343,430 99% 187 575 537 149 1,274 3
asterias 1,357,123 0.1% 1,336,895 98% 1,338 487 2,072 292 16,037 2

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.