Most directed user-agents

User-Agent Mentions
(%)
(%)
* 95,103,015 10.91% 10,575,045 11% 8,616,795 569,424 25,994,012 29,906,251 19,431,370 10,118
GetRight 8,701,716 1.0% 8,699,712 99% 75 335 545 26 1,014 9
Offline Explorer 7,169,560 0.82% 7,167,028 99% 109 352 548 81 1,436 6
Mozilla/4.0 (compatible; BullsEye; Windows 95) 7,371,121 0.85% 6,505,510 88% 2,358 2,340 4,102 3,356 853,450 5
Scooter/1.0 5,932,185 0.68% 5,926,958 99% 448 240 1,441 796 2,301 1
AhrefsBot 6,713,291 0.77% 2,273,210 33% 112,162 16,464 3,278,801 12,064 1,020,545 45
WebZIP 5,694,229 0.65% 5,690,537 99% 172 620 1,230 91 1,573 6
WebStripper 5,512,688 0.63% 5,510,132 99% 118 381 595 124 1,332 6
dotbot 5,885,580 0.68% 5,309,212 90% 3,167 16,616 24,900 5,697 525,965 23
Nutch 5,377,984 0.62% 5,322,575 98% 5,285 681 3,479 324 45,614 26
SmartDownload 4,338,673 0.5% 4,338,171 99% 23 80 121 12 264 2
Googlebot 5,335,773 0.61% 247,277 4% 790,101 60,048 2,559,636 638,116 1,038,443 2,152
larbin_2.6.2 larbin2.6.2@unspecified.mail 4,182,284 0.48% 4,181,977 99% 0 4 42 0 259 2
Wget 4,035,813 0.46% 4,030,122 99% 349 917 1,941 202 2,275 7
TeleportPro 3,812,426 0.44% 3,810,297 99% 97 273 583 30 1,137 9
adsbot-google 3,847,616 0.44% 37,736 0% 150,022 17,211 3,419,758 149,833 73,031 25
bingbot 4,605,519 0.53% 887,229 19% 232,214 35,106 2,074,714 306,427 1,069,704 125
AhrefsSiteAudit 3,339,964 0.38% 60,173 1% 18,183 80 3,213,962 21,592 25,971 3
Baiduspider 3,489,883 0.4% 1,008,706 28% 93,514 20,315 1,989,879 103,578 273,850 41
Xenu 3,128,914 0.36% 3,126,325 99% 209 536 654 44 1,143 3
Sosospider 3,175,626 0.36% 1,108,453 34% 28,259 112 1,876,014 1,372 161,415 1
ia_archiver 3,400,437 0.39% 852,321 25% 154,811 2,176 1,907,012 92,483 391,613 21
MJ12bot 7,175,972 0.82% 2,810,754 39% 7,563 1,127 101,663 9,228 4,245,588 49
NetAnts 2,898,254 0.33% 2,896,552 99% 86 350 444 67 752 3
Microsoft URL Control - 5.01.4511 2,851,570 0.33% 2,849,860 99% 104 515 362 103 625 1
SemrushBot 3,911,005 0.45% 2,702,159 69% 17,406 2,277 46,883 22,021 1,120,209 50
WebReaper 2,786,804 0.32% 2,784,772 99% 56 209 412 82 1,266 7
ichiro 2,774,679 0.32% 902,760 32% 118 18 1,870,060 144 1,579 0
NPbot 2,759,199 0.32% 2,758,183 99% 39 24 264 18 670 1
RepoMonkey Bait & Tackle/v1.01 2,710,120 0.31% 2,709,271 99% 72 254 198 13 311 1
SuperBot 2,701,074 0.31% 2,700,035 99% 37 124 276 96 504 2
aipbot 2,650,352 0.3% 2,650,010 99% 38 16 98 6 184 0
Mediapartners-Google 8,560,058 0.98% 40,624 0% 286,142 18,177 2,105,261 150,702 5,959,047 105
WebCopier v3.2a 2,545,760 0.29% 2,545,445 99% 15 15 47 0 236 2
WebCopier v2.8 2,516,231 0.29% 2,516,045 99% 0 6 21 0 156 3
ConveraCrawler 2,512,031 0.29% 2,511,827 99% 13 1 55 0 130 5
AltaVista Intranet V2.0 AVS EVAL search@freeit.com 2,502,990 0.29% 2,502,930 99% 2 0 43 0 15 0
YandexBot 2,347,721 0.27% 365,703 15% 34,833 16,318 1,898,353 10,521 21,956 37
MegaIndex.ru 2,342,678 0.27% 2,319,275 99% 109 94 1,136 158 21,906 0
GPTBot 2,619,624 0.3% 2,204,345 84% 9,536 53,109 31,869 10,024 310,697 44
True_Robot 2,238,049 0.26% 2,235,930 99% 135 551 587 108 737 1
MIIxpc 2,227,583 0.26% 2,225,500 99% 139 532 528 106 776 2
WebBandit 2,222,179 0.25% 2,220,098 99% 139 499 563 71 807 2
Exabot 2,220,303 0.25% 2,180,733 98% 4,588 489 6,439 24,382 3,671 1
lwp-trivial 2,212,770 0.25% 2,210,704 99% 136 536 567 101 724 2
TurnitinBot 2,191,336 0.25% 2,187,473 99% 308 106 643 259 2,544 3
sogou spider 3,153,120 0.36% 265,270 8% 19,651 2,890 1,873,539 1,517 990,250 3
Gigabot 2,191,750 0.25% 2,141,981 97% 2,838 338 2,211 415 43,967 0
WebAuto 2,142,352 0.25% 2,140,721 99% 136 349 392 67 685 2
YoudaoBot 2,127,689 0.24% 228,286 10% 22,473 53 1,874,129 1,104 1,631 13
Seznambot 2,120,711 0.24% 203,079 9% 4,881 626 1,878,565 26,231 7,317 12
Szukacz/1.4 2,102,099 0.24% 2,099,104 99% 92 311 412 62 2,117 1
larbin 2,081,582 0.24% 2,079,819 99% 107 328 433 40 852 3
CCBot 2,112,136 0.24% 1,999,193 94% 2,222 21,038 28,145 8,542 52,964 32
CherryPicker 2,044,450 0.23% 2,043,139 99% 80 285 364 57 524 1
BackDoorBot/1.0 2,044,002 0.23% 2,042,682 99% 80 286 382 57 514 1
LinkScan/8.1a Unix 2,043,769 0.23% 2,042,338 99% 82 291 398 60 599 1
libWeb/clsHTTP 2,041,227 0.23% 2,039,937 99% 90 282 349 55 513 1
Harvest/1.5 2,040,799 0.23% 2,039,345 99% 82 279 408 60 624 1
ProPowerBot/2.14 2,039,989 0.23% 2,038,722 99% 81 282 353 54 496 1
Bullseye/1.0 2,036,744 0.23% 2,035,392 99% 80 291 366 58 556 1
HTTrack 2,028,736 0.23% 2,024,391 99% 124 219 1,097 190 2,711 4
Keyword Density/0.9 2,016,528 0.23% 2,015,591 99% 82 254 264 12 324 1
Yahoo! Slurp 1,996,573 0.23% 67,894 3% 13,527 2,048 1,880,297 22,049 10,756 2
psbot 2,021,568 0.23% 1,967,518 97% 3,233 298 2,865 918 46,733 3
Microsoft URL Control 1,972,544 0.23% 1,971,776 99% 73 253 206 13 222 1
toCrawl/UrlDispatcher 1,900,552 0.22% 1,899,436 99% 74 274 274 55 438 1
ClaudeBot 1,959,449 0.22% 1,842,456 94% 2,016 32,109 18,535 1,293 63,021 19
YetiBot 1,872,710 0.21% 2,325 0% 135 1 1,869,432 79 738 0
rdfbot 1,869,561 0.21% 60 0% 37 0 1,869,374 2 88 0
Go!Zilla 1,849,028 0.21% 1,848,505 99% 22 82 116 14 286 3
website extractor 1,837,323 0.21% 1,836,768 99% 24 38 182 46 264 1
eCatch 1,836,962 0.21% 1,836,475 99% 22 89 130 13 231 2
Bytespider 1,886,903 0.22% 1,766,429 93% 6,325 50,626 5,255 6,078 52,171 19
Gaisbot 1,834,263 0.21% 1,833,527 99% 69 244 203 14 205 1
b2w/0.1 1,830,890 0.21% 1,830,152 99% 64 245 208 11 209 1
008 1,829,554 0.21% 1,827,729 99% 67 102 728 61 860 7
LWP 1,819,896 0.21% 1,819,711 99% 15 9 87 0 74 0
Express WebPictures 1,818,900 0.21% 1,818,566 99% 16 11 108 3 195 1
FrontPage 1,816,550 0.21% 1,816,227 99% 15 11 113 1 182 1
Enterprise_Search 1,812,709 0.21% 1,812,035 99% 23 445 73 11 120 2
Microsoft.URL.Control 1,786,601 0.2% 1,784,494 99% 41 67 251 39 1,706 3
BecomeBot 1,768,139 0.2% 1,763,438 99% 373 239 427 18 3,644 0
WebVac 1,753,486 0.2% 1,752,692 99% 14 231 106 10 433 0
ShopWiki 1,714,703 0.2% 1,713,436 99% 386 2 402 43 434 0
webcopy 1,692,908 0.19% 1,691,667 99% 12 28 561 47 593 0
DISCo Pump 3.1 1,683,897 0.19% 1,683,476 99% 8 32 114 44 222 1
WebReaper v9.7 - www.webreaper.net 1,681,533 0.19% 1,681,375 99% 0 6 21 0 128 3
Download Demon 1,678,730 0.19% 1,678,564 99% 1 6 32 2 122 3
WebEmailExtrac 1,678,044 0.19% 1,677,932 99% 0 2 24 0 85 1
WebSauger 1.20b 1,676,299 0.19% 1,676,080 99% 0 6 56 0 154 3
psycheclone 1,674,414 0.19% 1,674,393 99% 0 1 7 0 13 0
Website Quester - www.asona.org 1,673,957 0.19% 1,673,889 99% 0 2 11 0 54 1
Scooter-3.0.FS 1,669,316 0.19% 1,669,266 99% 0 0 0 8 42 0
Mister Pix II 2.02a 1,668,927 0.19% 1,668,815 99% 0 2 8 0 101 1
Linknzbot 1,668,717 0.19% 1,668,703 99% 0 0 0 0 14 0
NetZip-Downloader/1.0.62 (Win32; Dec 7 1998) 1,667,323 0.19% 1,667,245 99% 0 2 7 0 68 1
NetZippy+(http://www.innerprise.net/usp-spider.asp) 1,667,305 0.19% 1,667,226 99% 0 2 7 0 69 1
Scooter-ARS-1.1-ih 1,663,967 0.19% 1,663,963 99% 0 0 0 0 4 0
ChatGPT-User 1,545,351 0.18% 1,454,180 94% 9,603 838 19,181 9,359 52,150 40

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.