Most directed user-agents

User-Agent Mentions
(%)
(%)
* 256,393,205 17.74% 42,125,523 16% 70,515,663 435,291 44,833,719 64,169,484 34,003,626 309,899
Googlebot 23,612,583 1.63% 807,384 3% 2,931,365 87,552 14,070,011 1,379,116 4,334,303 2,852
bingbot 19,247,230 1.33% 1,206,207 6% 1,264,883 7,325,985 5,993,248 670,053 2,786,639 215
Baiduspider 16,488,264 1.14% 9,403,178 57% 784,196 27,077 5,561,104 207,242 505,374 93
GPTBot 15,524,330 1.07% 10,817,752 69% 509,469 90,464 146,078 3,503,362 456,854 351
AdsBot-Google 15,347,857 1.06% 115,842 0% 989,063 24,656 10,276,121 3,652,644 289,483 48
CCBot 13,976,200 0.97% 10,122,781 72% 255,789 30,274 104,666 3,400,523 62,031 136
ClaudeBot 14,076,037 0.97% 9,863,522 70% 404,832 64,462 95,984 3,454,447 192,643 147
Bytespider 13,598,507 0.94% 10,028,623 73% 215,390 85,689 57,900 3,120,772 89,972 161
Mediapartners-Google 21,006,990 1.45% 95,685 0% 492,267 23,171 12,602,708 226,979 7,566,053 127
Yahoo! Slurp 12,996,432 0.9% 199,780 1% 81,396 7,281,279 5,376,377 41,856 15,740 4
Amazonbot 13,036,362 0.9% 9,302,930 71% 73,335 34,071 99,345 3,436,714 89,773 194
Google-Extended 12,474,831 0.86% 8,350,031 66% 385,238 72,036 82,439 3,527,072 57,913 102
meta-externalagent 12,516,810 0.87% 8,537,071 68% 346,220 16,326 57,126 3,433,087 124,747 2,233
Applebot-Extended 11,516,255 0.8% 7,812,341 67% 143,717 61,661 45,816 3,404,650 48,017 53
BLEXBot 10,292,735 0.71% 10,064,585 97% 2,269 3,639 41,081 13,891 167,173 97
Yandex 10,483,754 0.73% 1,129,960 10% 270,818 7,299,359 607,585 752,213 423,602 217
AdsBot-Google-Mobile 9,636,486 0.67% 25,906 0% 104,891 24,520 5,969,936 3,481,452 29,765 16
AhrefsBot 16,517,691 1.14% 4,885,498 29% 320,445 22,709 4,157,625 83,864 7,047,376 174
PetalBot 9,046,089 0.63% 8,627,846 95% 92,306 2,426 67,237 46,579 209,543 152
Sogou 7,565,295 0.52% 7,453,999 98% 23,248 689 7,694 36,081 43,550 34
IPS-Agent 7,473,672 0.52% 191,186 2% 56 17 7,281,523 207 682 1
Pandalytics 7,452,657 0.52% 169,995 2% 60 449 7,281,105 264 782 2
IonCrawl 7,343,680 0.51% 62,517 0% 17 6 7,281,071 3 66 0
ia_archiver 18,168,396 1.26% 1,822,761 10% 184,386 3,175 5,078,298 124,329 10,955,346 101
YandexBot 6,992,360 0.48% 1,213,890 17% 233,361 21,648 5,422,622 49,152 51,608 79
Sosospider 7,041,651 0.49% 1,308,896 18% 97,108 196 5,384,107 2,271 249,072 1
anthropic-ai 6,753,095 0.47% 2,929,622 43% 307,356 7,014 55,335 3,427,917 25,723 128
MJ12bot 11,658,079 0.81% 6,159,943 52% 99,808 1,853 121,024 28,408 5,246,830 213
dotbot 12,849,274 0.89% 6,067,293 47% 93,124 21,796 119,182 28,979 6,518,795 105
ichiro 6,330,745 0.44% 958,829 15% 408 14 5,368,821 583 2,090 0
sogou spider 7,149,121 0.49% 564,853 7% 106,217 3,983 5,376,677 5,496 1,091,890 5
Seznambot 5,978,318 0.41% 531,894 8% 12,075 1,199 5,382,393 37,646 13,096 15
YoudaoBot 5,886,628 0.41% 426,420 7% 73,627 95 5,382,004 2,028 2,419 35
Nutch 5,616,888 0.39% 5,531,308 98% 13,343 1,534 5,091 560 65,013 39
cohere-ai 5,426,889 0.38% 1,874,845 34% 107,229 4,564 46,106 3,373,589 20,512 44
YetiBot 5,375,058 0.37% 5,421 0% 312 1 5,368,099 127 1,098 0
rdfbot 5,368,091 0.37% 22 0% 16 0 5,368,027 2 24 0
FacebookBot 5,360,911 0.37% 1,656,392 30% 100,583 71,574 50,395 3,391,894 89,960 113
omgili 5,137,084 0.36% 1,964,772 38% 5,283 23,929 41,413 3,082,220 19,425 42
SemrushBot 6,427,734 0.44% 4,767,431 74% 135,915 3,836 81,276 77,453 1,361,588 235
CloudflareBrowserRenderingCrawler 4,661,370 0.32% 4,660,746 99% 426 13 24 154 7 0
omgilibot 4,651,426 0.32% 1,470,431 31% 12,907 24,073 41,202 3,082,684 20,083 46
AhrefsSiteAudit 4,360,714 0.3% 119,268 2% 50,248 150 4,056,296 80,811 53,928 13
YouBot 4,351,228 0.3% 1,080,705 24% 85,512 3,092 40,189 3,093,058 48,619 53
GoogleOther 3,780,815 0.26% 486,645 12% 29,376 47,333 43,990 3,123,967 49,478 26
Quora-Bot 3,703,703 0.26% 351,617 9% 662 39 510 3,350,761 114 0
DuckAssistBot 3,672,558 0.25% 252,464 6% 24,471 1,294 26,591 3,365,499 2,201 38
aiHitBot 3,585,754 0.25% 501,050 13% 567 1,242 823 3,080,929 1,116 27
ChatGPT-User 3,666,370 0.25% 2,620,911 71% 421,792 16,429 110,005 413,992 83,147 94
cohere-training-data-crawler 3,566,722 0.25% 477,830 13% 2,861 1,594 1,480 3,081,816 1,112 29
Ai2Bot-Dolma 3,578,884 0.25% 440,234 12% 1,161 1,730 37,214 3,081,209 17,300 36
AI2Bot 3,538,439 0.24% 390,308 11% 6,350 1,429 38,507 3,083,015 18,794 36
AdsBot-Google-Mobile-Apps 3,529,060 0.24% 16,125 0% 30,258 20,707 9,940 3,441,517 10,504 9
img2dataset 3,458,729 0.24% 319,835 9% 935 1,186 37,746 3,081,669 17,323 35
GoogleOther-Image 3,353,359 0.23% 178,960 5% 3,862 2,347 38,720 3,085,528 43,923 19
GoogleOther-Video 3,350,974 0.23% 177,367 5% 3,589 2,333 38,560 3,085,328 43,778 19
TikTokSpider 3,251,424 0.23% 155,085 4% 915 934 1,016 3,092,559 887 28
MyCentralAIScraperBot 3,206,808 0.22% 124,956 3% 201 94 536 3,080,727 283 11
PerplexityBot 2,923,308 0.2% 1,833,124 62% 460,582 8,086 86,979 374,728 159,720 89
ImagesiftBot 2,908,098 0.2% 2,573,457 88% 3,925 2,962 41,320 6,281 280,094 59
rogerbot 2,464,692 0.17% 2,113,781 85% 92,421 21,401 7,600 101,165 128,158 166
OAI-SearchBot 2,445,882 0.17% 1,789,014 73% 294,657 2,935 103,604 94,423 161,188 61
Claude-Web 2,236,135 0.15% 1,630,527 72% 169,695 6,571 71,185 334,600 23,436 121
psbot 2,262,994 0.16% 2,181,219 96% 5,394 525 4,264 1,781 69,806 5
Googlebot-Image 2,997,320 0.21% 411,547 13% 1,215,812 34,034 262,465 268,395 804,982 85
magpie-crawler 2,123,368 0.15% 2,114,052 99% 714 990 805 3,911 2,894 2
DataForSeoBot 2,080,392 0.14% 1,952,471 93% 76,932 598 2,745 21,727 25,893 26
TurnitinBot 2,050,137 0.14% 2,041,697 99% 1,178 807 1,094 550 4,792 19
facebookexternalhit 2,182,188 0.15% 265,687 12% 1,224,441 27,260 105,483 389,682 169,483 152
MegaIndex.ru 1,940,416 0.13% 1,921,823 99% 479 787 1,289 303 15,728 7
Exabot 1,860,124 0.13% 1,780,011 95% 13,543 872 8,771 48,812 8,092 23
Scrapy 1,826,231 0.13% 1,719,299 94% 35,158 1,237 38,346 12,614 19,533 44
Gigabot 1,844,347 0.13% 1,773,284 96% 4,720 508 3,578 679 61,573 5
spbot 2,129,697 0.15% 1,622,132 76% 340 134 1,526 363 505,201 1
WebCopier 1,605,901 0.11% 1,599,222 99% 312 703 1,001 635 4,022 6
Teleport 1,588,839 0.11% 1,584,381 99% 263 619 749 624 2,196 7
WebZIP 1,587,389 0.11% 1,581,966 99% 285 622 1,465 622 2,423 6
Sogou web spider 2,565,698 0.18% 1,337,129 52% 191,811 482 19,303 16,545 1,000,423 5
WebStripper 1,552,017 0.11% 1,547,422 99% 269 613 750 618 2,341 4
TeleportPro 1,536,697 0.11% 1,532,774 99% 241 385 781 488 2,022 6
SiteSnagger 1,520,392 0.11% 1,515,899 99% 270 613 727 618 2,261 4
Offline Explorer 1,505,149 0.1% 1,500,224 99% 287 608 770 629 2,628 3
Applebot 1,727,387 0.12% 959,565 55% 245,383 4,398 118,022 172,309 227,546 164
Xenu 1,496,743 0.1% 1,491,791 99% 345 1,015 758 506 2,325 3
Teoma 1,567,758 0.11% 1,366,583 87% 14,457 1,364 10,510 86,559 88,277 8
Twiceler 1,454,551 0.1% 1,424,318 97% 1,863 17 2,128 233 25,992 0
EmailSiphon 1,426,625 0.1% 1,423,446 99% 192 601 597 151 1,635 3
larbin 1,423,130 0.1% 1,419,333 99% 224 481 638 500 1,950 4
NerdyBot 1,415,996 0.1% 1,414,885 99% 134 41 82 223 631 0
Zeus 1,408,248 0.1% 1,405,528 99% 198 608 553 152 1,206 3
Wget 1,404,467 0.1% 1,397,362 99% 520 715 2,223 740 2,901 6
MSIECrawler 1,392,925 0.1% 1,387,878 99% 313 408 756 496 3,070 4
EmailWolf 1,389,738 0.1% 1,387,152 99% 193 599 507 148 1,136 3
HTTrack 1,387,510 0.1% 1,379,797 99% 585 425 1,578 854 4,263 8
ExtractorPro 1,382,540 0.1% 1,379,487 99% 193 598 582 149 1,527 4
CherryPicker 1,378,448 0.1% 1,376,162 99% 167 504 504 142 967 2
WebBandit 1,377,167 0.1% 1,374,787 99% 178 496 534 144 1,026 2
CopyRightCheck 1,376,895 0.1% 1,374,567 99% 173 505 502 141 1,005 2
WebSauger 1,373,742 0.1% 1,371,155 99% 192 593 517 148 1,134 3

Download

The full dataset is available under a Creative Commons Attribution 4.0 International license.