Applebot
Avg Crawl Delay (when set) 43s
Mentions
1,238,831
0.1% of hostnames
Disallow All
759,321
61% of mentions
Allow
82,628
6% of mentions
Disallow All
+ Allow Some
+ Allow Some
3,279
0% of mentions
Disallow Some
68,151
5% of mentions
Disallow Some
+ Allow Some
+ Allow Some
93,836
7% of mentions
No Directives
231,554
18% of mentions
Average Crawl Delay
43
(when set)