Applebot
Avg Crawl Delay (when set) 40s
Mentions
1,190,486
0.1% of hostnames
Disallow All
739,106
62% of mentions
Allow
63,667
5% of mentions
Disallow All
+ Allow Some
+ Allow Some
2,455
0% of mentions
Disallow Some
67,099
5% of mentions
Disallow Some
+ Allow Some
+ Allow Some
88,677
7% of mentions
No Directives
229,418
19% of mentions
Average Crawl Delay
40
(when set)