CCBot
Avg Crawl Delay (when set) 21s
Mentions
8,071,142
0.68% of hostnames
Disallow All
4,761,976
59% of mentions
Allow
25,009
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
25,419
0% of mentions
Disallow Some
36,006
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
3,133,970
38% of mentions
No Directives
88,667
1% of mentions
Average Crawl Delay
21
(when set)