CCBot
Avg Crawl Delay (when set) 43s
Mentions
12,431,239
0.89% of hostnames
Disallow All
8,242,855
66% of mentions
Allow
155,943
1% of mentions
Disallow All
+ Allow Some
+ Allow Some
28,149
0% of mentions
Disallow Some
60,332
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
3,848,199
30% of mentions
No Directives
95,640
0% of mentions
Average Crawl Delay
43
(when set)