dotbot
Avg Crawl Delay (when set) 10s
Mentions
13,048,594
1.0% of hostnames
Disallow All
5,305,140
40% of mentions
Allow
28,185
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
20,969
0% of mentions
Disallow Some
36,254
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
10,134
0% of mentions
No Directives
7,647,833
58% of mentions
Average Crawl Delay
10
(when set)