dotbot
Avg Crawl Delay (when set) 11s
Mentions
12,090,109
1.02% of hostnames
Disallow All
4,923,265
40% of mentions
Allow
8,747
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
20,314
0% of mentions
Disallow Some
37,128
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
9,562
0% of mentions
No Directives
7,091,042
58% of mentions
Average Crawl Delay
11
(when set)