dotbot
Avg Crawl Delay (when set) 10s
Mentions
12,731,197
1.09% of hostnames
Disallow All
4,687,803
36% of mentions
Allow
8,491
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
20,032
0% of mentions
Disallow Some
39,300
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
9,454
0% of mentions
No Directives
7,966,070
62% of mentions
Average Crawl Delay
10
(when set)