dotbot
Avg Crawl Delay (when set) 10s
Mentions
12,634,982
1.1% of hostnames
Disallow All
4,652,125
36% of mentions
Allow
8,297
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
19,978
0% of mentions
Disallow Some
39,012
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
8,984
0% of mentions
No Directives
7,906,537
62% of mentions
Average Crawl Delay
10
(when set)