rogerbot
Avg Crawl Delay (when set) 194s
Mentions
2,458,351
0.17% of hostnames
Disallow All
2,108,077
85% of mentions
Allow
91,713
3% of mentions
Disallow All
+ Allow Some
+ Allow Some
21,462
0% of mentions
Disallow Some
7,579
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
101,204
4% of mentions
No Directives
128,150
5% of mentions
Average Crawl Delay
194
(when set)