rogerbot
Avg Crawl Delay (when set) 194s
Mentions
2,464,692
0.17% of hostnames
Disallow All
2,113,781
85% of mentions
Allow
92,421
3% of mentions
Disallow All
+ Allow Some
+ Allow Some
21,401
0% of mentions
Disallow Some
7,600
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
101,165
4% of mentions
No Directives
128,158
5% of mentions
Average Crawl Delay
194
(when set)