OAI-SearchBot
Avg Crawl Delay (when set) 9s
Mentions
2,431,910
0.17% of hostnames
Disallow All
1,788,995
73% of mentions
Allow
288,243
11% of mentions
Disallow All
+ Allow Some
+ Allow Some
2,956
0% of mentions
Disallow Some
97,983
4% of mentions
Disallow Some
+ Allow Some
+ Allow Some
92,528
3% of mentions
No Directives
161,142
6% of mentions
Average Crawl Delay
9
(when set)