OAI-SearchBot
Avg Crawl Delay (when set) 9s
Mentions
2,391,044
0.17% of hostnames
Disallow All
1,866,352
78% of mentions
Allow
201,347
8% of mentions
Disallow All
+ Allow Some
+ Allow Some
2,450
0% of mentions
Disallow Some
58,023
2% of mentions
Disallow Some
+ Allow Some
+ Allow Some
68,392
2% of mentions
No Directives
194,428
8% of mentions
Average Crawl Delay
9
(when set)