Yeti
Avg Crawl Delay (when set) 21s
Mentions
1,273,954
0.09% of hostnames
Disallow All
1,025,527
80% of mentions
Allow
69,976
5% of mentions
Disallow All
+ Allow Some
+ Allow Some
1,314
0% of mentions
Disallow Some
42,720
3% of mentions
Disallow Some
+ Allow Some
+ Allow Some
70,709
5% of mentions
No Directives
63,680
4% of mentions
Average Crawl Delay
21
(when set)