Facebot
Avg Crawl Delay (when set) 9s
Mentions
171,721
0.01% of hostnames
Disallow All
30,567
17% of mentions
Allow
42,769
24% of mentions
Disallow All
+ Allow Some
+ Allow Some
20,675
12% of mentions
Disallow Some
48,596
28% of mentions
Disallow Some
+ Allow Some
+ Allow Some
11,404
6% of mentions
No Directives
17,705
10% of mentions
Average Crawl Delay
9
(when set)