Facebot
Avg Crawl Delay (when set) 10s
Mentions
167,668
0.01% of hostnames
Disallow All
29,277
17% of mentions
Allow
43,680
26% of mentions
Disallow All
+ Allow Some
+ Allow Some
20,698
12% of mentions
Disallow Some
44,284
26% of mentions
Disallow Some
+ Allow Some
+ Allow Some
11,523
6% of mentions
No Directives
18,199
10% of mentions
Average Crawl Delay
10
(when set)