Facebot
Avg Crawl Delay (when set) 9s
Mentions
152,383
0.01% of hostnames
Disallow All
23,345
15% of mentions
Allow
38,936
25% of mentions
Disallow All
+ Allow Some
+ Allow Some
20,284
13% of mentions
Disallow Some
41,560
27% of mentions
Disallow Some
+ Allow Some
+ Allow Some
11,520
7% of mentions
No Directives
16,733
10% of mentions
Average Crawl Delay
9
(when set)