MyCentralAIScraperBot
Avg Crawl Delay (when set) 12s
Mentions
3,206,808
0.22% of hostnames
Disallow All
124,956
3% of mentions
Allow
201
0% of mentions
Disallow All
+ Allow Some
+ Allow Some
94
0% of mentions
Disallow Some
536
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
3,080,727
96% of mentions
No Directives
283
0% of mentions
Average Crawl Delay
12
(when set)