GPTBot
Avg Crawl Delay (when set) 16s
Mentions
15,017,462
1.05% of hostnames
Disallow All
10,364,500
69% of mentions
Allow
464,416
3% of mentions
Disallow All
+ Allow Some
+ Allow Some
90,343
0% of mentions
Disallow Some
143,414
0% of mentions
Disallow Some
+ Allow Some
+ Allow Some
3,498,552
23% of mentions
No Directives
455,931
3% of mentions
Average Crawl Delay
16
(when set)