User-agent: AhrefsBot Crawl-Delay: 30 User-agent: * Disallow: /ajax/