AWS resources are being used to crawl my website—what do I do?

Last updated: 2019-05-14

AWS resources are being used to crawl my website without my permission. How can I let AWS know?

Resolution

First, modify your robots.txt file. The robots.txt file lists all restrictions in place for crawlers. When the file is attached to the root domain of a website, it can stop or slow down the crawler:

By modifying your robots.txt file, you can impact the following:

  • Which crawlers can crawl your website.
  • Which pages these crawlers can crawl.
  • The rate at which pages can be crawled.

After modifying your robots.txt file, if you believe that a crawler running on AWS resources isn't abiding by your robots.txt, submit an abuse report.


Did this article help you?

Anything we could improve?


Need more help?