AWS resources are being used to crawl my website—what do I do?
Last updated: 2019-05-14
AWS resources are being used to crawl my website without my permission. How can I let AWS know?
First, modify your robots.txt file. The robots.txt file lists all restrictions in place for crawlers. When the file is attached to the root domain of a website, it can stop or slow down the crawler:
By modifying your robots.txt file, you can impact the following:
- Which crawlers can crawl your website.
- Which pages these crawlers can crawl.
- The rate at which pages can be crawled.
After modifying your robots.txt file, if you believe that a crawler running on AWS resources isn't abiding by your robots.txt, submit an abuse report.