Amazon Web Services
In this AWS re:Invent 2023 session, Jim Roskind and Ankit Chadha explore the phenomenon of congestion collapse in large distributed systems and how to prevent it. They discuss real-world examples, including the 2018 Amazon Prime Day incident, to illustrate how systems can reach 100% CPU utilization while providing zero productive work. The speakers delve into strategies for avoiding congestion collapse, such as implementing proper retry mechanisms, throttling upstream traffic, and utilizing AWS services like CloudWatch, WAF, and SQS. The session also covers testing methodologies, including crash testing and chaos engineering principles, to proactively identify and mitigate potential issues. This comprehensive talk provides valuable insights for developers and system architects looking to build more resilient and efficient distributed systems on AWS.