Category: Analytics

This post was co-authored by James Fogel, Staff Software Engineer on the Cloud Architecture Team at Pinterest Part 2: Spark on EKS network design at scale Introduction In this two-part series, my counterpart, James Fogel (Staff Cloud Architect at Pinterest), and I share Pinterest’s journey designing and implementing their networking topology for running large-scale Spark […]

This post was co-authored by James Fogel, Staff Software Engineer on the Cloud Architecture Team at Pinterest Part 1: Design process for Amazon EKS networking at scale Introduction Pinterest is a platform that helps inspire people to live a life they love. Big data and machine learning (ML) are core to Pinterest’s platform and product, […]

Introduction AWS Fargate is a serverless compute engine for running Amazon Elastic Kubernetes Service (Amazon EKS) and Amazon Elastic Container Service (Amazon ECS) workloads without managing the underlying infrastructure. AWS Fargate makes it easy to provision and scale secure, isolated, and right-sized compute capacity for containerized applications. As a result, teams are increasingly choosing AWS […]

Introduction Apache Spark revolutionized big data processing with its distributed computing capabilities, which enabled efficient data processing at scale. It offers the flexibility to run on traditional Central Processing Unit (CPUs) as well as specialized Graphic Processing Units (GPUs), which provides distinct advantages for various workloads. As the demand for faster and more efficient machine […]

Introduction When you purchase a packaged food item in your local grocery store, you probably check the list of ingredients written to understand what’s inside and make sure you aren’t consuming ingredients inadvertently that you don’t want to or are known to have adverse health effects. Do you think in a similar way when you […]

Introduction We are thrilled to introduce Data on EKS (DoEKS), a new open-source project aimed at streamlining and accelerating the process of building, deploying, and scaling data workloads on Amazon Elastic Kubernetes Service (Amazon EKS). With DoEKS, customers get access to a comprehensive range of resources including Infrastructure as Code (IaC) templates, performance benchmark reports, […]

Introduction Many customers use Amazon Elastic Kubernetes Service (Amazon EKS) to host their mission-critical applications. As a best practice, we ask our customers to spread their applications across multiple distinct availability zones (AZ). Because “everything fails all the time,” Werner Vogel, CTO, Amazon To achieve high availability, customers deploy Amazon EKS worker nodes (Amazon EC2 […]

Introduction Many organizations as part of their cloud journey into Amazon Web Services migrate and modernize their ETL (extract-transform-load) batch processing workloads running on on-premises Hadoop clusters to AWS. They often start their journey with the lift and shift approach, by hosting their Hadoop environment on Amazon Elastic Compute Cloud (Amazon EC2) or migrate to […]

AWS Fargate is a technology that provides on-demand capacity for running pods on EKS clusters. Fargate provides a more hands-off experience, helping you run container applications without needing to manage the EC2 instances underneath. AWS Fargate runs each Kubernetes pod in its own isolated security boundary. This means it has a slightly different operating model […]

Akshay Ram, Prithvi Ramesh, Michael Hausenblas In issue 701 of our containers roadmap we discussed supporting our CNCF Fluent Bit-based log router in the context of EKS on Fargate. In this blog post we provide you context on this new feature and walk you through the usage of it, shipping logs directly to CloudWatch with […]