Why Amazon Redshift Integration for Apache Spark?
Amazon Redshift Integration for Apache Spark simplifies and accelerates Apache Spark applications accessing Amazon Redshift data from AWS analytics services such as Amazon EMR, AWS Glue, and Amazon SageMaker. Using Amazon EMR, AWS Glue, and SageMaker, you can quickly build Apache Spark applications that read from and write to your Amazon Redshift data warehouse, without compromising performance or transactional consistency. Amazon Redshift Integration for Apache Spark also uses AWS Identity and Access Management (IAM)–based credentials to enhance security. With Amazon Redshift Integration for Apache Spark, there is no manual setup and maintenance of uncertified versions of third-party connectors. You can start with Apache Spark jobs using data in Amazon Redshift in seconds. This new integration improves the performance of Apache Spark applications using Amazon Redshift data.
Benefits of Amazon Redshift
How it works
The diagram is composed of three main columns. The far-left column includes three boxes stacked vertically reading (from top to bottom): AWS Glue, Amazon EMR, and Amazon SageMaker. These boxes all point to the center column/box, which reads “Access data in Amazon Redshift through Apache Spark apps.” The center box points to a far-right box that reads “Amazon Redshift."
Use cases
Customers
Huron is a global professional services firm that collaborates with clients to put possible into practice by creating sound strategies, optimizing operations, accelerating digital transformation, and empowering businesses and their people to own their future.
"We empower our engineers to build their data pipelines and applications with Apache Spark using Python and Scala. We wanted a tailored solution that simplified operations and delivered faster and more efficiently for our clients and that’s what we get with the new Amazon Redshift Integration for Apache Spark."
Corey Johnson, Data Architect Manager - Huron Consulting
GE Aerospace is a global provider of jet engines, components, and systems for commercial and military aircraft. The company has been designing, developing, and manufacturing jet engines since World War I.
“GE Aerospace uses AWS analytics and Amazon Redshift to enable critical business insights that drive important business decisions. With the support for auto-copy from Amazon S3, we can build simpler data pipelines to move data from Amazon S3 to Amazon Redshift. This accelerates our data product teams’ ability to access data and deliver insights to end users. We spend more time adding value through data and less time on integrations.”
Alcuin Weidus, Sr Principal Data Architect - GE Aerospace
The Goldman Sachs Group, Inc. is a leading global financial institution that delivers a broad range of financial services across investment banking, securities, investment management and consumer banking to a large and diversified client base that includes corporations, financial institutions, governments, and individuals.
"Our focus is on providing self-service access to data for all of our users at Goldman Sachs. Through Legend, our open source data management and governance platform, we enable users to develop data-centric applications and derive data-driven insights as we collaborate across the financial services industry. With Amazon Redshift integration for Apache Spark, our data platform team will be able to access Amazon Redshift data with minimal manual steps—allowing for zero-code ETL that will increase our ability to make it easier for engineers to focus on perfecting their workflow as they collect complete and timely information. We expect to see a performance improvement of applications and improved security as our users can now easily access the latest data in Amazon Redshift.”
Neema Raphael, Chief Data Officer - Goldman Sachs