AWS Big Data Blog

Set up and manage data ingestion easily with Amazon Redshift native console integration with partners

We’re excited to announce that Amazon Redshift console partner integration is now generally available. This new console integration provides rapid provisioning and seamless integration with AWS partners. You can onboard with data integration partner solutions in less than a minute directly on the Amazon Redshift console, and ingest data from multiple data sources using partners’ pre-built connectors. This allows you to speed up onboarding with partners and gain faster data-driven insights on your Amazon Redshift clusters. You also have the flexibility to experiment with different partner solutions to determine which one best fits your needs.

Furthermore, we’re introducing a new section on the Amazon Redshift console called Integrations, which is a centralized dashboard for all partners integrated with your Amazon Redshift cluster. This provides a single place to monitor all aspects of your cluster, including the status of data ingestion from your preferred solution providers.

Advantages of console partner integration

The console partner integration has several advantages:

  • Faster onboarding with partners – With minimal clicks and pre-built configurations, you can sign up with partners in under a minute. This can save you precious time and help you quickly load and access critical business data from different sources.
  • Automated connection and integrations – With the quick signup process, you also get an automated connection to Amazon Redshift, which offers an accelerated data loading experience and faster data-driven analytics. This eliminates the manual and tedious task of creating and testing your Amazon Redshift connection.
  • Explore partners – You can experiment with various partner solutions to find the best one for your needs. You can create a free trial account with the selected partner solution and gain insights to your data on your Amazon Redshift cluster within minutes for instant analysis. After you explore and implement proof of concepts with multiple partners, you can continue with your preferred partner solution.
  • One location for integration status – The new Integration section is a unified place to see a list of all your connected partners and their status, identify any issues, and see when the partner last performed a successful update to Amazon Redshift.

Integrate with an Amazon Redshift partner

The following diagram illustrates the flow to integrate with a partner solution on the Amazon Redshift console. You simply navigate to your cluster details, choose to add a partner, make your selection, and verify your connection. You’re redirected to the partner landing page to complete your registration, and the partner solution is now available in the Integrations section on the Amazon Redshift console.

Let’s walk through each step.

  1. On the Amazon Redshift console, choose Clusters in the navigation pane.
  2. Select your cluster and on the Actions menu, choose Add AWS Partner integration.

Alternatively, you can choose your individual cluster and on its details page, choose Add partner integration.

  1. Select your desired partner.
  2. Choose Next.

For this post, we demonstrate integrating with Etleap.

  1. Choose Add partner.

On the Amazon Redshift console, you should see a blue flashbar notification when the integration is complete.

Next, you log in to the partner landing page. For this post, we demonstrate integrating with Etleap, but each partner’s website is unique and looks different.

  1. On the partner landing page, enter your Redshift database password of the user shown in the image above.

Note: To connect to Redshift cluster, please make sure to enable “Publicly accessible” option and also whitelist IP’s in your Redshift Cluster’s security group.

  1. Enter your email (if you don’t have an account, you can sign up for a trial).
  2. Choose Next.
  3. Check your email for a verification link.
  4. When you choose the verification link from the email, you’re redirected to create your account.

After you register, you can see the Amazon Redshift connection in your partner portal.

You can also see the connection on the Amazon Redshift console in the Integrations section of your cluster details.

Congratulations! You’re now ready to start loading data from your partner solution.

Integrations section on the cluster details page

As part of this partner integration feature, we’re introducing a new section on the cluster details page called Integrations. In this section, you can track and monitor all the partner solutions integrated with your Amazon Redshift cluster. The section shows each partner’s last known connection status and the last connected time.

The Integrations section currently shows the four most commonly requested data points, but it’s designed to be easily extensible as we get additional customer feedback:

  • Active – Indicates you successfully integrated on the partner website or performed data reads or writes to the cluster.
  • Inactive – Indicates you removed your registration from the partner portal or the partner deleted your user account after the end of their trial period.
  • Runtime Failure – Indicates the solution failed to read or was unable to load. You also see a high-level error message.
  • Connection Failure – Indicates that a transaction failed due to a connection issue like network, authorization, or authentication with the partner solution.

Partner feedback

We’re delighted to share the following feedback from some of our partners.

“Data is growing exponentially and is more distributed and fragmented than ever. Connecting the right data with the right consumer is one of the biggest opportunities facing organizations that want to ensure all users have access to data they can trust. Informatica with its Intelligent Data Management Cloud (IDMC) has taken yet another step in its decade-long AWS relationship to make it easier for AWS customers on Amazon Redshift and other AWS services to seamlessly, easily, and confidently integrate, govern, and manage high-quality data to achieve business innovation and true digital transformation.”

—Rik Tamm-Daniels, Vice President, Strategic Ecosystems and Technology at Informatica

“Today’s organizations have an ever-growing amount of disparate software applications and data sources, each with their own best-of-breed capability or expertise, and each producing mountains of data. With SnapLogic, customers can connect these applications and load data into Amazon Redshift quickly and intuitively, harnessing their valuable data assets to drive accurate analysis and better decision-making.”

—Jason Wakeam, Vice President of Business Development and OEM Sales at SnapLogic

“We’re excited to launch our integration with the Amazon Redshift console. Using the Etleap integration, Amazon Redshift users can get their data integrated and ready for analysis in minutes.”

—Christian Romming, CEO of Etleap

“We are really excited about being integrated into the Amazon Redshift console. Through this integration, customers will experience the combined power of Datacoral’s CDC connectors and Amazon Redshift to seamlessly integrate data from different sources and extract value out of that data.”

—Raghu Murthy, CEO and Founder of Datacoral

Console integration launch partners

The following are our console integration launch partners.

Datacoral

Datacoral offers an AWS-native data integration and pipeline platform to integrate data from databases using change data capture into cloud warehouses. Datacoral’s award-winning architecture provides security and scalability with cloud on-premises deployments and serverless data connectors within the customer VPC. Datacoral customers can set up end-to-end data pipelines from ingestion to transformation to publishing using just SQL and a few clicks.

Etleap

Etleap is an AWS-native, enterprise-grade ETL solution for creating robust and flexible data pipelines from day one. It integrates with many data sources, and was built specifically for teams running Amazon Redshift warehouses and Amazon Simple Storage Service (Amazon S3) and AWS Glue data lakes. Etleap eliminates engineering work to set up, maintain, and scale ETL. Etleap also reduces setup time, automates maintenance work, and provides transparency and full control over data pipelines. It can run as a hosted software as a service (SaaS) solution or inside your VPC.

Fivetran

Fivetran automates data integration with zero-maintenance data pipelines for Amazon Redshift from sources like Salesforce, Marketo, Adwords, NetSuite, PostgreSQL, Amazon Aurora, and Oracle. After a quick setup, Fivetran automatically ingests and centralizes your data with over 150 connectors for applications, databases, events, and files into ready-to-analyze schemas. This enables you to concentrate on analytics rather than data engineering.

Informatica

Informatica Intelligent Data Management Cloud (IDMC) enables customers to visualize, analyze, and collaborate with their data regardless of location or platform. With cloud-native services, you can automatically discover, catalog, ingest, cleanse, integrate, optimize, govern, secure, prepare, and understand data on AWS. Informatica IDMC offers pre-built templates and integration wizards for native, high-volume, high-performance data integration on AWS.

Snaplogic

SnapLogic offers a solution for AWS that includes integration of applications, data warehouses, and data lakes, and big data and analytics acceleration through their Intelligent Integration Platform. This platform employs AI-powered workflows to automate all states of IT integration projects—design, development, deployment, and maintenance—for on-premises, cloud, and hybrid environments.

Console integration preview partners

The following are our console integration preview partners.

Matillion

Matillion ETL for Amazon Redshift provides cloud-native enterprise ELT for hundreds of customers, so you can easily and quickly ingest data from various on-premises and cloud sources into your cloud data warehouse. Matillion then transforms data into an analytics-ready format that enables you to make data-driven decisions for your business. You can use Matillion whether or not you know SQL, democratizing data for all.

Sisense

Sisense builds data pipelines with Amazon Redshift, performs advanced analysis with SQL, Python, and R, and delivers customized visual analytic experiences.

Stitch

Stitch is a fully managed data pipeline built to make analytics projects happen faster.

With connectivity to over 140 data sources such as Postgres, Salesforce, and Google Ads, Stitch helps analysts move data into a data warehouse without the need for engineering effort.

Console integration upcoming partner

The following is our console integration upcoming partner.

Upsolver

Upsolver is a no-code data lake engineering platform that lets you ingest batch and streaming data data into Amazon S3, perform complex stateful transformations, and continually deliver data to Amazon Redshift, Amazon OpenSearch Service, Amazon RDS and other stores, or query S3 data using Amazon Athena or Redshift Spectrum. It features native connectors, a visual UI for building SQL transform logic, and automatically optimizes performance. It deploys as a managed service in your AWS account and elastically scales, while decoupling S3 from EC2.

Conclusion

In this post, we discussed how the Amazon Redshift console partner integration provides a fast onboarding experience and allows you to create valuable business insights by integrating with select partner solutions.

If you’re an AWS partner and would like to integrate your product into the Amazon Redshift console, please contact redshift-partners@amazon.com for additional information and guidance.

This console partner integration functionality is available to new and existing customers at no additional cost. To get started and learn more, see Integrating Amazon Redshift with an AWS Partner.

About the Authors

Maneesh Sharma is a Senior Database Engineer with Amazon Redshift. He works and collaborates with various Redshift Partners to drive better integration. In his spare time, he likes running, playing ping pong and exploring new travel destinations.

Joe Yong is relentless pursuer of making complex database technologies easy and intuitive for the masses. In 20 years of building database engines and large scale data management systems, Joe has shipped dozens of features for on-premises and cloud-native databases. His work spans both SMP and MPP systems ranging from IOT devices to Petabyte-sized cloud data warehouses. Joe is currently a Product Manager with Amazon Web Services helping make Amazon Redshift – the world’s most popular, highest performance and most scalable cloud data warehouse – even better. Off keyboard, Joe tries to onsight 5.11s, hunt for good eats and seek a cure for his Australian Labradoodle’s obsession with squeaky tennis balls.