Matillion created the Data Productivity Cloud on Amazon Web Services (AWS) to give customers unlimited scalability for data analytics workloads and empower any data user to quickly and easily build data pipelines. The Data Productivity Cloud runs on Amazon Elastic Kubernetes Service (Amazon EKS) and takes advantage of AWS Fargate to automate deployment.
Opportunity | Addressing Customer Requirements for Scalability and Centralized Management
Matillion, a cloud analytics software company and AWS Partner, offers a platform to help data teams become more productive by empowering the entire data team to move, transform, and orchestrate data pipelines faster.
For the past several years, the company’s customers have been requesting more scalability from Matillion solutions. “Our customers want to be able to conduct operations such as data loading, transformation, and orchestration without being constrained by infrastructure,” says Naggi Asmar, chief engineering officer at Matillion. “Many customers have spiky workloads and need to scale up and down. They want to run jobs as quickly as possible and don’t want to have to deal with performance bottlenecks in data orchestration and movement.” Matillion customers also wanted unlimited use of the company’s visual designer for data pipelines, as well as centralized resource management. “Customers needed a single pane of glass to see their entire set of projects and data transformations at the same time,” Asmar says. “We knew the cloud would address these challenges.”
Our customers now have unlimited scalability for their users and projects with our new SaaS solution powered by AWS."
Chief Engineering Officer, Matillion
Solution | Building an Automated, Unified Data Solution on AWS
To address these three customer requirements, Matillion Data Productivity Cloud was built as a new software as a service (SaaS) solution on AWS. The solution simplifies and automates data movement and enables customers to run multiple data pipelines simultaneously instead of sequentially. It also gives customers flexibility by offering three deployment options: a fully managed SaaS, a hybrid SaaS, and a self-hosted application.
The Data Productivity Cloud supports scalability by running on Amazon Elastic Kubernetes Service (Amazon EKS), a managed Kubernetes service on AWS, and AWS Fargate, a serverless compute engine that automatically deploys data pipeline execution agents. Additionally, the Data Productivity Cloud aggregates data from hundreds of different enterprise system connectors and loads it into Amazon Redshift for analysis. “This helps customers integrate data and make it business-ready so they can run analytics in a unified way,” says Asmar. Matillion also uses AWS PrivateLink for connectivity between AWS and virtual private clouds (VPCs) without exposing the data to the internet. As a result, customers can have end-to-end data security with a Matillion-hosted agent.
Because Matillion had been using AWS services internally for many years, it was able to create the Data Productivity Cloud quickly. “Our route to market was more efficient because we already had the building blocks that AWS provides,” says Asmar. For example, Matillion developers relied on AWS CodeCommit, a fully managed source control service. “If we had to build our own source code control solution, it would have taken much longer,” says Asmar.
Outcome | Ensuring Unlimited Scalability for Data Analytics Workloads
With the Matillion Data Productivity Cloud, companies can scale their data analytics workloads up or down on demand. “Our customers now have unlimited scalability for their users and projects with our new SaaS solution powered by AWS,” Asmar says. “They no longer have to worry about adding more instances or upgrading systems. The automatic scalability of the architecture, based on Amazon EKS, takes care of it for them.”
Matillion customers using the Data Productivity Cloud can quickly create data pipelines on their own because the solution’s visual designer provides unlimited usage and gives rule-based access control. “Whether you’re an IT manager, a data engineer, or a data scientist, you can easily move data to or from any source and build and manage data movement, transformation, and orchestration without the need to code,” says Asmar.
Matillion plans to collaborate with AWS as it continues to enhance the Data Productivity Cloud, exploring AWS AI technologies and additional AWS services. “Our company is built on AWS, and that gives us confidence when we bring our solutions to customers because we can demonstrate that we’re both an AWS customer and partner,” Asmar says. “We’re still early in our journey with AWS, and we look forward to using AWS to deliver more value on top of our platform.”
Matillion, an AWS Partner founded in 2011, provides solutions that help anyone in the data team move, transform, and automate data pipelines, regardless of their coding capabilities. The company’s Data Productivity Cloud helps accelerate data work, including building data pipelines, and customers pay only for what they use.
AWS Services Used
Amazon Elastic Kubernetes Service (Amazon EKS) is a managed Kubernetes service to run Kubernetes in the AWS cloud and on-premises data centers.
AWS Fargate is a serverless, pay-as-you-go compute engine that lets you focus on building applications without managing servers.
Learn more »
Amazon Redshift uses SQL to analyze structured and semi-structured data across data warehouses, operational databases, and data lakes, using AWS-designed hardware and machine learning to deliver the best price performance at any scale.
Learn more »
AWS PrivateLink provides private connectivity between virtual private clouds (VPCs), supported AWS services, and your on-premises networks without exposing your traffic to the public internet.
Learn more »
Organizations of all sizes across all industries are transforming their businesses and delivering on their missions every day using AWS. Contact our experts and start your own AWS journey today.