Matillion Uses AWS to Help Data Teams Work Faster and Scale Data Pipelines on Demand

Matillion, a cloud analytics software company and AWS Partner, offers a platform to help data teams become more productive by empowering the entire data team to move, transform, and orchestrate data pipelines faster.

For the past several years, the company’s customers have been requesting more scalability from Matillion solutions. “Our customers want to be able to conduct operations such as data loading, transformation, and orchestration without being constrained by infrastructure,” says Naggi Asmar, chief engineering officer at Matillion. “Many customers have spiky workloads and need to scale up and down. They want to run jobs as quickly as possible and don’t want to have to deal with performance bottlenecks in data orchestration and movement.” Matillion customers also wanted unlimited use of the company’s visual designer for data pipelines, as well as centralized resource management. “Customers needed a single pane of glass to see their entire set of projects and data transformations at the same time,” Asmar says. “We knew the cloud would address these challenges.”

To address these three customer requirements, Matillion Data Productivity Cloud was built as a new software as a service (SaaS) solution on AWS. The solution simplifies and automates data movement and enables customers to run multiple data pipelines simultaneously instead of sequentially. It also gives customers flexibility by offering three deployment options: a fully managed SaaS, a hybrid SaaS, and a self-hosted application.

The Data Productivity Cloud supports scalability by running on Amazon Elastic Kubernetes Service (Amazon EKS), a managed Kubernetes service on AWS, and AWS Fargate, a serverless compute engine that automatically deploys data pipeline execution agents. Additionally, the Data Productivity Cloud aggregates data from hundreds of different enterprise system connectors and loads it into Amazon Redshift for analysis. “This helps customers integrate data and make it business-ready so they can run analytics in a unified way,” says Asmar. Matillion also uses AWS PrivateLink for connectivity between AWS and virtual private clouds (VPCs) without exposing the data to the internet. As a result, customers can have end-to-end data security with a Matillion-hosted agent.

Because Matillion had been using AWS services internally for many years, it was able to create the Data Productivity Cloud quickly. “Our route to market was more efficient because we already had the building blocks that AWS provides,” says Asmar. For example, Matillion developers relied on AWS CodeCommit, a fully managed source control service. “If we had to build our own source code control solution, it would have taken much longer,” says Asmar.