Amazon Managed Workflows for Apache Airflow (MWAA) Documentation

Amazon Managed Workflows for Apache Airflow (MWAA) is a managed orchestration service for Apache Airflow that makes it easier to set up and operate end-to-end data pipelines in the cloud at scale. Apache Airflow is an open-source tool used to programmatically author, schedule, and monitor sequences of processes and tasks referred to as “workflows.” With Managed Workflows, you can use Airflow and Python to create workflows without having to manage the underlying infrastructure for scalability, availability, and security. Managed Workflows scales its workflow execution capacity to meet your needs, and is integrated with AWS security services to help provide you with fast and secure access to data.

Easy Airflow deployment

Managed Workflows leverage the same open source Apache Airflow product you know, just made easier. You can deploy Managed Workflows from AWS Management Console, CLI, AWS CloudFormation, or AWS SDK - and leverage the same Airflow user experience you’re familiar with.

Automatic scaling

With Managed Workflows, there’s worker scaling with no configuration required. Worker monitoring is built in - when workers are over-burdened, additional workers are provisioned automatically, and then decommissioned when no longer needed.

Built-in security

Managed Workflows keep your data secure using Amazon’s Virtual Private Cloud (VPC), and data is encrypted using AWS Key Management Service (KMS), so your workflow environment is secure by default.

Workflow monitoring in AWS or on-premises

Managed Workflows sends Apache Airflow system metrics and logs to Amazon Cloudwatch, making it easier for you to view task execution delays and workflow errors across one or more environments without third party tools.

Low operational costs

Managed Workflows remove the operational load of running open source Apache Airflow at scale so you can reduce operational costs and engineering overhead while running a data pipeline orchestration at nearly scale.

Plug-in integration

Managed Workflows connect to the AWS resources required for your workflows including Athena, Batch, Cloudwatch, DynamoDB, DataSync, EMR, ECS/Fargate, EKS, Firehose, Glue, Lambda, Redshift, SQS, SNS, Sagemaker, and S3. You can use Managed Workflows to connect to your own on-premises resources.

Additional Information

For additional information about service controls, security features and functionalities, including, as applicable, information about storing, retrieving, modifying, restricting, and deleting data, please see https://docs.aws.amazon.com/index.html. This additional information does not form part of the Documentation for purposes of the AWS Customer Agreement available at http://aws.amazon.com/agreement, or other agreement between you and AWS governing your use of AWS’s services.