AWS Step Functions adds support for dynamic parallelism in workflows
AWS Step Functions now supports dynamic parallelism, so you can optimize the performance and efficiency of application workflows such as data processing and task automation. By running identical tasks in parallel, you can achieve consistent execution durations and improve utilization of resources to save on operating costs. Step Functions automatically scales resources in response to your input.
Step Functions lets you orchestrate multiple AWS services in fully-managed workflows so you can build and update apps quickly. Many organizations run batch processing workflows in parallel to use resources efficiently. However, it is difficult to predict the execution time of parallel workflow because the number of items to processes is often unknown. Building and debugging these parallel workflows is time consuming and hard.
Now, you can build dynamically parallel fanout and scatter-gather patterns in minutes with less code. Fanout patterns dispatch a list of identical tasks in parallel to simplify workflows such as order processing and instance patch management. Scatter-gather patterns leverage scalable compute on AWS to accelerate workflows such as file processing and report generation. For example, you can transcode ten 500 MB media files in parallel and then merge to create a 5 GB file. Step Functions parallel workflow visualization makes it easy to find the cause of defects in seconds.
You can get started by exploring a Sample Project in the Step Functions console. The console editor includes templates that you can use to easily add parallel steps to your workflows.
Dynamic parallelism is included in AWS Step Functions pricing at no additional cost and is available in all AWS public regions where Step Functions is available. For a complete list of regions where AWS Step Functions is offered, see AWS Regions.
To learn more about Step Functions, read the AWS Step Functions Developer Guide. To learn more about building workflows with dynamic parallelism, read about the AWS Step Functions Map state.