AWS Step Functions Use Cases

In this example of a simple banking system, a new bank account is created after validating a customer’s name and address. The workflow begins with two Lambda functions CheckName and CheckAddress executing in parallel as task states. Once both are complete, the workflow executes the ApproveApplication Lambda function. You can define retry and catch clauses to handle errors from task states. You can use predefined system errors or handle custom errors thrown by these Lambda functions in your workflow. Since your workflow code takes on error handling, the Lambda functions can focus on the business logic and have less code. Express workflows would be a better fit for this example as the Lambda functions are performing tasks that together take less than five minutes, with no external dependencies. 

Sometimes you might need a human to review and approve or reject a step in the business process so that the workflow can continue to the next step. We recommend using Standard  workflows when your workflow needs to wait for a human or for processes in which an external system might take more than five minutes to respond. Here we extend the new account opening process with a notify approver step in between.The workflow starts with the CheckName and CheckAddress task states executing in parallel. The next state ReviewRequired is a choice state that has two possible paths - either send an SNS notification email to the approver in the NotifyApprover task or proceed to the ApproveApplication state. The NotifyApprover task state sends an email to the approver, and waits for a response before proceeding to the next choice state “Approved”. Based on the approver’s decision, the account application is either Approved or Rejected through Lambda functions,

In this example, an event on a custom Eventbridge bus satisfies a rule and invokes a Step Functions workflow as a target. Suppose you have a customer service application that needs to handle expired customer subscriptions. An EventBridge rule listens for subscription expired events and invokes a target workflow in response. The subscription expired workflow will disable all the resources that the expired subscription owns without deleting them and email the customer to notify them about their expired subscription. These two actions can be done in parallel using Lambda functions. At the end of the workflow, a new event is sent to the event bus through a Lambda function indicating that the subscription expiration has been processed. For this example, we recommend using Express Workflows. As your business grows and you start to put more events on to the event bus, the capacity to invoke 100,000 workflow executions a second with Express Workflows is powerful. See the example in action with this Github repo.

You can use AWS Step Functions to create an automated security incident response workflow that includes a manual approval step. In this example, a Step Functions workflow is triggered when an IAM policy is created. The workflow compares the policy action against a customizable list of restricted actions. The workflow rolls back the policy temporarily, then notifies an administrator and waits for them to approve or deny. You can extend this workflow to remediate automatically such as applying alternative actions, or restricting actions to specific ARNs. See this example in action here.

You can reduce the operational overhead of maintaining your AWS cloud infrastructure by automating how you respond to operational events for your AWS resources. Amazon EventBridge provides a near real-time stream of system events that describe most changes and notifications for your AWS resources. From this stream, you can create rules to route specific events to AWS Step Functions, AWS Lambda, and other AWS services for further processing and automated actions. In this example, an AWS Step Functions workflow is triggered based on an event sourced from AWS Health. AWS proactively monitors popular code repository sites for IAM access keys that have been publicly exposed. Let’s suppose an IAM access key was exposed on GitHub. AWS Health generates an AWS_RISK_CREDENTIALS_EXPOSED event in the AWS account related to the exposed key. A configured Amazon Eventbridge rule detects this event and invokes a Step Functions workflow. With the help of AWS Lambda functions, the workflow then deletes the exposed IAM access key, summarizes the recent API activity for the exposed key, and sends the summary message to an Amazon SNS topic to notify the subscribers―in that order. See this example in action here.

You can use Amazon S3 to host a static website, and Amazon CloudFront to distribute the content worldwide. As a website owner, you might need two S3 buckets for uploading your website content: one for staging and testing, and one for production. You want to update the production bucket with all changes from the staging bucket, without having to create a new bucket from scratch every time you update your website. In this example, the Step Functions workflow performs tasks in two parallel and independent loops: one loop copies all objects from source bucket to destination bucket, but leaves out objects already present in the destination bucket. The second loop deletes any objects in the destination bucket not found in the source bucket. A set of AWS Lambda functions carry out the invididual steps: validate input, get the lists of objects from both source and destination buckets, and copy or delete objects in batches. See this example along with its code in detail here. Learn more about how to create parallel branches of execution in your state machine here.

In this example, Freebird built a data processing pipeline to process webhook data from multiple sources in real time and run Lambda functions that modify the data. In this use case, webhook data from several third party applications is sent through Amazon API Gateway to an Amazon Kinesis data stream. An AWS Lambda function pulls data off this Kinesis stream, and triggers an Express Workflow. This workflow goes through a series of steps to validate, process and normalize this data. In the end, a Lambda function updates the SNS topic that triggers messages to downstream Lambda functions for next steps through an SQS queue. You could have up to 100,000 invocations of this workflow per second to scale the data processing pipeline.

You can use Step Functions to orchestrate all steps of an ETL process, with different data sources and destinations. 

In this example, the Step Functions ETL workflow refreshes Amazon Redshift whenever new data is available in the source S3 bucket. The Step Functions state machine initiates an AWS Batch job and monitors its status for completion or errors. The AWS Batch job fetches the ETL workflow .sql script from the source i.e. Amazon S3 and refreshes the destination i.e. Amazon Redshift through a PL/SQL container. The .sql file contains the SQL code for each step in the data transformation.You can trigger the ETL workflow with an EventBridge event or manually through the AWS CLI or using AWS SDKs or even a custom automation script. You can alert an admin through SNS that triggers an email for failures at any step of the workflow or at the end of the workflow execution. This ETL workflow is an example where you can use Standard Workflows. See this example in detail here. You can learn more about submitting an AWS Batch job through a sample project here.

Extract, Load and Transform (ETL) operations transform raw data into useful data sets, turning data into actionable insight. 

You can use Step Functions to run multiple ETL jobs in parallel where your source datasets might be available at different times, and each ETL job is triggered only when its corresponding dataset becomes available. These ETL jobs can be managed by different AWS services, such as AWS, Glue, Amazon EMR, Amazon Athena or other non-AWS services.
In this example, you have two separate ETL jobs running on AWS Glue that process a sales dataset and a marketing dataset. Once both datasets are processed,a third ETL job combines the output from the previous ETL jobs to produce a combined dataset.The Step Functions workflow waits until data is available in S3. While the main workflow is kicked off on a schedule, an EventBridge event handler is configured on an Amazon S3 bucket, so that when Sales or Marketing dataset files are uploaded to the bucket, the State Machine can trigger the ETL job “ProcessSales Data” or “ProcessMarketingData” depending on which dataset became available. 

See this architecture in detail here to set up ETL orchestration architecture in your AWS account. Learn how to manage an AWS Batch job from Step Functions here.

You can use Step Functions to iterate over tens of millions of items in a dataset, such as a JSON array, a list of objects in S3 or a CSV file in an S3 bucket. You can then process the data in parallel with high concurrency.

In this example, the Step Functions workflow uses a Map state in Distributed mode to process a list of S3 objects in an S3 bucket. Step Functions iterates over the list of objects and then launches thousands of parallel workflows, running concurrently, to process the items. You can use compute services, such as Lambda, helping you write code in any language supported. You can also choose from over 220 purpose-built AWS services to include in the Map state workflow. Once executions of child workflows are complete, Step Functions can export the results to an S3 bucket making it available for review or for further processing.