New Express Workflows for AWS Step Functions

Today, AWS is introducing Express Workflows for AWS Step Functions. This is a new workflow type to orchestrate AWS services at a higher-throughput than existing workflows.

Developers have been using AWS Step Functions since December 2016 to assemble long running workflows to orchestrate AWS Lambda Functions and other serverless services. Customers were looking for better ways to optimize for workloads that require higher event rates and shorter durations.

The new AWS Step Functions Express Workflows type uses fast, in-memory processing for high-event-rate workloads of up to 100,000 state transitions per second, for a total workflow duration of up to 5 minutes. Express Workflows are suited to streaming data processing, IoT data ingestion, mobile backends, and other high-throughput use-cases. Existing workflows in AWS Step Functions are now called Standard Workflows.

Getting started

You can build and run Express Workflows using the AWS Management Console, AWS CLI, or AWS CloudFormation. The steps below explain how to build an Express Workflow from within the AWS Management Console:

1.    In the AWS Step Functions console, choose Create state machine.

Figure 1 Creating a state machine from the AWS Step Functions console

2.    In the Type section, choose Express.

Figure 2 New Express Workflow option

3.    Enter a name for the Express Workflow in the State machine name field. In the Definition section, you can generate code snippets from a selection of example use cases.

4.    Leave the default definition and choose Next.

Figure 3 Define your workflow with ASL

You can create a new role for the Express Workflow execution or use an existing role.

5.    Leave Create New Role selected, and give it a logical name in the Role name field.

A new section named Logging configuration appears in the AWS Management Console. Here you can choose what level of logging for CloudWatch Logs. For Express Workflows, you must enable logging to inspect and debug executions.

6.    Choose ALL for Log level and leave the defaults for Include execution data and CloudWatch log group fields.

Figure 4 Logging configuration

7.    Choose Create state machine.

Once the Express Workflow is created, in the Details section the type is Express.

Figure 5 New ‘type’ in details section

Executing an Express Workflow from within the AWS Management Console is the same process as with a Standard Workflow.

Combining workflows

Applications may require a combination of both long-running and high-event-rate workflows. For example, the initial step in a workflow may involve ingesting and processing IoT stream data (Express Workflow), followed by executing a long-running machine learning model to derive insights (Standard Workflow).

Applications can benefit from the wait state of a Step Functions Standard Workflow and the shorter duration cost efficiencies of an Express Workflow when used together. An example support ticket automation application shows the SaaS integration capabilities of Amazon EventBridge, using Amazon Comprehend for sentiment analysis on support tickets.

Figure 6 Support ticket automation application architecture

You can improve part of the workflow by moving the short duration Lambda ‘Set tags’, ’Set category’, and ‘Escalate priority’ functions into a nested Express Workflow to orchestrate the re-occurring ‘Process Ticket’ stage. The Step Functions Standard Workflow uses the ‘Wait’ task to pause for a set amount of time until the Express Workflow completes. The Express Workflow is a distinct child workflow with its own Success or Fail task state from within the parent workflow. The child workflow must complete within the parent’s duration limit and uses the parent retry policy.

Figure 7 Improved support ticket automation application

Monitoring and logging with Express Workflows

Standard Workflows show execution history and visual debugging in the Step Functions console. Express Workflows send execution history to CloudWatch Logs. Two new tabs, Monitoring and Logging, have been added to the AWS Step Functions console to gain visibility into Express Workflow executions.

The Monitoring tab shows six different graphs with CloudWatch metrics for Execution errors, Execution Succeeded, Execution Duration, Billed Duration, Billed Memory, and Executions Started.

The Logging tab shows the logging configuration with a quick link to CloudWatch Logs.

Figure 8 New logging and monitoring tabs

State execution details are visible in CloudWatch Logs. CloudWatch Logs Insights provides search and log analysis with a purpose-built query language.

API changes

The Step Functions API has been updated to integrate Express Workflows for both creating and updating a state machine.

New parameter: Type, String
State machines have a new field called Type. Which can be Standard or Express. The default type is Standard to maintain backwards compatibility.

New parameter: LoggingConfiguration, Object
Defines the log detail level for CloudWatch Logs.

New exception: InvalidStateMachineType
This is thrown when type is not Express or Standard, or when there’s an activity (.sync or .waitForTaskToken connector).

Creating state machine with these new parameters:

–        API Call without using Type returns a Standard Workflow state machine.

$ aws stepfunctions create-state-machine \
 --definition "{...}" \
 --name "FlightTicketHandler" \
 --role-arn "arn:aws:iam:<account_id>::role/roleName/user" \
 --logging-configuration "{\"level\": \"OFF\"}" \

–        API Call using Type Express

$ aws stepfunctions create-state-machine \
 --definition "{...}" \
 --name "FlightTicketHandler" \
 --role-arn "arn:aws:iam:<account_id>::role/roleName/user" \
 --logging-configuration "{\"level\": \"ON\"}" \
 --type "EXPRESS"

–        API Call using Type Standard

$ aws stepfunctions create-state-machine \
 --definition "{...}" \
 --name "FlightTicketHandler" \
 --role-arn "arn:aws:iam:<account_id>::role/roleName/user" \
 --logging-configuration "{\"level\": \"ON\"}" \
 --type "STANDARD"

–        Updating the log level of a state machine

$ aws stepfunctions update-state-machine \
 --state-machine-arn "arn:aws:states:us-east-1:123456789012:stateMachine:${StateMachineName}" \
--logging-configuration "{\"level\":\"ALL\",\"includeExecutionData\":true,\"destinations\":[{\"cloudWatchLogsLogGroup\":{\"logGroupArn\":\"arn:aws:logs:us-east-1:<account_id>:log-group:/aws/states/${StateMachineName}-logs:*\"}}]}" 

For more information regarding the API changes, refer to the API documentation.

Comparing Standard and Express Workflows

Both workflow types use the declarative code semantics of Amazon States Language to build the workflow definition. Here are some of the key differences:

Conclusion

Many customers are already using AWS Step Functions Standard Workflows to orchestrate long-running, auditable workloads. The addition of a new Express Workflow type adds a lower-priced, higher-throughput workflow capability to AWS Step Functions.

Express Workflows complement AWS Step Functions Standard Workflows. Developers now have the power to choose the workflow type that best suits their needs, or choose to mix and blend as appropriate.

You can get started with Express Workflows via the AWS Management Console, AWS CLI, or AWS CloudFormation. It is available in all AWS Regions where AWS Step Functions is available.

For more information on where AWS Step Functions is available, see the AWS Region Table. For pricing for Express Workflows, see pricing.