AWS Compute Blog
Resume AWS Step Functions from Any State
Update March, 5 2021 – Disclaimer: This blog precedes the introduction of map state to the Amazon States Language and requires modifications to work with the map state.
This post is written by Aaron Friedman, Partner Solutions Architect and Yash Pant, Solutions Architect.
When we discuss how to build applications with customers, we often align to the Well Architected Framework pillars of security, reliability, performance efficiency, cost optimization, and operational excellence. Designing for failure is an essential component to developing well architected applications that are resilient to spurious errors that may occur.
There are many ways you can use AWS services to achieve high availability and resiliency of your applications. For example, you can couple Elastic Load Balancing with Auto Scaling and Amazon EC2 instances to build highly available applications. Or use Amazon API Gateway and AWS Lambda to rapidly scale out a microservices-based architecture. Many AWS services have built in solutions to help with the appropriate error handling, such as Dead Letter Queues (DLQ) for Amazon SQS or retries in AWS Batch.
AWS Step Functions is an AWS service that makes it easy for you to coordinate the components of distributed applications and microservices. Step Functions allows you to easily design for failure, by incorporating features such as error retries and custom error handling from AWS Lambda exceptions. These features allow you to programmatically handle many common error modes and build robust, reliable applications.
In some rare cases, however, your application may fail in an unexpected manner. In these situations, you might not want to duplicate in a repeat execution those portions of your state machine that have already run. This is especially true when orchestrating long-running jobs or executing a complex state machine as part of a microservice. Here, you need to know the last successful state in your state machine from which to resume, so that you don’t duplicate previous work. In this post, we present a solution to enable you to resume from any given state in your state machine in the case of an unexpected failure.
Resuming from a given state
To resume a failed state machine execution from the state at which it failed, you first run a script that dynamically creates a new state machine. When the new state machine is executed, it resumes the failed execution from the point of failure. The script contains the following two primary steps:
- Parse the execution history of the failed execution to find the name of the state at which it failed, as well as the JSON input to that state.
- Create a new state machine, which adds an additional state to failed state machine, called “GoToState”. “GoToState” is a choice state at the beginning of the state machine that branches execution directly to the failed state, allowing you to skip states that had succeeded in the previous execution.
The full script along with a CloudFormation template that creates a demo of this is available in the aws-sfn-resume-from-any-state GitHub repo.
Diving into the script
In this section, we walk you through the script and highlight the core components of its functionality. The script contains a main function, which adds a command line parameter for the failedExecutionArn so that you can easily call the script from the command line:
python gotostate.py --failedExecutionArn '<Failed_Execution_Arn>'Identifying the failed state in your execution
First, the script extracts the name of the failed state along with the input to that state. It does so by using the failed state machine execution history, which is identified by the Amazon Resource Name (ARN) of the execution. The failed state is marked in the execution history, along with the input to that state (which is also the output of the preceding successful state). The script is able to parse these values from the log.
The script loops through the execution history of the failed state machine, and traces it backwards until it finds the failed state. If the state machine failed in a parallel state, then it must restart from the beginning of the parallel state. The script is able to capture the name of the parallel state that failed, rather than any substate within the parallel state that may have caused the failure. The following code is the Python function that does this.
def parseFailureHistory(failedExecutionArn):
'''
Parses the execution history of a failed state machine to get the name of failed state and the input to the failed state:
Input failedExecutionArn = A string containing the execution ARN of a failed state machine y
Output = A list with two elements: [name of failed state, input to failed state]
'''
failedAtParallelState = False
try:
#Get the execution history
response = client.get\_execution\_history(
executionArn=failedExecutionArn,
reverseOrder=True
)
failedEvents = response['events']
except Exception as ex:
raise ex
#Confirm that the execution actually failed, raise exception if it didn't fail.
try:
failedEvents[0]['executionFailedEventDetails']
except:
raise('Execution did not fail')
'''
If you have a 'States.Runtime' error (for example, if a task state in your state machine attempts to execute a Lambda function in a different region than the state machine), get the ID of the failed state, and use it to determine the failed state name and input.
'''
if failedEvents[0]['executionFailedEventDetails']['error'] == 'States.Runtime':
failedId = int(filter(str.isdigit, str(failedEvents[0]['executionFailedEventDetails']['cause'].split()[13])))
failedState = failedEvents[-1 \* failedId]['stateEnteredEventDetails']['name']
failedInput = failedEvents[-1 \* failedId]['stateEnteredEventDetails']['input']
return (failedState, failedInput)
'''
You need to loop through the execution history, tracing back the executed steps.
The first state you encounter is the failed state. If you failed on a parallel state, you need the name of the parallel state rather than the name of a state within a parallel state that it failed on. This is because you can only attach goToState to the parallel state, but not a substate within the parallel state.
This loop starts with the ID of the latest event and uses the previous event IDs to trace back the execution to the beginning (id 0). However, it returns as soon it finds the name of the failed state.
'''
currentEventId = failedEvents[0]['id']
while currentEventId != 0:
#multiply event ID by -1 for indexing because you're looking at the reversed history
currentEvent = failedEvents[-1 \* currentEventId]
'''
You can determine if the failed state was a parallel state because it and an event with 'type'='ParallelStateFailed' appears in the execution history before the name of the failed state
'''
if currentEvent['type'] == 'ParallelStateFailed':
failedAtParallelState = True
'''
If the failed state is not a parallel state, then the name of failed state to return is the name of the state in the first 'TaskStateEntered' event type you run into when tracing back the execution history
'''
if currentEvent['type'] == 'TaskStateEntered' and failedAtParallelState == False:
failedState = currentEvent['stateEnteredEventDetails']['name']
failedInput = currentEvent['stateEnteredEventDetails']['input']
return (failedState, failedInput)
'''
If the failed state was a parallel state, then you need to trace execution back to the first event with 'type'='ParallelStateEntered', and return the name of the state
'''
if currentEvent['type'] == 'ParallelStateEntered' and failedAtParallelState:
failedState = failedState = currentEvent['stateEnteredEventDetails']['name']
failedInput = currentEvent['stateEnteredEventDetails']['input']
return (failedState, failedInput)
#Update the ID for the next execution of the loop
currentEventId = currentEvent['previousEventId']
Create the new state machine
The script uses the name of the failed state to create the new state machine, with “GoToState” branching execution directly to the failed state.
To do this, the script requires the Amazon States Language (ASL) definition of the failed state machine. It modifies the definition to append “GoToState”, and create a new state machine from it.
The script gets the ARN of the failed state machine from the execution ARN of the failed state machine. This ARN allows it to get the ASL definition of the failed state machine by calling the DesribeStateMachine API action. It creates a new state machine with “GoToState”.
When the script creates the new state machine, it also adds an additional input variable called “resuming”. When you execute this new state machine, you specify this resuming variable as true in the input JSON. This tells “GoToState” to branch execution to the state that had previously failed. Here’s the function that does this:
def attachGoToState(failedStateName, stateMachineArn):
'''
Given a state machine ARN and the name of a state in that state machine, create a new state machine that starts at a new choice state called 'GoToState'. "GoToState" branches to the named state, and sends the input of the state machine to that state, when a variable called "resuming" is set to True.
Input failedStateName = A string with the name of the failed state
stateMachineArn = A string with the ARN of the state machine
Output response from the create_state_machine call, which is the API call that creates a new state machine
'''
try:
response = client.describe\_state\_machine(
stateMachineArn=stateMachineArn
)
except:
raise('Could not get ASL definition of state machine')
roleArn = response['roleArn']
stateMachine = json.loads(response['definition'])
#Create a name for the new state machine
newName = response['name'] + '-with-GoToState'
#Get the StartAt state for the original state machine, because you point the 'GoToState' to this state
originalStartAt = stateMachine['StartAt']
'''
Create the GoToState with the variable $.resuming.
If new state machine is executed with $.resuming = True, then the state machine skips to the failed state.
Otherwise, it executes the state machine from the original start state.
'''
goToState = {'Type':'Choice', 'Choices':[{'Variable':'$.resuming', 'BooleanEquals':False, 'Next':originalStartAt}], 'Default':failedStateName}
#Add GoToState to the set of states in the new state machine
stateMachine['States']['GoToState'] = goToState
#Add StartAt
stateMachine['StartAt'] = 'GoToState'
#Create new state machine
try:
response = client.create_state_machine(
name=newName,
definition=json.dumps(stateMachine),
roleArn=roleArn
)
except:
raise('Failed to create new state machine with GoToState')
return response
Testing the script
Now that you understand how the script works, you can test it out.
The following screenshot shows an example state machine that has failed, called “TestMachine”. This state machine successfully completed “FirstState” and “ChoiceState”, but when it branched to “FirstMatchState”, it failed.
Use the script to create a new state machine that allows you to rerun this state machine, but skip the “FirstState” and the “ChoiceState” steps that already succeeded. You can do this by calling the script as follows:
python gotostate.py --failedExecutionArn 'arn:aws:states:us-west-2:<AWS_ACCOUNT_ID>:execution:TestMachine-with-GoToState:b2578403-f41d-a2c7-e70c-7500045288595This creates a new state machine called “TestMachine-with-GoToState”, and returns its ARN, along with the input that had been sent to “FirstMatchState”. You can then inspect the input to determine what caused the error. In this case, you notice that the input to “FirstMachState” was the following:
{
"foo": 1,
"Message": true
}However, this state machine expects the “Message” field of the JSON to be a string rather than a Boolean. Execute the new “TestMachine-with-GoToState” state machine, change the input to be a string, and add the “resuming” variable that “GoToState” requires:
{
"foo": 1,
"Message": "Hello!",
"resuming":true
}When you execute the new state machine, it skips “FirstState” and “ChoiceState”, and goes directly to “FirstMatchState”, which was the state that failed:
Look at what happens when you have a state machine with multiple parallel steps. This example is included in the GitHub repository associated with this post. The repo contains a CloudFormation template that sets up this state machine and provides instructions to replicate this solution.
The following state machine, “ParallelStateMachine”, takes an input through two subsequent parallel states before doing some final processing and exiting, along with the JSON with the ASL definition of the state machine.
{
"Comment": "An example of the Amazon States Language using a parallel state to execute two branches at the same time.",
"StartAt": "Parallel",
"States": {
"Parallel": {
"Type": "Parallel",
"ResultPath":"$.output",
"Next": "Parallel 2",
"Branches": [
{
"StartAt": "Parallel Step 1, Process 1",
"States": {
"Parallel Step 1, Process 1": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:XXXXXXXXXXXX:function:LambdaA",
"End": true
}
}
},
{
"StartAt": "Parallel Step 1, Process 2",
"States": {
"Parallel Step 1, Process 2": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:XXXXXXXXXXXX:function:LambdaA",
"End": true
}
}
}
]
},
"Parallel 2": {
"Type": "Parallel",
"Next": "Final Processing",
"Branches": [
{
"StartAt": "Parallel Step 2, Process 1",
"States": {
"Parallel Step 2, Process 1": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:XXXXXXXXXXXXX:function:LambdaB",
"End": true
}
}
},
{
"StartAt": "Parallel Step 2, Process 2",
"States": {
"Parallel Step 2, Process 2": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:XXXXXXXXXXXX:function:LambdaB",
"End": true
}
}
}
]
},
"Final Processing": {
"Type": "Task",
"Resource": "arn:aws:lambda:us-west-2:XXXXXXXXXXXX:function:LambdaC",
"End": true
}
}
}First, use an input that initially fails:
{
"Message": "Hello!"
}This fails because the state machine expects you to have a variable in the input JSON called “foo” in the second parallel state to run “Parallel Step 2, Process 1” and “Parallel Step 2, Process 2”. Instead, the original input gets processed by the first parallel state and produces the following output to pass to the second parallel state:
{
"output": [
{
"Message": "Hello!"
},
{
"Message": "Hello!"
}
],
}Run the script on the failed state machine to create a new state machine that allows it to resume directly at the second parallel state instead of having to redo the first parallel state. This creates a new state machine called “ParallelStateMachine-with-GoToState”. The following JSON was created by the script to define the new state machine in ASL. It contains the “GoToState” value that was attached by the script.
{
"Comment":"An example of the Amazon States Language using a parallel state to execute two branches at the same time.",
"States":{
"Final Processing":{
"Resource":"arn:aws:lambda:us-west-2:XXXXXXXXXXXX:function:LambdaC",
"End":true,
"Type":"Task"
},
"GoToState":{
"Default":"Parallel 2",
"Type":"Choice",
"Choices":[
{
"Variable":"$.resuming",
"BooleanEquals":false,
"Next":"Parallel"
}
]
},
"Parallel":{
"Branches":[
{
"States":{
"Parallel Step 1, Process 1":{
"Resource":"arn:aws:lambda:us-west-2:XXXXXXXXXXXX:function:LambdaA",
"End":true,
"Type":"Task"
}
},
"StartAt":"Parallel Step 1, Process 1"
},
{
"States":{
"Parallel Step 1, Process 2":{
"Resource":"arn:aws:lambda:us-west-2:XXXXXXXXXXXX:LambdaA",
"End":true,
"Type":"Task"
}
},
"StartAt":"Parallel Step 1, Process 2"
}
],
"ResultPath":"$.output",
"Type":"Parallel",
"Next":"Parallel 2"
},
"Parallel 2":{
"Branches":[
{
"States":{
"Parallel Step 2, Process 1":{
"Resource":"arn:aws:lambda:us-west-2:XXXXXXXXXXXX:function:LambdaB",
"End":true,
"Type":"Task"
}
},
"StartAt":"Parallel Step 2, Process 1"
},
{
"States":{
"Parallel Step 2, Process 2":{
"Resource":"arn:aws:lambda:us-west-2:XXXXXXXXXXXX:function:LambdaB",
"End":true,
"Type":"Task"
}
},
"StartAt":"Parallel Step 2, Process 2"
}
],
"Type":"Parallel",
"Next":"Final Processing"
}
},
"StartAt":"GoToState"
}You can then execute this state machine with the correct input by adding the “foo” and “resuming” variables:
{
"foo": 1,
"output": [
{
"Message": "Hello!"
},
{
"Message": "Hello!"
}
],
"resuming": true
}This yields the following result. Notice that this time, the state machine executed successfully to completion, and skipped the steps that had previously failed.
Conclusion
When you’re building out complex workflows, it’s important to be prepared for failure. You can do this by taking advantage of features such as automatic error retries in Step Functions and custom error handling of Lambda exceptions.
Nevertheless, state machines still have the possibility of failing. With the methodology and script presented in this post, you can resume a failed state machine from its point of failure. This allows you to skip the execution of steps in the workflow that had already succeeded, and recover the process from the point of failure.
To see more examples, please visit the Step Functions Getting Started page.
If you have questions or suggestions, please comment below.