Schedule a Serverless Workflow with AWS Step Functions and Amazon EventBridge Scheduler
Automating how you respond to operational events for resources in your AWS account can help reduce the operational overhead of your AWS infrastructure. As a real-world example, if you are a DevOps engineer or IT manager, you might want to automate Data processing & ETL jobs or Security and IT operations.
To solve this challenge, you can run a serverless workflow on a time-based schedule. Amazon EventBridge is a serverless event bus that helps you receive, filter, transform, route, and deliver events from AWS services, your own applications, and software-as-a-service (SaaS) applications. Many AWS services generate events that EventBridge receives. Amazon EventBridge Scheduler is a serverless scheduler that allows you to create, run, and manage tasks at scale from one central, managed service. With AWS Step Functions, you can define state machines that describe your workflow as a series of steps, their relationships, and their inputs and outputs.
This tutorial shows you how to invoke a state machine using EventBridge Scheduler based on the schedule you define. While this tutorial invokes a workflow on a time-based schedule, it can be generalized to integrate with other event-generating AWS services.
You will use AWS Step Functions and Amazon EventBridge Scheduler in this tutorial, both of which are within the AWS Free Tier.
What you will accomplish
- Create and define a state machine in AWS Step Functions
- Create a schedule using Amazon EventBridge Scheduler
- Customize how your state machine handles input and output
- Confirm your state machine workflow is working as expected
a. Open the AWS Management Console in another browser window, so you can keep this step-by-step guide open. When the screen loads, enter your user name and password to get started. Then enter IAM in the search bar and select IAM to open the service console.
b. Select Roles in the left navigation pane, then choose Create role.
c. On the Select trusted entity page, under AWS service, select Step Functions from the list, and then choose Next.
d. On the Add permissions page, choose Next.
e. On the Review page, enter step_functions_basic_execution for Role name and choose Create role.
f. Your new IAM role is created.
a. AWS Step Functions offers various predefined state machines as templates. For this tutorial, you will create a state machine using the Hello World template. Navigate to the State machines section of the Step Functions console, and choose Create state machine.
c. In the Specify details section, enter the State machine name as ScheduledWorkflow, then, under Permissions > Execution role, select Choose an existing role. Under Existing roles, select the IAM role you created earlier, step_functions_basic_execution. Choose Create state machine.
d. Your state machine is now ready to be executed.
b. On the Schedules page, choose Create schedule.
c. On the Specify schedule detail page, do the following:
- Enter a Schedule name such as ExecuteStateMachine and a Description (optional) such as Execute state machine every minute.
- In the Schedule pattern section, choose Recurring schedule, then choose Rate-based schedule.
- In the Rate expression section, enter the number 1 for Value, then choose minutes for Unit from the dropdown options.
- For Flexible time window, choose Off to turn off the option. Leave all other options as default and choose Next.
d. On the Select target page, do the following:
- In the Target detail section, choose AWS Step Functions from the Frequently used APIs options.
- In the StartExecution section, choose ScheduledWorkflow from the State machine dropdown.
- Update the Input with the following JSON code.
"Payload": "This is a test."
4. Choose Next.
e. On the Settings page, leave all default values. Amazon EventBridge will automatically create a new role with required permissions. Choose Next.
f. On the Review and create schedule page, choose Create schedule.
g. A new execution of your state machine will now run every minute.
a. Back in the Step Functions console, your ScheduledWorkflow state machine should still be displayed. Choose Edit.
b. In the Definition window, refer to lines 7 and 12 of your state machine. The Result field in the Pass state specifies the output that should be passed on to the next state. Since your state machine consists of two states executing sequentially, the message “Hello!” will be passed from the first Hello state to the World state, and then subsequently the message “World” will be passed to the end state of your workflow.
c. You may recall that a Step Functions Pass state simply passes its input to its output. You can use the ResultPath field to modify the output of a state. For example, you can replace the state input with the result it produces (for example, output from a Task state’s Lambda function).
To combine the state’s input with its result, add new lines to both states as follows:
"Comment": "A Hello World example of the Amazon States Language using Pass states",
This will combine the details of the EventBridge Event that invoked the state machine execution with the output of each state. For more information, see Input and Output Processing in Step Functions.
d. Choose Save.
b. Under Executions, you can observe the executions that are fired by EventBridge Scheduler. You can select the refresh button to update the window to observe new executions of your state machine appear. Wait a couple of minutes to allow at least two instances of your workflow to be initiated.
c. Open the context (right-click) menu of one of the executions and open the link in a new browser tab. Select the Execution input and output tab.
You can see that the input event was combined with the output of the state machine.
a. Close the tabs for your individual state machine executions. In the upper left of the Step Functions window, choose State machines. In the State machines window, select the state machine you created for this tutorial and select Delete.
b. Confirm the action by entering delete in the text field and choosing Delete in the Delete state machine dialog box. Your state machine will be deleted in a minute or two once Step Functions has confirmed that any in-process executions have completed.
c. Next, you will delete your Amazon EventBridge Scheduler schedule. In the EventBridge Scheduler console, choose Actions > Delete. Confirm the deletion by choosing Delete again.
d. Choose Services in the upper left and then enter IAM in the search bar to navigate to the IAM console.
e. Select Roles in the left navigation pane.
f. Select both of the IAM roles that you created for this tutorial, then choose Delete. Confirm the deletion by entering delete in the text input field and choosing Delete in the dialog box.
You can now sign out of the AWS Management Console.
Congratulations! You have successfully scheduled a serverless workflow to run every minute. Using AWS Step Functions, you can inspect and audit execution of recurring tasks to confirm that they are executed consistently. By combining AWS Step Functions with Amazon EventBridge Scheduler, you can automate daily, weekly, and monthly tasks.