Schedule a Serverless Workflow

with AWS Step Functions and Amazon CloudWatch

Automating how you respond to operational events for resources in your AWS account can help reduce the operational overhead of your AWS Cloud infrastructure. As a real-world example, if you are a DevOps engineer or IT manager, you might want to automate patch management, ETL jobs, data synchronization, or security.

To solve this challenge, you can run a serverless workflow on a time-based schedule. Amazon CloudWatch Events provides a near real-time stream of system events that describe the changes and notifications for your AWS resources. From this stream, you can create rules to route specific events to AWS Step Functions and invoke a state machine to perform further processing. AWS Step Functions allows you to coordinate multiple services into serverless workflows so you can build and update automated processes quickly.

In this tutorial, you will use Step Functions to orchestrate a serverless workflow in response to a CloudWatch event. First, you’ll create a very simple state machine using AWS Step Functions. Then, you’ll use Amazon CloudWatch Events to set up a rule that runs your state machine every minute. While this tutorial invokes a workflow on a time-based schedule, it can be generalized to integrate with other event-generating AWS services.

You'll use AWS Step Functions and Amazon CloudWatch in this tutorial. Both services are within the AWS Free Tier.

This tutorial requires an AWS account

There are no additional charge for Step Functions or CloudWatch. The resources you create in this tutorial are Free Tier eligible. 

More about the Free Tier >>


Step 1. Create an AWS Identity and Access Management (IAM) role

AWS Step Functions can execute code and access other AWS resources (for example, data stored in Amazon S3 buckets). To maintain security, you must grant Step Functions access to these resources using AWS Identity and Access Management (IAM).


a. Open the AWS Management Console in another browser window, so you can keep this step-by-step guide open. When the screen loads, enter your user name and password to get started. Then type IAM in the search bar and select IAM to open the service console.

01a
01a

( click to enlarge )


b.  Click Roles, then choose Create Role.

01b
01b

( click to enlarge )


c. On the Select type of trusted entity page, under AWS service, select Step Functions from the list, and then choose Next: Permissions.

01c
01c

( click to enlarge )


d. On the Attach permissions policy page, choose Next: Review.

01d
01d

( click to enlarge )


e. On the Review page, type step_functions_basic_execution for Role name and click Create role.

01e
01e

( click to enlarge )


f. Your new IAM role is created.

01f
01f

( click to enlarge )


Step 2. Create an AWS Step Functions State Machine

In this step, you'll create a simple, independently running state machine using a Pass state. A Pass state simply passes its input to its output, and doesn’t actually perform any work.

In a real-world scenario, your serverless workflow can do whatever you need it to do. You could coordinate multiple Lambda functions to perform tasks, make decisions, and so on. For the sake of this tutorial though, we’re going to keep the workflow simple.


a. Click Services, and then type Step in the search bar. Select Step Functions to open the service console.

02a
02a

( click to enlarge )


b. If the Getting Started page appears, choose Get Started. Otherwise, click Create state machine.

02b
02b

( click to enlarge )


c. Step Functions offers various predefined state machines as templates. For this tutorial, you’ll create a state machine using the Hello World template. On the Create a state machine page, select Templates and then choose Hello world.

02c
02c

( click to enlarge )


d. In the Details section, enter the Name of your state machine as ScheduledWorkflow, then, under IAM Role, select I will use an existing role. Under Existing IAM roles, select the IAM role you created, step_functions_basic_execution.

02d
02d

( click to enlarge )


e. Step Functions fills in the name of the state machine automatically. It also populates the State machine definition pane with the Amazon States Language description of the state machine. For more information on how to define state machines, see State Machine Structure.

Let’s make some simple modifications to your state machine:

On line 2 and line 5, replace “HelloWorld” with “Do Something”
On line 7, replace “Hello World!” with “Work complete!”

Then, click the refresh button beside your visual workflow to update the state machine diagram to reflect your changes.

02e
02e

( click to enlarge )


f. Choose Create state machine. Your state machine is now ready to be executed.

02f
02f

( click to enlarge )


Step 3. Create an Amazon CloudWatch Events Rule

Now that you have created your state machine, you can create an Amazon CloudWatch Events rule that runs your state machine according to a schedule. Amazon CloudWatch Events delivers a near real-time stream of system events that describe changes in Amazon Web Services (AWS) resources. Using simple rules that you can quickly set up, you can match events and route them to one or more target functions or streams.


a. Open another browser tab. Navigate to the CloudWatch Events console by entering CloudWatch in the search bar.

03a
03a

( click to enlarge )


b. Choose Events, and then choose Create rule.  

03b
03b

( click to enlarge )


c. The Step 1: Create rule page is displayed. In the Event source section, select Schedule. In Fixed rate of enter 1 and leave the unit of “Minutes” selected.

03c
03c

( click to enlarge )


d. In the Targets section, choose Add target. Choose Step Functions state machine from the list and select your ScheduledWorkflow state machine.

Choose Configure input and select Matched event. This will pass the details of the CloudWatch Event as input to Step Functions each time it executes an instance of your state machine.

Select Create a new role for this specific resource and then click Configure details.

03d
03d

( click to enlarge )


e. The Step 2: Configure rule details page is displayed. Enter execute_state_machine as the Name for your rule, leave Enabled for State checked, and then choose Create rule.

The rule is created and the Rules page is displayed, listing all your CloudWatch Events rules.

A new execution of your state machine will now execute every minute.

03e
03e

( click to enlarge )


Step 4. Customize how your State Machine Handles Input and Output

Next, you’ll customize how your state machine handles input and output. Understanding how information flows from state to state, and learning how to filter and manipulate this data, will help you to effectively design and implement workflows in AWS Step Functions.


a. Back in the Step Functions console, your ScheduledWorkflow state machine should still be displayed. Click on Edit.

04a
04a

( click to enlarge )


b. In the State machine definition window, refer to line 7 of your state machine. The Result field in the Pass state specifies the output that should be passed on to the next state. Since your state machine only has one state, the message “Work complete!” will be passed to the end state of your workflow.

You may recall that a Step Functions Pass state simply passes its input to its output. You can use the ResultPath field to modify the output of a state. For example, you can replace the state input with the result it produces (for example, output from a Task state’s Lambda function).

To combine the state’s input with its result, add a new line after line 7 and type:

"ResultPath": "$.taskresult",

This will combine the details of the CloudWatch Event that invoked the state machine execution with the output of the Pass state, in this case, “Work complete!”. For more information, see Input and Output Processing in Step Functions.

04b
04b

( click to enlarge )


c. Click Save.

04c
04c

( click to enlarge )


Step 5. Confirm your Workflow is Working as Expected

Now that you have enabled your CloudWatch Events rule, CloudWatch is triggering an execution of your state machine workflow every minute. In this step, you will observe those executions and verify that the details of the CloudWatch Event are being captured by Step Functions.


a. Back in the Step Functions console, click on ScheduledWorkflow in the breadcrumb at the top of the page.

05a
05a

( click to enlarge )


b. Under Executions, you can observe the executions that are fired by CloudWatch Events. You can click the refresh button to update the window to observe new executions of your state machine appear. Wait a couple of minutes to allow at least two instances of your workflow to be triggered.

05b
05b

( click to enlarge )


c. Right-click on one of the executions and open the link in a new browser tab. Repeat for a different execution of your state machine.

On each tab, click the “Do Something” state in the Visual workflow pane to populate the Step details pane on the right.

By clicking Input, you can see the details of the matched event from CloudWatch. Note the differences in the events, such as the “id” and “time” fields.

05c
05c

( click to enlarge )


d. On each tab, expand Output under the Step details pane. You can see that the input was combined with the output of the "Do Something" state.

05d
05d

( click to enlarge )


Step 6. Terminate your Resources

In this step you will terminate your AWS Step Functions and Amazon CloudWatch resources.

Important: Terminating resources that are not actively being used reduces costs and is a best practice. Not terminating your resources can result in a charge.


a. Close the tabs for your individual state machine executions. At the top of the Step Functions window, click on State machines.

06a
06a

( click to enlarge )


b. In the State machines window, click on the state machine you created for this tutorial and select Delete. Confirm the action by selecting Delete state machine in the dialog box. Your state machine will be deleted in a minute or two once Step Functions has confirmed that any in process executions have completed.

06b
06b

( click to enlarge )


c. Next, you’ll delete your Amazon CloudWatch Events rule. In the CloudWatch console, click Actions and click Delete. Confirm the deletion by clicking Delete again.

06c
06c

( click to enlarge )


d. Click Services and then enter IAM in the search bar to navigate to the IAM console.

06d

( click to enlarge )


e. Click on Roles.

06e
06e

( click to enlarge )


f. Select both of the IAM roles that you created for this tutorial, then click Delete role. Confirm the delete by clicking Yes, Delete on the dialog box.

You can now sign out of the AWS Management console.

06f
06f

( click to enlarge )


Congratulations!

You have successfully scheduled a serverless workflow to run every minute.
AWS Step Functions allows you to inspect and audit execution of recurring tasks to confirm that they are executed consistently. By combining AWS Step Functions with Amazon CloudWatch Events, you can automate daily, weekly, and monthly tasks, or trigger execution of your Step Functions workflow when certain conditions are met.