Getting Started / Hands-on / ...
Orchestrate Queue-based Microservices with AWS Step Functions and Amazon SQS
TUTORIAL
Introduction
In this tutorial, you will learn how to use AWS Step Functions and Amazon SQS to design and run a serverless workflow that orchestrates a message queue-based microservice. Step Functions is a serverless orchestration service that lets you easily coordinate multiple AWS services into flexible workflows that are easy to debug and easy to change. Amazon SQS is the AWS service that allows application components to communicate in the cloud.
This tutorial will simulate inventory verification requests from incoming orders in an e-commerce application as part of an order processing workflow. Step Functions will send inventory verification requests to a queue on SQS. An AWS Lambda function will act as your inventory microservice that uses a queue to buffer requests. When it retrieves a request, it will check inventory and then return the result to Step Functions. When a task in Step Functions is configured this way, it is called a callback pattern. Callback patterns allow you to integrate asynchronous tasks in your workflow, such as the inventory verification microservice of this tutorial.
AWS Experience
Intermediate
Time to Complete
10 minutes
Cost to Complete
Free Tier
Services Used
This Tutorial
Requires an AWS Account
There are no additional charges for AWS Step Functions or Amazon SQS. The resources you create in this tutorial are Free Tier eligible.
Implementation
-
Enter the Amazon SQS Console
a. Open the AWS Management Console, so you can keep this step-by-step guide open. When the screen loads, enter your user name and password to get started. In the search text box, type SQS and select Simple Queue Service to open the service console.
b. If the SQS console landing page appears, as shown on by the screenshot, click Get Started Now. If you don't see this page, skip to the next step.
-
Create an Amazon SQS Queue
In this step, you will create and configure an Amazon SQS queue. A queue is a reliable, highly-scalable buffer that stores messages as they travel between distributed applications or microservices. Queues help to decouple applications, connect microservices, batch tasks, or store notifications.
Our use case for SQS in this tutorial will simulate the storage of inventory verification requests from incoming orders in an e-commerce application.
a. First, we will create a simple queue that stores orders that are placed on the store. Enter Orders in the Queue Name field.
b. For this tutorial, we do not require strict ordering, so we won’t make any changes to the queue type. Leave Standard Queue selected.
c. You can configure your queue to modify settings such as retention period, maximum message size and delivery delays. For this tutorial, we will keep the default parameters. Choose Quick-Create Queue.
d. Your new queue is created and selected in the queue list.
-
Create a Workflow with a State Machine
Your next step is to design a workflow that describes how you want e-commerce orders to be processed. Workflows describe a process as a series of discrete tasks that can be repeated again and again.
You’ll design your workflow in AWS Step Functions. Your workflow will request verification of inventory from a microservice. Many microservices uses a queue to receive requests. In this tutorial, you will use an AWS Lambda function to represent the microservice.
a. Open the AWS Step Functions console. Select Author with code snippets, then name your state machine InventoryStateMachine.
b. Replace the contents of the State machine definition window with the Amazon States Language (ASL) state machine definition below. Amazon States Language is a JSON-based, structured language used to define your state machine.
This state machine uses a task state to put a message on an SQS queue. This task state is configured for a callback pattern. When you append .waitForTaskToken to your resource, Step Functions will add a task token to the JSON payload and wait for a callback. The microservice can return a result to Step Functions by calling the Step Functions API.
{ "Comment": "An example of the Amazon States Language for starting a callback task.", "StartAt": "Check Inventory", "States": { "Check Inventory": { "Type": "Task", "Resource": "arn:aws:states:::sqs:sendMessage.waitForTaskToken", "Parameters": { "QueueUrl": "<INSERT SQS QUEUE URL HERE>", "MessageBody": { "MessageTitle": "Callback Task started by Step Functions", "TaskToken.$": "$$.Task.Token" } }, "Next": "Notify Success", "Catch": [ { "ErrorEquals": [ "States.ALL" ], "Next": "Notify Failure" } ] }, "Notify Success": { "Type": "Pass", "Result": "Callback Task started by Step Functions succeeded", "End": true }, "Notify Failure": { "Type": "Pass", "Result": "Callback Task started by Step Functions failed", "End": true } } }
c. Copy the URL of your SQS queue from the SQS console and paste into your State Machine definition.
d. Click the refresh button to have Step Functions translate the ASL state machine definition into a visual workflow. You are able to easily verify that the process is described correctly by reviewing the visual workflow. When the microservice returns a result, the state machine will progress down the success branch. If something goes wrong, you can catch the exception and go down the right branch to take corrective action.
Click Next.
e. Next, you will add an IAM role to your workflow. Select Create an IAM role for me and name it inventory-state-machine-role. Step Functions will analyze your workflow and generate an IAM policy that includes resources used by your workflow. Click Create state machine. You should see a green banner indicating your state machine was successfully created.
-
Create an AWS Identity and Access Management (IAM) Role
AWS IAM is a web service that helps you securely control access to AWS resources. In this step, you will create an IAM role that allows Step Functions to access SQS.
a. In another browser window, open the AWS Management Console. When the screen loads, type IAM in the search bar, then select IAM to open the service console.
b. Click Roles and then click Create Role.
c. On the Create Role screen, leave AWS Service selected, select Lambda then click Next: Permissions.
d. On the Create role screen, attach AmazonSQSFullAccess and AWSStepFunctionsFullAccess policies. Click Next: Tags and then click Next: Review.
e. Enter Role name as inventory-lambda-role and click Create role.
-
Create your Microservice with AWS Lambda Functions
In this step, you will create a Lambda function that will mock an inventory microservice. The Lambda function retrieves messages from SQS and will return a message to Step Functions that represents the result of the request.
a. Click on Services, type Lambda in the search bar, then select Lambda to open the service console.
b. Choose Create function.
c. Leave Author from scratch selected. Next, configure your first Lambda function as follows:
For Name, type Inventory.
For Runtime, choose Node.js 8.10.
For Role, select Use an existing role.Select inventory-lambda-role from the list
Click Create function.
d. Replace the contents of the Function code window with the following code, and then click Save.
console.log('Loading function'); const aws = require('aws-sdk'); exports.handler = (event, context, callback) => { const stepfunctions = new aws.StepFunctions(); for (const record of event.Records) { const messageBody = JSON.parse(record.body); const taskToken = messageBody.TaskToken; const params = { output: "\"Callback task completed successfully.\"", taskToken: taskToken }; console.log(`Calling Step Functions to complete callback task with params ${JSON.stringify(params)}`); stepfunctions.sendTaskSuccess(params, (err, data) => { if (err) { console.error(err.message); callback(err.message); return; } console.log(data); callback(null); }); } };
e. Select the SQS tigger. Set the toggle to Enable for your Orders queue. Click Add and then click Save.
-
Execute your workflow
Your serverless workflow is ready to be executed! A state machine execution is an instance of your workflow, and occurs each time a Step Functions state machine runs and performs its tasks. Each Step Functions state machine can have multiple simultaneous executions, which you can initiate from the Step Functions console (which is what you’ll do next), or using the AWS SDKs, the Step Functions API actions, or the AWS CLI. An execution receives JSON input and produces JSON output.
a. Click on Services, type Step in the search bar, then select Step Functions to open the service console.
b. On the State machines screen, click on your InventoryStateMachine.
c. Click on Start execution.
d. A new execution dialog box appears, where you can enter input for your state machine. This state machine does not depend on the input. You can use the default input. Click on Start execution.
e. As your workflow executes, each step will change color in the Visual workflow pane. Wait a few seconds for execution to complete. Then, in the Execution details pane, click Input and Output to view the inputs and results of your workflow.
f. Step Functions lets you inspect each step of your workflow execution, including the inputs and outputs of each state. Click on each task in your workflow and expand the Input and Output fields under Step details. You can see that the input payload into your state machine is passed from each step to the next, and that the payload is updated as each step completes.
g. Scroll down to the Execution event history section. Click through each step of execution to see how Step Functions called SQS and passed payloads between steps.
h. Depending on the output of your InventoryStateMachine, your workflow may have ended in success or failure.
In a real world scenario, you might to decide to retry the task or to take other actions, depending on the error message you received.
-
Terminate your Resources
In this step you will terminate your AWS Step Functions and AWS Lambda related resources.
Important: Terminating resources that are not actively being used reduces costs and is a best practice. Not terminating your resources can result in a charge.a. At the top of the AWS Step Functions console window, click on State machines.
b. In the State machines window, click on your InventoryStateMachine and select Delete. Confirm the action by selecting Delete state machine in the dialog box. Your state machine will be deleted in a minute or two once Step Functions has confirmed that any in process executions have completed.
c. Next, you’ll delete your Lambda functions. Click Services in the AWS Management Console menu, then select Lambda.
d. In the Functions screen, click the Inventory function you created for this tutorial and then select Actions and then Delete. Confirm the deletion by clicking Delete again.
e. Lastly, you’ll delete your IAM roles. Click Services in the AWS Management Console menu, then select IAM.
f. Select the IAM roles that you created for this tutorial, then click Delete role. Confirm the delete by clicking Yes, Delete on the dialog box.
g. In the queue list, select the Orders queue. Then, from Queue Actions, select Delete Queue.
h. The Delete Queues dialog box is displayed. You can delete your queue, even though you may still have messages in it. Choose Yes, Delete Queue. The queue is deleted.
Congratulations!
You have orchestrated a microservice in the cloud with a message queue using AWS Step Functions and Amazon SQS. Step Functions is a great fit when you need to coordinate your application resources where productivity and agility are key considerations.