Implementing the Saga Orchestration pattern with Amazon EKS and Amazon SNS

This blog post proposes an ecommerce scenario with an Orders microservice, an Orders Rollback microservice and an Inventory microservice that communicate with each other. This communication happens while raising an order successfully or rolling back an order when the Inventory microservice reports an error. This communication is orchestrated with Amazon Simple Notification Service (Amazon SNS) and Amazon Simple Queue Service (Amazon SQS) where the SQS queues serve as the subscriptions of Amazon SNS topics.

The Saga Orchestration pattern

The Saga patterns are used to bring multiple microservices together for concluding an end-to-end business process. The Saga patterns can be applied for all business processes that span across multiple microservices. The Saga pattern is described in two different ways – Choreography and Orchestration, which we’ll cover in this blog post.

In the Saga Orchestration pattern, microservices communicate with each other in a sequence with an intermediary, much like a conductor in an opera who conducts the musicians to play in a particular sequence.

A key feature of the Saga Orchestration pattern is the presence of one or more ‘compensating’ microservices. These microservices undo the effect on a state store made by a prior microservice. For example, in a business process for ordering books online, the microservice for inventory will adjust inventory database, MySQL. But, if the payment microservice, of this business process, was to fail, then a compensating microservice will readjust the inventory so that the overall state of library is restored.

Saga Orchestration – AWS architecture

The following diagram describes the AWS services in the architecture for the Saga Orchestration pattern.

1. The architecture is deployed in to a single Region with three VPCs where, the Control plane and Data Plane of Amazon EKS will be in separate VPCs. The Amazon Relational Database Service (Amazon RDS) VPC will host the MySQL database.
2. The Amazon EKS cluster is launched with following considerations.
   1. Worker nodes are launched in public subnets only for simplicity.
   2. Launched with m5.large Amazon Elastic Compute Cloud (Amazon EC2) instances, as general-purpose instances are sufficient for this demo.
3. The Amazon RDS for MySQL instance is launched with the following considerations.
   1. Launched with public access turned off.
   2. Launched with the default security group whose inbound rules are edited to accept traffic from the public IP address of the EC2 instances. These EC2 instances are the worker nodes of Amazon EKS launched in public subnets. See the previous step 2.
   3. The MySQL database will create a user with AWSAuthenticationPlugin as Amazon RDS so that the containers can connect with IAM role then password. Read Creating a database account using IAM authentication for setting up authentication in Amazon RDS. Read IAM roles for service accounts to learn how to enable containers to connect Amazon RDS with an IAM role.
4. The Amazon SQS queues are standard queues as strict order of messages is not mandatory for this demo. The messages are stored in the queues for 300 seconds to avoid piling up of messages.
5. Policies for connecting with Elastic Load Balancing, Amazon SNS, Amazon SQS, and Amazon RDS are defined in AWS IAM.
6. Application load balancing on Amazon EKS exposes pod outside of the cloud over internet for receiving HTTP traffic.

Saga Orchestration – Kubernetes architecture

This section describes the Kubernetes artifacts that were developed for the Saga Orchestration pattern.

1. There are six pods in all.
   1. Orders microservice receives JSON input over HTTP POST on /orders with JSON payload. A successful order is inserted into the orders table of the MySQL database. After processing the JSON input, a message is written to Amazon SNS topics for success or failure as the case maybe.
   2. Orders Rollback microservice receives failed inventory update, as input, over Amazon SQS queue. This Amazon SQS queue is written by the Orchestrator microservice. The input message is processed to delete the order from the orders table of the MySQL database.
   3. Inventory microservice receives successful orders via Amazon SQS queue. The Orchestrator microservice writes this queue to Amazon SQS. The input message is used to adjust the inventory based on order quantity. If the resulting stock level is too low, the inventory level is not updated and a message is published into Amazon SNS topic for inventory failure. Else, the inventory level is updated with the new value and a message is published into Amazon SNS topic for inventory success.
   4. Orchestrator microservice processes success and failure messages from Orders and Inventory microservices. These messages are read from the Amazon SQS queue for Orchestrator input where this queue is a subscription of Amazon SNS topic for successful orders and failed inventory update. Based on the routing logic in the JSON document for orchestration, the Orchestrator microservice will forward the message from the ‘sending’ microservice to the ‘receiving’ microservice by writing into the appropriate Amazon SQS queue. The Amazon SQS queue URL is provided in the JSON document for orchestration. The JSON document is itself provided as the value of a key named ORCHESTRATION in ConfigurationMap/eks-saga-orchestrationobject.
   5. Audit microservice receives inputs over Amazon SQS queue of all success and failed updates from Orders, Orders Rollback, and Inventory microservice.
   6. Trail microservice receives input over HTTP GET on /trail/{orderId} to respond with current status of order.
2. Since Orders and Trail microservice expose HTTP endpoints, these microservices use Service and Ingressobjects. The Ingress object uses AWS Load Balancer Controller to route the traffic to pods. The Ingress objects use an Ingress group so that a single AWS ALB instance can cater to both the microservices.
3. All of the microservices use ConfigurationMap objects to parameterize the database details (host, port, and database name), SNS topics, and SQS queues as applicable.
4. All microservices use a ServiceAccount object to connect to Amazon RDS, Amazon SQS, and Amazon SNS. This ServiceAccount object also allows the usage of IAM role for service accounts for the microservices to connect to Amazon RDS with an IAM role rather than a password.

Demo

For installation instructions, please visit our GitHub.

Running the demo – success path

After the installation is completed, run the following commands to simulate the success path.

1. Run the following command to get the HTTP endpoint of the Ingress object.

INGRESS_ADDR=`kubectl -n eks-saga get ing/eks-saga-orders -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'`

2. Create a new order with the following command. Ensure that the order quantity is less than 40 so that we can also test the failure scenario.

curl -X POST -H "Content-Type: application/json" -d '{"order_sku_id": 2,"order_qty": 10,"order_price": 100.50}'  http://${INGRESS_ADDR}/eks-saga/orders | jq . 

3. An HTTP response would be received with status code 201. The response payload would look like this. {"poll":"/eks-saga/trail/3"}
4. Check the status of the order with the following command replacing {orderId} with the order ID as received in the preceding HTTP response.

curl -X GET -H "Accept: application/json" http://${INGRESS_ADDR}/eks-saga/trail/3 | jq .

5. The response from the Trail microservice clearly shows that the Order microservice and Inventory microservice ran in sequence to conclude the creation of order.

6. Get logs from the Orders microservice pod.
   1. Run this command to get the pod name of Orders microservice. kubectl -n eks-saga get po
   2. Run this command to get logs of the Orders microservice pod. kubectl -n eks-saga logs eks-saga-orders-6569895-xhhq5. The command output maybe filtered for message to reduce display with grep message.
   3. The following screenshot shows the logs from Orders microservice pod.
7. Analysis of logs from Orders microservice.
   1. The Request ID in all the log lines is the trace ID (X-Amzn-Trace-Id) inserted by AWS ALB.
   2. The first two log lines indicate that the IAM token was obtained and used to connect to MySQL successfully. This demonstrates the IAM role for service accounts feature.
   3. The third log line indicates that an order was raised with order number 5.
   4. The final log line shows the message ID that was published to the Amazon SNS success topic for Orders microservice.
8. Get logs from Inventory microservice.
   1. Run this command to get the pod name of Inventory microservice. kubectl -n eks-saga get po
   2. Run this command to get logs of the Inventory microservice pod. kubectl -n eks-saga logs eks-saga-inventory-77f54b8697-6fsdl. The command output maybe filtered for message to reduce display with grep message.
   3. The following screenshot shows the logs from Inventory microservice pod.

9. Analysis of logs from Inventory microservice.
   1. The RequestId in all the log lines is the trace ID (X-Amzn-Trace-Id) inserted by AWS ALB and published by Orders microservice into Amazon SNS success topic. The Inventory microservice receives this ID through an Amazon SQS queue, which is a subscription of the Amazon SNS success topic.
   2. The first log line describes that a successful order has been received. This message is read from the Amazon SQS queue for inventory input. This queue was written to by the Orchestrator microservice as it is subscribed to Amazon SNS topic for orders success. Note the Request ID and MessageId – they are the same as published by the Orders microservice.
   3. The second and third log lines indicate that the IAM token was obtained and used to connect to MySQL successfully. This demonstrates the IAM role for service accounts feature.
   4. The fourth log line indicates that the inventory was correctly adjusted.
   5. The fifth log line indicates that the Inventory microservice published a success message on the Amazon SNS topic for success of Inventory microservice.
   6. The sixth log line indicates that the input message was deleted so that the processing is now complete.
10. Get logs from Orchestrator microservice.
    1. Run this command to get the pod name of Orchestrator microservice. kubectl -n eks-saga get po
    2. Run this command to get logs of the Orchestrator microservice pod. kubectl -n eks-saga logs eks-saga-orchestrator-6b969d4599-nd4hg. The command output maybe filtered for message to reduce display with grep message.
    3. This screenshot shows the logs from Orchestrator microservice pod.
11. Analysis of logs from Orchestrator microservice.
    1. The RequestID in all the log lines is the trace ID (X-Amzn-Trace-Id) inserted by AWS ALB and published by Orders microservice into Amazon SNS success topic. The Orchestrator microservice receives this ID through an Amazon SQS queue, which is a subscription of the Amazon SNS success topic for orders.
    2. The first log line describes that a successful order has been received by the Orchestrator microservice, as it is subscribed to Amazon SNS topic for orders success. Note the Request ID and MessageId – they are the same as published by the Orders microservice.
    3. The second log line indicates that the message was published on to the Amazon SQS queue for Inventory input. The name of the queue is derived from the JSON configuration file for orchestration.
    4. The third log line indicates that the input message was deleted so that the processing is now complete.

Running the demo – failure path

The ‘failure path’ is demonstrated in a scenario where the inventory level is too low to fulfill an order requirement. In such a scenario, the Inventory microservice raises an error, which is consumed by the Orders Rollback microservice to delete the order from MySQL table. To simulate this failure, multiple orders might have to be raised (explained in the following steps) to reach the low level of inventory.

Run the following commands to simulate the failure path.

1. Run the following command to get the HTTP endpoint of the Ingress object.

INGRESS_ADDR=`kubectl -n eks-saga get ing/eks-saga-orders -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'`

2. Create a new order with the following command. Ensure that the order quantity is less than 40 so that we can test the failure scenario.

curl -X POST -H "Content-Type: application/json" -d '{"order_sku_id": 2,"order_qty": 40,"order_price": 100.50}'  http://${INGRESS_ADDR}/eks-saga/orders  | jq .

3. An HTTP response would be received with status code 201. The response payload would look like this. {"poll":"/eks-saga/trail/5"}
4. Check the status of the order with the following command replacing {orderId} with the order ID as received in the preceding HTTP response.

curl -X GET -H "Accept: application/json" http://${INGRESS_ADDR}/eks-saga/trail/5  | jq .

5. The response from the Trail microservice clearly shows that the Order microservice and Inventory microservice ran in sequence. While the Order microservice went through successfully, the Inventory microservice responded with an error. Therefore, the Orders Rollback microservice took over to delete the Order.
6. Get logs from the Orders microservice pod.
   1. Run this command to get the pod name of Orders microservice. kubectl -n eks-saga get po
   2. Run this command to get logs of the Orders microservice pod. kubectl -n eks-saga logs po/eks-saga-orders-6569895-xhhq5. The command output maybe filtered for message to reduce display with grep message.
7. Analysis of logs from Orders microservice.
   1. The Request ID in all the log lines is the trace ID (X-Amzn-Trace-Id) inserted by AWS ALB.
   2. The first two log lines indicate that the IAM token was obtained and used to connect to MySQL successfully. This demonstrates the IAM role for service accounts feature.
   3. The third log line indicates that an order was raised with order number 5.
   4. The final log line shows the message ID that was published to the Amazon SNS success topic for Orders microservice.
8. Get logs from Inventory microservice.
   1. Run this command to get the pod name of Inventory microservice. kubectl -n eks-saga get po
   2. Run this command to get logs of the Inventory microservice pod. kubectl -n eks-saga logs po/ eks-saga-inventory-77f54b8697-6fsdl. The command output maybe filtered for message to reduce display with grep message.
9. Analysis of logs from Inventory microservice.
   1. The RequestID in all the log lines is the trace ID (X-Amzn-Trace-Id) inserted by AWS ALB and published by Orders microservice into Amazon SNS success topic. The Inventory microservice receives this ID through an Amazon SQS queue, which is a subscription of the Amazon SNS success topic.
   2. The first log line describes that a successful order has been received. This message is read from the Amazon SQS queue for inventory input. This queue was written to by the Orchestrator microservice as it is subscribed to Amazon SNS topic for orders success. Note the Request ID and MessageId – they are the same as published by the Orders microservice.
   3. The second and third log lines indicate that the IAM token was obtained and used to connect to MySQL successfully. This demonstrates the IAM role for service accounts feature.
   4. The fourth log line indicates that the inventory could not be adjusted because the level of inventory, for that SKU, was too low.
   5. The fifth log line indicates that the Inventory microservice published a failure message on the Amazon SNS topic for failure of Inventory microservice.
   6. The sixth log line indicates that the input message was deleted so that the processing is now complete.
10. Get logs from the Orders Rollback microservice.
    1. Run this command to get the pod name of Orders Rollback microservice. kubectl -n eks-saga get po
    2. Run this command to get logs of the Inventory microservice pod. kubectl -n eks-saga logs po/eks-saga-orders-rb-77f69f476-6d9kl. The command output maybe filtered for message to reduce display with grep message.
11. Analysis of logs from Orders Rollback microservice.
    1. The RequestID in all the log lines is the trace ID (X-Amzn-Trace-Id) inserted by AWS ALB and published by Orders microservice into Amazon SNS success topic. The Inventory microservice receives this ID through an Amazon SQS queue, which is a subscription of the Amazon SNS success topic.
    2. The first log line describes that an order for rolling back has been received. Note the Request ID and MessageId – they are the same as published by the Orders microservice and processed by the Inventory microservice.
    3. The second and third log lines indicate that the IAM token was obtained and used to connect to MySQL successfully. This demonstrates the IAM role for service accounts feature.
    4. The fourth log line indicates that the order was deleted.
    5. The fifth log line indicates that the Orders Rollback microservice published a success message on the Amazon SNS topic for success of Orders Rollback microservice.
    6. The sixth log line indicates that the input message was deleted so that the processing is now complete.
12. Get logs from Orchestrator microservice.
    1. Run this command to get the pod name of Orchestrator microservice. kubectl -n eks-saga get po
    2. Run this command to get logs of the Orchestrator microservice pod. kubectl -n eks-saga logs eks-saga-orchestrator-6b969d4599-nd4hg. The command output maybe filtered for message to reduce display with grep message.
    3. The following screenshot shows the logs from Orchestrator microservice pod.
13. Analysis of logs from Orchestrator microservice.
    1. The RequestID in all the log lines is the trace ID (X-Amzn-Trace-Id) inserted by AWS ALB and published by Orders microservice into Amazon SNS success topic. The Orchestrator microservice receives this ID through an Amazon SQS queue, which is a subscription of the Amazon SNS success topic for orders.
    2. The first log line describes that a successful order has been received. This message is read from the Amazon SQS queue for inventory input. This queue was written to by the Orchestrator microservice as it is subscribed to Amazon SNS topic for orders success. Note the Request ID and MessageId – they are the same as published by the Orders microservice.
    3. The second log line indicates that the message was published on to the Amazon SQS queue for Inventory input. The name of the queue is derived from the JSON configuration file for orchestration.
    4. The third log line indicates that the input message was deleted so that the processing is now complete.
    5. The fourth log line indicates that message was received.
    6. The fifth log line indicates that the message was published to the Orders Rollback Amazon SQS queue because the message was received from the Inventory microservice and is a failure message. The name of the queue is derived from the JSON configuration file for orchestration.
    7. The sixth log line indicates that the input message was deleted so that the processing is now complete.

Conclusion

This blog described the Saga Orchestration pattern for coordinating Orders, Orders Rollback, and Inventory microservices with Amazon SNS passing messages between microservices. This pattern can be applied in digital transformation projects where features are implemented as microservices. As a next step, consider other options for passing messages between microservices, such as Amazon Managed Streaming for Apache Kafka (Amazon MSK) or a fully Kubernetes native solution with gRPC.