AWS Lambda is a compute service that lets you run code without provisioning or managing servers. It enables your code to be triggered by many different event sources, including Amazon Simple Notification Service (Amazon SNS), Amazon Simple Store Service (Amazon S3), and Amazon Simple Queue Service (Amazon SQS). In this tutorial, you learn how to use Amazon SQS as an event source for Lambda.
To get started, you need an AWS account and an AWS Identity and Access Management (IAM) user with permission to use Amazon SQS, Lambda, and AWS Key Management Service (AWS KMS). These services are within the AWS Free Tier.
- In this example, you’re a developer who has been asked to create a simple banking application where all bank transactions are quickly and reliably reconciled. Because of the sensitive nature of the application, all data must be encrypted, and the application must be able to withstand message loss.
- To solve this challenge, you decide to use Amazon SQS and Lambda. Amazon SQS is ideal for transmitting a large volume of data, at any throughput, without losing messages or requiring other services to be available.
- Create an AWS KMS key to encrypt each message body using Amazon SQS server-side encryption (SSE). With the direct integration of Amazon SQS and Lambda, you no longer have to be concerned about how to read messages from queues, or how to scale your processing to ensure that all transactions are reconciled in a timely manner. Lambda processes the transactions at scale and performs the reconciliation.
- Create an SQS queue to capture the transactions. You will also create a dead-letter-queue to ensure that you don’t lose any transactions that might be corrupted or fail reconciliation. Dead-letter queues are important because they provide a way to deal with failures, debug applications, and reprocess failed transactions.
- Create a Lambda function that processes each message to complete the reconciliation. You will configure the function to use the source transaction queue as the event source.
Level: Beginner, Intermediate
Time to Complete Module: 25 Minutes
Services used: Amazon SQS, AWS Lambda
Follow the step-by-step instructions below to complete the tasks outlined above. Click on each step number to expand the section.
-
Step 1. Create an AWS Key Management Service (AWS KMS) Customer Master Key
Security first! Before you get started with Amazon SQS and Lambda, you create an AWS KMS customer master key (CMK) for the application. This key is used to configure encryption across Amazon SQS and Lambda, and ensure that all of your data is safely encrypted.
1. Open the AWS Management Console, so you can keep this step-by-step guide open. Enter your user name and password. In the search text box, enter KMS and choose Key Management Service to open the service console.
2. Choose Create a key.
3. In the first step, create an alias and description. For the alias, enter banking-key. In the description, enter Tutorial - Using Amazon Simple Queue Service as an Event Source for AWS Lambda.
4. Under the Advanced options, ensure that KMS is chosen under Key material origin.
5. Choose Next.
6. In Step 2 - Add tags, if desired, enter an optional tag key and value to help you better organize your encryption keys.
7. Choose Next.
8. In Step 3 - Define key administrative permissions, choose a user and/or a role that can administer the key. Choose your IAM user (assuming that you have Administrator permissions).
9. In Step 4 - Define key usage permissions, ensure that your user and/or role is selected, and choose Next.
10. In the final Step 5 - Review and edit key policy, choose Finish. You have now created a KMS key to secure your messages.
-
Step 2. Create a Dead-Letter Queue
Your next step is to create the dead-letter queue for your application. You use a dead-letter queue to set aside transactions that couldn’t be processed. This could happen if the messages are poorly formed or corrupted, or the transaction reconciliation failed for some other unknown reason.
1. Open the AWS Management Console, so you can keep this step-by-step guide open. Enter your user name and password. In the search text box, enter SQS and choose Simple Queue Service to open the service console.
2. If you’ve never created a queue before, choose the Get Started Now button on the service console home page. Otherwise, choose Create New Queue.
3. Enter the queue name banking-dl-queue.
4. Choose Standard Queue for the type of queue you want to configure because this queue doesn’t need ordering (FIFO) capability.
5. Choose Configure Queue.
6. On the following configuration page, set these queue attributes for Server-Side Encryption (SSE) Settings:
- Use SSE check box = Selected
- AWS KMS Customer Master Key (CMK) = banking-key
7. Choose Create Queue.
-
Step 3. Create a Banking Transaction Queue
In this step, you create your primary processing queue. You use this queue as the event source for your Lambda function.
1. Choose Create New Queue.
2. For Queue Name enter banking-queue.
3. Choose Standard Queue for the type of queue that you want to configure.
4. Choose Configure Queue.
5. On the configuration page shown after step 7, set the following Queue Attributes:
- Default Visibility Timeout = 30 seconds (This is the amount of time that a message is invisible so that it isn’t processed by another functions).
- Message Retention Period = 1 day (This is the amount of time that Amazon SQS retains a message before it gets deleted).
- Receive Message Wait Time = 20 seconds (This is the amount of time that a long poller waits before it returns an empty response to the function).
As the developer, you expect the reconciliation to take less than a second. Following the guidance for configuring a queue for use with Lambda in the AWS Lambda Developer Guide, you set the visibility timeout to at least six times the timeout that you configure on your function. You set the timeout (5 seconds) for your function in Step 6. Configure KMS and Timeout Settings for banking-reconciliation-function.
6. Configure Dead Letter Queue Settings.
- Use Redrive Policy check box = Selected
- Dead Letter Queue = The name of the dead-letter queue that you created earlier (banking-dl-queue)
- Maximum Receives = 5 (This is the number of times a message can be received or attempted before it’s sent to the dead-letter queue).
7. Configure Server-Side Encryption (SSE) Settings.
- Use SSE check box = Selected
- AWS KMS Customer Master Key (CMK) = banking-key
8. Choose Create Queue.
-
Step 4. Create a Lambda Function
Now that you’ve created your queues, you create a Lambda function that processes the transactions that arrive on the banking-queue. The function code simulates a reconciliation. The main focus is on understanding how to implement the AWS SDK for Python (Boto3) to process messages, not the fictional business logic.
1. Open the AWS Lambda console in a new tab or window.
2. Choose Create function.
3. Choose Author from scratch.
4. Set the Basic information for the function.
- Function Name = banking-reconciliation-function
- Runtime = Python 3.6
5. Set the execution role by choosing Create a new role from AWS policy templates.
6. For Role name, choose banking-reconciliation-role.
7. From the Policy templates dropdown, choose both Amazon SQS poller permissions and AWS KMS decryption permissions.
8. Choose Create function.
-
Step 5. Configure the Event Source for banking-reconciliation-function
Your function Designer panel should now have the AWS Key Management Service policy listed in the function’s execution role. You now create the Amazon SQS event source that triggers function execution.
1. Choose the Add trigger button to the left of the execution roles. The Add trigger page opens.
2. Under Trigger configuration, in the dropdown menu of triggers, choose SQS.
3. For SQS queue, choose the Amazon Resource Name (ARN) of the banking-queue.
4. For Batch size, choose 10. This is the largest number of records that can be read from your queue at once.
5. Ensure that Enable trigger is selected.
6. Choose Add.
-
Step 6. Configure KMS and Timeout Settings for banking-reconciliation-function
You can now set the rest of the function configuration.
1. On the banking-reconciliation-function screen, scroll down to to the Basic settings panel.
2. Set the following properties:
- Description = Tutorial - Using Amazon Simple Queue Service as an Event Source for AWS Lambda.
- Memory = 128 MB.
- Timeout = 5 seconds.
3. In the Environment Variables panel, expand the function's Encryption configuration.
a. Under AWS KMS key to encrypt at rest, choose Use a customer master key.
b. Choose the banking-key.
4. To save the function, choose Save.
-
Step 7. Add the Code for banking-reconciliation-function
Now that you’ve configured the function, it’s time to add some code.
Under normal circumstances, you might be deploying this code through a CI/CD pipeline, using frameworks such as the AWS Serverless Application Model (AWS SAM), and using services such as AWS CodeCommit, AWS CodeBuild and AWS CodePipeline. However, for this tutorial, you’re editing the code directly in the browser.
1. Copy and paste the code that follows step 2 into the Function code panel for the banking-reconciliation-function.
2. Save the function.
import json import boto3 import random def lambda_handler(event, context): """Sample pure Lambda function Parameters ---------- event: dict, required SQS Message Event doc: https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html context: object, required Lambda Context runtime methods and attributes Context doc: https://docs.aws.amazon.com/lambda/latest/dg/python-context-object.html """ try: for record in event['Records']: body = json.loads(record["body"]) bank_schedule = body["bank_schedule"] account = record['messageAttributes']['Account']['stringValue'] reconcile_transaction(account, bank_schedule) except Exception as e: # Send some context about this error to Lambda Logs print(e) # throw exception, do not handle. Lambda will make message visible again. raise e def reconcile_transaction(account, bank_schedule): """reconciles the bank schedule for a specified account Arguments: account {number} -- account number to reconcile bank_schedule {dict} -- the schedule for the bank statement Raises: ArithmeticError: Cannot compute negative number Exception: Custom exception """ books_schedule = get_books_schedule_for_account(account) balance_per_books = books_schedule["balance_per_books"] balance_per_statement = bank_schedule['balance_per_statement'] print(f"Balance per Statement={balance_per_statement}; Balance per Books={balance_per_books}") if balance_per_statement < 0: raise ArithmeticError("Cannot be negative") elif balance_per_statement > balance_per_books: raise Exception("Schedules cannot be reconciled") else: print(f"Successfully reconciled {account}") def get_books_schedule_for_account(account): print(f"Getting Books Schedule for account {account}") # Generate some random figures balance_per_books = random.randint(-10000, 10000) nfs_checks_and_fees = random.randint(-100, 0) bank_service_charges = random.randint(-100, 0) check_printing_charges = random.randint(-100, 0) interest_earned = random.randint(0, 100) note_receivable = random.randint(0, 1000) errors = random.randint(1, 5) books_schedule = { "books_schedule": { "balance_per_books": balance_per_books, "adjustments": { "bank_service_charges": bank_service_charges, "nfs_checks_and_fees": nfs_checks_and_fees, "check_printing_charges":check_printing_charges, "interest_earned": interest_earned, "note_receivable": note_receivable, "errors": errors } } } return books_schedule["books_schedule"] -
Step 8. Test banking-reconciliation-function
Now that the function code has been saved, you’re ready to test your function.
1. On the top menu, choose Configure test events.
2. In the Configure test event pop-up window:
a. Choose Create new test event.
b. For Event name, enter bankreconciliationtes.
c. In the event body, copy and paste the following test SQS event.
{ "Records": [ { "messageId": "43f49972-fe80-4e96-a9ff-1cd8046bff65", "receiptHandle": "AQEBLkHyJdCs103/Rvg4urDeQ0r8i114NGsbU8jQAm8iWu1LZ9CDS0ZrLbFGSYDCKrKnx/Bb5uordJ5CrqZ+YQe7nMWXxPTIhAgasLQj2oCibAmjqUAR2w4rWLHgDdSUeIx8YzKMlHX9WgrCZEnxX+txgGa5Ip8i7lcGiKpx083TqpzgbXuY3S6NpDTTpNKN4537h5BnxhoroIynz5bc5xFcJN7GzEVbMSYkYacgjyQjSVhomVXFL/WsHOOSJ+zyBvxfmm2QBXo6w6LEmYhwUdpUj56hLYiosL1ZYbUc4B9JM/QTMJsSFxnjxZchTkkI/N/9OgPUyJI6zL4rs99yx1xGojKEYYZdRa3JYPA6TAJytOV4lx78QIU2lU1pG38ZMj3a", "body": "{\"bank_schedule\": {\"balance_per_statement\": 4412, \"adjustments\": {\"deposits_in_transit\": 669, \"outstanding_checks\": -5652, \"bank_errors\": 3}}}", "attributes": { "ApproximateReceiveCount": "1", "SentTimestamp": "1561995379044", "SenderId": "AROAXEIPK2XYZTIRFXOZ4:sqs-producer", "ApproximateFirstReceiveTimestamp": "1561995379045" }, "messageAttributes": { "Account": { "stringValue": "8956156996094", "stringListValues": [], "binaryListValues": [], "dataType": "Number" } }, "md5OfMessageAttributes": "3110763adbb3585c7093bc2bbd8d07f1", "md5OfBody": "fa3492dfd04f085068a4fa426056df80", "eventSource": "aws:sqs", "eventSourceARN": "arn:aws:sqs:us-east-1:123456789012:banking-queue", "awsRegion": "us-east-1" } ] }
3. Choose Create.
4. To invoke the function, choose Test.
You should receive an Execution result that looks like the following. You might also get an error if the Balance per Books is greater than the Balance per Statement. This is expected. Choose the Test button a few times to see the variable results that are returned by the function.
-
Step 9. Create a Lambda Function to Produce Transactions
To test your application end to end, you create another Lambda function to produce some transactions to put onto the queue. Before you do this, you need resource information for the SQS queue and the AWS KMS key that you created earlier.
1. Open the Amazon SQS console and navigate to the banking-queue. On the Details page, copy and save the queue URL and Amazon Resource Name (ARN).
2. Open the AWS KMS console and navigate to the banking-key that you created in Step 1. Copy and save the ARN.
3. Return to the AWS Lambda console, and choose Create function.
4. Choose Author from scratch.
5. Set the Basic information for the function.
- Function Name = banking-reconciliation-function-tester
- Runtime = Python 3.6
6. Set the execution role by choosing Create a new role with basic Lambda permissions.
7. Choose Create function.
8. Scroll to the Execution role panel and choose View the banking-reconciliation-function-tester-xxx role (this opens the AWS IAM console for the role).
9. In the role summary, choose Add inline policy.
10. Open the JSON tab.
11. Paste the policy into the editor.
12. Replace the AWS KMS ARN in the Resource section with the ARN of your banking-key.
13. Replace the Amazon SQS ARN in the Resource section with the ARN of your banking-queue.
14. Choose Review policy.
{ "Version": "2012-10-17", "Statement": [ { "Sid": "WriteToQueuePolicy", "Effect": "Allow", "Action": [ "kms:Decrypt", "kms:Encrypt", "kms:GenerateDataKey", "sqs:SendMessageBatch", "sqs:SendMessage" ], "Resource": [ "arn:aws:kms:us-east-1:123456789010:key/b5b22a1e-d171-2505-b516-60a67c6e28cc", "arn:aws:sqs:us-east-1:123456789010:banking-queue" ] } ] }15. Under Review policy, for Name, enter the policy name WriteToBankingQueuePolicy.
16. Choose Create policy.
17. Return to the banking-reconciliation-function-tester in the Lambda console.
18. Verify that the AWS KMS and Amazon SQS appear in the Designer panel. You might need to refresh the page.
19. Add the following into the Function code panel:
import boto3 import os import random import json # Create SQS client sqs = boto3.client('sqs') queue_url = os.environ['BANKING_QUEUE_URL'] def lambda_handler(event, context): for x in range(100): # Generating some random numbers to simulate unique transactions account_number = random.randint(100000000000, 9000000000000) balance = random.randint(-100, 1000) deposits_in_transit = random.randint(-10000, 10000) outstanding_checks = random.randint(-10000, 10000) bank_errors = random.randint(1,5) schedule = { "bank_schedule": { "balance_per_statement": balance, "adjustments": { "deposits_in_transit": deposits_in_transit, "outstanding_checks": outstanding_checks, "bank_errors": bank_errors } } } # Send message to SQS queue response = sqs.send_message( QueueUrl=queue_url, MessageAttributes={ 'Account': { 'DataType': 'Number', 'StringValue': str(account_number) } }, MessageBody= json.dumps(schedule) ) print(response['MessageId'])20. In the Environment variables panel, add a new key called BANKING_QUEUE_URL.
21. Set the value to the URL for your banking-queue—for example, https://sqs.us-east-1.amazonaws.com/123456789010/banking-queue.
22. In the Basic settings panel, set the function timeout to 30 seconds.
23. Choose Save.
Tip: If you change the range in the code (see line 12, for x in range(100)), you can alter the number of messages that you’re sending to the queue. Note that the more messages you send, the longer the function needs to run. So you might need to adjust the function timeout setting.
-
Step 10. Execute the Lambda Function to Produce Transactions
Now that the function code has been saved, you’re ready to test your function.
1. On the top menu, choose Configure test events.
2. Configure a new test event with these settings, and then choose Create.
- Event template = Hello World (Note: You don’t need to make changes to the default payload that’s loaded by the template. You’re not processing these values—you just need to create a dummy event to execute the function).
- Event name = NoEvent
3. Choose Test.
The banking-reconciliation-function-tester simply loops to create 100 different dynamically generated transactions, and sends them to the banking-queue.
The banking-reconciliation-function receives the messages and processes them. The functional processing logic has been intentionally written to throw exceptions to illustrate retries and dead-letter queue functionality. You should expect there to be many failed messages. You can find the messages that haven’t been processed in the banking-dl-queue.
Lambda automatically monitors Lambda functions on your behalf, and reports metrics through Amazon CloudWatch.
4. Open the Monitoring tab for the banking-reconciliation-function. On this page, you see CloudWatch metrics and logs about the function execution. Use this information as a starting point to start investigating the behavior of the function. For example, you can track the number of requests, the execution duration per request, and the number of requests that result in an error.
Similarly, Amazon SQS and CloudWatch are integrated so you can use CloudWatch to view and analyze metrics for your Amazon SQS queues. You can view and analyze metrics for your queues by using the Amazon SQS console, the CloudWatch console, the AWS CLI, or the CloudWatch API. For information on how to access CloudWatch metrics for Amazon SQS, see the Amazon Simple Queue Service Developer Guide.
You can also see a full list of available Amazon SQS metrics.
Well done! You just created a Lambda function with an SQS event source. You used AWS KMS to encrypt the messages as they passed between Amazon SQS and Lambda. Then you created a function to push messages to the queue to test the event source.
Now that you’ve learned to use Amazon SQS as an event source for Lambda, you can move on to the next tutorial, Send Fanout Event Notifications with Amazon Simple Queue Service (SQS) and Amazon Simple Notification Service (SNS). Together with Lambda, Amazon SQS and Amazon SNS can make a powerful messaging pattern for your next serverless application.