Amazon Aurora is a MySQL and PostgreSQL-compatible relational database that combines the performance and availability of traditional enterprise databases with the simplicity and cost-effectiveness of open source databases. Amazon Aurora Serverless is an on-demand, auto-scaling configuration for Amazon Aurora (MySQL-compatible and PostgreSQL-compatible editions), where the database will automatically start up, shut down, and scale capacity up or down based on your application's needs. It enables you to run your database in the cloud without managing any database instances. It's a simple, cost-effective option for infrequent, intermittent, or unpredictable workloads.
In this tutorial, you will learn how to create a serverless message processing application with Amazon Aurora Serverless (PostgreSQL-compatible edition), Data API for Aurora Serverless, Amazon Lambda, and Amazon Simple Notification Service (SNS). The tutorial will provide step-by-step instructions to create an Aurora Serverless database, use Data API to connect it with an Amazon Lambda funtion that consumes messages from Amazon SNS, and stores the messages in Aurora Serverless database.
|About this Tutorial|
|Cost||Less than $1|
|Products||Amazon Aurora, Amazon SNS, AWS Lambda|
|Last Updated||September 24, 2019|
Step 1: Create your Aurora serverless database
1.1 — Open a browser and navigate to Amazon RDS console. If you already have an AWS account, login to the console. Otherwise, create a new AWS account to get started.
Already have an account? Log in to your account
1.2 — On the top right corner, select the region where you want to launch the Aurora DB cluster.
1.3 — Click on “Create database” in the Amazon Aurora window.
Before continuing, switch to the new database creation flow:
1.4 — On Database engine, select “Amazon Aurora”.
1.5 — For edition, select "Amazon Aurora with PostgreSQL compatibility".
1.6 — Select "Serverless".
1.7 — Choose an identifier for your Aurora DB cluster, e.g. "database-1".
1.8 — Select a username and password for your database.
1.9 — For maximum Aurora capacity unit, select 2.
1.10 — Select the VPC where you want to create the database.
Note that once created, a database can't be migrated to a different VPC.
1.11 — Click on “Additional connectivity configuration”.
1.12 — Select the default value for Subnet group.
1.13 — On VPC security group, select “create new”. If you happen to have a security group that allows incoming TCP connections on port 3306, you can choose it instead.
1.14 — In the new VPC security group name, type “aurora-tutorial”.
1.15 — Enable the Data API.
1.16 — On “Deletion protection” uncheck “Enable deletion protection”.
The best practice is to enable the Deletion protection, but if you want to delete the database at the end of the tutorial you can leave the option unchecked.
Review and create
After a quick review of all the fields in the form, you can proceed.
1.17 — Click on “Create database”.
While the instances are being created, you will see a banner explaining how to obtain your credentials. This is a good opportunity to save the credentials somewhere, as this is the only time you will be able to view this password.
Retrieve the Cluster ARN
1.18 — Click on the database name.
1.19 — In the “Configuration” tab, copy the Cluster ARN and keep it handy. You will need it later.
Connect to the database
1.20 — Open the left panel and click on “Query Editor”.
1.21 — Select the “database-1”, enter “postgres” as the database username and input the database password you created earlier, then type “postgres” for the database name.
1.22 — Click on “Connect to database”.
1.23 — You can now click “Run” and execute the sample query.
1.24 — Create a database by running the following query:
CREATE DATABASE tutorial;
1.25 — Click on “Change database”.
1.26 — Change the database to the one you just created.
1.27 — Create a table with this query:
CREATE TABLE sample_table(received_at TIMESTAMP, message VARCHAR(255));
By connecting to the database with the Query Editor, a Secret is created that you will use later on in your Lambda function. Leave this tab open, as you will need to run some queries at the end of the tutorial.
Copy the secret ARN
Open a new tab and head to the AWS Secrets Manager. Then follow the steps below to retrieve the Secret ARN.
1.28 — Find the secret containing the “RDS database postgres credentials for database-1”.
1.29 — After clicking on the Secret name, copy the Secret ARN and keep it handy.
Step 2: Create an AWS Cloud9 environment
Open a new tab and head to the AWS Cloud9 console. Then follow the steps below to create a new environment.
2.1 — Click on “Create environment”:
2.2 — Enter a name for your new environment and click on “Next step”.
2.3 — Scroll down and click on “Next step”.
2.4 — Click on “Create environment”.
2.5 — Once the environment is ready, click on “Create Lambda Function”.
2.6 — As the name of the function, enter “auroratest”. Then click “Next”.
2.7 — Select the “empty-python” blueprint and click “Next”.
2.8 — In the following screens, click “Next”, “Next”, then “Finish”.
Now the environment is ready and you will see a blueprint for a Lambda Function.
Before you start coding you need to configure some permissions as described in Step 3. Keep this environment open and use other tabs for completing the upcoming steps.
Step 3: Configure permissions
Open a new tab and head to the AWS IAM console. Then follow the steps below to assign permissions to your Lambda function.
3.1 — Click “Roles” and search for a role starting with “cloud9-”.
3.2 — Click on the role name, then click on “Attach policies”.
3.3 — Select the “AmazonRDSDataFullAccess“ policy, then click on “Attach policy”.
You will see a screen with a light green banner explaining that the policy was successfully attached.
Step 4: Create your AWS Lambda function
Go back to the Cloud9 environment you created in step 2, then follow the instructions below.
4.1 — Go to the auroratest directory by running this command in the console:
$ cd auroratest
4.2 — Install the boto3 dependency by running this command in the console:
$ pip install boto3 -t boto3
4.4 — Replace the cluster_arn and secret_arn values with the Cluster ARN and Secret ARN values obtained in previous steps.
4.5 — Click on “File > Save”, then select you Lambda Function on the right panel.
4.6 — Deploy your Lambda Function by clicking on the up arrow.
Step 5: Create an Amazon SNS topic
Your Lambda Function will process messages from Amazon Simple Notification Service (SNS), which offers pub/sub messaging for microservices and serverless applications.
In a new tab, visit the SNS Dashboard and follow these instructions:
5.1 — In “Topic name” enter aurora-lambda-sns-test, then click on “Next step”.
5.2 — Leave all the default values and click on “Create topic”.
You will see a green banner indicating that the topic was successfully created.
5.3 — Copy the SNS ARN and keep it handy.
Keep this tab open, as you will use it to publish a message once the Lambda Function is configured to read from the topic you created.
Step 6: Subscribe AWS Lambda Function to Amazon SNS topic
Go to the AWS Lambda Management Console and follow these instructions:
6.1 — Click on the name of your Lambda Function you created in step 4.
6.2 — Click on “Add trigger”.
6.3 — Type “SNS” and select the “SNS” services from the dropdown menu.
6.4 — In the “SNS Topic” field, enter the SNS ARN.
6.5 — Enable the trigger and click on “Add”.
7.1 — Click on “Publish message”.
7.2 — Enter any value for the “Subject” field.
7.3 — Enter any value for the “Body” field.
7.4 — Scroll down and click on “Publish message”.
Once the message is published, your Lambda Function will consume it and process it. In the next section, you will verify how the data was written to your Aurora database.
Verify database changes
7.5 — Go to the tab where you left the Query Editor open. If you closed it, visit the RDS Dashboard and connect to the Query Editor as explained in 1.20.
7.6 — Connect to the “tutorial” database.
7.7 — Select all the records from sample_table:
SELECT * FROM sample_table;
7.8 — Click on “Run” and scroll down to see the results.
All should be working now. You can experiment by changing the messages you send via SNS, or you can alter the lambda_handler function any way you want.
Step 8: Cleanup
To finish this tutorial, you will learn how to delete your Aurora DB cluster when it's not needed anymore, along with the Lambda Function, the Cloud9 environment, the Secret for connecting to the database, and any other leftovers.
Delete your Lambda Function
8.1 — Go to the AWS Lambda Management Console and select your Lambda Function.
8.2 — Click on “Actions > Delete”.
8.4 — Select the topic you created in Step 5 and click on “Delete”.
8.5 — You will be asked for confirmation. Type “delete me” to confirm and click on “Delete”.
Delete your Cloud9 environment
8.6 — Head to the AWS Cloud9 console and select the environment you created for this tutorial.
8.7 — Click on “Delete”.
8.8 — When asked for confirmation, type “Delete” and click the “Delete” button.
Delete your Aurora Serverless database
8.9 — Go to the Amazon RDS console and select the database you created for this tutorial.
8.10 — Click on “Actions > Delete”.
You will be asked if you want to create a final snapshot. That's usually a good idea, but it's not necessary for this tutorial.
8.11 — When asked for confirmation, select “No” to the question of whether to create a final snapshot. Then check “I acknowledge…” and click on “Delete DB cluster”.
Delete your secret
8.12 — Go to the AWS Secrets Manager, find the secret containing the “RDS database admin credentials for database-1”, and click on the name of the secret.
8.13 — Click on “Actions > Delete secret”.
8.14 — The secrets can't be deleted immediately. The minimum waiting period for a scheduled deletion is 7 days. Select 7 days and click on “Schedule deletion”.
Delete your IAM role
8.15 — Go to the AWS IAM console. Click “Roles” and search for a role starting with “cloud9-”.
8.16 — Select the role and click on “Delete role”.
8.17 — When asked for confirmation, click on “Yes, delete”.
You have created an Aurora Serverless database and connected it with an AWS Lambda Function via Aurora's Data API. You configured Amazon Simple Notification Service (SNS) as a trigger for your Lambda Function, and the messages you sent via SNS were processed and stored in your Aurora Serverless database.
Recommended next steps
Learn more about Amazon Aurora features
Find out more about the features of Amazon Aurora with the Amazon Aurora User Guide.
Best practices with Amazon Aurora
Learn about general best practices and options for using or migrating data to an Amazon Aurora DB cluster.
Learn more about Serverless
If you want to know more about serverless applications, check the AWS Lambda Documentation, as well as the User Guide for Aurora.