Category: AWS Lambda

Running Express Applications on AWS Lambda and Amazon API Gateway

by Jeff Barr | on | in Amazon API Gateway, AWS Lambda | | Comments

Express is a web framework for Node.js . It simplifies the development of “serverless” web sites, web applications, and APIs. In a serverless environment, most or all of the backend logic is run on-demand in a stateless fashion (see Mike Roberts’ article on Serverless Architectures for a more detailed introduction). AWS Lambda, when used in conjunction with the new Amazon API Gateway features that I blogged about earlier this month (API Gateway Update – New Features Simplify API Development), allows existing Express applications to be run in serverless fashion. When you use API Gateway you have the opportunity to take advantage of additional features such as Usage Plans which allow you to build a developer ecosystem around your APIs, and caching which allows you to build applications that are responsive and cost-effective.

In order to help you to migrate your Express applications to Lambda and API Gateway, we have created the aws-serverless-express package. This package contains a working example that you can use as a starting point for your own work.

I have two resources that you can use to migrate your Express code and applications to API Gateway and Lambda:


Yemeksepeti: Our Shift to Serverless Architecture

by Jeff Barr | on | in AWS Lambda, Customer Success, Guest Post | | Comments

AWS Community Hero Onur Salk wrote the guest post below in order to tell you how he helped his employer to move to a serverless architecture.


I’m Onur Salk, AWS Community Hero, AWS Certified Solutions Architect – Professional, and organizer of the AWS user group in Turkey. As a Hero, I like to share my AWS experience and knowledge with the community on my personal blog and through meetups with the community. Today I want to share the story behind Yemeksepeti and our shift to serverless architecture.

The Story Behind Yemeksepeti
Yemeksepeti is the biggest online food ordering company in Turkey. It lets users place food orders from affiliated network restaurants without charging any extra fees. At Yemeksepeti, we needed to set up a globally distributed service that is scalable, high-performing, and cost-effective. Our belief is that by designing a serverless architecture, we won’t have to worry about managing our servers and can remove a lot of operational burdens from our team. This means we can focus on running our code at scale.

At, we developed a real-time discount system called Joker about four years ago. The purpose of this system is to suggest discounts to customers that they normally cannot find for restaurants. The original Joker platform was developed in .NET and then integrated with the website and mobile devices using its REST API. We were asked to open the platform’s API to our sister companies operating in 34 countries, so that they can also provide real-time Joker discounts to their customers.

Initially, we thought we would share our code and let them integrate their applications. However, most of other countries were using different technology stack (programming languages, database, and so on). Although using our code might accelerate their development at first,  they would have to maintain an unfamiliar system. We needed to find an integration method that was easier to implement and cheaper to maintain.

Our Requirements
This was a global project, and these were our five focus areas:

  • Ease of management
  • High availability
  • Scalability
  • Use in several regions
  • Cost advantage

We evaluated these focus areas against several different processing models and came up with the following matrix:

Ease of Management High Availability Scalability Use in Several Regions Cost Advantage

We could spin up some EC2 instances running IIS on top of Microsoft Windows Server and connected to an RDS DB instance.
 No. We need to take care of our servers.  Yes. We can distribute our servers to different AZs.  Yes. We can use Auto Scaling  Yes. We can use AMIs and copy between regions  Partially. There will be license fees and costs for running EC2 instances .

We could use AWS Elastic Beanstalk.
 Partially. We need to take care of our servers.  Yes. We can distribute our servers to different AZs.  Yes. We can use Auto Scaling.  Yes. We can use environment configurations, AMIs, etc.  Partially. There will be license fees and costs for running EC2 instances.

We could use AWS Lambda.
 Yes. AWS takes care of the services.  Yes. It is already highly available  Yes. It performs at any scale  Yes. We can export/import/upload our configurations easily.  Yes. There are no licenses and we pay only for what we use.

We decided to use Faas (Functions as a Service). We started our project in the EU (Ireland) regions using the following services:

Our architecture looks like this:

Amazon VPC: We use Amazon VPC to launch our resources in our private network.

Amazon API Gateway: During the development phase, we started to develop the service in the EU (Ireland) region. At that time, AWS Lambda was not available in EU (Frankfurt). We created two APIs: one for web integration and the other for the admin interface. We used custom authorizers with JSON Web Tokens (JWT) to enable token-based authorization for our APIs. We used mapping templates to pass our variables to our Lambda functions.

In the development phase, there was only a test stage for each API.

During the production phase, AWS Lambda became available in Frankfurt. We decided to move the service there to benefit from low latency access from Turkey. We used the API Gateway Export API feature to export our configuration in Swagger format, and then imported it into Frankfurt. (Before the import, we changed the region definitions in the exported file to eu-central-1.) After that, we created a production stage and used stage variables to parameterize our database definitions of the Amazon RDS instances (like host, username, and so on). We also wanted to use our custom domain name. After we bought an SSL certificate for our domain, we created a custom domain name in the Amazon API Gateway console and created an alias for our CloudFront distribution name (Amazon API Gateway uses Amazon CloudFront in the background). Finally, we created an IAM role to enable Amazon CloudWatch logging for API calls, latency, and more.

Metrics for Get_Joker_offer resource:

AWS Lambda: During the development phase, we used Python to develop our service and created 65 functions for integrating our API methods and scheduled tasks using CloudWatch Events Lambda triggers. Lambda VPC integration became available during the production phase, so we uploaded our functions to the Frankfurt region and integrated them with VPC.

Invocation count of Get_joker_offer Lambda function (The peaks correspond to lunch and dinner times (when people are hungry)):

Amazon RDS: During the development phase, we chose to use Amazon RDS for PostgreSQL. We created a single-AZ RDS instance to test our service. During the production phase, we needed to move our database because we migrated our APIs and functions to Frankfurt. We created a snapshot of our instance and using the Copy snapshot feature of RDS, we successfully moved our database. We launched two instances in our VPC: a multi-AZ instance for production and a single-AZ instance for test purposes. In our API stage variables, we defined the endpoint names of our RDS instances to map the staging to the appropriate instance. We also enabled automated backups for both instances.

Amazon S3: The Joker platform has an admin panel that’s used for managing and reporting Joker offers. To host this administration interface, which is basically a Single Page Application (SPA) with AngularJS, we used the static website hosting feature of Amazon S3. All of the logic and functionality is provided by methods running on Lambda, so we didn’t need a server for the admin interface:

Amazon CloudWatch: We use the service to monitor the usage of our important assets and to get alerts if something goes wrong. We created a custom dashboard to monitor the CPU of our production database, connection count, critical API latencies and function counts and durations.

In our Python code, we log the durations of each inner method in CloudWatch to track performance and find any bottlenecks:

Here’s our CloudWatch dashboard:

Amazon ElasticSearch: During the development phase, Cloudwatch Logs streaming to Amazon ES became available in the Ireland region. Using this feature, we created a Kibana dashboard to monitor some other metrics from the logs we generate from our code. As soon as Amazon ES is available in the Frankfurt region, we will use it again.

Initial Results
The Joker system is in production now, as a pilot for a small region of a country. As you can see from the following chart, the growth of the number of orders is promising. By leveraging serverless architecture, we didn’t have to install and manage an operating system and dependencies. Using Amazon API Gateway, AWS Lambda, Amazon S3, and Amazon RDS, our architecture runs in a highly available environment. We don’t need to learn and manage any master-slave replication features or third-party tools. As our service gets more requests, AWS Lambda adds more Lambda instance, so it runs at any scale. We are able to copy our service to another region using the features of AWS services as we did before going into production. Finally, we don’t run any servers, so we benefit from the cost advantage of serverless architecture.

Here is a representation of the number of orders placed through Joker:

What’s Next
We hope this service will spread to all 34 of the sister companies within Delivery Hero Holding. As the service is rolled out globally, we will deploy to other AWS regions. We plan to choose the region nearest to the company. To optimize our costs, we will purchase reserved instances for our RDS instances. Also, as we monitor our inner methods’ duration, we can re-factor and optimize our code and so that we can decrease our Lambda functions’ execution times.

We believe the future of the cloud is FaaS. We would like to experiment more as other features, services,  and functions become available.

As an AWS Community Hero, I look forward to sharing the Yemeksepeti story with the AWS user group in Turkey. I’d like to help people explore and leverage serverless architecture.

— Onur Salk


Enter the AWS Serverless Chatbot Competition!

by Jeff Barr | on | in Amazon API Gateway, AWS Lambda, Developers | | Comments

Earlier this month I was asked to be a judge for the AWS Serverless Chatbot Competition and I was happy to accept!

Build a Chatbot
We want you to build a chatbot for Slack using AWS Lambda and Amazon API Gateway. You can also use other APIs (the Slack Events API will be helpful), additional services (AWS or otherwise),  and other data sources.  Your entry should be creative and original, and should provide genuine value to Slack users.

The AWS Free Tier provides you with access to Lambda, API Gateway, and other AWS services at no charge. New and existing AWS users get 1 million Lambda requests and 400,000 GB-seconds of compute time for free. New AWS users get 1 million API calls per month to the API Gateway for up to 12 months after signing up.

Your Entry
After you build your chatbot, I would encourage you to spend some time on the packaging and the marketing. Be sure to supply the following items as part of your entry:

  • Demonstration video of your chatbot in action.
  • Brief explanation of what it does and what makes it unique.
  • Link to your public or private GitHub repo (include all files needed to run your chatbot).
  • Instructions for testing and using your chatbot.

Your chatbot can be a new or existing application, but it must use Lambda and API Gateway. The code must function as shown in the video, and the submission must be in English. It also has to be substantially different from any other app submitted by the same submitter, team, or organization.

Individuals, teams, and organizations with 50 or less employees are eligible for the Serverless Chatbot Hero Award. This award includes one ticket to AWS re:Invent and access to discounted hotel room rates, along with recognition at the Serverless State of the Union address, some cool swag, $100 in AWS credits, and publicity opportunities for their bot. We will make up to eight of these awards.

Larger organizations are eligible for a non-cash, recognition-only award.

See the FAQ for more information on rules and eligibility.

Here’s the timeline for the competition:

  • August 10 – September 29, 2016 – Submission period.
  • October 3-7 – AWS Judging.
  • October 15 – Winners announced.

Get Started Now
Here’s what you need to do to get started:

  1. Read the Rules and Eligibility Guidelines.
  2. Register for the AWS Serverless Chatbot Competition.
  3. Create AWS and Slack developer accounts.
  4. Visit the Resources Page to learn more about the APIs and services.
  5. Build your chatbot. Our sample code (aws-serverless-chatbot-sample) is a good place to start.
  6. Create your demo video and your other materials for the submission.
  7. Submit your materials before 5 PM ET on September 29, 2016.

I’m looking forward to seeing, using, and judging the entries!

Relevant Webinars
Here are some upcoming webinars that will help you to get started:

You may also find these recordings of past webinars to be helpful:


Arduino Web Editor and Cloud Platform – Powered by AWS

by Jeff Barr | on | in Amazon Internet of Things, Announcements, AWS Lambda | | Comments

Last night I spoke with Luca Cipriani from Arduino to learn more about the new AWS-powered Arduino Web Editor and Arduino Cloud Platform offerings. Luca was en-route to the Bay Area Maker Faire and we had just a few minutes to speak, but that was enough time for me to learn a bit about what they have built.

If you have ever used an Arduino, you know that there are several steps involved. First you need to connect the board to your PC’s serial port using a special cable (you can also use Wi-Fi if you have the appropriate add-on “shield”), ensure that the port is properly configured, and establish basic communication. Then you need to install, configure, and launch your development environment, make sure that it can talk to your Arduino, tell it which make and model of Arduino that you are using, and select the libraries that you want to call from your code. With all of that taken care of, you are ready to write code, compile it, and then download it to the board for debugging and testing.

Arduino Code Editor
Luca told me that the Arduino Code Editor was designed to simplify and streamline the setup and development process. The editor runs within your browser and is hosted on AWS (although we did not have time to get in to the details, I understand that they made good use of AWS Lambda and several other AWS services).

You can write and modify your code, save it to the cloud and optionally share it with your colleagues and/or friends. The editor can also detect your board (using a small native plugin) and configure itself accordingly; it even makes sure that you can only write code using libraries that are compatible with your board. All of your code is compiled in the cloud and then downloaded to your board for execution.

Here’s what the editor looks like (see Sneak Peek on the New, Web-Based Arduino Create for more):

Arduino Cloud Platform
Because Arduinos are small, easy to program, and consume very little power, they work well in IoT (Internet of Things) applications. Even better, it is easy to connect them to all sorts of sensors, displays, and actuators so that they can collect data and effect changes.

The new Arduino Cloud Platform is designed to simplify the task of building IoT applications that make use of Arduino technology. Connected devices will be able to be able to connect to the Internet, upload information derived from sensors, and effect changes upon command from the cloud. Building upon the functionality provided by AWS IoT, this new platform will allow devices to communicate with the Internet and with each other. While the final details are still under wraps, I believe that this will pave the wave for sensors to activate Lambda functions and for Lambda functions to take control of displays and actuators.

I look forward to learning more about this platform as the details become available!



Machine Learning, Recommendation Systems, and Data Analysis at Cloud Academy

by Jeff Barr | on | in Amazon RDS, Amazon S3, AWS Lambda, Guest Post | | Comments

In today’s guest post, Alex Casalboni and Giacomo Marinangeli of Cloud Academy discuss the design and development of their new Inspire system.


Our Challenge
Mixing technology and content has been our mission at Cloud Academy since the very early days. We are builders and we love technology, but we also know content is king. Serving our members with the best content and creating smart technology to automate it is what kept us up at night for a long time.

Companies are always fighting for people’s time and attention and at Cloud Academy, we face those same challenges as well. Our goal is to empower people, help them learn new Cloud skills every month, but we kept asking ourselves: “How much content is enough? How can we understand our customer’s goals and help them select the best learning paths?”

With this vision in mind about six months ago we created a project called Inspire which focuses on machine learning, recommendation systems and data analysis. Inspire solves our problem on two fronts. First, we see an incredible opportunity in improving the way we serve our content to our customers. It will allow us to provide better suggestions and create dedicated learning paths based on an individual’s skills, objectives and industries. Second, Inspire represented an incredible opportunity to improve our operations. We manage content that requires constant updates across multiple platforms with a continuously growing library of new technologies.

For instance, getting a notification to train on a new EC2 scenario that you’re using in your project can really make a difference in the way you learn new skills. By collecting data across our entire product, such as when you watch a video or when you’re completing an AWS quiz, we can gather that information to feed Inspire. Day by day, it keeps personalising your experience through different channels inside our product. The end result is a unique learning experience that will follow you throughout your entire journey and enable a customized continuous training approach based on your skills, job and goals.

Inspire: Powered by AWS
Inspire is heavily based on machine learning and AI technologies, enabled by our internal team of data scientists and engineers. Technically, this involves several machine learning models, which are trained on the huge amount of collected data. Once the Inspire models are fully trained, they need to be deployed in order to serve new predictions, at scale.

Here the challenge has been designing, deploying and managing a multi-model architecture, capable of storing our datasets, automatically training, updating and A/B testing our machine learning models, and ultimately offering a user-friendly and uniform interface to our website and mobile apps (available for iPhone and Android).

From the very beginning, we decided to focus high availability and scalability. With this in mind, we designed an (almost) serverless architecture based on AWS Lambda. Every machine learning model we build is trained offline and then deployed as an independent Lambda function.

Given the current maximum execution time of 5 minutes, we still run the training phase on a separate EC2 Spot instance, which reads the dataset from our data warehouse (hosted on Amazon RDS), but we are looking forward to migrating this step to a Lambda function as well.

We are using Amazon API Gateway to manage RESTful resources and API credentials, by mapping each resource to a specific Lambda function.

The overall architecture is logically represented in the diagram below:

Both our website and mobile app can invoke Inspire with simple HTTPS calls through API Gateway. Each Lambda function logically represents a single model and aims at solving a specific problem. More in detail, each Lambda function loads its configuration by downloading the corresponding machine learning model from Amazon S3 (i.e. a serialized representation of it).

Behind the scenes, and without any impact on scalability or availability, an EC2 instance takes care of periodically updating these S3 objects, as outcome of the offline training phase.

Moreover, we want to A/B test and optimize our machine learning models: this is transparently handled in the Lambda function itself by means of SixPack, an open-source A/B testing framework which uses Redis.

Data Collection Pipeline
As far as data collection is concerned, we use as data hub: with a single API call, it allows us to log events into multiple external integrations, such as Google Analytics, Mixpanel, etc. We also developed our own custom integration (via webhook) in order to persistently store the same data in our AWS-powered data warehouse, based on Amazon RDS.

Every event we send to is forwarded to a Lambda function – passing through API Gateway – which takes care of storing real-time data into an SQS queue. We use this queue as a temporary buffer in order to avoid scalability and persistency problems, even during downtime or scheduled maintenance. The Lambda function also handles the authenticity of the received data thanks to a signature, uniquely provided by

Once raw data has been written onto the SQS queue, an elastic fleet of EC2 instances reads each individual event – hence removing it from the queue without conflicts – and writes it into our RDS data warehouse, after performing the required data transformations.

The serverless architecture we have chosen drastically reduces the costs and problems of our internal operations, besides providing high availability and scalability by default.

Our Lambda functions have a pretty constant average response time – even during load peaks – and the SQS temporary buffer makes sure we have a fairly unlimited time and storage tolerance before any data gets lost.

At the same time, our machine learning models won’t need to scale up in a vertical or distributed fashion since Lambda takes care of horizontal scaling. Currently, they have an incredibly low average response time of 1ms (or less):

We consider Inspire an enabler for everything we do from a product and content perspective, both for our customers and our operations. We’ve worked to make this the core of our technology, so that its contributions can quickly be adapted and integrated by everyone internally. In the near future, it will be able to independently make decisions for our content team while focusing on our customers’ need.  At the end of the day, Inspire really answers our team’s doubts on which content we should prioritize, what works better and exactly how much of it we need. Our ultimate goal is to improve our customer’s learning experience by making Cloud Academy smarter by building real intelligence.

Join our Webinar
If you would like to learn more about Inspire, please join our April 27th webinar – How we Use AWS for Machine Learning and Data Collection.

Alex Casalboni, Senior Software Engineer, Cloud Academy
Giacomo Marinangeli, CTO, Cloud Academy

PS – Cloud Academy is hiring – check out our open positions!

Using Enhanced RDS Monitoring with Datadog

by Jeff Barr | on | in Amazon CloudWatch, AWS Lambda, Guest Post | | Comments

Today’s guest post comes from K Young, Director of Strategic Initiatives at Datadog!


AWS recently announced enhanced monitoring for Amazon RDS instances running MySQL, MariaDB, and Aurora. Enhanced monitoring includes over 50 new CPU, memory, file system, and disk I/O metrics which can be collected on a per-instance basis as frequently as once per second.

AWS and Datadog
AWS worked closely with Datadog to help customers send this new high-resolution data to Datadog for monitoring. Datadog is an infrastructure monitoring platform that is very popular with AWS customers—you can see historical trends with full granularity and also visualize and alert on live data from any part of your stack.

With a few minutes of work your enhanced RDS metrics will immediately begin populating a pre-built, customizable dashboard in Datadog:

Connect RDS and Datadog
The first step is to send enhanced RDS metrics to CloudWatch Logs. You can enable the metrics during instance creation, or on an existing RDS instance by selecting it in the RDS Console and then choosing Instance OptionsModify:

Set Granularity to 1–60 seconds; every 15 seconds is often a good choice. Once enabled, enhanced metrics will be sent to CloudWatch Logs.

The second step is to send the CloudWatch Log data to Datadog. Begin by setting up a Lambda function to process the logs and send the metrics:

  1. Create a role for your Lambda function. Name it something like lambda-datadog-enhanced-rds-collector and select AWS Lambda as the role type.
  2. From the Encryption Keys tab on the IAM Management Console, create a new encryption key. Enter an Alias for the key like lambda-datadog-key. On the next page, add the appropriate administrators for the key. Next you’ll be prompted to add users to the key. Add at least two: yourself (so that you can encrypt the Datadog API key from the AWS CLI in the next step), and the role created above, e.g. lambda-datadog-enhanced-rds-collector (so that it can decrypt the API key and submit metrics to Datadog). Finish creating the key.
  3. Encrypt the token using the AWS Command Line Interface (CLI), providing the Alias of your just-created key (e.g. lambda-datadog-key) as well as your Datadog keys, available here. Use KMS to encrypt your key, like this:
    $ aws kms encrypt --key-id alias/ALIAS_KEY_NAME --plaintext '{"api_key":"DATADOG_API_KEY", "app_key":"DATADOG_APP_KEY"}'

    Save the output of this command; you will need it for the next step.

  4. From the Lambda Management Console, create a new Lambda Function. Filter blueprints by datadog, and select the datadog-process-rds-metrics blueprint.
  5. Choose RDSOSMetrics from the Log Group dropdown, enter the Filter Name of your choice, and go to the next page. If you have not yet enabled enhanced monitoring, you must do so before RDSOSMetrics will be presented an as option (see the instructions under Connect RDS and Datadog above):
  6. Give your function a name like send-enhanced-rds-to-datadog. In the Lambda function code area, replace the string after KMS_ENCRYPTED_KEYS with the ciphertext blob part of the CLI command output above.
  7. Under Lambda function handler and role, choose the role you created in step 2, e.g. lambda-datadog-enhanced-rds-collector. Go to the next page, select the Enable Now radio button, and create your function.

That’s It
Once you have enabled RDS in Datadog’s AWS integration tile, Datadog will immediately begin displaying your enhanced RDS metrics. Your RDS instances will be individually identifiable in Datadog via automatically-created tags of the form dbinstanceidentifier:YOUR_DB_INSTANCE_NAME, as well as any tags you added through the RDS console.

You can clone the pre-built dashboard and customize it however you want: add RDS metrics that are not displayed by default, or start correlating RDS metrics with the performance of the rest of your stack.

— K Young, Director of Strategic Initiatives


New – Notifications for AWS CodeCommit

by Jeff Barr | on | in Amazon CodeCommit, Amazon Simple Notification Service, AWS Lambda | | Comments

AWS CodeCommit is a fully-managed source control service that makes it easy for you to host a secure and highly scalable private Git repository.Today we are making CodeCommit even more useful by adding support for repository triggers. You can use these triggers to integrate your existing unit tests and deployment tools into your source code management workflow. Because triggers are efficient and scalable, they are more broadly applicable than a model that is built around polling for changes. I believe that you will find these triggers to be helpful as you move toward a development methodology based on  Continuous Integration and Continuous Delivery.

All About Notifications
You can create up to 10 triggers for each of your CodeCommit repositories. The triggers are activated in response to actions on the repository including code pushes, branch/tag creation, and branch/tag deletion. Triggers can be set to run for a specific branch of a repository or for all branches.

Triggers can send a notification to an Amazon Simple Notification Service (SNS) topic or can invoke a AWS Lambda function. Each trigger can also be augmented with custom data (an uninterpreted string) that you can use to distinguish the trigger from others that run for the same event. You can use triggers to subscribe to repository events through email or SMS. You can wire up SNS to SQS and queue up jobs for your CI/CD tools, or you can use SNS to activate webhooks provided by your tools. In any case, the actions you designate will be triggered by the changes in your CodeCommit repository. You can also use Lambda functions to trigger builds, check syntax, capture code complexity metrics, measure developer producitivity (less is more, of course), and so forth. My colleagues have also come up with some off-unusual ideas that you can find at the end of this post!

You can create, view, and manage your triggers from the AWS Management Console, AWS Command Line Interface (CLI), or via the CodeCommit API.  I used the Console. The navigation column on the left now includes an entry for Triggers:

I simply click on Create Trigger to get started. Then I select the event (or events), pick the branch (or all branches), and fill in the details that are needed to publish a notification or invoke a Lambda function:

Here’s how I choose the events and branches of interest:


Then I point to my SNS topic or Lambda function (after ensuring that the proper permissions are in place), use Test Trigger to make sure that it all works as expected, and click on Create.

You can use Test Trigger to verify that your IAM permissions are working as expected. For example, here’s an error that I triggered on purpose:

I fixed this by reading How to Allow AWS CodeCommit to run the Function in the documentation!

Available Now
This new functionality is available now and you can start using it today.To learn more, read about Managing Triggers for an AWS CodeCommit Repository.

My colleague Clare Liguori suggested some creative uses for CodeCommit triggers, above and beyond the usual integration with your CI/CD process:

  • Video Deployment – Have your Lambda function check to see if a new video or a new version of an existing video has been committed, and deploy the video to YouTube.
  • Party Time – Automatically throw and cater a party (using APIs for sandwiches, pizza, and beer) when you deploy a new release.
  • Advertise Releases – When a new release is ready, automatically generate and run a Facebook ad and publicize the release on social media.

I am looking forward to hearing about the creative ways that you make use of these triggers within your development process. Leave me a comment and let me know!



New – Access Resources in a VPC from Your Lambda Functions

by Jeff Barr | on | in AWS Lambda | | Comments

A few months ago I announced that you would soon be able to access resources in a VPC from your AWS Lambda functions. I am happy to announce that this much-wanted feature is now available and that you can start using it today!

Your Lambda functions can now access Amazon Redshift data warehouses, Amazon ElastiCache clusters, Amazon Relational Database Service (RDS) instances, and service endpoints that are accessible only from within a particular VPC. In order to do this, you simply select one of your VPCs and identify the relevant subnets and security groups. Lambda uses this information to set up elastic network interfaces (ENIs) and private IP addresses (drawn from the subnet or subnets that you specified) so that your Lambda function has access to resources in the VPC.

Accessing Resources in a VPC
You can set this up when you create a new function. You can also update an existing function so that it has VPC access.  You can configure this feature from the Lambda Console or from the CLI.  Here’s how you set it up from the Console:

That’s all you need to do! Be sure to read Configuring a Lambda Function to Access Resources in an Amazon VPC in the Lambda documentation if you have any questions.

Things to Know
Here are a couple of things that you should know about this new feature:

ENI & IP Address Resources – Because Lambda automatically scales based on the number of events that is needs to process, your VPC must have an adequate supply of free IP addresses on the designated subnets.

Internet Access – As soon as you enable this functionality for a particular function, the function no longer has access to the Internet by default. If your function requires this type of access, you will need to set up a Managed NAT Gateway in your VPC (see New – Managed NAT (Network Address Translation) Gateway for AWS for more information) or run your own NAT (see NAT Instances).

Security Groups – The security groups that you choose for a function will control the function’s access to the resources in the subnets and on the Internet.

S3 Endpoints – You can also use this feature to access S3 endpoints within a VPC (consult New – VPC Endpoint for Amazon S3 to learn more).

Webinar – To learn more about this new feature, join our upcoming webinar, Essentials: Introducing AWS VPC Support for AWS Lambda.


New – Slack Integration Blueprints for AWS Lambda

by Jeff Barr | on | in AWS Lambda | | Comments

Does your operations team practice ChatOps? This brand-new term refers to the practice of conversation-driven operations using one or more “bots” that have the ability to insert notifications & status reports into the conversation and to respond to commands.

The chat environment provides real-time communication, a coherent shared view, multi-user access from web and mobile devices, access to previous messages, and so forth. Integrating bots into the conversation gives operations teams the ability to work collaboratively to understand, diagnose, and address emerging issues while simultaneously tracking changes made to the target system by way of commands directed to a bot.

New Slack Integration
In order to make it even easier for AWS customers to manage their environments in this new and innovative way, we recently launched a collection of Slack Integration Blueprints for AWS Lambda:

You can use these blueprints to build chat-based tools that participate in your own Slack conversations. The slack-echo- blueprints will help you to write bots that respond to commands; the cloudwatch-alarm-to-slack- blueprints will help you to write bots that emit status reports and notifications. Because you have the ability to give the bots access to any desired AWS APIs, you can interact with your AWS resources in any desired way. You can query their status, look for error conditions, change settings, or even create new resources.

Suppose you want to monitor an Auto Scaling group using a CloudWatch alarm, and generate a message to your ChatOps team (a Slack channel) when a limit is exceeded. The team can take a closer look at the situation, and then decide to remedy the issue by scaling up. Using the new Slack integration and various AWS services, the overall flow would look like this (red arrows represent the notification; green arrows represent the response):

In order to implement a system like this you would use a Slack webhook to post messages to the channel, along with the cloudwatch-alarm-to-slack-python blueprint. Following the directions in the blueprint, you would create a AWS Key Management Service (KMS) key, use it to encrypt the URL of the webhook, and base-64 encode it before pasting it in to the code. Then you would create an IAM Role and give it permission to call the KMS Decrypt function on the key. The event handler would then send messages to the Slack channel of your choice like this:

slack_message = {
    'channel': SLACK_CHANNEL,
    'text': "%s state is now %s: %s" % (alarm_name, new_state, reason)

req = Request(HOOK_URL, json.dumps(slack_message))

The handler would also need to handle any exceptions raised by the request (this is all spelled out in the blueprint).

You can also create functions that implement custom Slack commands. In order to do this, you use the Amazon API Gateway to create an HTTP endpoint for each function, and then configure your Slack channel to POST to the endpoint when the command is invoked. For example, here’s the configuration for commands named /scale and /forcealarm:

The handler function has access to the user, command, channel, and command text:

def lambda_handler(event, context):
    req_body = event['body']
    params = parse_qs(req_body)

    user = params['user_name'][0]
    command = params['command'][0]
    channel = params['channel_name'][0]
    command_text = params['text'][0]

You will need to configure the endpoint with a POST method, and set the authorization to NONE. You’ll also need to map the body of the request to JSON. The comment block at the top of each of the new blueprints contains more information on this aspect of the integration.

Share your Functions
I would love to see what kinds of cool functions you come up with. Please feel free to post your work as a comment to this post.


Learn About the newest AWS Services – Attend our October Webinars

by Jeff Barr | on | in Amazon EC2, Amazon Elasticsearch Service, Amazon Internet of Things, Amazon Kinesis, Amazon QuickSight, AWS Import/Export, AWS Lambda, AWS WAF, Mobile, Webinars | | Comments

If you attended AWS re:Invent, you were among the first to know about Amazon QuickSight, AWS IoT, Kinesis Firehose, and our other new offerings. Perhaps you had time to attend a session to learn more about the new service or services that were of interest to you. If you didn’t attend re:Invent or missed a session or two and are ready to learn more, I’ve got you covered. We will be running nine new-product webinars later this month. Each webinar is designed to provide you with the information that you need to have in order to be up and running as quickly as possible.

Here’s what we have for you! The webinars are free but “seating” is limited and you should definitely sign up ahead of time if you want to attend (all times are Pacific):

Tuesday, October 27
QuickSight is a fast, cloud-powered business intelligence tool. You can build visualizations, perform ad-hoc analysis, and get business insights from your data.

AWS IoT is a managed cloud platform that lets connected devices easily and securely interact with cloud applications and other devices.

Amazon Kinesis Firehose is the easiest way to load streaming data into AWS.

Wednesday, October 28
Spot Blocks allow you to launch Spot instances that will run for a finite duration (1 to 6 hours).

AWS WAF is a web application firewall that helps protect your web applications from common exploits.

Amazon Elasticsearch Service is a managed service that makes it easy to deploy, operate, and scale Elasticsearch in the AWS Cloud.

Thursday, October 29
AWS Lambda lets you run code in the cloud without provisioning or managing servers.

AWS Mobile Hub provides an integrated console that helps you build, test, and monitor your mobile apps.

AWS Import/Export Snowball is a petabyte-scale data transport solution that uses secure appliances to transfer large amounts of data into and out of AWS.