Category: Python


Serverless Service Discovery: Part 3: Registration

by Magnus Bjorkman | on | in Python | Permalink | Comments |  Share

In this, the third part of our serverless service discovery series, we will show how to configure Amazon API Gateway to require AWS Identity and Access Management (IAM) for authentication and how to create a V4 signature to call our register and deregister methods.

We have created all the functions required to manage our API and code, so we can jump directly into creating our new functions.

Registering and Deregistering Services

We start by creating a Lambda function for registering a service:


def lambda_handler(api_parameters, context):
    """Lambda hander for registering a service."""
    logger.info("lambda_handler - service_name: %s"
                " service_version: %s"
                % (api_parameters["service_name"],
                   api_parameters["service_version"]))

    table = boto3.resource('dynamodb',
                           region_name='us-east-1').Table('Services')

    table.put_item(
           Item={
                'name': api_parameters["service_name"],
                'version': api_parameters["service_version"],
                'endpoint_url': api_parameters["endpoint_url"],
                'ttl': int(api_parameters["ttl"]),
                'status': api_parameters["status"],
            }
        )

This function takes the input and stores it in Amazon DynamoDB. If you call the function with the same service name and version (our DynamoDB key), then it will overwrite the existing item.

Followed by the function to deregister:


def lambda_handler(api_parameters, context):
    """Lambda hander for deregistering a service."""
    logger.info("lambda_handler - service_name: %s"
                " service_version: %s"
                % (api_parameters["service_name"],
                   api_parameters["service_version"]))

    table = boto3.resource('dynamodb',
                           region_name='us-east-1').Table('Services')

    table.delete_item(
            Key={
                'name': api_parameters["service_name"],
                'version': api_parameters["service_version"]
            }
        )

The function removes the item from the DynamoDB table based on the service name and version.

We need to add the new functions and API methods to the Swagger file:


{
  "swagger": "2.0",
  "info": {
    "title": "catalog_service",
    "version": "1.0.0"
  },
  "basePath": "/v1",
  "schemes": ["https"],
  "consumes": ["application/json"],
  "produces": ["application/json"],
  "paths": {
    "/catalog/{serviceName}/{serviceVersion}": {
      "parameters": [{
        "name": "serviceName",
        "in": "path",
        "description": "The name of the service to look up.",
        "required": true,
        "type": "string"
      },
      {
        "name": "serviceVersion",
        "in": "path",
        "description": "The version of the service to look up.",
        "required": true,
        "type": "string"
      }],
      "get": {
        "responses": {
          "200": {
            "description": "version information"
          },
          "404": {
            "description": "service not found"
          }
        },
        "x-amazon-apigateway-integration": {
          "type": "aws",
          "uri": "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/$catalog_serviceARN$/invocations",
          "httpMethod": "POST",
          "requestTemplates": {
            "application/json": "{\"service_name\": \"$input.params('serviceName')\",\"service_version\": \"$input.params('serviceVersion')\"}"
          },
          "responses": {
            "default": {
              "statusCode": "200",
              "schema": {
                "$ref": "#/definitions/CatalogServiceModel"
              }
            },
            ".*NotFound.*": {
              "statusCode": "404",
              "responseTemplates" : {
                 "application/json": "{\"error_message\":\"Service Not Found\"}"
                } 
            } 
          }
        }
      }
    },
    "/catalog/register": {
      "post": {
        "responses": {
          "201": {
            "description": "service registered"
          }
        },
        "parameters": [{
          "name": "body",
          "in": "body",
          "description": "body object",
          "required": true,
          "schema": {
            "$ref":"#/definitions/CatalogRegisterModel"
          }
        }],
        "x-amazon-apigateway-auth" : {
          "type" : "aws_iam" 
        },
        "x-amazon-apigateway-integration": {
          "type": "aws",
          "uri": "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/$catalog_registerARN$/invocations",
          "httpMethod": "POST",
          "requestTemplates": {
            "application/json": "$input.json('$')"
          },
          "responses": {
            "default": {
              "statusCode": "201"
            } 
          }
        }
      } 
    },
    "/catalog/deregister": {
      "post": {
        "responses": {
          "201": {
            "description": "service deregistered"
          }
        },
        "parameters": [{
          "name": "body",
          "in": "body",
          "description": "body object",
          "required": true,
          "schema": {
            "$ref":"#/definitions/CatalogDeregisterModel"
          }
        }],
        "x-amazon-apigateway-auth" : {
          "type" : "aws_iam" 
        },
        "x-amazon-apigateway-integration": {
          "type": "aws",
          "uri": "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/$catalog_deregisterARN$/invocations",
          "httpMethod": "POST",
          "requestTemplates": {
            "application/json": "$input.json('$')"
          },
          "responses": {
            "default": {
              "statusCode": "201"
            } 
          }
        }
      } 
    }
  },
  "definitions": {
    "CatalogServiceModel": {
      "type": "object",
      "properties": {
        "endpoint_url": {
          "type": "string"
        },
        "ttl": {
          "type": "integer"
        },
        "status": {
          "type": "string"
        }
      },
      "required": ["endpoint_url", "ttl", "status"]
    },
    "CatalogRegisterModel": {
      "type": "object",
      "properties": {
        "service_name": {
          "type": "string"
        },
        "service_version": {
          "type": "string"
        },
        "endpoint_url": {
          "type": "string"
        },
        "ttl": {
          "type": "integer"
        },
        "status": {
          "type": "string"
        }
      },
      "required": ["service_name","service_version","endpoint_url", "ttl", "status"]
    },
    "CatalogDeregisterModel": {
      "type": "object",
      "properties": {
        "service_name": {
          "type": "string"
        },
        "service_version": {
          "type": "string"
        }
      },
      "required": ["service_name","service_version"]
    }
  }
}

The new methods will be POST-based, so we need to define models (CatalogRegisterModel and CatalogDeregisterModel) for the data passed through the method body. After API Gateway processes the models, the JSON objects will be passed, as is, to the Lambda functions.

We set the x-amazon-apigateway-auth element to the type of aws_iam for the register and deregister methods, so API Gateway will require a V4 signature when we access them.

We can now deploy our new functions:


ACCOUNT_NUMBER = _your account number_

create_deployment_package("/tmp/catalog_register.zip", ["catalog_register.py"])
catalog_register_arn = create_lambda_function(
                       "/tmp/catalog_register.zip",
                       "catalog_register",
                       "arn:aws:iam::"+ACCOUNT_NUMBER+":role/lambda_s3",
                       "catalog_register.lambda_handler",
                       "Registering a service.",
                       ACCOUNT_NUMBER)
replace_instances_in_file("swagger.json",
                          "/tmp/swagger_with_arn.json",
                          "$catalog_registerARN$", catalog_register_arn)
create_deployment_package("/tmp/catalog_deregister.zip",
                          ["catalog_deregister.py"])
catalog_deregister_arn = create_lambda_function(
                       "/tmp/catalog_deregister.zip",
                       "catalog_deregister",
                       "arn:aws:iam::"+ACCOUNT_NUMBER+":role/lambda_s3",
                       "catalog_deregister.lambda_handler",
                       "Deregistering a service.",
                       ACCOUNT_NUMBER)
replace_instances_in_file("/tmp/swagger_with_arn.json",
                          "/tmp/swagger_with_arn.json",
                          "$catalog_deregisterARN$", catalog_deregister_arn)
catalog_service_arn = get_function_arn("catalog_service")
replace_instances_in_file("/tmp/swagger_with_arn.json",
                          "/tmp/swagger_with_arn.json",
                          "$catalog_serviceARN$", catalog_service_arn)
api_id = update_api("/tmp/swagger_with_arn.json")
deploy_api(api_id, "/tmp/swagger_with_arn.json", "dev")

We can try out the new register service like this:


json_body = {
            "service_name": "registerservice3",
            "service_version": "1.0",
            "endpoint_url": "notarealurlregister3",
            "ttl": "300",
            "status": "healthy"
            }
request_url = "https://yourrestapi.execute-api.us-east-1.amazonaws.com/"\
              "dev/catalog/register"
response = requests.post(
            request_url,
            data=json.dumps(json_body))
if(not response.ok):
    logger.error("Error code: %i" % (response.status_code,))


We should get something like this:


ERROR:root:Error code: 403

Signing a Request with Signature Version 4

To successfully call our new services, we need to implement a client that will sign the request to the API with a Version 4 signature. First we implement the functions that creates the signature:


from botocore.credentials import get_credentials
from botocore.session import get_session
import requests
import json
import logging
import sys
import datetime
import hashlib
import hmac
import urlparse
import urllib
from collections import OrderedDict

def sign(key, msg):
    """Sign string with key."""
    return hmac.new(key, msg.encode('utf-8'), hashlib.sha256).digest()


def getSignatureKey(key, dateStamp, regionName, serviceName):
    """Create signature key."""
    kDate = sign(('AWS4' + key).encode('utf-8'), dateStamp)
    kRegion = sign(kDate, regionName)
    kService = sign(kRegion, serviceName)
    kSigning = sign(kService, 'aws4_request')
    return kSigning


def create_canonical_querystring(params):
    """Create canonical query string."""
    ordered_params = OrderedDict(sorted(params.items(), key=lambda t: t[0]))
    canonical_querystring = ""
    for key, value in ordered_params.iteritems():
        if len(canonical_querystring) > 0:
            canonical_querystring += ","
        canonical_querystring += key+"="+value
    return canonical_querystring


def sign_request(method, url, credentials, region, service, body=''):
    """Sign a HTTP request with AWS V4 signature."""
    ###############################
    # 1. Create a Canonical Request
    ###############################
    t = datetime.datetime.utcnow()
    amzdate = t.strftime('%Y%m%dT%H%M%SZ')
    # Date w/o time, used in credential scope
    datestamp = t.strftime('%Y%m%d')

    # Create the different parts of the request, with content sorted
    # in the prescribed order
    parsed_url = urlparse.urlparse(url)
    canonical_uri = parsed_url.path
    canonical_querystring = create_canonical_querystring(
                              urlparse.parse_qs(parsed_url.query))
    canonical_headers = ("host:%s\n"
                         "x-amz-date:%s\n" %
                         (parsed_url.hostname, amzdate))
    signed_headers = 'host;x-amz-date'
    if (not (credentials.token is None)):
        canonical_headers += ("x-amz-security-token:%s\n") % (credentials.token,)
        signed_headers += ';x-amz-security-token'

    payload_hash = hashlib.sha256(body).hexdigest()
    canonical_request = ("%s\n%s\n%s\n%s\n%s\n%s" %
                         (method,
                          urllib.quote(canonical_uri),
                          canonical_querystring,
                          canonical_headers,
                          signed_headers,
                          payload_hash))

    #####################################
    # 2. Create a String to Sign
    #####################################
    algorithm = 'AWS4-HMAC-SHA256'
    credential_scope = ("%s/%s/%s/aws4_request" % 
                        (datestamp,
                         region,
                         service))
    string_to_sign = ("%s\n%s\n%s\n%s" %
                       (algorithm,
                        amzdate,
                        credential_scope,
                        hashlib.sha256(canonical_request).hexdigest()))
    #####################################
    # 3. Create a Signature
    #####################################
    signing_key = getSignatureKey(credentials.secret_key,
                                  datestamp, region, service)
    signature = hmac.new(signing_key, (string_to_sign).encode('utf-8'),
                         hashlib.sha256).hexdigest()

    ######################################################
    # 4. Assemble request to it can be used for submission
    ######################################################
    authorization_header = ("%s Credential=%s/%s, "
                            "SignedHeaders=%s, "
                            "Signature=%s" %
                            (algorithm,
                             credentials.access_key,
                             credential_scope,
                             signed_headers,
                             signature))
    headers = {'x-amz-date': amzdate, 'Authorization': authorization_header}
    if (not (credentials.token is None)):
        headers['x-amz-security-token'] = credentials.token
    request_url = ("%s://%s%s" % 
                   (parsed_url.scheme,parsed_url.netloc,canonical_uri))
    if (len(canonical_querystring) > 0):
        request_url += ("?%s" % (canonical_querystring,))

    return request_url, headers, body

The main function, sign_request, can sign requests for both POST and GET methods. It also works with both short and long term credentials. For more information about creating Signature Version 4 requests, see Signing Requests

We implement the following method to submit a POST request:


def signed_post(url, region, service, data, **kwargs):
    """Signed post with AWS V4 Signature."""
    credentials = get_credentials(get_session())

    request_url, headers, body = sign_request("POST", url, credentials, region,
                                              service, body=data)

    return requests.post(request_url, headers=headers, data=body, **kwargs)

We are using botocore functionality to get the configured keys on the instance we are running. If we are running this on an Amazon EC2 instance or AWS Lambda, botocore will use the configured IAM role.

We can now test the service by calling register:


json_body = {
            "service_name": "registerservice6",
            "service_version": "1.0",
            "endpoint_url": "notarealurlregister6",
            "ttl": "300",
            "status": "healthy"
            }
request_url = "https://yourrestapiid.execute-api.us-east-1.amazonaws.com/"\
              "dev/catalog/register"
response = signed_post(
            request_url,
            "us-east-1",
            "execute-api",
            json.dumps(json_body))
if(not response.ok):
    logger.error("Error code: %i" % (response.status_code,))
else:
    logger.info("Successfully registered the service.")

The test should complete without a failure. To test, look up this item:


request_url="https://your_rest_api_id.execute-api.us-east-1.amazonaws.com/"\
            "dev/v1/catalog/registerservice6/1.0"
response = requests.get(request_url)
json_response = json.loads(response.content)
logging.info("Endpoint URL: %s" % (json_response['endpoint_url'],))
logging.info("TTL: %i" % (json_response['ttl'],))
logging.info("Status: %s" % (json_response['status'],))

You should get the following output:


INFO:root:Endpoint URL: notarealurlregister6
INFO:root:TTL: 300
INFO:root:Status: healthy

Serverless Service Discovery: Part 2: Lookup

by Magnus Bjorkman | on | in Python | Permalink | Comments |  Share

In this, the second part of our serverless service discovery series, we will use Amazon DynamoDB to store information about the services in our service discovery service and update our AWS Lambda function to read information from the DynamoDB table.

Updating Amazon API Gateway and AWS Lambda

As part of adding new functionality to what we are implementing, we need to update the Service Discovery API and the code in API Gateway and AWS Lambda. First, we update the code in AWS Lambda:


def get_function_arn(function_name):
    """Return ARN given the Function Name.

    :param function_name: The name of the Lambda function.
    :return: The ARN for the Lambda function.
    """
    client = boto3.client('lambda')

    response = client.get_function(
        FunctionName=function_name
    )

    return response['Configuration']['FunctionArn']


def update_lambda_function(package_name, function_name):
    """Update a Lambda function from zip-file.

    :param package_name: The name of the package. Full or relative path.
    :param function_name: The name of the Lambda function.
    :return: The ARN for the Lambda function.
    """
    with open(package_name, "rb") as package_file:
        package_data = package_file.read()

    # connect to Lambda API
    client = boto3.client('lambda')

    # update the function code
    client.update_function_code(
        FunctionName=function_name,
        ZipFile=package_data,
        Publish=True
    )

    # get function configuration to get top level ARN
    return get_function_arn(function_name)

The get_function_arn returns the top level ARN for the Lambda function. When we update the code, we get the ARN for the version that was uploaded, but we need to use the top level ARN with Swagger.

The update_lambda_function updates the code of the Lambda function only. There are other functions in the Lambda API to update other configurations for Lambda functions.

Next we update the API with Swagger:


def update_api(swagger_file_name):
    """Update an API defined in Swagger.

    :param swagger_file_name: The name of the swagger file.
                              Full or relative path.
    :return: The id of the REST API.
    """
    # get the API Gateway ID of the existing API
    rest_api_name = get_rest_api_name(swagger_file_name)
    client = boto3.client('apigateway')
    paginator = client.get_paginator('get_rest_apis')
    rest_api_id = ""
    for response in paginator.paginate():
        for item in response["items"]:
            if (rest_api_name == item["name"]):
                rest_api_id = item["id"]

    with open(swagger_file_name, "r") as swagger_file:
        swagger_data = swagger_file.read()

    response = client.put_rest_api(restApiId=rest_api_id,
                                   body=swagger_data)

    return response['id']

We first query API Gateway for the REST API identifier based on the name in the Swagger file. We need to submit this identifier for the update to work.

We can use the same deployment method we used in the first blog post to deploy the update to the API Gateway stage. If the stage already exists, it will just be updated.

Creating a DynamoDB Table

We need to be able to persistently store the information about a service, and we need to be able to create, update, and query that information. DynamoDB is a fully managed serverless service that works well with AWS Lambda. We will start by creating a table for our discovery service:


dynamodb = boto3.resource('dynamodb', region_name = 'us-east-1')  

# create the table
table = dynamodb.create_table(
    TableName='Services',
    KeySchema=[ { 'AttributeName': 'name',
                  'KeyType': 'HASH' }, 
                { 'AttributeName': 'version',
                  'KeyType': 'RANGE' } ],
    AttributeDefinitions=[ { 'AttributeName': 'name',
                             'AttributeType': 'S' },
                           { 'AttributeName': 'version',
                             'AttributeType': 'S' }, ],
    ProvisionedThroughput={ 'ReadCapacityUnits': 10,
                            'WriteCapacityUnits': 10 } )

# wait for the table to be ready
# this will block until the table is ACTIVE
table = boto3.resource('dynamodb').Table('Services')
table.wait_until_exists()

# insert some test data
with table.batch_writer() as batch:
    batch.put_item(Item={
                'name': 'testservice1', 
                'version': '1.0', 
                'endpoint_url': 'notarealurl1',
                'ttl': 300,
                'status': 'healthy' })
    batch.put_item(Item={
                'name': 'testservice2', 
                'version': '1.0', 
                'endpoint_url': 'notarealurl2',
                'ttl': 600,
                'status': 'healthy' })

The only attributes that we need to define are the keys. We will store additional attributes, but we can add them as we store items in the table.
We are using the name and version as the key because that matches our access pattern.
We are provisioning 10 read and write capacity units for the table. The number can be adjusted, depending on the amount of traffic the service receives and the effectiveness of the client caching.
After the table is created and active, we then prepare for the testing of our lookup service by inserting a couple of test records.

Looking Up Service Information from a DynamoDB Table

We are now ready to update our Lambda function so we can start using our DynamoDB table:


def lambda_handler(api_parameters, context):
    """Lambda hander for service lookup."""
    logger.info("lambda_handler - service_name: %s"
                " service_version: %s"
                % (api_parameters["service_name"],api_parameters["service_version"]))

    table = boto3.resource('dynamodb',region_name='us-east-1').Table('Services')

    dynamodb_response = table.get_item(
                    Key={
                        'name': str(api_parameters["service_name"]),
                        'version': str(api_parameters["service_version"])
                    }
                )

    if ('Item' in dynamodb_response):
        logger.info("found service with: %s" %
                     (dynamodb_response['Item']['endpoint_url'],))
        return {
            "endpoint_url": dynamodb_response['Item']['endpoint_url'],
            "ttl": dynamodb_response['Item']['ttl'],
            "status": dynamodb_response['Item']['status']
            }
    else:
        raise Exception('NotFound')

The function gets the item from the table, and then returns a JSON object with the information to the client.

Notice that we throw an exception if we don’t find a record in DynamoDB. We can use this exception in API Gateway to map to a 404 HTTP code. We update the two response sections in the Swagger file to make that happen:


{
  "swagger": "2.0",
  "info": {
    "title": "catalog_service",
    "version": "1.0.0"
  },
  "basePath": "/v1",
  "schemes": ["https"],
  "consumes": ["application/json"],
  "produces": ["application/json"],
  "paths": {
    "/catalog/{serviceName}/{serviceVersion}": {
      "parameters": [{
        "name": "serviceName",
        "in": "path",
        "description": "The name of the service to look up.",
        "required": true,
        "type": "string"
      },
      {
        "name": "serviceVersion",
        "in": "path",
        "description": "The version of the service to look up.",
        "required": true,
        "type": "string"
      }],
      "get": {
        "responses": {
          "200": {
            "description": "version information"
          },
          "404": {
            "description": "service not found"
          }
        },
        "x-amazon-apigateway-integration": {
          "type": "aws",
          "uri": "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/$catalog_serviceARN$/invocations",
          "httpMethod": "POST",
          "requestTemplates": {
            "application/json": "{\"service_name\": \"$input.params('serviceName')\",\"service_version\": \"$input.params('serviceVersion')\"}"
          },
          "responses": {
            "default": {
              "statusCode": "200",
              "schema": {
                "$ref": "#/definitions/CatalogServiceModel"
              }
            },
            ".*NotFound.*": {
              "statusCode": "404",
              "responseTemplates" : {
                 "application/json": "{\"error_message\":\"Service Not Found\"}"
                } 
            } 
          }
        }
      }
    }
  },
  "definitions": {
    "CatalogServiceModel": {
      "type": "object",
      "properties": {
        "endpoint_url": {
          "type": "string"
        },
        "ttl": {
          "type": "integer"
        },
        "status": {
          "type": "string"
        }
      },
      "required": ["endpoint_url", "ttl", "status"]
    }
  }
}

We use a regular expression (.*NotFound.*) under x-amazon-apigateway-integration -> responses to catch our exception and map it to a static JSON message.

We can now put everything together and update the code and API:


create_deployment_package("/tmp/catalog_service.zip", ["catalog_service.py"])
function_arn = update_lambda_function(
                       "/tmp/catalog_service.zip",
                       "catalog_service")
replace_instances_in_file("swagger.json",
                          "/tmp/swagger_with_arn.json",
                          "$catalog_serviceARN$", function_arn)
api_id = update_api("/tmp/swagger_with_arn.json")
deploy_api(api_id, "/tmp/swagger_with_arn.json", "dev")

We can use the following to test our deployment:


request_url="https://yourrestapiid.execute-api.us-east-1.amazonaws.com/"\
            "dev/catalog/testservice2/1.0"
response = requests.get(request_url)
json_response = json.loads(response.content)
logging.info("Endpoint URL: %s" % (json_response['endpoint_url'],))
logging.info("TTL: %i" % (json_response['ttl'],))
logging.info("Status: %s" % (json_response['status'],))

That should give us the following results:


INFO:root:Endpoint URL: notarealurl2
INFO:root:TTL: 600
INFO:root:Status: healthy

Serverless Service Discovery – Part 1: Get Started

by Magnus Bjorkman | on | in Python | Permalink | Comments |  Share

AWS provides a lot of features and services for writing serverless architectures with Python. In this four-part series, we will show how you can use Python to manage and implement Amazon API Gateway, AWS Lambda, and Amazon DynamoDB. We will use a common use case, service discovery, to showcase a simple way to do this with Python and boto3. Service discovery is a foundational service for microservices. There are many implementations running on servers or in containers, including Consul by HashiCorp and ZooKeeper from Apache.

This four-part series will cover the following topics:

  • Part 1: Get Started: Using Python and Swagger to Deploy to Amazon API Gateway and AWS Lambda
  • Part 2: Lookup: Looking Up Service Information in Amazon DynamoDB from AWS Lambda
  • Part 3: Registration: Using Signature Version 4 Authentication to API Gateway and AWS Lambda
  • Part 4: Registrar: Using a Registrar Agent in AWS Lambda to Manage Service Registration

By the end of the series, we will have built the system shown in this diagram:

Then we will be able to use a client that can look up a Hello World service in the discovery service, and call the Hello World service. We will also implement a registrar agent with the Hello World service that will keep the information about the Hello World service up-to-date in the discovery service.

Today’s post will cover these areas of our overall design:

We will create the basics of setting up a service running on API Gateway and Lambda. So we can do something easy to get us started, for this first step, we will return hard-coded values in the service.

We will create a few functions to make it easy to manage our serverless architecture. These are all of the imports used by management functions in this series:

import json
import os
import logging
import zipfile
import boto3

We set the log level to INFO:


logging.basicConfig()
logger = logging.getLogger()
logger.setLevel(logging.INFO)

Creating an AWS Lambda Function

We start with a couple of utility methods that will help us package a list of files or directories into a zip file that can be used with AWS Lambda:

def zipdir(path, ziph):
    """Add directory to zip file.

    :param path: The top level path to traverse to discover files to add.
    :param ziph: A handle to a zip file to add files to.
    """
    for root, dirs, files in os.walk(path):
        for file in files:
            ziph.write(os.path.join(root, file))


def create_deployment_package(package_name, file_names):
    """Create a deployment package for Lambda.

    :param package_name: The name of the package. Full or relative path.
    :param file_names: Files or folders to add to the package.
    """
    ziph = zipfile.ZipFile(package_name, "w", zipfile.ZIP_DEFLATED)
    for file_name in file_names:
        if (os.path.isdir(file_name)):
            zipdir(file_name, ziph)
        else:
            ziph.write(file_name)
    ziph.close()

The next function will use the package we just created to create the Lambda function:


def create_lambda_function(package_name, function_name, role,
                           handler, description, account_number):
    """Create a Lambda function from zip-file.

    :param package_name: The name of the package. Full or relative path.
    :param function_name: The name of the Lambda function to create.
    :param role: The Role ARN to use when executing Lambda function
    :param handler: The handler to execute when the Lambda function is called.
    :param description: The description of the Lambda function.
    :param: account_number: The Account number of the API Gateway using this
                            function.
    :return: The ARN for the Lambda function.
    """
    with open(package_name, "rb") as package_file:
        package_data = package_file.read()

    # connect to Lambda API
    client = boto3.client('lambda')

    # create the function
    response = client.create_function(
        FunctionName=function_name,
        Runtime="python2.7",
        Role=role,
        Handler=handler,
        Code={'ZipFile': package_data},
        Description=description,
        Timeout=60,
        MemorySize=128,
        Publish=True
    )

    # store away the name and arn for later use
    function_arn = response['FunctionArn']
    function_name = response['FunctionName']

    # add permissions for the function to be called by API Gateway
    response = client.add_permission(
        FunctionName=response['FunctionArn'],
        StatementId=response['FunctionName']+"-invoke",
        Action="lambda:InvokeFunction",
        Principal="apigateway.amazonaws.com",
        SourceArn='arn:aws:execute-api:us-east-1:'+account_number+':*'
    )

    return function_arn

We read the package into memory and provide it directly to the create_function method that creates our Lambda function. You might want to put a large package in Amazon S3 and then submit a reference to the package.

We need to give permissions to API Gateway to call our Lambda function. We do that using the AWS Lambda resource policies, adding the ARN of the API Gateway service for our account to the Lambda permissions.

Creating an API with Swagger

We again start with a couple of utility methods.


def replace_instances_in_file(filename_source, filename_target, old, new):
    """Replace string occurence in file.

    :param filename_source: The name of the file to read in.
    :param filename_target: The name of the file to write to.
    :param old: The string to find in the file.
    :param new: The string to replace any found occurrences with.
    """
    with open(filename_source, 'r') as f:
        newlines = []
        for line in f.readlines():
            newlines.append(line.replace(old, new))
    with open(filename_target, 'w') as f:
        for line in newlines:
            f.write(line)


def get_rest_api_name(swagger_file):
    """Get Rest API Name from Swagger file.

    :param swagger_file: The name of the swagger file. Full or relative path.
    :return: The name of the API defined in the Swagger file.
    """
    with open(swagger_file) as json_data:
        api_def = json.load(json_data)
        json_data.close()
        rest_api_name = api_def["info"]["title"]
        return rest_api_name

The replace_instances_in_file function allows us to take Lambda function ARNs and put them into specific places in the Swagger file. We will put in a marker string in the Swagger file. This function finds the marker and replaces it with the Lambda ARN.

The get_rest_api_name function allows us to get the name of the REST API specified in the Swagger file so we can use it with calls to the API Gateway API.

In the following function, we are using the newly released API function to import an API defined in Swagger:


def create_api(swagger_file_name):
    """Create an API defined in Swagger.

    :param swagger_file_name: The name of the swagger file.
                              Full or relative path.
    :return: The id of the REST API.
    """
    with open(swagger_file_name, "r") as swagger_file:
        swagger_data = swagger_file.read()

    client = boto3.client('apigateway')
    response = client.import_rest_api(body=swagger_data)

    return response['id']

Like the creation of the Lambda function, we read the Swagger file into memory and submit it directly to the function.

The last management function deploys the API to an API Gateway stage so we have a public host name that we can use to access the API:


def deploy_api(api_id, swagger_file, stage):
    """Deploy API to the given stage.

    :param api_id: The id of the API.
    :param swagger_file: The name of the swagger file. Full or relative path.
    :param stage: The name of the stage to deploy to.
    :return: Tuple of Rest API ID, stage and Enpoint URL.
    """
    client = boto3.client('apigateway')

    with open(swagger_file) as json_data:
        api_def = json.load(json_data)
        json_data.close()
        logger.info("deploying: "+api_id+" to "+stage)
        client.create_deployment(restApiId=api_id,
                                 stageName=stage)

        # print the end points
        logger.info("--------------------- END POINTS (START) ---------------")
        for path, path_object in api_def["paths"].iteritems():
            logger.info("End Point: https://%s"
                        ".execute-api.us-east-1.amazonaws.com/"
                        "%s%s" % (api_id, stage, path))
        logger.info("--------------------- END POINTS (END) -----------------")

        enpoint_url = ("https://%s"
                       ".execute-api.us-east-1.amazonaws.com/"
                       "%s" % (api_id, stage))
        return api_id, stage, enpoint_url

Deploying a Skeleton Service

We are now ready to test the functions with a simple skeleton of our service lookup function. The function is minimal and includes hard-coded values:


def lambda_handler(api_parameters, context):
    """Lambda hander for service lookup."""
    logger.info("lambda_handler - service_name: %s"
                " service_version: %s"
                % (api_parameters["service_name"],
                   api_parameters["service_version"]))

    response = {
            "endpoint_url": "notarealurl",
            "ttl": "300",
            "status": "healthy"
         }

    return response

Given a service name and a service version, the function will return three values:

  • The endpoint URL from which the service can be accessed.
  • The time to live (TTL) for this information so that a client knows for how long to cache this information and can avoid unnecessary calls to the service.
  • The status of the service, either healthy or unhealthy.

We define the API in a Swagger file for the preceeding Lambda function:


{
  "swagger": "2.0",
  "info": {
    "title": "catalog_service",
    "version": "1.0.0"
  },
  "schemes": ["https"],
  "consumes": ["application/json"],
  "produces": ["application/json"],
  "paths": {
    "/catalog/{serviceName}/{serviceVersion}": {
      "parameters": [{
        "name": "serviceName",
        "in": "path",
        "description": "The name of the service to look up.",
        "required": true,
        "type": "string"
      },
      {
        "name": "serviceVersion",
        "in": "path",
        "description": "The version of the service to look up.",
        "required": true,
        "type": "string"
      }],
      "get": {
        "responses": {
          "200": {
            "description": "version information"
          }
        },
        "x-amazon-apigateway-integration": {
          "type": "aws",
          "uri": "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/$catalog_serviceARN$/invocations",
          "httpMethod": "POST",
          "requestTemplates": {
            "application/json": "{\"service_name\": \"$input.params('serviceName')\",\"service_version\": \"$input.params('serviceVersion')\"}"
          },
          "responses": {
            "default": {
              "statusCode": "200",
              "schema": {
                "$ref": "#/definitions/CatalogServiceModel"
              }
            } 
          }
        }
      }
    }
  },
  "definitions": {
    "CatalogServiceModel": {
      "type": "object",
      "properties": {
        "endpoint_url": {
          "type": "string"
        },
        "ttl": {
          "type": "integer"
        },
        "status": {
          "type": "string"
        }
      },
      "required": ["endpoint_url", "ttl", "status"]
    }
  }
}

We define our service method as a GET method that will take the service name and service version as part of the path. We have also defined a response model (CatalogServiceModel) that specifies our return properties as the endpoint URL, the TTL, and the status.

The x-amazon-apigateway-integration element specifies how Amazon API Gateway will be integrated with AWS Lambda. The marker $catalog_serviceARN$ will be replaced with the AWS Lambda function ARN when this service is deployed.

We can now use all of the above to deploy our service to Lambda and API Gateway:


ACCOUNT_NUMBER = _your AWS account number_

create_deployment_package("/tmp/catalog_service.zip", ["catalog_service.py"])
function_arn = create_lambda_function(
                       "/tmp/catalog_service.zip",
                       "catalog_service",
                       "arn:aws:iam::"+ACCOUNT_NUMBER+":role/lambda_s3",
                       "catalog_service.lambda_handler",
                       "Looking up service information.",
                       ACCOUNT_NUMBER)
replace_instances_in_file("swagger.json",
                          "/tmp/swagger_with_arn.json",
                          "$catalog_serviceARN$", function_arn)
api_id = create_api("/tmp/swagger_with_arn.json")
deploy_api(api_id, "/tmp/swagger_with_arn.json", "dev")

We can use this to test our new deployment:


import requests
import json
import logging

request_url = "https://yourrestapiid.execute-api.us-east-1.amazonaws.com/"\
              "dev/catalog/testservice1/1.0"
response = requests.get(request_url)
json_response = json.loads(response.content)
logging.info("Endpoint URL: %s" % (json_response['endpoint_url'],))
logging.info("TTL: %i" % (json_response['ttl'],))
logging.info("Status: %s" % (json_response['status'],))

That should give us the following results:


Endpoint URL: notarealurl
TTL: 300
Status: healthy

Migrating to Boto3

by Michael Dowling | on | in Python | Permalink | Comments |  Share

Boto3, the latest version of the AWS SDK for Python, was released earlier this year. Since its release, we’ve seen more and more customers migrating to the latest major version of Boto. Boto3 provides many significant improvements over Boto:

  • Faster availability of API updates and consistency in exposed interfaces
  • Collections that provide an iterable interface to a collection of resources, including batch actions that can be used to perform an API operation on an entire collection
  • Waiters that make it easier to poll resources for status changes
  • Data-driven resource abstractions that provide an object-oriented API while still allowing a rapid relase cadence with significantly reduced maintenance overhead

We understand that migrating to a new major version of a project can be a big undertaking. It can take significant developer time to migrate; it requires more testing to ensure the successful migration of your application; and it can often involve wrestling with your dependency graph to ensure everything is compatible.

Using Boto and Boto3 side-by-side

To make the process of migrating to Boto3 easier, we released Boto3 under the boto3 namespace so that you can use Boto and Boto3 in the same project without conflicts. This allows you to continue to use Boto for legacy applications Boto3 for new development.


pip install boto
pip install boto3


import boto3
import boto.ec2

# For new development, use boto3. In this case, S3
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():
    print(bucket.name)

# Feel free to use boto for legacy code. In this case, EC2
conn = boto.ec2.connect_to_region('us-west-2')
conn.run_instances('ami-image-id')

If your legacy applications or individual application components are currently running without issue, you might not have much motiviation to migrate from Boto to Boto3. After all, if it ain’t broke, dont’ fix it. However, if you have new applications or applications that need to use newer services and service features, you are strongly encouraged to upgrade to Boto3. Boto3 is the future of Boto. It is where most of our development will be focused.

Anything holding you back?

We want to make Boto3 as good as it can be. Your input and feedback is crucial in helping us decide how to allocate developer time and which features to develop. We’d like to know if there’s something holding you back from migrating or using Boto3 for new application development. Maybe there’s a feature you relied on in Boto that is not present in Boto3. Or perhpas you find something confusing in the documentation or the way a client is used. Please open an issue on the Boto3 issue tracker and let us know. We’ll apply the appropriate issue labels to make it easier to find and +1 existing issues.

Ready to migrate?

Ready to migrate now? Our migration guide covers some of the high-level concepts to keep in mind. And again, feel free to use the the Boto3 issue tracker for questions or feature requests.

Welcome to the AWS Developer Blog for Python

by Jordon Phillips | on | in Python | Permalink | Comments |  Share

Hi everyone! Welcome to the AWS Developer Blog for Python. I’m Jordon Phillips, and I work on the AWS SDK for Python. This blog will be the place to go for information about:

  • Tips and tricks for using the AWS SDK for Python
  • New feature announcements
  • Deep dives into AWS SDK for Python features
  • Guest posts from AWS service teams

In the meantime, here are a few links to get you started:

We’re excited to get this blog started, and we hope to see you again real soon. Stay tuned!