Serverless Service Discovery – Part 1: Get Started

AWS provides a lot of features and services for writing serverless architectures with Python. In this four-part series, we will show how you can use Python to manage and implement Amazon API Gateway, AWS Lambda, and Amazon DynamoDB. We will use a common use case, service discovery, to showcase a simple way to do this with Python and boto3. Service discovery is a foundational service for microservices. There are many implementations running on servers or in containers, including Consul by HashiCorp and ZooKeeper from Apache.

This four-part series will cover the following topics:

Part 1: Get Started: Using Python and Swagger to Deploy to Amazon API Gateway and AWS Lambda
Part 2: Lookup: Looking Up Service Information in Amazon DynamoDB from AWS Lambda
Part 3: Registration: Using Signature Version 4 Authentication to API Gateway and AWS Lambda
Part 4: Registrar: Using a Registrar Agent in AWS Lambda to Manage Service Registration

By the end of the series, we will have built the system shown in this diagram:

Then we will be able to use a client that can look up a Hello World service in the discovery service, and call the Hello World service. We will also implement a registrar agent with the Hello World service that will keep the information about the Hello World service up-to-date in the discovery service.

Today’s post will cover these areas of our overall design:

We will create the basics of setting up a service running on API Gateway and Lambda. So we can do something easy to get us started, for this first step, we will return hard-coded values in the service.

We will create a few functions to make it easy to manage our serverless architecture. These are all of the imports used by management functions in this series:

import json
import os
import logging
import zipfile
import boto3

We set the log level to INFO:


logging.basicConfig()
logger = logging.getLogger()
logger.setLevel(logging.INFO)

Creating an AWS Lambda Function

We start with a couple of utility methods that will help us package a list of files or directories into a zip file that can be used with AWS Lambda:

def zipdir(path, ziph):
    """Add directory to zip file.

    :param path: The top level path to traverse to discover files to add.
    :param ziph: A handle to a zip file to add files to.
    """
    for root, dirs, files in os.walk(path):
        for file in files:
            ziph.write(os.path.join(root, file))


def create_deployment_package(package_name, file_names):
    """Create a deployment package for Lambda.

    :param package_name: The name of the package. Full or relative path.
    :param file_names: Files or folders to add to the package.
    """
    ziph = zipfile.ZipFile(package_name, "w", zipfile.ZIP_DEFLATED)
    for file_name in file_names:
        if (os.path.isdir(file_name)):
            zipdir(file_name, ziph)
        else:
            ziph.write(file_name)
    ziph.close()

The next function will use the package we just created to create the Lambda function:


def create_lambda_function(package_name, function_name, role,
                           handler, description, account_number):
    """Create a Lambda function from zip-file.

    :param package_name: The name of the package. Full or relative path.
    :param function_name: The name of the Lambda function to create.
    :param role: The Role ARN to use when executing Lambda function
    :param handler: The handler to execute when the Lambda function is called.
    :param description: The description of the Lambda function.
    :param: account_number: The Account number of the API Gateway using this
                            function.
    :return: The ARN for the Lambda function.
    """
    with open(package_name, "rb") as package_file:
        package_data = package_file.read()

    # connect to Lambda API
    client = boto3.client('lambda')

    # create the function
    response = client.create_function(
        FunctionName=function_name,
        Runtime="python2.7",
        Role=role,
        Handler=handler,
        Code={'ZipFile': package_data},
        Description=description,
        Timeout=60,
        MemorySize=128,
        Publish=True
    )

    # store away the name and arn for later use
    function_arn = response['FunctionArn']
    function_name = response['FunctionName']

    # add permissions for the function to be called by API Gateway
    response = client.add_permission(
        FunctionName=response['FunctionArn'],
        StatementId=response['FunctionName']+"-invoke",
        Action="lambda:InvokeFunction",
        Principal="apigateway.amazonaws.com",
        SourceArn='arn:aws:execute-api:us-east-1:'+account_number+':*'
    )

    return function_arn

We read the package into memory and provide it directly to the create_function method that creates our Lambda function. You might want to put a large package in Amazon S3 and then submit a reference to the package.

We need to give permissions to API Gateway to call our Lambda function. We do that using the AWS Lambda resource policies, adding the ARN of the API Gateway service for our account to the Lambda permissions.

Creating an API with Swagger

We again start with a couple of utility methods.


def replace_instances_in_file(filename_source, filename_target, old, new):
    """Replace string occurence in file.

    :param filename_source: The name of the file to read in.
    :param filename_target: The name of the file to write to.
    :param old: The string to find in the file.
    :param new: The string to replace any found occurrences with.
    """
    with open(filename_source, 'r') as f:
        newlines = []
        for line in f.readlines():
            newlines.append(line.replace(old, new))
    with open(filename_target, 'w') as f:
        for line in newlines:
            f.write(line)


def get_rest_api_name(swagger_file):
    """Get Rest API Name from Swagger file.

    :param swagger_file: The name of the swagger file. Full or relative path.
    :return: The name of the API defined in the Swagger file.
    """
    with open(swagger_file) as json_data:
        api_def = json.load(json_data)
        json_data.close()
        rest_api_name = api_def["info"]["title"]
        return rest_api_name

The replace_instances_in_file function allows us to take Lambda function ARNs and put them into specific places in the Swagger file. We will put in a marker string in the Swagger file. This function finds the marker and replaces it with the Lambda ARN.

The get_rest_api_name function allows us to get the name of the REST API specified in the Swagger file so we can use it with calls to the API Gateway API.

In the following function, we are using the newly released API function to import an API defined in Swagger:


def create_api(swagger_file_name):
    """Create an API defined in Swagger.

    :param swagger_file_name: The name of the swagger file.
                              Full or relative path.
    :return: The id of the REST API.
    """
    with open(swagger_file_name, "r") as swagger_file:
        swagger_data = swagger_file.read()

    client = boto3.client('apigateway')
    response = client.import_rest_api(body=swagger_data)

    return response['id']

Like the creation of the Lambda function, we read the Swagger file into memory and submit it directly to the function.

The last management function deploys the API to an API Gateway stage so we have a public host name that we can use to access the API:


def deploy_api(api_id, swagger_file, stage):
    """Deploy API to the given stage.

    :param api_id: The id of the API.
    :param swagger_file: The name of the swagger file. Full or relative path.
    :param stage: The name of the stage to deploy to.
    :return: Tuple of Rest API ID, stage and Enpoint URL.
    """
    client = boto3.client('apigateway')

    with open(swagger_file) as json_data:
        api_def = json.load(json_data)
        json_data.close()
        logger.info("deploying: "+api_id+" to "+stage)
        client.create_deployment(restApiId=api_id,
                                 stageName=stage)

        # print the end points
        logger.info("--------------------- END POINTS (START) ---------------")
        for path, path_object in api_def["paths"].iteritems():
            logger.info("End Point: https://%s"
                        ".execute-api.us-east-1.amazonaws.com/"
                        "%s%s" % (api_id, stage, path))
        logger.info("--------------------- END POINTS (END) -----------------")

        enpoint_url = ("https://%s"
                       ".execute-api.us-east-1.amazonaws.com/"
                       "%s" % (api_id, stage))
        return api_id, stage, enpoint_url

Deploying a Skeleton Service

We are now ready to test the functions with a simple skeleton of our service lookup function. The function is minimal and includes hard-coded values:


def lambda_handler(api_parameters, context):
    """Lambda hander for service lookup."""
    logger.info("lambda_handler - service_name: %s"
                " service_version: %s"
                % (api_parameters["service_name"],
                   api_parameters["service_version"]))

    response = {
            "endpoint_url": "notarealurl",
            "ttl": "300",
            "status": "healthy"
         }

    return response

Given a service name and a service version, the function will return three values:

The endpoint URL from which the service can be accessed.
The time to live (TTL) for this information so that a client knows for how long to cache this information and can avoid unnecessary calls to the service.
The status of the service, either healthy or unhealthy.

We define the API in a Swagger file for the preceeding Lambda function:


{
  "swagger": "2.0",
  "info": {
    "title": "catalog_service",
    "version": "1.0.0"
  },
  "schemes": ["https"],
  "consumes": ["application/json"],
  "produces": ["application/json"],
  "paths": {
    "/catalog/{serviceName}/{serviceVersion}": {
      "parameters": [{
        "name": "serviceName",
        "in": "path",
        "description": "The name of the service to look up.",
        "required": true,
        "type": "string"
      },
      {
        "name": "serviceVersion",
        "in": "path",
        "description": "The version of the service to look up.",
        "required": true,
        "type": "string"
      }],
      "get": {
        "responses": {
          "200": {
            "description": "version information"
          }
        },
        "x-amazon-apigateway-integration": {
          "type": "aws",
          "uri": "arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/$catalog_serviceARN$/invocations",
          "httpMethod": "POST",
          "requestTemplates": {
            "application/json": "{\"service_name\": \"$input.params('serviceName')\",\"service_version\": \"$input.params('serviceVersion')\"}"
          },
          "responses": {
            "default": {
              "statusCode": "200",
              "schema": {
                "$ref": "#/definitions/CatalogServiceModel"
              }
            } 
          }
        }
      }
    }
  },
  "definitions": {
    "CatalogServiceModel": {
      "type": "object",
      "properties": {
        "endpoint_url": {
          "type": "string"
        },
        "ttl": {
          "type": "integer"
        },
        "status": {
          "type": "string"
        }
      },
      "required": ["endpoint_url", "ttl", "status"]
    }
  }
}

We define our service method as a GET method that will take the service name and service version as part of the path. We have also defined a response model (CatalogServiceModel) that specifies our return properties as the endpoint URL, the TTL, and the status.

The x-amazon-apigateway-integration element specifies how Amazon API Gateway will be integrated with AWS Lambda. The marker $catalog_serviceARN$ will be replaced with the AWS Lambda function ARN when this service is deployed.

We can now use all of the above to deploy our service to Lambda and API Gateway:


ACCOUNT_NUMBER = _your AWS account number_

create_deployment_package("/tmp/catalog_service.zip", ["catalog_service.py"])
function_arn = create_lambda_function(
                       "/tmp/catalog_service.zip",
                       "catalog_service",
                       "arn:aws:iam::"+ACCOUNT_NUMBER+":role/lambda_s3",
                       "catalog_service.lambda_handler",
                       "Looking up service information.",
                       ACCOUNT_NUMBER)
replace_instances_in_file("swagger.json",
                          "/tmp/swagger_with_arn.json",
                          "$catalog_serviceARN$", function_arn)
api_id = create_api("/tmp/swagger_with_arn.json")
deploy_api(api_id, "/tmp/swagger_with_arn.json", "dev")

We can use this to test our new deployment:


import requests
import json
import logging

request_url = "https://yourrestapiid.execute-api.us-east-1.amazonaws.com/"\
              "dev/catalog/testservice1/1.0"
response = requests.get(request_url)
json_response = json.loads(response.content)
logging.info("Endpoint URL: %s" % (json_response['endpoint_url'],))
logging.info("TTL: %i" % (json_response['ttl'],))
logging.info("Status: %s" % (json_response['status'],))

That should give us the following results:


Endpoint URL: notarealurl
TTL: 300
Status: healthy

Select your cookie preferences

AWS Developer Tools Blog

Serverless Service Discovery – Part 1: Get Started

Creating an AWS Lambda Function

Creating an API with Swagger

Deploying a Skeleton Service

Resources

Follow