Monetize your custom HTTP APIs via AWS Data Exchange

AWS Data Exchange recently released support for API-based datasets, a new feature that enables customers to find, subscribe to, and use third-party API products from providers on AWS Data Exchange. With AWS Data Exchange for APIs, customers can use AWS-native authentication and governance, explore consistent API documentation and use supported AWS Software Development Kits (SDKs) to make API calls. Data providers can make their APIs discoverable by millions of AWS customers by listing their Amazon API Gateway APIs in the AWS Data Exchange catalog. However, many data providers have APIs built on different platforms such as Apigee and Kong, or are simply Amazon Elastic Compute Cloud (Amazon EC2) hosted HTTP services. In this blog post, I show how data providers can front your custom data APIs with an API Gateway endpoint. This helps you to list and monetize APIs using AWS Data Exchange.

Background on AWS Data Exchange for APIs

AWS Data Exchange for APIs simplifies the lives of developers and IT administrators who have to integrate and secure access to multiple third-party APIs. With AWS Data Exchange for APIs, data subscribers can make RESTful or GraphQL API calls directly to AWS Data Exchange. You then receive synchronous responses that contain the information you need, using the AWS SDK in the supported programming language of your choice. AWS Data Exchange integrates with the API provider, handles authentication, manages the API subscription, and ensures charges appear on your data subscriber’s AWS bill. Data subscribers can manage API access centrally with AWS Identity and Access Management (IAM). You can learn more about the feature at the AWS News Blog.

Solution overview: Monetize your custom HTTP APIs via AWS Data Exchange

AWS Data Exchange for APIs requires the backend API to be an Amazon API Gateway REST API endpoint. If the API you want to monetize is not based on Amazon API Gateway, you can simply create a wrapper API Gateway endpoint with an API Gateway endpoint with HTTP integration that forwards to your API. Once your API has responded with the data, it forwards the response containing data to the caller.

Here is how this solution works, as shown in the following diagram:

1. The data subscriber subscribes to the product and sends an HTTP request to an endpoint hosted by AWS Data Exchange, via AWS CLI or AWS SDK.
2. The AWS Data Exchange hosted API endpoint validates whether the caller is authorized to call the API endpoint.
3. AWS Data Exchange forwards the request to the wrapper API Gateway endpoint data providers create, while adding custom metadata if specified by the provider. The API on AWS Data Exchange passes all headers (except for the headers listed in Header forwarding section in the documentation), body, HTTP method, path, and query strings as-is from the customer request. It appends the headers identified here. When you configure the wrapper endpoint, you create an HTTP integration with your custom data API while specifying a resource policy that enables AWS Data Exchange service to call your custom API.
   •  The wrapper API Gateway endpoint forwards the HTTP request sent by subscribers to your custom API.
   • When the response is available, it is sent back to the wrapper API endpoint.
4. The wrapper API Gateway endpoint forwards the response to the AWS Data Exchange.
5. AWS Data Exchange forwards the response back to the caller.

To successfully create an API-based Data Exchange product, review the prerequisites outlined in the documentation.

Solution walkthrough

Step 1: Identify an API you want to monetize

As a data provider, you may already have an API that you want to monetize. If you don’t, proceed with this step. Otherwise, skip to Step 2.

For this blog post, I am going to use the pet-store public API. You can find more information about the API at the Amazon API Developer Guide.

Here is the API’s endpoint: http://petstore.execute-api.eu-west-1.amazonaws.com/petstore/pets

The API supports GET method /pets/{petId}: for read access to a pet identified by a petId value specified as a path variable of the incoming request URL. So when I invoke the URL http://petstore-demo-endpoint.execute-api.com/petstore/pets/1, I get the following response.

{
"id": 1,
"type": "dog",
"price": 249.99
}

Step 2: Create wrapper API and integrate it with custom API

AWS Data Exchange for APIs requires an Amazon API Gateway REST API that is able to authenticate and authorize calls from the AWS Data Exchange service principal. Since your custom data API was not originally set up with API Gateway, you must proxy all calls from API Gateway to your HTTP API directly. To do this, you must set up an API Gateway endpoint and complete an HTTP integration. You also must configure your Gateway API to use IAM authentication, set up the resource policy, and (optionally) import an OpenAPI schema describing your API. The more detailed your schema, the better the subscriber experience, as they can better understand how to call your API. If you are new to API Gateway, check out the Amazon API Gateway tutorials and workshops.

In this step, you create a simple wrapper REST API around your custom API. To do that, follow these steps:

1. Create the API
   • Open Amazon API Gateway. If this is your first time using API Gateway, you see a page that introduces you to the features of the service.
   • Choose Create API.
   • Under REST API, choose Build.
   • Under Create new API, choose New API.
   • Under Settings:
     • For API name, enter PetStoreWrapper.
     • If desired, enter a description in the Description field; otherwise, leave it empty.
     • Leave Endpoint Type set to Regional.
     • Choose Create API.
2. Create resource

To create the pets resource, do the following:

• • In the Resources tree, choose the root resource (/).
  • From the Actions dropdown menu, choose Create Resource.
  • Leave Configure as proxy resource unchecked.
  • For Resource Name, enter pets.
  • Leave Resource Path set to /pets.
  • Leave Enable API Gateway CORS unchecked.
  • Choose Create Resource.

3. Create method

To create the method, do the following:

• • In the Resources list, choose /pets.
  • In the Actions menu, choose Create Method.
  • From the dropdown menu, choose GET, and choose the checkmark icon.
  • Choose the Integration type set to HTTP.
  • Specify the endpoint URL to be http://petstore-demo-endpoint.execute-api.com/petstore/pets
  • Leave Use Default Timeout checked.
  • Choose Save.

4. Add AWS_IAM as the authorization method to your wrapper API

Your wrapper API must have the AWS_IAM as the authorization method. To do that, perform these steps:

• • Choose GET method.
  •  In the Method Execution pane, choose Method Request.
  •  Under Settings, for Authorization, choose the pencil icon (Edit).
  • Choose AWS_IAM from the dropdown list, and then choose the check mark icon (Update).
  • Choose Save.

5. Add policy

AWS Data Exchange for APIs uses your API’s resource policy in order to have access to invoke your API. Add the following policy to your API’s resource policy. To add this policy, do the following:

• • Open Resource Policies corresponding to your PetStoreWrapper API.
  • You also must update the account ID in the following snippet and then paste the same in the Resource Policy text field. If you have an existing resource policy, you can simply add the statement in bold.

{
"Version": "2012-10-17",
"Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "dataexchange.amazonaws.com"
            },
            "Action": "execute-api:Invoke",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "aws:SourceAccount": "<account_id>"
                }
            }
        }
    ]
}

• • Choose Save.

This grants AWS Data Exchange the ability to invoke your API when a subscriber makes a request and to get your API’s schema. It restricts these permissions to calls made by AWS Data Exchange’s service principal and requires that only your account can authorize AWS Data Exchange to integrate your API with AWS Data Exchange for APIs.

Now that you have built the REST API, you can deploy it.

6. Deploy the API

To deploy the API, do the following:

• • Open Resources corresponding to your PetStoreWrapper API.
  • Select /pets API.
  • From the Actions dropdown menu, choose Deploy API.
    • For Deployment stage, choose [new stage].
    • For Stage name, enter dev.
    • If desired, enter a Stage description.
    • If desired, enter a Deployment description.
  • Choose Deploy.

Data Providers may incur additional charges from API Gateway when using AWS Data Exchange for APIs. Visit Amazon API Gateway pricing to learn more.

Step 3: List API as an API dataset

To list the wrapper API as an API dataset using AWS Data Exchange, follow the steps outlined in our documentation in Step 3 to create a revison. Then follow the steps in step 4 to add API assets to the revison. You can use either the AWS CLI or the console to complete these steps, as outlined in the documentation. Make note of the asset id, revision id and dataset id for the created API asset. You use them to test your API.

Step 4: Test your API

To test your API, invoke your API Asset. You can invoke the API via the AWS CLI. I invoked the API via AWS CloudShell using the following command:

$aws dataexchange send-api-asset  --data-set-id 80a90c5fc527b5d57918c9510ee8dc94  --revision-id 5340eab51f082157d260d121dfdd3f18 --asset-id bad0a2ee4f519fbf5dc2e5223714a99e --method GET --path "pets"

When I run the above command, I receive a response containing the following JSON.

[
    {
        "id": 1,
        "type": "dog",
        "price": 249.99
    },
    {
        "id": 2,
        "type": "cat",
        "price": 124.99
    },
    {
        "id": 3,
        "type": "fish",
        "price": 0.99
    }
]

Congratulations! You just performed an HTTP integration with your custom APIs and created an API dataset that you can add to your private or public AWS Data Exchange product!

Next steps: Monetize your API by publishing it

The next step is to publish an AWS DataExchange product containing APIs as outlined in Step 5: Publish a new product containing APIs of the AWS Data Exchange documentation.

Cleanup

To avoid recurring charges from deploying the solution outlined in this post, perform the following steps:

1. Navigate to the API Gateway Console.
2. On the APIs page, select the API, Choose Actions, and then choose
3. Choose Delete.

Conclusion

In this blog post, I showed you how to create custom HTTP APIs using AWS Data Exchange for APIs. This process enables you to list and monetize your APIs using AWS Data Exchange. To learn more about the service, see AWS Data Exchange.

About the authors

Kanchan Waikar is a Principal Specialist Solutions Architect at Amazon Web Services with AWS Marketplace for machine learning group. She has over 15 years of experience building, architecting, and managing natural language processing (NLP) and software development projects. She has a masters degree in computer science (data science major) and enjoys helping customers build solutions backed by AI/ML based AWS services and partner solutions.

Esa Laine is a Senior Solutions Architect at Amazon Web Services with AWS Data Exchange. He has over 20 years of financial industry experience architecting and building mission critical software systems.