Access data in Amazon Timestream with AWS AppSync

AWS AppSync is a fully managed serverless GraphQL API service that simplifies application development by providing a single endpoint to securely query or update data from multiple databases, microservices, and APIs.

Many organizations across different industry verticals (e.g., health care, manufacturing, energy generation, and transportation) use a stream of data to improve efficiencies of business processes as well as generate long tail revenue opportunities. Many AWS customers keep data in a serverless time series Amazon Timestream database. Analytics in the cloud transform this data into information. Ultimately, this information must be delivered to an application. Use cases often require near real time access to information as it changes. Several data ingestion architectural patterns are available to deliver data in near real time.

This blog post shows how to access time series data in the Amazon Timestream database via AWS AppSync service using direct REST API via the endpoint discovery pattern.

Overview

AWS AppSync allows you to respond to GraphQL requests by performing operations on your data sources. For each GraphQL field on which you wish to run a query, mutation, or subscription, a resolver must be attached. In AWS AppSync code example published on Github, I configure two resolver functions that communicate with data sources. Resolvers are the connectors between GraphQL and a data sources.

You can write your own AWS AppSync resolver functions that directly interact with a data source API using the AWS AppSync HTTP data source via the JavaScript runtime (APPSYNC_JS) or through a Lambda proxy  that interacts with data source API.

An AWS AppSync function using APPSYNC_JS  runtime: a custom-built JavaScript runtime that lives within the AppSync service is ideal for use cases that require direct interaction with their target data source, including:

• Amazon DynamoDB, Amazon Aurora Serverless, and Amazon OpenSearch Service
• An arbitrary HTTP APIs endpoint as data source.
• AWS services using HTTP data sources (with AWS AppSync automatically signing requests with the provided data source role).

AWS AppSync functions that use a AWS Lambda data source as a proxy are ideal for use cases, including:

• Using a language other than JavaScript or Velocity Template Language (VTL)
• Adjusting and controlling CPU or memory to optimize performance
• Importing third-party libraries or requiring unsupported features in APPSYNC_JS
• Making multiple network requests and/or getting file system access to fulfil a query
• Batching requests using batching configuration.

In this post, I present an AWS AppSync code example that provides a single endpoint powered by JavaScrpt resolver and Direct Lambda resolvers to access the Amazon Timestream database. You can refer section for choosing between direct data source access and proxying via a Lambda data source in the AWS AppSync developer guide.

In the following sections, I walk through different parts of the solution.

Architecture

This architecture shows the flow of the use case.

1. You can publish IoT time series data using the AWS IoT Device SDK to the AWS IoT Core MQTT (Message Queuing Telemetry Transport) topic.
2. OR you can use the AWS IoT Greengrass edge runtime to collect data and export to AWS IoT Core.
3. You can set up MQTT topics on AWS IoT Core to route messages to subscribing clients.
4. AWS IoT core rule engine
   1. evaluates the rule of reading MQTT messages from subscribed topic
   2. then executes the action to Ingest data into Amazon Timestream
5. You then store IoT data in Timestream using the flat model, which is a default model for querying data.
6. AWS AppSync provides a single endpoint with two resolver options to choose from:
   1. A function that runs using JavaScript runtime
   2. A function that proxies requests to a lambda
7. To monitor your AppSync GraphQL API and help debug issues related to the requests, you can turn on logging to Amazon CloudWatch logs. You can also enable authorization and authentication using Amazon Cognito User Pools.
8. You can use AppSync APIs to implement web application, mobile apps faster. In this post, I test GraphQL using the AppSync AWS Console.

Ingest data into Amazon Timestream

You can ingest IoT time series data into Amazon Timestream through the built in IoT core rule. In order to publish data to IoT core, you use the MQTT test client available in IoT core. You can configure an IoT Core rule to subscribe to the MQTT topic and take an action on published events to ingest data into Amazon timestream database.

The screenshot below shows what the IoT core rule looks like in the AWS IoT Core console.

Creating AppSync GraphQL API to access Amazon Timestream

Let us create a public AppSync API protected with a simple API key to query the Amazon Timestream database. You can use either:

• The direct Lambda resolver function that uses AWS SDK to access Timestream and execute a query against the database and return the requested data.
• JavaScript function resolver function uses Timestream direct REST API via the region endpoint discovery pattern.

To show both resolver options, create schema with two separate query functions to retrieve the list of IoT events from timestream database table.

Create GraphQL Schema needed for our example

The example in this blog post only contains one Query field: getSensorData. The getSensorData retrieves the data for a duration in minutes. I keep it very simple with no pagination support.

schema {
          query: Query
        }

        type IOT {
          fleet: String
          fuel_capacity_in_litres: String
          load_capacity_in_tons: String
          make: String
          current_fuel_lvl_in_litres: String
          gps_location_latlong: String
          model: String
          truck_id: String
        }

        type Query {
          getSensorDataUsingLambdaResolver (durationInMinutes: Int): [IOT]
          getSensorDataUsingJsResolver (durationInMinutes: Int): [IOT]
        }

Prerequisites

To deploy the solution, you will need

1. An AWS Account
2. The AWS Command Line Interface (AWS CLI)
3. The timestream cell API endpooint
   • Acquire the endpoint for the API you would like to make calls against (Write or Query) using the DescribeEndpoints request.
     • Write call : ingest.timestream.<region>.amazonaws.com
     • Query call: query.timestream.<region>.amazonaws.com
   • An example CLI call for region us-east-1 as follows.
     • REGION_ENDPOINT=”https://query.timestream.us-east-1.amazonaws.com” REGION=us-east-1 aws timestream-write describe-endpoints \ –endpoint-url $REGION_ENDPOINT \ –region $REGION.
     • Verify the cell number in the Address. In the following example, note that cell2 which you will need in “Deploy the example section step 2”.
     • For example: query-cell2.timestream.us-east-1.amazonaws.com

Deploy the example

• Clone the GitHub repository to your local machine.

git clone https://github.com/aws-samples/aws-appsync-access-amazon-timestream-example

• Deploy the solution

aws cloudformation deploy --template-file cfn/main.template --stack-name appsync-timestream-api --parameter-overrides ParameterKey=TimestreamCellEndpoint,ParameterValue="<Refer point 3 in prerequisites>" --capabilities CAPABILITY_IAM CAPABILITY_AUTO_EXPAND

• Retrieve the details. Please note down the GraphQL endpoint and API key for testing purpose.

aws cloudformation describe-stacks --stack-name appsync-timestream-api --query "Stacks[0].Outputs" --output table

Test the example

You can test using AppSync API console.

AppSync Console

• Navigate to AppSync console and select the API name appsync-timestream-api  to view the dashboard for your API.
• Next click on Queries in the left-hand menu to view the query editor. From here, we can test out the API by running the following queries:

• Select query getSensorDataUsingJsResolver(durationInMinutes: 10) and choose the fields as shown in the screen below. Press the red color arrow button to execute the query. You can see the result in JSON format on the right side.

• Next, select the query getSensorDataUsingJsResolver(durationInMinutes: 10) and choose the fields as shown in screen below. Press the red color arrow button to execute the query. You can see the result in JSON format on the right side.

Clean up

To avoid incurring future charges, clean up the resources created. To delete the stack, use the following command.

aws cloudformation delete-stack --stack-name appsync-timestream-api

Conclusion

In this post, you learned how to create an AppSync API to connect to the Timestream database and query the data. In doing so, I presented two code examples, one with APPSYNC_JS runtime and one with a Lambda function. You can write your own code that implements your custom business logic by using AWS AppSync functions to access your data sources. This makes it easy for you to directly interact with data sources like Amazon DynamoDB, Aurora Serverless, OpenSearch Service, HTTP APIs, and other AWS services. AWS AppSync also makes it easy to interact with an AWS Lambda function by configuring a Lambda data source. Lambda data sources allow you to run complex business logic using AWS Lambda’s full set of capabilities to resolve a GraphQL request. In situations where you need to implement complex business logic that is not supported by the APPSYNC_JS runtime, you can use a Lambda data source as a proxy to interact with your target data source. To get started with AWS AppSync, try it on the AWS Free Tier.

About the Authors