Q/Kdb+ on AWS Lambda: Serverless Time-Series Analytics at Scale

By Eric Corcoran, Cloud Solutions Architect at Kx
By Balaji Gopalan, Sr. Solutions Architect at AWS

For many financial institutions, high-performance computing (HPC) grids are a key infrastructure component of their financial and risk modelling. These model computations are becoming increasingly more complex, and a growing number of financial institutions are evolving to leverage serverless architectures.

AWS Lambda is a particularly desirable environment for HPC applications because of the high level of parallelization it supports. However, a serverless architecture only solves part of the problem.

The q language has been widely adopted by the financial industry because it’s capable of rapidly performing computationally dense arithmetic operations in parallel. Unfortunately, those q applications are written for traditional cloud architectures, not for AWS Lambda functions.

Kx, an AWS Partner Network (APN) Advanced Technology Partner, created a q/kdb+ runtime that enables financial institutions to optimize their applications for the serverless environment of AWS Lambda.

Q/kdb+ has been widely adopted by the financial services industry because of its small footprint, high performance, and high volume time-series analytics capabilities.

In this post, you will learn how to use the q/kdb+ runtime within the AWS Lambda environment. As an example, we’ll use a typical financial services application that calculates the option price of a stock using the Black Scholes pricing model.

The kdb+ Database and q Language

Kdb+ is a columnar high performance time-series database with in-memory analytics and streaming capabilities. It also has a fully-featured programming language called q, which is a vector-based functional programming language that can be further extended with qSQL, a superset of SQL.

One feature that sets q/kdb+ apart is its ability to combine streaming, in-memory, and historical data in one simple and unified platform. It supports time-series data types as well as joins, windowing, and temporal and bitemporal aggregation and arithmetic.

Serverless Applications and AWS Lambda

You can set up AWS Lambda to run your code in response to events like changes to data in an Amazon Simple Storage Service (Amazon S3) bucket or Amazon DynamoDB table, or in response to HTTP requests using Amazon API Gateway. You can also set up Lambda to invoke your code using API calls from an AWS Software Development Kit (SDK).

Setting up Lambda in this way allows you to easily build data processing triggers for AWS services, process streaming data stored in Amazon Kinesis, or create your own backend that operates with AWS scale, performance, and security.

Figure 1 – Typical serverless application event flow.

AWS Lambda Runtimes

The code you run on AWS Lambda is uploaded as a Lambda function. When you create a Lambda function, you specify configuration information, such as the amount of memory and maximum execution time you want to allow.

When a Lambda function is invoked, AWS Lambda launches an execution context based on the configuration settings you provide. The execution context is a temporary runtime environment that initializes any external dependencies of your Lambda function code, such as database connections or HTTP endpoints.

A runtime is a program that runs a Lambda function’s handler method when the function is invoked. A runtime is responsible for running the function’s setup code, reading the handler name from an environment variable, and reading invocation events from the Lambda runtime API. The runtime passes the event data to the function handler, and posts the response from the handler back to Lambda.

To support a range of languages, AWS Lambda uses the concept of custom runtimes. To use q/kdb+ with AWS Lambda, Kx created a custom runtime for q/kdb+.

Custom Runtimes and the Bootstrap File for q/kdb+

The custom runtime program is executed by a file named bootstrap. Once the Lambda function is invoked by an event, the bootstrap file passes the event data to the function handler, and posts the response from the handler back to the Lambda.

The q-specific bootstrap file begins by setting environment variables for QHOME, QLIC, and PATH. It also takes advantage of FIFO pipes in Linux to handle large payloads, as shown in the following code snippet.

#!/bin/sh

set -eu

export QHOME=/opt
export QLIC=/tmp
export PATH="$PATH:$QHOME/l64:$LAMBDA_TASK_ROOT"

while true; do
	WORKDIR=$(mktemp -d)
	cd "$WORKDIR"
	curl -sSLf -D headers -o event_data http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/next 
	

	AWS_LAMBDA_REQUEST_ID=$(sed -ne 's/^Lambda-Runtime-Aws-Request-Id:\s*\(.*\)\s*\r$/\1/ p' headers)
	# exported incase we want these in q-land
	export AWS_LAMBDA_DEADLINE_MS=$(sed -ne 's/^Lambda-Runtime-Deadline-Ms:\s*\(.*\)\s*\r$/\1/ p' headers)
	export AWS_LAMBDA_TRACE_ID=$(sed -ne 's/^Lambda-Runtime-Trace-Id:\s*\(.*\)\s*\r$/\1/ p' headers)

	# we stream to support large responses
	mkfifo response
     # we call q executable and q code and send response to response pipe
     q /var/task/$_HANDLER > response &

	# we want to stop just before the deadline
	WATCHDOG=$(((AWS_LAMBDA_DEADLINE_MS - $(date -u +%s) * 1000 - 1000) / 1000))
	[ $WATCHDOG -ge 3 ] || WATCHDOG=3

	curl -sSf -X POST --max-time $WATCHDOG \
		-H 'Expect:' \
		-H 'Content-Type: application/octet-stream' \
		-T response \
		http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/${AWS_LAMBDA_REQUEST_ID}/response &
	PID_C=$!
    
	q -q <&- > response &
	PID_Q=$!

	wait $PID_Q && RC_Q=0 || RC_Q=$?
	[ $RC_Q -eq 0 ] || error $AWS_LAMBDA_REQUEST_ID q

	# curl may have not finished yet
	wait $PID_C || true

	# we tidy up here in case some external resources were never actually used by our q code
	PIDS=$(ps --no-headers -o pid --ppid $$)
	echo "$PIDS" | xargs -r kill -9 2>/dev/null || true
	
	cd - >/dev/null
	unset AWS_LAMBDA_REQUEST_ID AWS_LAMBDA_DEADLINE_MS AWS_LAMBDA_TRACE_ID
	rm -rf "$WORKDIR"
done

exit 0

Using AWS Custom Runtime with q/kdb+

To execute our q code on the kdb+ database, we combined all of the preceding concepts into a q/kdb+ custom runtime program. You can try it out yourself by following these steps.

Step 1: Download the q/kdb+ Lambda Runtime

The q/kdb+ runtime program for AWS Lambda is available by searching for “Kx” in the AWS Serverless Application Repository. Instructions for deploying the latest version of Kx’s q/kdb+ runtime program to your Lambda environment are included in the preceding link.

Once you deploy the q/kdb+ runtime program to your Lambda environment, two files appear on the Lambda console:

bootstrap
script.q

Figure 2 – AWS Lambda console with q/kdb+ custom runtime.

The script.q file contains the q code to execute on the kdb+ database. As mentioned earlier, you specify the name of the file that contains the q code in the Handler field of the AWS Lambda user interface:

Kx-AWS-Lambda-2

The bootstrap file extracts the value of the Handler with the $_HANDLER function:

q /var/task/$_HANDLER > response &

When the $_HANDLER function is invoked, the JSON event object is passed into a file called event_data, like this:

curl -sSLf -D headers -o event_data http://${AWS_LAMBDA_RUNTIME_API}/2018-06-01/runtime/invocation/next

Step 2: Register the Lambda Function

After downloading the q/kdb+ runtime program, register your AWS account at serverless.kx.com.

Example: Option Pricing Using Black Scholes Model

One use case for a custom runtime q/kdb+ runtime is calculating option prices using the Black Scholes Pricing Model.

To set up the function to price a call option using the Black Scholes Pricing Model written in q, follow these steps:

Create the q file.
Set the Handler field to the name of the q file.
Invoke the Lambda function and decode the response.

Step 1: Create the q File

On the AWS Lambda console, we created a file called Black_Scholes.q with this content:

\c 1000 1000 
tbl:.j.k raze read0 hsym `$"event_data"

// For simplicity, an approximation for the Cumulative Normal Dist function 
cndf:{abs(x>0)-(exp[-.5*x*x]%sqrt 2*3.14159265358979323846)*t*.31938153+t*-.356563782+t*1.781477937+t*-1.821255978+1.330274429*t:1%1+.2316419*abs x}
BlackScholes:{[s;x;r;t;v;cp]
 d1:((log s%x)+(r+0.5*(v*v))*t)%v*sqrt t;
 d2:((log s%x)+(r-0.5*(v*v))*t)%v*sqrt t;
 $[cp~"c";(s*cndf[d1])-x*(exp neg r*t)*cndf[d2];
 cp~"p";(x*(exp neg r*t)*cndf[neg d2])-s*cndf[neg d1];
 '"(c)all/(p)ut error"]}
tbl[`BlackScholes_Option_Price]:BlackScholes[
    tbl.EquityPrice;
    tbl.Strike;
    tbl.Rate;
    tbl.Time;
    tbl.Volatility;
    -10h$tbl.CallPut
    ]
.j.j tbl

In the Black_Scholes.q script, the event_data is parsed into a table using:

tbl:.j.k raze read0 hsym `$"event_data"

Each parameter (Price, Strike, etc.) is then passed to the BlackScholes q function. The j.j function outputs the result in JSON format.

Step 2: Set the Handler to Black_Scholes.q

On the AWS Lambda user interface, we set the Handler field to Black_Scholes.q.

Kx-AWS-Lambda-4

When the bootstrap file is invoked, it calls the $_HANDLER variable to execute the Black Scholes q code.

Step 3: Invoke the Lambda Function and Decode the Response

The Lambda function is invoked by calling the aws lambda invoke command on the AWS Command Line Interface (CLI). That command passes the input parameters to our Black Scholes Pricing Model as a JSON message using the --payload parameter:

$ aws lambda invoke \
--invocation-type RequestResponse \
--function-name q_Black_Scholes \
--region us-east-1 \
--log-type Tail \
--payload '{ "Ticker": "AAPL","EquityPrice": 200,"Strike": 220,"Rate": 0.025,"Time": 0.25,"Volatility": 0.2673,"CallPut": "c"}' \
--profile kx_lambda_user output.txt \
--query 'LogResult' \
--output text |  base64 -D

The bootstrap file then passes the result object to the response pipe and the output is returned to your Lambda invoke CLI call.

The CLI command should return this response in the output.txt file:

{\"Ticker\":\"AAPL\",\"EquityPrice\":200,\"Strike\":220,\"Rate\":0.025,\"Time\":0.25,\"Volatility\":0.2673,\"CallPut\":\"c\",\"BlackScholes_Option_Price\":4.220232}

From the response, you can see the function was successful, and that it calculated the Black Scholes Option Price as $4.22. Instead of using the CLI interface, you can also use the AWS Lambda testing feature to test your function from the AWS Lambda console.

For simplicity, one option price was calculated, but this function can be easily extended to scale and process multiple option price calculations/events. One way would be using Amazon S3 object updates as event triggers to your q/kdb+ Lambda function. In this manner, a q/kdb+ Lambda function can scale to process multiple objects containing Black Scholes input parameters.

Summary

This post provides a glimpse into what’s possible using q/kdb+ within a serverless AWS Lambda framework. By using q/kdb+ in conjunction with Lambda, you get improved efficiency, easy scaling, and high availability to reliably meet your computational demands.

From here, you can choose to incorporate other AWS services, such as fronting your application with Amazon API Gateway, or processing data from Amazon Simple Queue Service (SQS), Amazon Simple Notification Service (SNS), or Amazon Kinesis Data Streams. You could also set up a CI/CD pipeline using AWS CodePipeline.

Serverless q/kdb+ has many promising capabilities that can be applied not only to finance, but also to Internet of Things (IoT) devices and applications, artificial intelligence and machine learning, and more.

As more applications evolve to use serverless frameworks, we look forward to seeing what applications are built using q/kdb+ on AWS Lambda.

.

.

Kx – APN Partner Spotlight

Kx is an APN Advanced Technology Partner. Kx is a suite of enterprise-level products and solutions centered around the kdb+ time series database. Kx technology combines streaming, in-memory, and historical data in a unified platform that’s optimized for ingesting, analyzing, and storing massive amounts of data.

Contact Kx | Solution Overview | AWS Marketplace

*Already worked with Kx? Rate this Partner

*To review an APN Partner, you must be an AWS customer that has worked with them directly on a project.

AWS Partner Network (APN) Blog