New Logstash Plugin – Search DynamoDB Content using Elasticsearch

When I take a look at our recent service releases and combine it with some of the AWS-related repos on GitHub, an interesting trend becomes apparent. It seems to me that connecting AWS services to each other and to third party tools is becoming more and more common. For example, in a recent post, I showed you how to combine CloudWatch Logs, Elasticsearch, and Kibana to visualize event data. In another post I showed you how AWS OpsWorks can provision and manage ECS container instances. As I noted in that post, I think of this as “peanut butter and chocolate” — combining two good flavors in to another that is even better.

DynamoDB + Elasticsearch
Today I would like to tell you about another way to connect several interesting pieces of technology together: DynamoDB and Elasticsearch by way of a DynamoDB Streams connector (plugin) for Logstash.

You can run Logstash on an Amazon Elastic Compute Cloud (Amazon EC2) instance or on-premises. After you configure it to take input from the DynamoDB tables and streams that you designate, it will track changes (inserts, updates, and deletions) to the tables and update your Elasticsearch cluster accordingly. You can also configure the plugin to scan the table (with checkpoints along the way) to bring your cluster in to sync with the table.

Once your cluster is up, running, and tracking changes to your tables, you can perform efficient queries (structured, full-text, and multifield) using Elasticsearch. Your queries can make use of proximity matching and partial matching, and you can also control relevance using Elasticsearch’s scoring infrastructure (you can learn about these topics and more in Elasticsearch: The Definitive Guide).

Changes made to your DynamoDB tables are reflected in the stream very quickly (generally a second or less). The plugin will have access to these changes and will update your cluster as expeditiously as possible.

Download, Install, and Run
You can download the plugin from our new DynamoDB Community page, install it on an EC2 instance or on-premises, point it at your Elasticsearch cluster, and start searching your DynamoDB content today! Start by reading the documentation and following the directions in the README.

— Jeff;