Amazon DynamoDB integrations

With Amazon DynamoDB, you can easily integrate your workloads with Amazon DynamoDB Accelerator (DAX) for up to10 times performance improvement, Amazon OpenSearch Service to enable real-time search and analytics capabilities, and a variety of other AWS services. DynamoDB provides you the flexibility to integrate with a broad range of AWS services that meet your use case requirements.

Amazon DynamoDB Accelerator (DAX) is a fully managed, highly available caching service built for Amazon DynamoDB. DAX delivers up to a 10 times performance improvement—from milliseconds to microseconds—even at millions of requests per second. DAX manages all the cache invalidation and data population to add in-memory acceleration to your DynamoDB tables. DAX is an inline cache so if the cached item doesn’t exist or is stale, DAX will automatically pass through to the underlying table.

Now you can focus on building great applications for your customers without worrying about performance at scale. As DAX is compatible with existing DynamoDB API calls, no application changes are required making it easy to get started. 

Amazon DynamoDB zero-ETL Integration with Amazon OpenSearch Service provides a no-code, fully managed ETL pipeline with near-real time replication from DynamoDB to OpenSearch Service. This zero-ETL integration enables customers to seamlessly synchronize their data from DynamoDB to OpenSearch Service within seconds, eliminating the need to write custom code to build and maintain complex data pipelines for extract, transform and load (ETL) operations. This integration reduces the operational burden and cost involved in keeping the data in sync between transactional and search databases, enabling customers to focus on their core business problems. 

You can quickly create your first pipeline from the Integrations tab in the DynamoDB console. The integration leverages Amazon OpenSearch Ingestion pipelines to define and process the transforms, routing, and mapping of your DynamoDB data to specific OpenSearch Service indexes.

This zero-ETL integration makes it is easy to create indexes to do full-text search, geo-spatial search, vector search, and more. OpenSearch Service searches and aggregations work together with DynamoDB queries to enable interactive, performant applications. DynamoDB customers can now leverage OpenSearch Service’s extensive search and machine learning capabilities to offer new experiences that boost user engagement and improve satisfaction with their applications. Learn more in this zero-ETL tutorial.

AWS Lambda triggers for Amazon DynamoDB enable you to easily set up custom logic to run in response to any changes to an item in DynamoDB. You can quickly create your first trigger from the Exports and Streams tab in the DynamoDB console, after enabling DynamoDB Streams. After configuring your trigger, your Lambda code can trigger within seconds of any changes to the data in your table.

With AWS Lambda triggers for DynamoDB, you can easily build database triggers to validate and enrich data, integrate with other services by replicating the data, or build entire event-driven architectures, such as order fulfillment or media processing, that react to data modification in DynamoDB tables. The Lambda function can perform any actions you specify, such as sending a notification or initiating a workflow.  

DynamoDB zero-ETL integration with OpenSearch Service abstracts away the operational complexity in orchestrating the replication of data from a transactional datastore to a search datastore. Data pipelines that are used to keep transactional and search datastores in sync can be challenging and costly to build and manage, and suffer from intermittent errors which are difficult to track. 

This integration enables Amazon DynamoDB customers to obtain near real-time search results from their transactional data by offering a fully managed solution for making transactional data from DynamoDB available in OpenSearch Service within seconds of being written. Customers simply choose the DynamoDB tables containing the data they want to analyze with OpenSearch Service, and this zero-ETL integration seamlessly replicates the schema and data into OpenSearch Service using OpenSearch Ingestion pipelines. Customers can replicate data from multiple DynamoDB tables into a single OpenSearch Service managed domain or serverless collection to derive holistic insights across several applications, while also consolidating their core analytics assets, gaining significant cost savings and operational efficiencies. 

Customers can get started by using the AWS Management Console for DynamoDB, OpenSearch Service, AWS CLI or AWS SDK or AWS CloudFormation. To enable an integration, customers first choose the DynamoDB table whose data needs to be replicated. Customers then choose either DynamoDB Streams for near real-time replication or DynamoDB Incremental Exports for delayed replication as the CDC mechanism to keep the data between the two systems in sync. 

This zero-ETL integration sets up an OpenSearch Ingestion pipeline in the customer’s account that takes care of replicating the data to an OpenSearch Service managed cluster or serverless collection. OpenSearch Ingestion understands the structure of the DynamoDB tables and then creates an equivalent OpenSearch Service managed domain or serverless collection and bootstraps the destination with the existing data from the DynamoDB tables. Optionally, customers can specify a schema for the indices that will be created in OpenSearch Service. 

This zero-ETL integration provides you a dashboard where you can monitor the state of your end-to-end integration with real time metrics and logs Amazon CloudWatch. You can set up alerting in case of breach of user defined thresholds. This integration also continually monitors the state of the DynamoDB tables and OpenSearch Service indices and immediately notifies users in case of regressions with any of these entities.

In order to ensure that OpenSearch Ingestion has the necessary permissions to replicate data across both these systems, DynamoDB zero-ETL integration with OpenSearch Service creates an IAM role with the necessary permissions to read data from DynamoDB tables and writes to an OpenSearch domain or collection. This role is then assumed by OpenSearch Ingestion pipelines to ensure that the right security posture is always maintained when moving the data from source to destination.

This zero-ETL integration uses the native data transformational capabilities of OpenSearch Ingestion pipelines to aggregate and filter the data while it is in motion. When moving the data from a DynamoDB table, customers may want to drop a few fields or create new fields based on aggregations across existing fields. 

Optionally, customers can also write custom logic for OpenSearch Ingestion to achieve bespoke transformational capability. For other users, who just want to move their entire data from source to sink, this zero-ETL integration will provide out-of-the box OpenSearch Ingestion blueprints so that they can perform the integrations with just a few button clicks.

This zero-ETL integration provides customers options to specify their custom data schema along with index mappings that is used to by OpenSearch Ingestion when writing data from DynamoDB to OpenSearch Service. This experience is added to the UI console within DynamoDB so that customers have full control on the format of indices that are created on OpenSearch Service.

There is no additional cost to use DynamoDB zero-ETL integration with OpenSearch Service apart from the cost of the existing underlying components. This zero-ETL integration uses Amazon OpenSearch Ingestion to read data in DynamoDB tables and replicates to OpenSearch Service. The cost involved in using DynamoDB zero-ETL integration with OpenSearch Service is the cost of OpenSearch Compute Units (OCUs) needed for OpenSearch Ingestion to replicate the data across the systems. Furthermore, customers have an option to choose either DynamoDB streams or incremental exports as the choice of CDC. For incremental exports, there is cost associated with writing data to S3 buckets. For DynamoDB streams, customers would be charged the standard charges for using DynamoDB streams.

DynamoDB zero-ETL integration with OpenSearch Service allows cross account ingestion of data. Users will be able to read data from DynamoDB tables in one account and write to OpenSearch Service domains or serverless collections in another account. OpenSearch Ingestion already allows cross account ingestion with other sources that it reads data from like Amazon S3 buckets and Amazon MSK clusters, and will enable cross account access for Amazon DynamoDB tables using the same IAM paradigms.

DynamoDB zero-ETL integration with Amazon OpenSearch Service is available in all the regions that OpenSearch Ingestion is available in today.