Posted On: Nov 24, 2020
Amazon Elasticsearch Service now offers anomaly detection for high cardinality datasets. This new feature enables you to sift through thousands of metrics from millions of events to accurately pinpoint individual entities with abnormal patterns. By leveraging machine learning, Amazon Elasticsearch Service now provides reliable and actionable insights to drastically reduce the time to isolate and remediate issues. High cardinality anomaly detection can be invaluable for a number of operational, security and business use cases like identifying hosts with high CPU and memory consumption, finding services with most error rates, isolating suspicious users or IP addresses accessing sensitive information, or detecting outliers in sales by region.
Supporting high cardinality data streams is challenging due to its computational and scale demands to handle the complexity of the data. This problem is further exacerbated when operators need to pinpoint anomalous entities from massive datasets in near real-time. Amazon Elasticsearch Service uses a distributed architecture to address the challenges with performance and scalability by disseminating the load across the instances in your domain. Anomaly detection for high cardinality datasets is powered by the Random Cut Forest (RCF) algorithm, a light weight unsupervised machine learning algorithm, proven in production to handle multi-dimensional data streams at scale. The feature also provides Kibana dashboards and visualizations to quickly spot issues at a glance, thereby making anomaly detection accessible to all users, regardless of their machine learning knowledge.
The high cardinality anomaly detection feature is powered by Open Distro for Elasticsearch, an Apache 2.0-licensed distribution of Elasticsearch. To learn more about Open Distro for Elasticsearch and its anomaly detection plugin, visit the project website. Anomaly detection is available on all domains running Elasticsearch 7.9. To learn more, see the documentation.
The high cardinality anomaly detection feature for Amazon Elasticsearch Service is now available in 24 regions globally: US East (N. Virginia, Ohio), US West (Oregon, N. California), AWS GovCloud (US-Gov-East, US-Gov-West), Canada (Central), South America (Sao Paulo), EU (Ireland, London, Frankfurt, Paris, Stockholm, Milan), Asia Pacific (Singapore, Sydney, Tokyo, Seoul, Mumbai, Hong Kong), Middle East (Bahrain), China (Beijing – operated by Sinnet, Ningxia – operated by NWCD) and Africa (Cape Town). Please refer to the AWS Regional Services List for more information about Amazon Elasticsearch Service availability.