AWS Open Source Blog

The OKTA login screen for logging in to Open Distro For Elasticsearch Kibana

Add Single Sign-On (SSO) to Open Distro for Elasticsearch Kibana Using SAML and Okta

Open Distro for Elasticsearch Security implements the web browser single sign-on (SSO) profile of the SAML 2.0 protocol. This enables you to configure federated access with any SAML 2.0 compliant identity provider (IdP). In a prior post, I discussed setting up SAML-based SSO using Microsoft Active Directory Federation Services (ADFS). In this post, I’ll cover […]

Read More
Diagram of six elasticsearch nodes with three indexes showing uneven, skewed CPU, RAM, JVM, and I/O usage.

Demystifying Elasticsearch Shard Allocation

At the core of Open Distro for Elasticsearch’s ability to provide a seamless scaling experience, lies its ability distribute its workload across machines. This is achieved via sharding. When you create an index you set a primary and replica shard count for that index. Elasticsearch distributes your data and requests across those shards, and the […]

Read More
diagram: Network load balancer in front of the Ingress resource.

Using a Network Load Balancer with the NGINX Ingress Controller on Amazon EKS

Kubernetes Ingress is an API object that provides a collection of routing rules that govern how external/internal users access Kubernetes services running in a cluster. An ingress controller is responsible for reading the ingress resource information and processing it appropriately. As there are different ingress controllers that can do this job, it’s important to choose the right one for the type […]

Read More

AWS API Gateway for HPC Job Submission

AWS ParallelCluster simplifies the creation and the deployment of HPC clusters. AWS API Gateway is a fully managed service that makes it easy for developers to create, publish, maintain, monitor, and secure APIs at any scale. In this post we combine AWS ParallelCluster and AWS API Gateway to allow an HTTP interaction with the scheduler. […]

Read More
Open Distro for Elasticsearch logo

Open Distro for Elasticsearch 1.1.0 Released

We are happy to announce that Open Distro for Elasticsearch 1.1.0 is now available for download! Version 1.1.0 includes the upstream open source versions of Elasticsearch 7.1.1, Kibana 7.1.1, and the latest updates for alerting, SQL, security, performance analyzer, and Kibana plugins, as well as the SQL JDBC driver. You can find details on enhancements, […]

Read More
An Open Distro for Elasticsearch cluster with balanced resource usage

Use Elasticsearch’s _rollover API For Efficient Storage Distribution

Many Open Distro for Elasticsearch users manage data life cycle in their clusters by creating an index based on a standard time period, usually one index per day. This pattern has many advantages: ingest tools like Logstash support index rollover out of the box; defining a retention window is straightforward; and deleting old data is […]

Read More

Add Single Sign-On to Open Distro for Elasticsearch Kibana Using SAML and ADFS

Open Distro for Elasticsearch Security (Open Distro Security) comes with authentication and access control out of the box. Prior posts have discussed LDAP integration with Open Distro for Elasticsearch and JSON Web Token authentication with Open Distro for Elasticsearch. Security Assertion Markup Language 2.0 (SAML) is an open standard for exchanging identity and security information […]

Read More
Diagram showing where PartiQL fits with other data sources.

Announcing PartiQL: One query language for all your data

Data is being gathered and created at rates unprecedented in history. Much of this data is intended to drive business outcomes but, according to the Harvard Business Review, “…on average, less than half of an organization’s structured data is actively used in making decisions…” The root of the problem is that data is typically spread […]

Read More
AWS Parallel Cluster graphic

AWS ParallelCluster with AWS Directory Services Authentication

AWS ParallelCluster simplifies the creation and deployment of HPC clusters. In this post we combine ParallelCluster with AWS Directory Services to create a multi-user, POSIX-compliant system with centralized authentication and automated home directory creation. To grant only the minimum permissions to the nodes in the cluster, no AD configuration parameters or permissions are stored directly […]

Read More

Best Practices for Running Ansys Fluent Using AWS ParallelCluster

Using HPC (high performance computing) to solve Computational Fluid Dynamics (CFD) challenges has become common practice. As the growth from HPC workstation to supercomputer has slowed over the last decade or two, compute clusters have increasingly taken the place of single, big SMP (shared memory processing) supercomputers, and have become the ‘new normal’. Another, more […]

Read More