AWS Open Source Blog

LDAP Integration for Open Distro for Elasticsearch

中文版 – Open Distro for Elasticsearch’s security plugin comes with authentication and access control out of the box. In prior posts we showed how you can change your admin password in Open Distro for Elasticsearch and how you can add your own SSL certificates to Open Distro for Elasticsearch.

One of the key steps to using the Security plugin is to decide on an authentication backend. The plugin has an internal user database, but many people prefer to use an existing authentication backend, such as an LDAP server, or some combination of the two. In this post we will talk about integrating the Security plugin with your LDAP or Active Directory and configuring the mapping between your backend user roles and Elasticsearch Security roles to provide granular access control.

The main configuration file for authentication and authorization modules isplugins/opendistro_security/securityconfig/config.yml. It defines how the Security plugin retrieves the user credentials, how it verifies these credentials, and how additional user roles are fetched from backend systems. Details on the structure of this configuration file are documented in the Open Distro for Elasticsearch docs.

Create Users and Assign to Groups in LDAP/AD

I’m using Active Directory for my LDAP server. Using the “Active Directory Users and Computers” administrative tool, I’ve created three users – esuser1, esuser2 & esuser3 and two groups – ESAdmins and ES-Read-Grp. I’ve made esuser1 a member of ESAdmins group, esuser2 a member of ES-Read-Grp and esuser3 is not part of any group.

 

Microsoft Active Directory configuration of users for Open Distro for Elasticsearch

Applying Changes to Security Plugin Configuration

The Security plugin stores its configuration — including users, roles, and permissions — in an index on the Elasticsearch cluster (.opendistro_security). Storing these settings in an index lets you change settings without restarting the cluster and eliminates the need to place configuration files on any node.

After changing any of the configuration files in plugins/opendistro_security/securityconfig, however, you must run plugins/opendistro_security/tools/securityadmin.sh to load these new settings into the index. This script identifies itself against a cluster through an admin TLS certificate, in .pem, .jks, .p12, or .pfx format.

If the .opendistro_security index is already initialized, you can also use Kibana to change users, roles, and permissions. However, you need to run securityadmin.sh at least once to initialize the index and configure your authentication and authorization methods.

For additional details on how to use securityadmin.sh, please refer to the Open Distro for Elasticsearch documentation.

Configure LDAP Details in Security Plugin

LDAP can be used for authentication and authorization and thus can be used both in the authc and authz sections of the configuration. The authc section is used for configuring authentication, which means to check whether the user has entered the correct credentials. authz is used for authorization, which defines how the role(s) for an authenticated user are retrieved and mapped.

For a new setup, you can use plugins/opendistro_security/securityconfig/config.yml to update the LDAP configuration details. In an established setup, make sure you retrieve the current Security plugin configuration from a running cluster (securityadmin.sh -r ...) and use those files to avoid losing any changes.

Authentication

Add the following lines to authc section of the config.yml file to enable LDAP authentication:

  ...  
    authc:
      ldap:
        http_enabled: true
        transport_enabled: false
        order: 1
        http_authenticator:
          type: "basic"
          challenge: false
        authentication_backend:
          type: "ldap"
          config:
   ...

In the config section, we’ll configure LDAP connection settings: hostname & port (hosts:), BindDN (bind_dn:), and password (password). Here is my copy of the section – replace "<< password >>" with your actual password for the bind user in plain text.

          config:
            enable_ssl: false
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: true
            hosts:
                - "ad.example.com:389"
            bind_dn: "cn=esuser1,OU=Users,OU=AD,dc=ad,dc=example,dc=com"
            password: "<< password >>"
            userbase: "OU=Users,OU=AD,dc=ad,dc=example,dc=com"
            usersearch: "(sAMAccountName={0})"
            username_attribute: "cn"

Authentication works by issuing an LDAP query containing the username against the user subtree of the LDAP tree.

The Security plugin first takes the configured LDAP query and replaces the placeholder {0} with the username from the user’s credentials.

usersearch: "(sAMAccountName={0})"

Then it issues this query against the user subtree. Currently, the whole subtree below the configured userbase is searched:

userbase: "OU=Users,OU=AD,dc=ad,dc=example,dc=com"

If the query was successful, the Security plugin retrieves the username from the LDAP entry. You can specify which attribute from the LDAP entry the Security plugin should use as the username:

username_attribute: "cn"

If this key is not set or null, then the Distinguished Name (DN) of the LDAP entry is used. I’ve set my to “CN”.

Authorization

Add the following lines to authz section of the config.yml file to enable LDAP authorization:

...
    authz:
      roles_from_myldap:
        http_enabled: true
        transport_enabled: false
        authorization_backend:
          type: "ldap"
          config:
...

Authorization is the process of retrieving backend roles for an authenticated user from an LDAP server. This is typically the same server(s) you use for authentication, but you can also use a different server. The only requirement is that the desired user actually exists on the LDAP server. Since the Security plugin always checks if a user exists in the LDAP server, you need to configure userbase, usersearch, and username_attribute in the authz section. Authorization works similarly to authentication. The Security plugin issues an LDAP query containing the username against the role subtree of the LDAP tree. Here is my copy of the section:

        config:
            enable_ssl: false
            enable_start_tls: false
            enable_ssl_client_auth: false
            verify_hostnames: true
            hosts:
            - "ad.example.com:389"
            bind_dn: "cn=esuser1,OU=Users,OU=AD,dc=ad,dc=example,dc=com"
            password: "Test1234"
            userbase: "OU=Users,OU=AD,dc=ad,dc=example,dc=com"
            usersearch: "(uid={0})"
            rolebase: "OU=Groups,OU=AD,dc=ad,dc=example,dc=com"
            rolesearch: "(member={0})"
            userrolename: "memberOf"
            rolename: "cn"

The Security plugin first takes the LDAP query for fetching roles (“rolesearch”) and substitutes any variables found in the query. For example, for a standard Active Directory installation, you would use the following role search. Here {0} is substituted with the DN of the user:

rolesearch: '(member={0})'

The Security plugin then issues the substituted query against the configured role subtree. The whole subtree below rolebase is searched.

rolebase: 'ou=groups,dc=example,dc=com'

After all roles have been fetched, the Security plugin extracts the final role names from a configurable attribute of the role entries:

rolename: 'cn'

If this is not set, the DN of the role entry is used.

Update the Security plugin Config

Now run securityadmin.sh to update the Security Plugin config as shown below:

$ /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh \
    -cacert /etc/elasticsearch/root-ca.pem \
-cert /etc/elasticsearch/kirk.pem \ 
-key /etc/elasticsearch/kirk-key.pem \
-h <Cluster-IP-or-FQDN> \
-f <Path-to-config>/config.yml -t config

Here I’m specifying the paths for the Root CA certificate(-cacert), Admin Certificate(-cert), and Admin Private Key (-key) files. The DN of the Admin certificate needs to be configured in the elasticsearch.yml file under opendistro_security.authcz.admin_dn . I’m restricting this update to the config file by explicitly specifying the config file location.

Create Elasticsearch Security Roles

Open Distro for Elasticsearch’s Security Plugin provides built-in Default Actions groups. We’ll create new roles and assign cluster- and index-level permissions using these default action groups. Please refer to the documentation for further details on creating new custom action groups combining the default action groups and single permissions, and also how to add Document and Field Level security as part of your role definitions.

Kibana

Using Kibana as the “admin” user, I created two custom roles, one with Full Read/Write permissions and another with just read-only permissions.

Select Security and click on Roles:
Open Distro for Elasticsearch Security plugin's main Kibana panel

Click on the + button to add a new role:

Showing Open Distro for Elasticsearch's security plugin role management page

Enter a new Role Name:

Kibana panel for setting the role name for a new backend role in Open Distro for Elasticsearch

Select Cluster Permissions and add an Action Group and/or Single Permissions:

Adding permissions for a new backend role in Open Distro for Elasticsearch's security plugin

Select Index Permissions and add an Index Pattern (supports wildcards) or select an index.

For each Index Pattern, add an Action Group and/or Single Permissions.

When you’re finished, click Save Role Definition.

Set index permissions for a new backend role in Open Distro for Elasticsearch

Map LDAP Backend Roles to Elasticsearch Security Roles

You can map Elasticsearch Roles to usernames, backend roles and/or hosts. Backend roles are determined as part of the authentication and authorization process and can come from an internal user configuration, LDAP, or JSON Web Token. We will map our roles to LDAP Group names.

In Kibana, select Security and click on Role Mappings:

 

Open Distro for Elasticsearch main Kibana panel for the security plugin, showing the location of the role mappings button

Click on the + button to add a new role mapping:

Manage role mappings panel in Open Distro for Elasticsearch's security plugin

Select the new Role from the dropdown (only unmapped roles are available for selection).

Add the appropriate Backend Role. (I’m using the CN of the LDAP Group name as I had configured the rolename: 'cn' in the security config. Without this configuration, the full DN needs be specified here.)

Click Submit to save the mapping:

Dialog for creating a new role mapping in Open Distro for Elasticsearch

Role Creation and Mapping Using the CLI

We’ll create the read-only role and mapping using the command-line interface (CLI).

In a new setup, you can also use the default security configuration files in plugins/opendistro_security/securityconfig/ to create roles and role mappings from the command line. In established setups, retrieve the current Security plugin configuration from a running cluster (securityadmin.sh -r ...) and use those output files to ensure you are updating the current configuration.

$ /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh \
    -cacert /etc/elasticsearch/root-ca.pem \
-cert /etc/elasticsearch/kirk.pem \ 
-key /etc/elasticsearch/kirk-key.pem \
-h <Cluster-IP-or-FQDN> \
-r

This creates five files with datestamps in the local folder – config, roles, roles_mapping, internal_users, action_groups

Copy the roles file to roles.yml and roles_mapping file to roles_mapping.yml

Append the configuration shown below to the roles.yml file. This creates a new role called CustomReadOnly with Read permissions

CustomReadOnly:
  cluster:
  - "CLUSTER_COMPOSITE_OPS_RO"
  indices:
    '*':
      '*':
      - "READ"

Append the configuration shown below to roles_mapping.yml. This maps the role CustomReadOnly to the LDAP group ES-Read-Grp.

CustomReadOnly:
  backendroles:
  - "ES-Read-Grp" hosts: [] users: []

Now run securityadmin.sh to update the Security Plugin config as shown below, to push one update at a time:

This first command-line pushes the roles.

$ /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh \
    -cacert /etc/elasticsearch/root-ca.pem \
-cert /etc/elasticsearch/kirk.pem \ 
-key /etc/elasticsearch/kirk-key.pem \
-h <Cluster-IP-or-FQDN> \
-f ./roles.yml -t roles

This command-line pushes the role mappings.

$ /usr/share/elasticsearch/plugins/opendistro_security/tools/securityadmin.sh \
    -cacert /etc/elasticsearch/root-ca.pem \
-cert /etc/elasticsearch/kirk.pem \ 
-key /etc/elasticsearch/kirk-key.pem \
-h <Cluster-IP-or-FQDN> \
-f ./roles_mapping.yml -t rolesmapping

Test Logging in as Different Users

As a Read-Write User esuser1

User esuser1 is part of ESAdmins LDAP group and is mapped to security role ESAdminRole. As part of this role, this user is allowed to perform read and write operations.

Create an index (succeeds as expected):

$ curl -XPUT -k "https://odfe-node1:9200/my-index" -u esuser1
Enter host password for user 'esuser1':
{
    "acknowledged":true,
    "shards_acknowledged":true,
    "index":"my-index"
 }

Add a document (succeeds as expected):

$ curl -XPOST -k "https://odfe-node1:9200/my-index/doc1/?pretty=true" -H 'Content-Type: application/json' -d'
        { 
         "text":"Hello World!"
        }' -u esuser1
Enter host password for user 'esuser1':
{
  "_index" : "my-index",
  "_type" : "doc1",
  "_id" : "GPxOLGoB9R98haWaBOPk",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 2,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

Run a search query (succeeds as expected):

$ curl -XPOST -k "https://odfe-node1:9200/_search?pretty=true"  -H 'Content-Type: application/json' -d' 
{ 
 "query":{
    "query_string":{
        "query":"hello"
        }
    }
}' -u esuser1
Enter host password for user 'esuser1':
{
  "took" : 99,
  "timed_out" : false,
  "_shards" : {
    "total" : 90,
    "successful" : 90,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 0.2876821,
    "hits" : [
      {
        "_index" : "my-index",
        "_type" : "doc1",
        "_id" : "GPxOLGoB9R98haWaBOPk",
        "_score" : 0.2876821,
        "_source" : {
          "text" : "Hello World!"
        }
      }
    ]
  }
}

As Read-only User esuser2

User esuser2 is part of ES-Read-Grp LDAP group and is mapped to security role CustomReadOnly. As part of this role, this user is allowed to perform only read operations.

Create an index (fails as expected. esuser2is not a member of a group that allows writing):

$ curl -XPUT -k "https://odfe-node1:9200/my-index" -u esuser2
Enter host password for user 'esuser2':
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "no permissions for [indices:data/write/index] and User [name=esuser2, roles=[Domain Users, Users, AWS Delegated Add Workstations To Domain Users], requestedTenant=null]"
      }
    ],
    "type" : "security_exception",
    "reason" : "no permissions for [indices:data/write/index] and User [name=esuser2, roles=[Domain Users, Users, AWS Delegated Add Workstations To Domain Users], requestedTenant=null]"
  },
  "status" : 403
}

Run a search query (succeeds as expected):

$ curl -XPOST -k "https://odfe-node1:9200/_search?pretty=true"  -H 'Content-Type: application/json' -d' 
{ 
 "query":{
    "query_string":{
        "query":"hello"
        }
    }
}' -u esuser2
Enter host password for user 'esuser2':
{
  "took" : 101,
  "timed_out" : false,
  "_shards" : {
    "total" : 90,
    "successful" : 90,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : 1,
    "max_score" : 0.2876821,
    "hits" : [
      {
        "_index" : "my-index",
        "_type" : "doc1",
        "_id" : "GPxOLGoB9R98haWaBOPk",
        "_score" : 0.2876821,
        "_source" : {
          "text" : "Hello World!"
        }
      }
    ]
  }
}

As User esuser3 – Not in Any Groups

User esuser3 is not part of any LDAP groups and is not mapped to any security roles. So, this user is not allowed to perform any operations.

Add a document (fails as expected. esuser3 is not a member of any group):

$ curl -XPOST -k "https://odfe-node1:9200/my-index/doc1/?pretty=true" -H 'Content-Type: application/json' -d'
        { 
         "text":"Hello World!"
        }' -u esuser3
Enter host password for user 'esuser3':
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "no permissions for [indices:data/write/index] and User [name=esuser3, roles=[], requestedTenant=null]"
      }
    ],
    "type" : "security_exception",
    "reason" : "no permissions for [indices:data/write/index] and User [name=esuser3, roles=[], requestedTenant=null]"
  },
  "status" : 403
}

Run a search query (fails as expected. esuser3 is not a member of any group):

$ curl -XPOST -k "https://odfe-node1:9200/_search?pretty=true"  -H 'Content-Type: application/json' -d' 
{ 
 "query":{
    "query_string":{
        "query":"hello"
        }
    }
}' -u esuser3
Enter host password for user 'esuser3':
{
  "error" : {
    "root_cause" : [
      {
        "type" : "security_exception",
        "reason" : "no permissions for [indices:data/read/search] and User [name=esuser3, roles=[], requestedTenant=null]"
      }
    ],
    "type" : "security_exception",
    "reason" : "no permissions for [indices:data/read/search] and User [name=esuser3, roles=[], requestedTenant=null]"
  },
  "status" : 403
}

Conclusion

In this post I covered integrating the Security plugin with your LDAP or Active Directory server for user authentication. I showed how to configure authorization and mapping between your backend user groups and Elasticsearch Security roles to provide granular role based access control.

Please refer to the Open Distro for Elasticsearch documentation for additional configuration options for Security Plugin’s Active Directory and LDAP integration.

Have an issue or question? Want to contribute? You can get help and discuss Open Distro for Elasticsearch on our forums. You can file issues here.

TAGS: , , ,
Jagadeesh Pusapadi

Jagadeesh Pusapadi

Jagadeesh Pusapadi is a Principal Solutions Architect for Amazon Web Services based in the UK. He is a quantum technologies enthusiast. He is passionate about helping customers build innovative solutions on AWS Cloud by providing architectural guidance. He also guides them to explore the world of quantum computing using Amazon Braket.