How can I renew an expired Kerberos ticket that I'm using for Amazon EMR authentication?

Last updated: 2019-11-21

I'm using Kerberos for authentication on an Amazon EMR cluster. My Kerberos ticket expired. When I try to run commands on the cluster, I get an error message like this:

19/11/04 22:12:16 WARN ipc.Client: Exception encountered while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
ls: Failed on local exception: java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]; Host
Details : local host is: "ip-xxx-x-x-x.ec2.internal/192.x.x.x";destination host is: "ip-xxx-x-x-x.ec2.internal":8020; 

Short Description

To renew an expired Kerberos ticket:

1.    Run the klist command to show the credentials issued by the key distribution center (KDC).

2.    To get a new ticket, run the kinit command and either specify a keytab file that contains credentials, or enter the password for your principal.

Resolution

1.    Connect to the master node using SSH.

2.    To confirm that the ticket is expired, run the klist command. This command checks for a credentials cache. If no credentials are cached, the ticket is expired.

[hadoop@ip-xxx-x-x-xxx ~]$ klist
klist: No credentials cache found (filename: /tmp/krb5cc_498)

3.    To confirm the Kerberos principal name, list the contents of the keytab file:

[hadoop@ip-xxx-x-x-xxx ~]$ klist -kt /etc/hadoop.keytab
Keytab name: FILE:/etc/hadoop.keytab
KVNO Timestamp           Principal
---- ------------------- ------------------------------------------------------
   2 07/04/2019 21:48:46 hadoop/ip-xxx-x-x-xxx.ec2.internal@EC2.INTERNAL
   2 07/04/2019 21:48:46 hadoop/ip-xxx-x-x-xxx.ec2.internal@EC2.INTERNAL
   2 07/04/2019 21:48:46 hadoop/ip-xxx-x-x-xxx.ec2.internal@EC2.INTERNAL

4.    To renew the Kerberos ticket, run kinit and specify both the keytab file and the principal:

[hadoop@ip-xxx-x-x-xxx ~]$ kinit -kt /etc/hadoop.keytab hadoop/ip-xxx-x-x-xxx.ec2.internal@EC2.INTERNAL

5.    Confirm that the credentials are cached:

[hadoop@ip-xxx-x-x-xxx ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_498
Default principal: hadoop/ip-xxx-x-x-xxx.ec2.internal@EC2.INTERNAL
Valid starting       Expires              Service principal
11/04/2019 22:13:47  11/05/2019 08:13:47  krbtgt/EC2.INTERNAL@EC2.INTERNAL
renew until 11/06/2019 22:13:47

6.    Run an HDFS command to confirm that the Kerberos ticket is working:

[hadoop@ip-XXX-XX-XX-XXX ~]$ hdfs dfs -ls /
Found 4 items
drwxr-xr-x   - hdfs hadoop          0 2019-11-05 22:45 /apps
drwxrwxrwt   - hdfs hadoop          0 2019-11-05 22:46 /tmp
drwxr-xr-x   - hdfs hadoop          0 2019-11-05 22:45 /user
drwxr-xr-x   - hdfs hadoop          0 2019-11-05 22:45 /var

Did this article help you?

Anything we could improve?


Need more help?