spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vinay Kashyap <>
Subject Error in Spark Executors when trying to read HBase table from Spark with Kerberos enabled
Date Wed, 13 Jan 2016 13:09:40 GMT
Hi all,

I am using  *Spark 1.5.1 in YARN cluster mode in CDH 5.5.*
I am trying to create an RDD by reading HBase table with kerberos enabled.
I am able to launch the spark job to read the HBase table, but I notice
that the executors launched for the job cannot proceed due to an issue with
Kerberos and they are stuck indefinitely.

Below is my code to read a HBase table.

*Configuration configuration = HBaseConfiguration.create();*
*      configuration.set(TableInputFormat.INPUT_TABLE,
*      String hbaseKerberosUser = "sparkUser";*
*      String hbaseKerberosKeytab = "";*
*      if (!hbaseKerberosUser.trim().isEmpty() &&
!hbaseKerberosKeytab.trim().isEmpty()) {*
*        configuration.set("", "kerberos");*
*        configuration.set("", "kerberos");*
*        configuration.set("", "true");*
*        configuration.set("", "authentication");*
*        configuration.set("hbase.master.kerberos.principal",
*        configuration.set("hbase.regionserver.kerberos.principal",
*        configuration.set("",
*        configuration.set("hbase.thrift.kerberos.principal",
*        configuration.set("hbase.master.keytab.file",
*        configuration.set("hbase.regionserver.keytab.file",
*        configuration.set("",
*        configuration.set("hbase.thrift.keytab.file",
*        UserGroupInformation.setConfiguration(configuration);*
*        if (UserGroupInformation.isSecurityEnabled()) {*
*          UserGroupInformation ugi = UserGroupInformation*
*              .loginUserFromKeytabAndReturnUGI(hbaseKerberosUser,
*          TokenUtil.obtainAndCacheToken(configuration, ugi);*
*        }*
*      }*

*      System.out.println("loading HBase Table RDD ...");*
*      JavaPairRDD<ImmutableBytesWritable, Result> hbaseTableRDD =
*          configuration, TableInputFormat.class,
ImmutableBytesWritable.class, Result.class);*
*      JavaRDD<Row> tableRDD = getTableRDD(hbaseTableRDD, dataFrameModel);*
*  System.out.println("Count :: " + tableRDD.count());*
Following is the error which I can see in the container logs

*16/01/13 10:01:42 WARN security.UserGroupInformation:
PriviledgedActionException as:sparkUser (auth:SIMPLE) GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to
find any Kerberos tgt)]*
*16/01/13 10:01:42 WARN ipc.RpcClient: Exception encountered while
connecting to the server : GSS initiate
failed [Caused by GSSException: No valid credentials provided (Mechanism
level: Failed to find any Kerberos tgt)]*
*16/01/13 10:01:42 ERROR ipc.RpcClient: SASL authentication failed. The
most likely cause is missing or invalid credentials. Consider 'kinit'.*
* GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to
find any Kerberos tgt)]*
* at*
* at*
* at
* at
* at
* at
* at Method)*
* at*

Have valid Kerberos Token as can be seen below:

sparkUser@infra:/ebs1/agent$ klist
Ticket cache: FILE:/tmp/krb5cc_1001
Default principal: sparkUser@CERT.LOCAL

Valid starting    Expires           Service principal
13/01/2016 12:07  14/01/2016 12:07  krbtgt/CERT.LOCAL@CERT.LOCAL

Also, I confirmed that only reading from HBase is giving this problem.
Because I can read a simple file in HDFS and I am able to create the RDD as
After digging through some contents in the net, found that there is a
ticket in JIRA which is logged which is similar to what I am experiencing

Wanted to know if the issue is the same as I am facing..??
And any workaround for the same so that I can proceed with my requirement
reading from HBase table.??

*Thanks and regards*
*Vinay Kashyap*

View raw message