hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <els...@apache.org>
Subject Re: HBase 2.1 client intermittent Kerberos failure in IBM JVM
Date Mon, 10 Feb 2020 14:43:04 GMT
There have been multiple issues filed in Hadoop relating to the 
implementation differences of IBM Java compared to Oracle Java and 
OpenJDK [1]. Make sure that you're not running into any of them as a 
first step.

After that, you'd want to compare the differences of the Java platforms, 
with krb5 JVM level debugging *and* org.apache.hadoop.security debugging 
enabled, and understand what is fundamentally different in their 
runtimes. From there, hopefully it becomes obvious what the solution is. 
Sometimes it's just different JAAS options that will need to be added 
into UserGroupInformation.

Having done it before, it's not a super-fun experience. Your other 
solution is to just not use IBM Java. Good luck.

[1] https://www.google.com/search?q=hadoop+jaas+ibm+site%3Aissues.apache.org

On 2/7/20 10:57 AM, kyip wrote:
> Hi,
> 
> I have an application  that has been working with HBase 1.x servers using
> Kerberos authentication for a while.
> 
> I upgraded the application to support HBase 2.1 servers recently. The
> application is working fine in Oracle JVM but not in IBM JVM (both Java
> 1.8).
> 
> In IBM JVM, after the successful UserGroupInformation.loginUserFromKeytab(),
> it always fails to find the javax.security.auth.Subject during the
> PROCESS_TGS step and the TGS_REQ was never sent for the /hbase service. So,
> in order to address this, I made use of
> UserGroupInformation.getCurrentUser().doAs(<my HBase operation>) where <my
> HBase operation> can be HBase available check, connection creation, get
> table names, table scan, put, get, etc. This approach seems to work except I
> am facing intermittent failures where the following error is logged:
> 
> [2/7/20 6:50:20:682 GMT] 0000014e SystemErr
> R javax.security.sasl.SaslException: Call to
> eng-bigbang-hadoop01.rpega.com/10.20.204.19:16020 failed on local exception:
> javax.security.sasl.SaslException: Failure to initialize security context
> [Caused by org.ietf.jgss.GSSException, major code: 11, minor code: 0
>          major string: General failure, unspecified at GSSAPI level
>          minor string: Cannot get credential for principal default principal]
> [Caused by javax.security.sasl.SaslException: Failure to initialize security
> context [Caused by org.ietf.jgss.GSSException, major code: 11, minor code: 0
>          major string: General failure, unspecified at GSSAPI level
>          minor string: Cannot get credential for principal default
> principal]]
> 
> This is the same error that consistently happens before I used the
> UserGroupInformation.getCurrentUser().doAs(<my HBase operation>) technique.
> It seems to me somehow the "login context" was lost occasionally and that is
> why the logged in Subject cannot be found.
> 
> Not sure how this is relevant to the issue here. From my debugging sessions,
> I notice is that HBase 1.x performs the PROCESS_TGS step in the same thread
> as the initial steps while HBase 2.1 performs the step in a separate thread.
> 
> Since my application has been working with HBase 1.x servers (in both Oracle
> and IBM JVM's) and my application also works properly with HDFS services in
> Kerberos configuration in both Oracle and IBM JVM's, this seems to be a
> HBase 2.x issue. (I also tried HBase 2.2 client jars which did not help.)
> 
> Any suggestion on how to address or troubleshoot this issue is greatly
> appreciated.
> 
> 
> Best Regards,
> 
> Kai
> 
> 
> 
> 
> 
> --
> Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-f4020416.html
> 

Mime
View raw message