hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kyip <kyip1...@yahoo.com.INVALID>
Subject HBase 2.1 client intermittent Kerberos failure in IBM JVM
Date Fri, 07 Feb 2020 15:57:42 GMT
Hi,

I have an application  that has been working with HBase 1.x servers using
Kerberos authentication for a while.

I upgraded the application to support HBase 2.1 servers recently. The
application is working fine in Oracle JVM but not in IBM JVM (both Java
1.8).

In IBM JVM, after the successful UserGroupInformation.loginUserFromKeytab(),
it always fails to find the javax.security.auth.Subject during the
PROCESS_TGS step and the TGS_REQ was never sent for the /hbase service. So,
in order to address this, I made use of
UserGroupInformation.getCurrentUser().doAs(<my HBase operation>) where <my
HBase operation> can be HBase available check, connection creation, get
table names, table scan, put, get, etc. This approach seems to work except I
am facing intermittent failures where the following error is logged:

[2/7/20 6:50:20:682 GMT] 0000014e SystemErr                                              
    
R javax.security.sasl.SaslException: Call to
eng-bigbang-hadoop01.rpega.com/10.20.204.19:16020 failed on local exception:
javax.security.sasl.SaslException: Failure to initialize security context
[Caused by org.ietf.jgss.GSSException, major code: 11, minor code: 0
        major string: General failure, unspecified at GSSAPI level
        minor string: Cannot get credential for principal default principal]
[Caused by javax.security.sasl.SaslException: Failure to initialize security
context [Caused by org.ietf.jgss.GSSException, major code: 11, minor code: 0
        major string: General failure, unspecified at GSSAPI level
        minor string: Cannot get credential for principal default
principal]]

This is the same error that consistently happens before I used the
UserGroupInformation.getCurrentUser().doAs(<my HBase operation>) technique.
It seems to me somehow the "login context" was lost occasionally and that is
why the logged in Subject cannot be found.

Not sure how this is relevant to the issue here. From my debugging sessions,
I notice is that HBase 1.x performs the PROCESS_TGS step in the same thread
as the initial steps while HBase 2.1 performs the step in a separate thread.

Since my application has been working with HBase 1.x servers (in both Oracle
and IBM JVM's) and my application also works properly with HDFS services in
Kerberos configuration in both Oracle and IBM JVM's, this seems to be a
HBase 2.x issue. (I also tried HBase 2.2 client jars which did not help.)

Any suggestion on how to address or troubleshoot this issue is greatly
appreciated.


Best Regards,

Kai





--
Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-User-f4020416.html

Mime
View raw message