hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yifeng Jiang <uprushwo...@gmail.com>
Subject Re: Secure HBase setup
Date Tue, 22 May 2012 13:34:36 GMT
Hi Laxman,

Have you obtained a Kerberos ticket before connecting to the cluster?
Can you try the following from your client and then reconnect to the cluster?
$ kinit testuser/your-client-hostname

-Yifeng

On May 22, 2012, at 6:51 PM, Laxman wrote:

> We got stuck with a problem while verifying client authentication in a secure HBase cluster.
> We are able to start a secure HBase cluster successfully. 
> 
> However, clients are not able to establish secure connection with HBase server successfully.
> 
> Other details:
> HBase version: 0.94.0
> Hadoop version: 0.23.1
> Kerberos version: 1.10.1
> Java version: 1.6.0_31, 64 bit
> Linux version: SuSE 11.1 [Kernel version : 2.6.32.12-0.7-default x86_64 GNU/Linux]
> 
> We had gone thru the solutions available @
> http://docs.oracle.com/javase/1.5.0/docs/guide/security/jgss/tutorials/Troubleshooting.html
> https://ccp.cloudera.com/display/CDHDOC/Appendix+A+-+Troubleshooting#AppendixA-Troubleshooting-Problem2%3AJavaisunabletoreadtheKerberoscredentialscachecreatedbyversionsofMITKerberos1.8.1orhigher.
> 
> But none of then seems to work. Any clue?
> 
> There are no change in server logs as client is failing is failing even before it communicates
with server.
> Exception we are hitting (Client side logs):
> 
> 2012-05-22 09:42:22,627 WARN org.apache.hadoop.ipc.SecureClient: Exception encountered
while connecting to the server : javax.security.sasl.SaslException: GSS initiate failed [Caused
by GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)]
> 2012-05-22 09:42:22,627 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:testuser (auth:KERBEROS) cause:java.io.IOException: javax.security.sasl.SaslException:
GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]
> 2012-05-22 09:42:22,630 DEBUG org.apache.hadoop.ipc.SecureClient: closing ipc connection
to HOST-10-18-40-19/10.18.40.19:60020: javax.security.sasl.SaslException: GSS initiate failed
[Caused by GSSException: No valid credentials provided (Mechanism level: Failed to find any
Kerberos tgt)]
> java.io.IOException: javax.security.sasl.SaslException: GSS initiate failed [Caused by
GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos
tgt)]
> 	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClient.java:227)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
> 	at org.apache.hadoop.hbase.security.User.call(User.java:586)
> 	at org.apache.hadoop.hbase.security.User.access$700(User.java:50)
> 	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:440)
> 	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.handleSaslConnectionFailure(SecureClient.java:194)
> 	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:274)
> 	at org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.java:485)
> 	at org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.java:69)
> 	at org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:897)
> 	at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine.java:164)
> 	at $Proxy6.getProtocolVersion(Unknown Source)
> 	at org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.java:208)
> 	at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:303)
> 	at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:280)
> 	at org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:332)
> 	at org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:236)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1284)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1240)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getHRegionConnection(HConnectionManager.java:1227)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:936)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:832)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegionInMeta(HConnectionManager.java:933)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:836)
> 	at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.locateRegion(HConnectionManager.java:801)
> 	at org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
> 	at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
> 	at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
> 	at hbase.test.Hbasetest.main(Hbasetest.java:37)
> Caused by: javax.security.sasl.SaslException: GSS initiate failed [Caused by GSSException:
No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)]
> 	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:194)
> 	at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:138)
> 	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnection(SecureClient.java:176)
> 	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$500(SecureClient.java:84)
> 	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:267)
> 	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClient.java:264)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:396)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1177)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
> 	at org.apache.hadoop.hbase.security.User.call(User.java:586)
> 	at org.apache.hadoop.hbase.security.User.access$700(User.java:50)
> 	at org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:440)
> 	at org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(SecureClient.java:263)
> 	... 23 more
> Caused by: GSSException: No valid credentials provided (Mechanism level: Failed to find
any Kerberos tgt)
> 	at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:130)
> 	at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:106)
> 	at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:172)
> 	at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:209)
> 	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195)
> 	at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162)
> 	at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:175)
> 	... 40 more
> 2012-05-22 09:42:22,636 DEBUG org.apache.hadoop.ipc.SecureClient: IPC Client (1778276127)
connection to HOST-10-18-40-19/10.18.40.19:60020 from testuser: closed
> 2012-05-22 09:42:22,638 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-ROOT-,,0.70236052, hostname=HOST-10-18-40-19,
port=60020}, attempt=0 of 120 failed; retrying after sleep of 1000 because: javax.security.sasl.SaslException:
GSS initiate failed [Caused by GSSException: No valid credentials provided (Mechanism level:
Failed to find any Kerberos tgt)]
> 2012-05-22 09:42:22,640 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@6ecf829d;
serverName=HOST-10-18-40-19,60020,1337574445438
> 2012-05-22 09:42:23,641 DEBUG org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation:
Looked up root region location, connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@6ecf829d;
serverName=HOST-10-18-40-19,60020,1337574445438
> 2012-05-22 09:42:23,642 DEBUG org.apache.hadoop.ipc.SecureClient: RPC Server Kerberos
principal name for protocol=org.apache.hadoop.hbase.ipc.HRegionInterface is hbase/hadoop@HADOOP.COM
> 
> 
> --
> Regards,
> Laxman
> 


Mime
View raw message