Return-Path: X-Original-To: apmail-hbase-dev-archive@www.apache.org Delivered-To: apmail-hbase-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AD3D19D94 for ; Tue, 22 May 2012 13:35:09 +0000 (UTC) Received: (qmail 85618 invoked by uid 500); 22 May 2012 13:35:09 -0000 Delivered-To: apmail-hbase-dev-archive@hbase.apache.org Received: (qmail 85569 invoked by uid 500); 22 May 2012 13:35:09 -0000 Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hbase.apache.org Delivered-To: mailing list dev@hbase.apache.org Received: (qmail 85560 invoked by uid 99); 22 May 2012 13:35:09 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 May 2012 13:35:09 +0000 X-ASF-Spam-Status: No, hits=-0.7 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of uprushworld@gmail.com designates 209.85.160.41 as permitted sender) Received: from [209.85.160.41] (HELO mail-pb0-f41.google.com) (209.85.160.41) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 May 2012 13:35:02 +0000 Received: by pbbrp2 with SMTP id rp2so12184909pbb.14 for ; Tue, 22 May 2012 06:34:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=content-type:mime-version:subject:from:in-reply-to:date :content-transfer-encoding:message-id:references:to:x-mailer; bh=gQk4xcl1D/ox0DzWhw3AcUF6IVNiMIKXyRxDjBM0m6Q=; b=TS1joIr4Qtp05ZdXusiQi2Lq0JfdBuPEYACw8gk9LnnC+BI6vsxucteMp8PK9ylr8s mB6umKguMEAxti7Ni7PLq31fJKmMWAlycb2WBVCg7F8OqmgvZfluaSYWEe8NhQ0LYjbp sxeWqBWq9q5xu52uh3YWQc9PwdY48tS9w0wK4aHbk7OeVin0LRIKmbhaE7lfWraq+wtV cmBNP7+XqrrlMV44tp5PPtORM3STs6uITwELvCYG7RN7BsmTk+Y4C94MhYgEEoknoXFq 66MkNWBUOJ2enxzmXFji+YV3iNFy380xiPS6A8ptL0+Tfbokq3X6N1W+uir2X7X2gOmb 5Hmg== Received: by 10.68.217.233 with SMTP id pb9mr18651012pbc.59.1337693680520; Tue, 22 May 2012 06:34:40 -0700 (PDT) Received: from [192.168.11.6] (ab152063.dynamic.ppp.asahi-net.or.jp. [183.76.152.63]) by mx.google.com with ESMTPS id ub8sm26726870pbc.44.2012.05.22.06.34.38 (version=TLSv1/SSLv3 cipher=OTHER); Tue, 22 May 2012 06:34:39 -0700 (PDT) Content-Type: text/plain; charset=us-ascii Mime-Version: 1.0 (Apple Message framework v1278) Subject: Re: Secure HBase setup From: Yifeng Jiang In-Reply-To: <000001cd3800$76a4a6f0$63edf4d0$%ch@huawei.com> Date: Tue, 22 May 2012 22:34:36 +0900 Content-Transfer-Encoding: quoted-printable Message-Id: References: <000001cd3800$76a4a6f0$63edf4d0$%ch@huawei.com> To: dev@hbase.apache.org, lakshman.ch@huawei.com X-Mailer: Apple Mail (2.1278) Hi Laxman, Have you obtained a Kerberos ticket before connecting to the cluster? Can you try the following from your client and then reconnect to the = cluster? $ kinit testuser/your-client-hostname -Yifeng On May 22, 2012, at 6:51 PM, Laxman wrote: > We got stuck with a problem while verifying client authentication in a = secure HBase cluster. > We are able to start a secure HBase cluster successfully.=20 >=20 > However, clients are not able to establish secure connection with = HBase server successfully. >=20 > Other details: > HBase version: 0.94.0 > Hadoop version: 0.23.1 > Kerberos version: 1.10.1 > Java version: 1.6.0_31, 64 bit > Linux version: SuSE 11.1 [Kernel version : 2.6.32.12-0.7-default = x86_64 GNU/Linux] >=20 > We had gone thru the solutions available @ > = http://docs.oracle.com/javase/1.5.0/docs/guide/security/jgss/tutorials/Tro= ubleshooting.html > = https://ccp.cloudera.com/display/CDHDOC/Appendix+A+-+Troubleshooting#Appen= dixA-Troubleshooting-Problem2%3AJavaisunabletoreadtheKerberoscredentialsca= checreatedbyversionsofMITKerberos1.8.1orhigher. >=20 > But none of then seems to work. Any clue? >=20 > There are no change in server logs as client is failing is failing = even before it communicates with server. > Exception we are hitting (Client side logs): >=20 > 2012-05-22 09:42:22,627 WARN org.apache.hadoop.ipc.SecureClient: = Exception encountered while connecting to the server : = javax.security.sasl.SaslException: GSS initiate failed [Caused by = GSSException: No valid credentials provided (Mechanism level: Failed to = find any Kerberos tgt)] > 2012-05-22 09:42:22,627 ERROR = org.apache.hadoop.security.UserGroupInformation: = PriviledgedActionException as:testuser (auth:KERBEROS) = cause:java.io.IOException: javax.security.sasl.SaslException: GSS = initiate failed [Caused by GSSException: No valid credentials provided = (Mechanism level: Failed to find any Kerberos tgt)] > 2012-05-22 09:42:22,630 DEBUG org.apache.hadoop.ipc.SecureClient: = closing ipc connection to HOST-10-18-40-19/10.18.40.19:60020: = javax.security.sasl.SaslException: GSS initiate failed [Caused by = GSSException: No valid credentials provided (Mechanism level: Failed to = find any Kerberos tgt)] > java.io.IOException: javax.security.sasl.SaslException: GSS initiate = failed [Caused by GSSException: No valid credentials provided (Mechanism = level: Failed to find any Kerberos tgt)] > at = org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureClie= nt.java:227) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at = org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.= java:1177) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at = sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:= 39) > at = sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37) > at org.apache.hadoop.hbase.security.User.call(User.java:586) > at = org.apache.hadoop.hbase.security.User.access$700(User.java:50) > at = org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:440= ) > at = org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.handleSaslConnec= tionFailure(SecureClient.java:194) > at = org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(S= ecureClient.java:274) > at = org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.java:4= 85) > at = org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.java:6= 9) > at = org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:897) > at = org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEngine= .java:164) > at $Proxy6.getProtocolVersion(Unknown Source) > at = org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.java:= 208) > at = org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:303) > at = org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:280) > at = org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:332) > at = org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:236) > at = org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio= n.getHRegionConnection(HConnectionManager.java:1284) > at = org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio= n.getHRegionConnection(HConnectionManager.java:1240) > at = org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio= n.getHRegionConnection(HConnectionManager.java:1227) > at = org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio= n.locateRegionInMeta(HConnectionManager.java:936) > at = org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio= n.locateRegion(HConnectionManager.java:832) > at = org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio= n.locateRegion(HConnectionManager.java:801) > at = org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio= n.locateRegionInMeta(HConnectionManager.java:933) > at = org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio= n.locateRegion(HConnectionManager.java:836) > at = org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio= n.locateRegion(HConnectionManager.java:801) > at = org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) > at org.apache.hadoop.hbase.client.HTable.(HTable.java:174) > at org.apache.hadoop.hbase.client.HTable.(HTable.java:133) > at hbase.test.Hbasetest.main(Hbasetest.java:37) > Caused by: javax.security.sasl.SaslException: GSS initiate failed = [Caused by GSSException: No valid credentials provided (Mechanism level: = Failed to find any Kerberos tgt)] > at = com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Clien= t.java:194) > at = org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslR= pcClient.java:138) > at = org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConnect= ion(SecureClient.java:176) > at = org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$500(Secur= eClient.java:84) > at = org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClie= nt.java:267) > at = org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureClie= nt.java:264) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at = org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.= java:1177) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at = sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:= 39) > at = sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorIm= pl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37) > at org.apache.hadoop.hbase.security.User.call(User.java:586) > at = org.apache.hadoop.hbase.security.User.access$700(User.java:50) > at = org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:440= ) > at = org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstreams(S= ecureClient.java:263) > ... 23 more > Caused by: GSSException: No valid credentials provided (Mechanism = level: Failed to find any Kerberos tgt) > at = sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.j= ava:130) > at = sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactor= y.java:106) > at = sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory= .java:172) > at = sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:2= 09) > at = sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195) > at = sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162) > at = com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Clien= t.java:175) > ... 40 more > 2012-05-22 09:42:22,636 DEBUG org.apache.hadoop.ipc.SecureClient: IPC = Client (1778276127) connection to HOST-10-18-40-19/10.18.40.19:60020 = from testuser: closed > 2012-05-22 09:42:22,638 DEBUG = org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio= n: locateRegionInMeta parentTable=3D-ROOT-, = metaLocation=3D{region=3D-ROOT-,,0.70236052, hostname=3DHOST-10-18-40-19, = port=3D60020}, attempt=3D0 of 120 failed; retrying after sleep of 1000 = because: javax.security.sasl.SaslException: GSS initiate failed [Caused = by GSSException: No valid credentials provided (Mechanism level: Failed = to find any Kerberos tgt)] > 2012-05-22 09:42:22,640 DEBUG = org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio= n: Looked up root region location, = connection=3Dorg.apache.hadoop.hbase.client.HConnectionManager$HConnection= Implementation@6ecf829d; serverName=3DHOST-10-18-40-19,60020,1337574445438= > 2012-05-22 09:42:23,641 DEBUG = org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementatio= n: Looked up root region location, = connection=3Dorg.apache.hadoop.hbase.client.HConnectionManager$HConnection= Implementation@6ecf829d; serverName=3DHOST-10-18-40-19,60020,1337574445438= > 2012-05-22 09:42:23,642 DEBUG org.apache.hadoop.ipc.SecureClient: RPC = Server Kerberos principal name for = protocol=3Dorg.apache.hadoop.hbase.ipc.HRegionInterface is = hbase/hadoop@HADOOP.COM >=20 >=20 > -- > Regards, > Laxman >=20