Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 4D5CA1660D7 for ; Tue, 22 Aug 2017 04:04:11 +0200 (CEST) Received: (qmail 2385 invoked by uid 500); 22 Aug 2017 02:04:10 -0000 Mailing-List: contact issues-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@hive.apache.org Delivered-To: mailing list issues@hive.apache.org Received: (qmail 2376 invoked by uid 99); 22 Aug 2017 02:04:10 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 22 Aug 2017 02:04:10 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id EE6041806E2 for ; Tue, 22 Aug 2017 02:04:09 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -99.202 X-Spam-Level: X-Spam-Status: No, score=-99.202 tagged_above=-999 required=6.31 tests=[KAM_ASCII_DIVIDERS=0.8, RP_MATCHES_RCVD=-0.001, SPF_PASS=-0.001, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id D_oel4mIP5Zi for ; Tue, 22 Aug 2017 02:04:07 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTP id 92E965FACE for ; Tue, 22 Aug 2017 02:04:06 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id EC94DE0114 for ; Tue, 22 Aug 2017 02:04:04 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 0EC952537E for ; Tue, 22 Aug 2017 02:04:02 +0000 (UTC) Date: Tue, 22 Aug 2017 02:04:00 +0000 (UTC) From: "Vihang Karajgaonkar (JIRA)" To: issues@hive.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Updated] (HIVE-17368) DBTokenStore fails to connect in Kerberos enabled remote HMS environment MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/HIVE-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-17368: --------------------------------------- Attachment: HIVE-17368.01.patch Adding the first version of the patch. Modified the existing test {{TestJdbcWithDBTokenStore}} so that it now uses a secure remote HMS. > DBTokenStore fails to connect in Kerberos enabled remote HMS environment > ------------------------------------------------------------------------ > > Key: HIVE-17368 > URL: https://issues.apache.org/jira/browse/HIVE-17368 > Project: Hive > Issue Type: Bug > Affects Versions: 1.1.0, 2.0.0, 2.1.0, 2.2.0 > Reporter: Vihang Karajgaonkar > Assignee: Vihang Karajgaonkar > Attachments: HIVE-17368.01.patch > > > In setups where HMS is running as a remote process secured using Kerberos, and when {{DBTokenStore}} is configured as the token store, the HS2 Thrift API calls like {{GetDelegationToken}}, {{CancelDelegationToken}} and {{RenewDelegationToken}} fail with exception trace seen below. HS2 is not able to invoke HMS APIs needed to add/remove/renew tokens from the DB since it is possible that the user which is issue the {{GetDelegationToken}} is not kerberos enabled. > Eg. Oozie submits a job on behalf of user "Joe". When Oozie opens a session with HS2 it uses Oozie's principal and creates a proxy UGI with Hive. This principal can establish a transport authenticated using Kerberos. It stores the HMS delegation token string in the sessionConf and sessionToken. Now, lets say Oozie issues a {{GetDelegationToken}} which has {{Joe}} as the owner and {{oozie}} as the renewer in {{GetDelegationTokenReq}}. This API call cannot instantiate a HMSClient and open transport to HMS using the HMSToken string available in the sessionConf, since DBTokenStore uses server HiveConf instead of sessionConf. It tries to establish transport using Kerberos and it fails since user Joe is not Kerberos enabled. > I see the following exception trace in HS2 logs. > {noformat} > 2017-08-21T18:07:19,644 ERROR [HiveServer2-Handler-Pool: Thread-61] transport.TSaslTransport: SASL negotiation failure > javax.security.sasl.SaslException: GSS initiate failed > at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211) ~[?:1.8.0_121] > at org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) ~[libthrift-0.9.3.jar:0.9.3] > at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) [libthrift-0.9.3.jar:0.9.3] > at org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) [libthrift-0.9.3.jar:0.9.3] > at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121] > at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121] > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) [hadoop-common-2.7.2.jar:?] > at org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:488) [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:255) [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:70) [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:1.8.0_121] > at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) [?:1.8.0_121] > at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) [?:1.8.0_121] > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) [?:1.8.0_121] > at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1699) [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:83) [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133) [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104) [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3595) [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3647) [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3627) [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121] > at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121] > at org.apache.hadoop.hive.thrift.DBTokenStore.invokeOnTokenStore(DBTokenStore.java:157) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hadoop.hive.thrift.DBTokenStore.addToken(DBTokenStore.java:74) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.createPassword(TokenStoreDelegationTokenSecretManager.java:142) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hadoop.hive.thrift.TokenStoreDelegationTokenSecretManager.createPassword(TokenStoreDelegationTokenSecretManager.java:56) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hadoop.security.token.Token.(Token.java:59) [hadoop-common-2.7.2.jar:?] > at org.apache.hadoop.hive.thrift.DelegationTokenSecretManager.getDelegationToken(DelegationTokenSecretManager.java:109) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager$1.run(HiveDelegationTokenManager.java:123) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager$1.run(HiveDelegationTokenManager.java:119) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121] > at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121] > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) [hadoop-common-2.7.2.jar:?] > at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.getDelegationToken(HiveDelegationTokenManager.java:119) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hadoop.hive.thrift.HiveDelegationTokenManager.getDelegationTokenWithService(HiveDelegationTokenManager.java:130) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hive.service.auth.HiveAuthFactory.getDelegationToken(HiveAuthFactory.java:261) [hive-service-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hive.service.cli.session.HiveSessionImplwithUGI.getDelegationToken(HiveSessionImplwithUGI.java:174) [hive-service-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_121] > at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_121] > at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_121] > at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_121] > at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:78) [hive-service-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:36) [hive-service-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:63) [hive-service-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_121] > at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121] > at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) [hadoop-common-2.7.2.jar:?] > at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:59) [hive-service-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at com.sun.proxy.$Proxy36.getDelegationToken(Unknown Source) [?:?] > at org.apache.hive.service.cli.CLIService.getDelegationToken(CLIService.java:589) [hive-service-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hive.service.cli.thrift.ThriftCLIService.GetDelegationToken(ThriftCLIService.java:254) [hive-service-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetDelegationToken.getResult(TCLIService.java:1737) [hive-service-rpc-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.hive.service.rpc.thrift.TCLIService$Processor$GetDelegationToken.getResult(TCLIService.java:1722) [hive-service-rpc-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39) [libthrift-0.9.3.jar:0.9.3] > at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:39) [libthrift-0.9.3.jar:0.9.3] > at org.apache.hadoop.hive.thrift.HadoopThriftAuthBridge$Server$TUGIAssumingProcessor.process(HadoopThriftAuthBridge.java:621) [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT] > at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286) [libthrift-0.9.3.jar:0.9.3] > at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_121] > at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_121] > at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] > Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt) > at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[?:1.8.0_121] > at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122) ~[?:1.8.0_121] > at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[?:1.8.0_121] > at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224) ~[?:1.8.0_121] > at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[?:1.8.0_121] > at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[?:1.8.0_121] > at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192) ~[?:1.8.0_121] > ... 65 more > {noformat} > On HMS side I see a exception saying > {noformat} > 2017-08-17 11:45:13,655 ERROR org.apache.thrift.server.TThreadPoolServer: [pool-7-thread-34]: Error occurred during processing of message. > java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: DIGEST-MD5: IO error acquiring password > {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)