Mailing-List: contact issues-help@ambari.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@ambari.apache.org
Date: Mon, 15 May 2017 22:29:04 +0000 (UTC)
From: "Weiqing Yang (JIRA)" <jira@apache.org>
To: issues@ambari.apache.org
Message-ID: <JIRA.13072218.1494882107000.215751.1494887344212@Atlassian.JIRA>
In-Reply-To: <JIRA.13072218.1494882107000@Atlassian.JIRA>
References: <JIRA.13072218.1494882107000@Atlassian.JIRA> <JIRA.13072218.1494882107276@jira-lw-us.apache.org>
Subject: [jira] [Updated] (AMBARI-21028) The credential cache for livy is
 messed up
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Mon, 15 May 2017 22:29:09 -0000


     [ https://issues.apache.org/jira/browse/AMBARI-21028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Weiqing Yang updated AMBARI-21028:
----------------------------------
    Attachment: AMBARI-21028_v2.patch

> The credential cache for livy is messed up
> ------------------------------------------
>
>                 Key: AMBARI-21028
>                 URL: https://issues.apache.org/jira/browse/AMBARI-21028
>             Project: Ambari
>          Issue Type: Bug
>          Components: ambari-server
>    Affects Versions: trunk
>            Reporter: Weiqing Yang
>            Assignee: Weiqing Yang
>             Fix For: trunk
>
>         Attachments: AMBARI-21028_v0.patch, AMBARI-21028_v1.patch, AMBARI-21028_v2.patch
>
>
> This issue was reported by [~kbadani]. 
> Steps to reproduce this issue:
> * Kdestroy and kinit as 'livy' user
> * Do spark-submit with --proxy-user as 'hrt_1'
> * In the console output, you can see that 'ambari-qa' is trying to impersonate as 'hrt_1' and its failing
> * Cancel the running job and do klist again - it will show credentials for 'ambari-qa' user and not the 'livy' user with which it was kinited
> {code:java}
> [livy@ctr-e133-1493418528701-6489-01-000003 spark]$ kinit -kt /etc/security/keytabs/livy.service.keytab livy/ctr-e133-1493418528701-6489-01-000003.hwx.site
> [livy@ctr-e133-1493418528701-6489-01-000003 spark]$ klist
> Ticket cache: FILE:/tmp/krb5cc_1808
> Default principal: livy/ctr-e133-1493418528701-6489-01-000003.hwx.site@EXAMPLE.COM
> Valid starting       Expires              Service principal
> 05/02/2017 23:52:14  05/03/2017 23:52:14  krbtgt/EXAMPLE.COM@EXAMPLE.COM
> [livy@ctr-e133-1493418528701-6489-01-000003 spark]$ spark-submit --class org.apache.spark.examples.SparkPi --master yarn-cluster --num-executors 3 --driver-memory 512m --executor-memory 512m --proxy-user hrt_1 --executor-cores 1 /usr/hdp/current/spark-client/lib/spark-examples-1.6.3.2.6.1.0-45-hadoop2.7.3.2.6.1.0-45.jar 10
> Multiple versions of Spark are installed but SPARK_MAJOR_VERSION is not set
> Spark1 will be picked by default
> 17/05/02 23:53:10 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> 17/05/02 23:53:12 INFO AHSProxy: Connecting to Application History server at ctr-e133-1493418528701-6489-01-000004.hwx.site/172.27.22.136:10200
> 17/05/02 23:53:12 INFO RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
> 17/05/02 23:53:12 WARN RequestHedgingRMFailoverProxyProvider: Invocation returned exception on [rm1] : org.apache.hadoop.security.authorize.AuthorizationException: User: ambari-qa@EXAMPLE.COM is not allowed to impersonate hrt_1, so propagating back to caller.
> 17/05/02 23:53:12 INFO RequestHedgingRMFailoverProxyProvider: Connection lost with rm1, trying to fail over.
> 17/05/02 23:53:12 INFO RequestHedgingRMFailoverProxyProvider: Looking for the active RM in [rm1, rm2]...
> 17/05/02 23:53:12 WARN RequestHedgingRMFailoverProxyProvider: Invocation returned exception on [rm1] : org.apache.hadoop.security.authorize.AuthorizationException: User: ambari-qa@EXAMPLE.COM is not allowed to impersonate hrt_1, so propagating back to caller.
> 17/05/02 23:53:12 INFO RetryInvocationHandler: org.apache.hadoop.security.authorize.AuthorizationException: User: ambari-qa@EXAMPLE.COM is not allowed to impersonate hrt_1, while invoking $Proxy9.getClusterMetrics over Failover proxy for [rm1, rm2] after 1 failover attempts. Trying to failover after sleeping for 10442ms.
> [livy@ctr-e133-1493418528701-6489-01-000003 spark]$ klist
> Ticket cache: FILE:/tmp/krb5cc_1808
> Default principal: ambari-qa@EXAMPLE.COM
> Valid starting       Expires              Service principal
> 05/02/2017 23:53:03  05/03/2017 23:53:03  krbtgt/EXAMPLE.COM@EXAMPLE.COM
> 05/02/2017 23:53:03  05/03/2017 23:53:03  HTTP/ctr-e133-1493418528701-6489-01-000003.hwx.site@EXAMPLE.COM
> {code}
> Root cause is: 
> The livy smoke test launched by Ambari is run as livy user, but kinits as ambari-qa, and therefore messes up the credential cache for livy.


--
This message was sent by Atlassian JIRA
(v6.3.15#6346)