spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Parente (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-19105) yarn/Client.scala copyToRemote does not include keytab destination name
Date Fri, 06 Jan 2017 13:29:58 GMT

    [ https://issues.apache.org/jira/browse/SPARK-19105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15804545#comment-15804545
] 

Peter Parente commented on SPARK-19105:
---------------------------------------

>From chat in the PR, the keytab on AM does have the proper UUID suffix. The HDFS staging
area filename is a red herring and a misunderstanding on my part about where the files are
first distributed.

> yarn/Client.scala copyToRemote does not include keytab destination name
> -----------------------------------------------------------------------
>
>                 Key: SPARK-19105
>                 URL: https://issues.apache.org/jira/browse/SPARK-19105
>             Project: Spark
>          Issue Type: Bug
>    Affects Versions: 2.1.0
>         Environment: YARN in client mode
>            Reporter: Peter Parente
>
> When I specify --principal user@REALM and --keytab /some/path/user.keytab, I see the
following in my app staging directory on HDFS:
> {code}
> -rw-r--r--   3 user supergroup         68 2017-01-06 03:59 user.keytab
> -rw-r--r--   3 user supergroup      73502 2017-01-06 03:59 __spark_conf__.zip
> -rw-r--r--   3 user supergroup  189767340 2017-01-06 03:59 __spark_libs__4440821503780683972.zip
> -rw-r--r--   3 user supergroup      91275 2017-01-06 03:59 py4j-0.10.3-src.zip
> -rw-r--r--   3 user supergroup     440385 2017-01-06 03:59 pyspark.zip
> {code}
> I also see that my spark.yarn.keytab config value has changed to user.keytab-54ee5192-43d0-41b5-ba50-1181ece26961
by the yarn client to ensure the keytab is unique within the app staging directory. However,
from the directory listing above, it's clear that the file written does not match this new
name. As a result, when it comes time to renew the Kerberos ticket, AMDelegationTokenRenewer
fails to find the keytab under the UUID-suffixed name and also fails to renew the tickets.
> The problem looks to be in one call to [copyFileToRemote in yarn/Client.java|https://github.com/apache/spark/blob/fe1c895e16c475a6f271ce600a42a8d0dc7986e5/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L482]
that leaves off the destination filename param. The other calls in that object which use copyFileToRemote
and have a custom destination name all provide this parameter (e.g., https://github.com/apache/spark/blob/fe1c895e16c475a6f271ce600a42a8d0dc7986e5/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala#L652).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message