spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Saisai Shao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-18840) HDFSCredentialProvider throws exception in non-HDFS security environment
Date Wed, 14 Dec 2016 01:34:58 GMT

    [ https://issues.apache.org/jira/browse/SPARK-18840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15746877#comment-15746877
] 

Saisai Shao commented on SPARK-18840:
-------------------------------------

[~vanzin], is it necessary to fix it in old version (2.0/1.6), seems this problem seldom happens.

> HDFSCredentialProvider throws exception in non-HDFS security environment
> ------------------------------------------------------------------------
>
>                 Key: SPARK-18840
>                 URL: https://issues.apache.org/jira/browse/SPARK-18840
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 1.6.3, 2.1.0
>            Reporter: Saisai Shao
>            Assignee: Saisai Shao
>            Priority: Minor
>             Fix For: 2.1.1, 2.2.0
>
>
> Current in {{HDFSCredentialProvider}}, the code logic assumes HDFS delegation token should
be existed, this is ok for HDFS environment, but for some cloud environment like Azure, HDFS
is not required, so it will throw exception:
> {code}
> java.util.NoSuchElementException: head of empty list
>         at scala.collection.immutable.Nil$.head(List.scala:337)
>         at scala.collection.immutable.Nil$.head(List.scala:334)
>         at org.apache.spark.deploy.yarn.Client.getTokenRenewalInterval(Client.scala:627)
> {code}
> We should also consider this situation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message