spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Apache Spark (JIRA)" <j...@apache.org>
Subject [jira] [Assigned] (SPARK-9833) Add options to explicitly disable delegation token retrieval for non-HDFS
Date Wed, 12 Aug 2015 19:23:46 GMT

     [ https://issues.apache.org/jira/browse/SPARK-9833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Apache Spark reassigned SPARK-9833:
-----------------------------------

    Assignee: Apache Spark

> Add options to explicitly disable delegation token retrieval for non-HDFS
> -------------------------------------------------------------------------
>
>                 Key: SPARK-9833
>                 URL: https://issues.apache.org/jira/browse/SPARK-9833
>             Project: Spark
>          Issue Type: Improvement
>          Components: YARN
>    Affects Versions: 1.4.0, 1.5.0
>            Reporter: Marcelo Vanzin
>            Assignee: Apache Spark
>            Priority: Minor
>
> In 1.4, code was added to fetch delegation tokens for Hive metastores and HBase masters.
That code is run regardless of whether the user app actually needs those tokens, since there's
no way for Spark to know otherwise.
> In the best case, that's just unnecessary overhead. In some cases, the Hadoop / Hive
/ HBase configuration being loaded may contain references to things that Spark doesn't have
access to (e.g. Sentry classes), and the Spark application might fail to start.
> Adding explicit configuration options to disable those would help, since currently the
only way is to set some specific Hive / HBase configuration values, which is kinda brittle.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message