spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Thomas Graves (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-21888) Cannot add stuff to Client Classpath for Yarn Cluster Mode
Date Fri, 01 Sep 2017 13:38:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150530#comment-16150530
] 

Thomas Graves edited comment on SPARK-21888 at 9/1/17 1:37 PM:
---------------------------------------------------------------

Putting things into SPARK_CONF_DIR will work, the question is more about convenience for users.
 In hosted/multitenant environments there is probably a generic SPARK_CONF_DIR shared by everyone
(at least this is how our env works), for the user to add hbase-site.xml they would have to
copy, add files  and then export SPARK_CONF_DIR.  If that user continues to use the copied
version they might miss changes to the cluster version, etc.  Previously they didn't have
to do this, they just had to set SPARK_CLASSPATH, of course even that doesn't always work
if your cluster env (spark_env.sh) had SPARK_CLASSPATH set in it.

So the question is more of what we think about this for convenience for users.  Personally
I think it would be nice to have a config that would allow users to set an extra classpath
on the client side without having to modify the SPARK_CONF_DIR. thoughts from others?
 
I think we can move this to an improvement jira, if other people here don't agree or see the
usefulness then we can just close.




was (Author: tgraves):
Putting things into SPARK_CONF_DIR will work, the question is more about convenience for users.
 In hosted/multitenant environments there is probably a generic SPARK_CONF_DIR shared by everyone
(at least this is how our env works), for the user to add hbase-site.xml they would have to
copy, add files  and then export SPARK_CONF_DIR.  If that user continues to use the copied
version they might miss changes to the cluster version, etc.  Previously they didn't have
to do this, they just had to set SPARK_CLASSPATH, of course even that doesn't always work
if your cluster env (spark_env.sh) had SPARK_CLASSPATH set in it.

So the question is more of what we think about this for convenience for users.  Personally
I think it would be nice to have a config that would allow users to set an extra classpath
on the client side without having to modify the SPARK_CONF_DIR? 
 
I think we can move this to an improvement jira, if other people here don't agree or see the
usefulness then we can just close.



> Cannot add stuff to Client Classpath for Yarn Cluster Mode
> ----------------------------------------------------------
>
>                 Key: SPARK-21888
>                 URL: https://issues.apache.org/jira/browse/SPARK-21888
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.2.0
>            Reporter: Parth Gandhi
>            Priority: Minor
>
> While running Spark on Yarn in cluster mode, currently there is no way to add any config
files, jars etc. to Client classpath. An example for this is that suppose you want to run
an application that uses hbase. Then, unless and until we do not copy the necessary config
files required by hbase to Spark Config folder, we cannot specify or set their exact locations
in classpath on Client end which we could do so earlier by setting the environment variable
"SPARK_CLASSPATH".



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message