Mailing-List: contact issues-help@spark.apache.org; run by ezmlm
Precedence: bulk
Date: Fri, 1 Sep 2017 13:38:00 +0000 (UTC)
From: "Thomas Graves (JIRA)" <jira@apache.org>
To: issues@spark.apache.org
Message-ID: <JIRA.13098904.1504195241000.8841.1504273080688@Atlassian.JIRA>
In-Reply-To: <JIRA.13098904.1504195241000@Atlassian.JIRA>
References: <JIRA.13098904.1504195241000@Atlassian.JIRA> <JIRA.13098904.1504195241639@jira-lw-us.apache.org>
Subject: [jira] [Comment Edited] (SPARK-21888) Cannot add stuff to Client
 Classpath for Yarn Cluster Mode
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
archived-at: Fri, 01 Sep 2017 13:38:11 -0000


    [ https://issues.apache.org/jira/browse/SPARK-21888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150530#comment-16150530 ] 

Thomas Graves edited comment on SPARK-21888 at 9/1/17 1:37 PM:
---------------------------------------------------------------

Putting things into SPARK_CONF_DIR will work, the question is more about convenience for users.  In hosted/multitenant environments there is probably a generic SPARK_CONF_DIR shared by everyone (at least this is how our env works), for the user to add hbase-site.xml they would have to copy, add files  and then export SPARK_CONF_DIR.  If that user continues to use the copied version they might miss changes to the cluster version, etc.  Previously they didn't have to do this, they just had to set SPARK_CLASSPATH, of course even that doesn't always work if your cluster env (spark_env.sh) had SPARK_CLASSPATH set in it.

So the question is more of what we think about this for convenience for users.  Personally I think it would be nice to have a config that would allow users to set an extra classpath on the client side without having to modify the SPARK_CONF_DIR. thoughts from others?
 
I think we can move this to an improvement jira, if other people here don't agree or see the usefulness then we can just close.


was (Author: tgraves):
Putting things into SPARK_CONF_DIR will work, the question is more about convenience for users.  In hosted/multitenant environments there is probably a generic SPARK_CONF_DIR shared by everyone (at least this is how our env works), for the user to add hbase-site.xml they would have to copy, add files  and then export SPARK_CONF_DIR.  If that user continues to use the copied version they might miss changes to the cluster version, etc.  Previously they didn't have to do this, they just had to set SPARK_CLASSPATH, of course even that doesn't always work if your cluster env (spark_env.sh) had SPARK_CLASSPATH set in it.

So the question is more of what we think about this for convenience for users.  Personally I think it would be nice to have a config that would allow users to set an extra classpath on the client side without having to modify the SPARK_CONF_DIR? 
 
I think we can move this to an improvement jira, if other people here don't agree or see the usefulness then we can just close.


> Cannot add stuff to Client Classpath for Yarn Cluster Mode
> ----------------------------------------------------------
>
>                 Key: SPARK-21888
>                 URL: https://issues.apache.org/jira/browse/SPARK-21888
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Core
>    Affects Versions: 2.2.0
>            Reporter: Parth Gandhi
>            Priority: Minor
>
> While running Spark on Yarn in cluster mode, currently there is no way to add any config files, jars etc. to Client classpath. An example for this is that suppose you want to run an application that uses hbase. Then, unless and until we do not copy the necessary config files required by hbase to Spark Config folder, we cannot specify or set their exact locations in classpath on Client end which we could do so earlier by setting the environment variable "SPARK_CLASSPATH".


--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org