spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kannan Rajah <kra...@maprtech.com>
Subject Re: Is SPARK_CLASSPATH really deprecated?
Date Fri, 27 Feb 2015 01:12:02 GMT
There is a usability concern I have with the current way of specifying
--jars. Imagine a use case like hbase where a lot of jobs need it in its
classpath. This needs to be set every time. If we use
spark.executor.extraClassPath,
then we just need to set it once But there is no programmatic way to set
this value, like picking up from an environment variable or by running a
script that generates classpath.  You need to hard code the jars in
spark-defaults.conf.

Also, I would like to know if there is a localization overhead when we use
spark.executor.extraClassPath. Again, in the case of hbase, these jars
would be typically available on all nodes. So there is no need to localize
them from the node where job was submitted. I am wondering if we use the
SPARK_CLASSPATH approach, then it would not do localization. That would be
an added benefit.
Please clarify.




--
Kannan

On Thu, Feb 26, 2015 at 4:15 PM, Marcelo Vanzin <vanzin@cloudera.com> wrote:

> SPARK_CLASSPATH is definitely deprecated, but my understanding is that
> spark.executor.extraClassPath is not, so maybe the documentation needs
> fixing.
>
> I'll let someone who might know otherwise comment, though.
>
> On Thu, Feb 26, 2015 at 2:43 PM, Kannan Rajah <krajah@maprtech.com> wrote:
> > SparkConf.scala logs a warning saying SPARK_CLASSPATH is deprecated and
> we
> > should use spark.executor.extraClassPath instead. But the online
> > documentation states that spark.executor.extraClassPath is only meant for
> > backward compatibility.
> >
> >
> https://spark.apache.org/docs/1.2.0/configuration.html#execution-behavior
> >
> > Which one is right? I have a use case to submit a hbase job from
> spark-shell
> > and make it run using YARN. In this case, I need to somehow add the hbase
> > jars to the classpath of the executor. If I add it to SPARK_CLASSPATH and
> > export it it works fine. Alternatively, if I set the
> > spark.executor.extraClassPath in spark-defaults.conf, it works fine. But
> the
> > reason I don't like spark-defaults.conf is that I need to hard code it
> > instead of relying on a script to generate the classpath. I can use a
> script
> > in spark-env.sh and set SPARK_CLASSPATH.
> >
> > Given that compute-classpath uses SPARK_CLASSPATH variable, why is it
> marked
> > as deprecated?
> >
> > --
> > Kannan
>
>
>
> --
> Marcelo
>

Mime
View raw message