spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandy Ryza <>
Subject Re: Question about running spark on yarn
Date Tue, 22 Apr 2014 16:38:50 GMT
Hi Gordon,

We recently handled this in SPARK-1064.  As of 1.0.0, you'll be able to
pass -Phadoop-provided to Maven and avoid including Hadoop and its
dependencies in the assembly jar.


On Tue, Apr 22, 2014 at 2:43 AM, Gordon Wang <> wrote:

> In this page
> We have to use spark assembly to submit spark apps to yarn cluster.
> And I checked the assembly jars of spark. It contains some yarn classes
> which are added during compile time. The yarn classes are not what I want.
> My question is that is it possible to use other jars to submit spark app
> to yarn cluster.
> I do not want to use the assembly jar because it has yarn classes which
> may overwrite the yarn class in HADOOP_CLASSPATH. If the yarn cluster is
> upgraded, even if the YARN apis are same, spark has to be recompiled
> against to the new version of yarn.
> Any help is appreciated ! Thanks.
> --
> Regards
> Gordon Wang

View raw message