spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Corey Nolet <cjno...@gmail.com>
Subject Re: “mapreduce.job.user.classpath.first” for Spark
Date Wed, 04 Feb 2015 12:47:19 GMT
Bo yang-

I am using Spark 1.2.0 and undoubtedly there are older Guava classes which
are being picked up and serialized with the closures when they are sent
from the driver to the executors because the class serial version ids don't
match from the driver to the executors. Have you tried doing this? Guava
works fine for me when this is not the case- but as soon as a Guava class
which was changed from versions <15.0 is serialized, it fails. See [1] fore
info- we did fairly extensive testing last night. I've isolated the issue
to Hadoop's really old version of Guava being picked up. Again, this is
only noticeable when classes are used from Guava 15.0 that were changed
from previous versions and those classes are being serialized on the driver
and shipped to the executors.


[1] https://github.com/calrissian/mango/issues/158

On Wed, Feb 4, 2015 at 1:31 AM, bo yang <bobyangbo@gmail.com> wrote:

> Corey,
>
> Which version of Spark do you use? I am using Spark 1.2.0, and  guava
> 15.0. It seems fine.
>
> Best,
> Bo
>
>
> On Tue, Feb 3, 2015 at 8:56 PM, M. Dale <medale94@yahoo.com.invalid>
> wrote:
>
>>  Try spark.yarn.user.classpath.first (see
>> https://issues.apache.org/jira/browse/SPARK-2996 - only works for YARN).
>> Also thread at
>> http://apache-spark-user-list.1001560.n3.nabble.com/netty-on-classpath-when-using-spark-submit-td18030.html
>> .
>>
>> HTH,
>> Markus
>>
>> On 02/03/2015 11:20 PM, Corey Nolet wrote:
>>
>> I'm having a really bad dependency conflict right now with Guava versions
>> between my Spark application in Yarn and (I believe) Hadoop's version.
>>
>>  The problem is, my driver has the version of Guava which my application
>> is expecting (15.0) while it appears the Spark executors that are working
>> on my RDDs have a much older version (assuming it's the old version on the
>> Hadoop classpath).
>>
>>  Is there a property like "mapreduce.job.user.classpath.first' that I
>> can set to make sure my own classpath is extablished first on the executors?
>>
>>
>>
>

Mime
View raw message