Return-Path: X-Original-To: apmail-spark-user-archive@minotaur.apache.org Delivered-To: apmail-spark-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id C3BF710F6B for ; Wed, 4 Feb 2015 12:50:58 +0000 (UTC) Received: (qmail 93485 invoked by uid 500); 4 Feb 2015 12:50:23 -0000 Delivered-To: apmail-spark-user-archive@spark.apache.org Received: (qmail 93405 invoked by uid 500); 4 Feb 2015 12:50:22 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 93395 invoked by uid 99); 4 Feb 2015 12:50:22 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Feb 2015 12:50:22 +0000 X-ASF-Spam-Status: No, hits=2.8 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,URI_HEX X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of cjnolet@gmail.com designates 209.85.213.170 as permitted sender) Received: from [209.85.213.170] (HELO mail-ig0-f170.google.com) (209.85.213.170) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 04 Feb 2015 12:49:57 +0000 Received: by mail-ig0-f170.google.com with SMTP id l13so31243449iga.1 for ; Wed, 04 Feb 2015 04:47:40 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc:content-type; bh=rUFlpozQteAtywUOTzKXMeEZWQ1T4RfDbslrbkx9DVc=; b=p0VRKZ83eAa/uNH587j7AAzNYFbGj74P+dBlGJLaIUv9J25GJE2ib3+zWMKoC0JvHa jRUS5E8HOCz2jgLkrDUZuegbsQpd6rBgRr6//Ay4cxZamASIfJq3zEjyc6X21GIUCKoU 3oGAg7U8tt0UtuI26iUHT8rdT0Cs2M5E37WxJ2WvrSOdNwj53QVZ+JM5jqrW1HnSxMhM yCe7AGbnE7l6LKDiNRLAnqMvdh3+BhVzrRQzXKsceZ8Zab/KaW5p3s5TmZINRImG4vbV +yoRu4WuMt8Cemx7zijcNag4W7amFtuIBv9u6kVgBo6qUnXIqYMjjCYegqCxckGQ54o8 8qwA== X-Received: by 10.42.151.197 with SMTP id f5mr1474591icw.27.1423054060151; Wed, 04 Feb 2015 04:47:40 -0800 (PST) MIME-Version: 1.0 Received: by 10.64.232.146 with HTTP; Wed, 4 Feb 2015 04:47:19 -0800 (PST) In-Reply-To: References: <54D1A67A.2030609@yahoo.com> From: Corey Nolet Date: Wed, 4 Feb 2015 07:47:19 -0500 Message-ID: Subject: =?UTF-8?B?UmU6IOKAnG1hcHJlZHVjZS5qb2IudXNlci5jbGFzc3BhdGguZmlyc3TigJ0gZm9yIFNwYQ==?= =?UTF-8?B?cms=?= To: bo yang Cc: medale@acm.org, user Content-Type: multipart/alternative; boundary=90e6ba613592c08e1f050e42973d X-Virus-Checked: Checked by ClamAV on apache.org --90e6ba613592c08e1f050e42973d Content-Type: text/plain; charset=UTF-8 Bo yang- I am using Spark 1.2.0 and undoubtedly there are older Guava classes which are being picked up and serialized with the closures when they are sent from the driver to the executors because the class serial version ids don't match from the driver to the executors. Have you tried doing this? Guava works fine for me when this is not the case- but as soon as a Guava class which was changed from versions <15.0 is serialized, it fails. See [1] fore info- we did fairly extensive testing last night. I've isolated the issue to Hadoop's really old version of Guava being picked up. Again, this is only noticeable when classes are used from Guava 15.0 that were changed from previous versions and those classes are being serialized on the driver and shipped to the executors. [1] https://github.com/calrissian/mango/issues/158 On Wed, Feb 4, 2015 at 1:31 AM, bo yang wrote: > Corey, > > Which version of Spark do you use? I am using Spark 1.2.0, and guava > 15.0. It seems fine. > > Best, > Bo > > > On Tue, Feb 3, 2015 at 8:56 PM, M. Dale > wrote: > >> Try spark.yarn.user.classpath.first (see >> https://issues.apache.org/jira/browse/SPARK-2996 - only works for YARN). >> Also thread at >> http://apache-spark-user-list.1001560.n3.nabble.com/netty-on-classpath-when-using-spark-submit-td18030.html >> . >> >> HTH, >> Markus >> >> On 02/03/2015 11:20 PM, Corey Nolet wrote: >> >> I'm having a really bad dependency conflict right now with Guava versions >> between my Spark application in Yarn and (I believe) Hadoop's version. >> >> The problem is, my driver has the version of Guava which my application >> is expecting (15.0) while it appears the Spark executors that are working >> on my RDDs have a much older version (assuming it's the old version on the >> Hadoop classpath). >> >> Is there a property like "mapreduce.job.user.classpath.first' that I >> can set to make sure my own classpath is extablished first on the executors? >> >> >> > --90e6ba613592c08e1f050e42973d Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Bo yang-=C2=A0

I am using Spark 1.2.0 a= nd undoubtedly there are older Guava classes which are being picked up and = serialized with the closures when they are sent from the driver to the exec= utors because the class serial version ids don't match from the driver = to the executors. Have you tried doing this? Guava works fine for me when t= his is not the case- but as soon as a Guava class which was changed from ve= rsions <15.0 is serialized, it fails. See [1] fore info- we did fairly e= xtensive testing last night. I've isolated the issue to Hadoop's re= ally old version of Guava being picked up. Again, this is only noticeable w= hen classes are used from Guava 15.0 that were changed from previous versio= ns and those classes are being serialized on the driver and shipped to the = executors.



On Wed, Feb 4, 2015 at 1:31 AM, bo yang <= ;bobyangbo@gmail.c= om> wrote:
Corey,

Which version of Spark do you use? I am using= Spark 1.2.0, and =C2=A0guava 15.0. It seems fine.

Best,
Bo

On Tue, Feb 3, 2015 at 8:56 PM, M. Dale <medale94@yahoo.com.invalid> wrote:
=20 =20 =20
Try spark.yarn.user.classpath.first (see https://issues.apache.org/jira/browse/SPARK-2996 - only works f= or YARN). Also thread at http://apach= e-spark-user-list.1001560.n3.nabble.com/netty-on-classpath-when-using-spark= -submit-td18030.html.

HTH,
Markus

=20
On 02/03/2015 11:20 PM, Corey Nolet wrote:
I'm having a really bad dependency conflict righ= t now with Guava versions between my Spark application in Yarn and (I believe) Hadoop's version.=C2=A0

The problem is, my driver has the version of Guava which my application is expecting (15.0) while it appears the Spark executors that are working on my RDDs have a much older version (assuming it's the old version on the Hadoop classpath).=C2=A0

Is there a property like "mapreduce.job.user.classpath.first' that I can set to m= ake sure my own classpath is extablished first on the executors?



--90e6ba613592c08e1f050e42973d--