spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Cody Koeninger <c...@koeninger.org>
Subject Re: guava version conflicts
Date Mon, 22 Sep 2014 19:46:07 GMT
We're using Mesos, is there a reasonable expectation that
spark.files.userClassPathFirst will actually work?

On Mon, Sep 22, 2014 at 1:42 PM, Marcelo Vanzin <vanzin@cloudera.com> wrote:

> Hi Cody,
>
> I'm still writing a test to make sure I understood exactly what's
> going on here, but from looking at the stack trace, it seems like the
> newer Guava library is picking up the "Optional" class from the Spark
> assembly.
>
> Could you try one of the options that put the user's classpath before
> the Spark assembly? (spark.files.userClassPathFirst or
> spark.yarn.user.classpath.first depending on which master you're
> running)
>
> People seem to have run into issues with those options in the past,
> but if they work for you, then Guava should pick its own Optional
> class (instead of the one shipped with Spark) and things should then
> work.
>
> I'll investigate a way to fix it in Spark in the meantime.
>
>
> On Fri, Sep 19, 2014 at 10:30 PM, Cody Koeninger <cody@koeninger.org>
> wrote:
> > After the recent spark project changes to guava shading, I'm seeing
> issues
> > with the datastax spark cassandra connector (which depends on guava 15.0)
> > and the datastax cql driver (which depends on guava 16.0.1)
> >
> > Building an assembly for a job (with spark marked as provided) that
> > includes either guava 15.0 or 16.0.1, results in errors like the
> following:
> >
> > scala> session.close
> >
> > scala> s[14/09/20 04:56:35 ERROR Futures$CombinedFuture: input future
> > failed.
> > java.lang.IllegalAccessError: tried to access class
> > org.spark-project.guava.common.base.Absent from class
> > com.google.common.base.Optional
> >         at com.google.common.base.Optional.absent(Optional.java:79)
> >         at com.google.common.base.Optional.fromNullable(Optional.java:94)
> >         at
> >
> com.google.common.util.concurrent.Futures$CombinedFuture.setOneValue(Futures.java:1608)
> >         at
> >
> com.google.common.util.concurrent.Futures$CombinedFuture.access$400(Futures.java:1470)
> >         at
> >
> com.google.common.util.concurrent.Futures$CombinedFuture$2.run(Futures.java:1548)
> >         at
> >
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
> >         at
> >
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> >         at
> >
> com.google.common.util.concurrent.ExecutionList.add(ExecutionList.java:101)
> >         at
> >
> com.google.common.util.concurrent.AbstractFuture.addListener(AbstractFuture.java:170)
> >         at
> >
> com.google.common.util.concurrent.Futures$CombinedFuture.init(Futures.java:1545)
> >         at
> >
> com.google.common.util.concurrent.Futures$CombinedFuture.<init>(Futures.java:1491)
> >         at
> > com.google.common.util.concurrent.Futures.listFuture(Futures.java:1640)
> >         at
> > com.google.common.util.concurrent.Futures.allAsList(Futures.java:983)
> >         at
> >
> com.datastax.driver.core.CloseFuture$Forwarding.<init>(CloseFuture.java:73)
> >         at
> >
> com.datastax.driver.core.HostConnectionPool.closeAsync(HostConnectionPool.java:398)
> >         at
> >
> com.datastax.driver.core.SessionManager.closeAsync(SessionManager.java:157)
> >         at
> > com.datastax.driver.core.SessionManager.close(SessionManager.java:172)
> >         at
> >
> com.datastax.spark.connector.cql.CassandraConnector$.com$datastax$spark$connector$cql$CassandraConnector$$destroySession(CassandraConnector.scala:180)
> >         at
> >
> com.datastax.spark.connector.cql.CassandraConnector$$anonfun$5.apply(CassandraConnector.scala:151)
> >         at
> >
> com.datastax.spark.connector.cql.CassandraConnector$$anonfun$5.apply(CassandraConnector.scala:151)
> >         at com.datastax.spark.connector.cql.RefCountedCache.com
> >
> $datastax$spark$connector$cql$RefCountedCache$$releaseImmediately(RefCountedCache.scala:86)
> >         at
> >
> com.datastax.spark.connector.cql.RefCountedCache$ReleaseTask.run(RefCountedCache.scala:26)
> >         at
> >
> com.datastax.spark.connector.cql.RefCountedCache$$anonfun$com$datastax$spark$connector$cql$RefCountedCache$$processPendingReleases$2.apply(RefCountedCache.scala:150)
> >         at
> >
> com.datastax.spark.connector.cql.RefCountedCache$$anonfun$com$datastax$spark$connector$cql$RefCountedCache$$processPendingReleases$2.apply(RefCountedCache.scala:147)
> >         at
> >
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
> >         at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> >         at
> > scala.collection.concurrent.TrieMapIterator.foreach(TrieMap.scala:922)
> >         at
> > scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> >         at scala.collection.concurrent.TrieMap.foreach(TrieMap.scala:632)
> >         at
> >
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
> >         at com.datastax.spark.connector.cql.RefCountedCache.com
> >
> $datastax$spark$connector$cql$RefCountedCache$$processPendingReleases(RefCountedCache.scala:147)
> >         at
> >
> com.datastax.spark.connector.cql.RefCountedCache$$anon$1.run(RefCountedCache.scala:157)
> >         at
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> >         at
> >
> java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
> >         at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
> >         at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
> >         at
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> >         at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> >         at java.lang.Thread.run(Thread.java:722)
>
>
>
> --
> Marcelo
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message