spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Soren Macbeth <so...@yieldbot.com>
Subject Re: bug using kryo as closure serializer
Date Mon, 05 May 2014 15:56:42 GMT
I just took a peek at KryoSerializer and it looks like you're already using
all the scala stuff from chill in there, so I would imagine that scala
things should serialize pretty well.

Seems like the readonly bytebuffer thing might be some other sort of
downstream bug.


On Sun, May 4, 2014 at 10:24 PM, Reynold Xin <rxin@databricks.com> wrote:

> Technically you only need to change the build file, and change part of a
> line in SparkEnv so you don't have to break your oath :)
>
>
>
> On Sun, May 4, 2014 at 10:22 PM, Soren Macbeth <soren@yieldbot.com> wrote:
>
> > that would violate my personal oath of never writing a single line of
> > scala, but I might be able to do that if I can get past the issue this
> > issue I'm struggling with in this thread.
> >
> > On Sunday, May 4, 2014, Reynold Xin <rxin@databricks.com> wrote:
> >
> > > Thanks. Do you mind playing with chill-scala a little bit and see if it
> > > actually works well for closures? One way to try is to hard code the
> > > serializer to use Kryo with chill-scala, and then run through all the
> > unit
> > > tests.
> > >
> > > If it works well, we can incorporate that in the next release (probably
> > not
> > > 1.0, but after that).
> > >
> > >
> > > On Sun, May 4, 2014 at 9:08 PM, Soren Macbeth <soren@yieldbot.com>
> > wrote:
> > >
> > > > fwiw, it seems like it wouldn't be very difficult to integrate
> > > chill-scala,
> > > > since you're already chill-java and probably get kryo serialization
> of
> > > > closures and all sorts of other scala stuff for free. All that would
> be
> > > > needed would be to include the dependency and then update
> > KryoSerializer
> > > to
> > > > register the stuff in chill-scala.
> > > >
> > > > In that case, you could probably safely make kryo the default
> > serializer,
> > > > which I think would be desirable in general.
> > > >
> > > >
> > > > On Sun, May 4, 2014 at 8:48 PM, Reynold Xin <rxin@databricks.com>
> > wrote:
> > > >
> > > > > Good idea. I submitted a pull request for the doc update here:
> > > > > https://github.com/apache/spark/pull/642
> > > > >
> > > > >
> > > > > On Sun, May 4, 2014 at 3:54 PM, Soren Macbeth <soren@yieldbot.com>
> > > > wrote:
> > > > >
> > > > > > Thanks for the reply!
> > > > > >
> > > > > > Ok, if that's the case, I'd recommend a note to that affect
in
> the
> > > docs
> > > > > at
> > > > > > least.
> > > > > >
> > > > > > Just to give some more context here, I'm working on a Clojure
DSL
> > for
> > > > > Spark
> > > > > > called Flambo, which I plan to open source shortly. If I could
> I'd
> > > like
> > > > > to
> > > > > > focus on the initial bug that I hit.
> > > > > >
> > > > > > Exception in thread "main" org.apache.spark.SparkException:
Job
> > > > aborted:
> > > > > > Exception while deserializing and fetching task:
> > > > > > com.esotericsoftware.kryo.KryoException:
> > > > > > java.lang.IllegalArgumentException: Can not set final
> > > > > > scala.collection.convert.Wrappers field
> > > > > > scala.collection.convert.Wrappers$SeqWrapper.$outer to
> > > > > > clojure.lang.PersistentVector
> > > > > > Serialization trace:
> > > > > > $outer (scala.collection.convert.Wrappers$SeqWrapper)
> > > > > >         at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1020)
> > > > > >         at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1018)
> > > > > >         at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> > > > > >         at
> > > > > >
> scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> > > > > >         at org.apache.spark.scheduler.DAGScheduler.org
> > > > > >
> > > >
> > $apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1018)
> > > > > >         at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
> > > > > >         at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
> > > > > >         at scala.Option.foreach(Option.scala:236)
> > > > > >         at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:604)
> > > > > >         at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:190)
> > > > > >         at
> akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
> > > > > >         at akka.actor.ActorCell.invoke(ActorCell.scala:456)
> > > > > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message