spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Soren Macbeth <so...@yieldbot.com>
Subject Re: bug using kryo as closure serializer
Date Mon, 05 May 2014 05:22:02 GMT
that would violate my personal oath of never writing a single line of
scala, but I might be able to do that if I can get past the issue this
issue I'm struggling with in this thread.

On Sunday, May 4, 2014, Reynold Xin <rxin@databricks.com> wrote:

> Thanks. Do you mind playing with chill-scala a little bit and see if it
> actually works well for closures? One way to try is to hard code the
> serializer to use Kryo with chill-scala, and then run through all the unit
> tests.
>
> If it works well, we can incorporate that in the next release (probably not
> 1.0, but after that).
>
>
> On Sun, May 4, 2014 at 9:08 PM, Soren Macbeth <soren@yieldbot.com> wrote:
>
> > fwiw, it seems like it wouldn't be very difficult to integrate
> chill-scala,
> > since you're already chill-java and probably get kryo serialization of
> > closures and all sorts of other scala stuff for free. All that would be
> > needed would be to include the dependency and then update KryoSerializer
> to
> > register the stuff in chill-scala.
> >
> > In that case, you could probably safely make kryo the default serializer,
> > which I think would be desirable in general.
> >
> >
> > On Sun, May 4, 2014 at 8:48 PM, Reynold Xin <rxin@databricks.com> wrote:
> >
> > > Good idea. I submitted a pull request for the doc update here:
> > > https://github.com/apache/spark/pull/642
> > >
> > >
> > > On Sun, May 4, 2014 at 3:54 PM, Soren Macbeth <soren@yieldbot.com>
> > wrote:
> > >
> > > > Thanks for the reply!
> > > >
> > > > Ok, if that's the case, I'd recommend a note to that affect in the
> docs
> > > at
> > > > least.
> > > >
> > > > Just to give some more context here, I'm working on a Clojure DSL for
> > > Spark
> > > > called Flambo, which I plan to open source shortly. If I could I'd
> like
> > > to
> > > > focus on the initial bug that I hit.
> > > >
> > > > Exception in thread "main" org.apache.spark.SparkException: Job
> > aborted:
> > > > Exception while deserializing and fetching task:
> > > > com.esotericsoftware.kryo.KryoException:
> > > > java.lang.IllegalArgumentException: Can not set final
> > > > scala.collection.convert.Wrappers field
> > > > scala.collection.convert.Wrappers$SeqWrapper.$outer to
> > > > clojure.lang.PersistentVector
> > > > Serialization trace:
> > > > $outer (scala.collection.convert.Wrappers$SeqWrapper)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1020)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.spark.scheduler.DAGScheduler$$anonfun$org$apache$spark$scheduler$DAGScheduler$$abortStage$1.apply(DAGScheduler.scala:1018)
> > > >         at
> > > >
> > > >
> > >
> >
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> > > >         at
> > > > scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
> > > >         at org.apache.spark.scheduler.DAGScheduler.org
> > > >
> > $apache$spark$scheduler$DAGScheduler$$abortStage(DAGScheduler.scala:1018)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.spark.scheduler.DAGScheduler$$anonfun$processEvent$10.apply(DAGScheduler.scala:604)
> > > >         at scala.Option.foreach(Option.scala:236)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:604)
> > > >         at
> > > >
> > > >
> > >
> >
> org.apache.spark.scheduler.DAGScheduler$$anonfun$start$1$$anon$2$$anonfun$receive$1.applyOrElse(DAGScheduler.scala:190)
> > > >         at akka.actor.ActorCell.receiveMessage(ActorCell.scala:498)
> > > >         at akka.actor.ActorCell.invoke(ActorCell.scala:456)
> > > >

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message