spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Soren Macbeth (JIRA)" <>
Subject [jira] [Commented] (SPARK-2878) Inconsistent Kryo serialisation with custom Kryo Registrator
Date Thu, 07 Aug 2014 03:42:11 GMT


Soren Macbeth commented on SPARK-2878:

My particular case _feels_ a bit different, although I suspect the root
case is likely the same. In my case, intermittently, some of the scala
collection classes that are part of chill and part of the default kryo
serializer wouldn't get registered in my mesos slave tasks. I fixed it my
explicitly registering them in my custom serializer to ensure the the class
in question, scala's `Some`, always had a consistent id in kryo.

I haven't commented out my hack in >= 1.0.0 to see if it's still an issue
or not.

On Wed, Aug 6, 2014 at 7:23 PM, Graham Dennis (JIRA) <>


> Inconsistent Kryo serialisation with custom Kryo Registrator
> ------------------------------------------------------------
>                 Key: SPARK-2878
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.0.0, 1.0.2
>         Environment: Linux RedHat EL 6, 4-node Spark cluster.
>            Reporter: Graham Dennis
> The custom Kryo Registrator (a class with the org.apache.spark.serializer.KryoRegistrator
trait) is not used with every Kryo instance created, and this causes inconsistent serialisation
and deserialisation.
> The Kryo Registrator is sometimes not used because of a ClassNotFound exception that
only occurs if it *isn't* the Worker thread (of an Executor) that tries to create the KryoRegistrator.
> A complete description of the problem and a project reproducing the problem can be found
> I have currently only tested this with Spark 1.0.0, but will try to test against 1.0.2.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message