spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Soren Macbeth (JIRA)" <>
Subject [jira] [Commented] (SPARK-2878) Inconsistent Kryo serialisation with custom Kryo Registrator
Date Thu, 07 Aug 2014 02:19:11 GMT


Soren Macbeth commented on SPARK-2878:

Great find! 

This has definitely bitten me. I found a lame workround, but It would be fantastic to have
this well and truly fixed. Flambo needs kryo to function (not to mention that kryo is much
faster than java serialization almost all the time).  

> Inconsistent Kryo serialisation with custom Kryo Registrator
> ------------------------------------------------------------
>                 Key: SPARK-2878
>                 URL:
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 1.0.0, 1.0.2
>         Environment: Linux RedHat EL 6, 4-node Spark cluster.
>            Reporter: Graham Dennis
> The custom Kryo Registrator (a class with the org.apache.spark.serializer.KryoRegistrator
trait) is not used with every Kryo instance created, and this causes inconsistent serialisation
and deserialisation.
> The Kryo Registrator is sometimes not used because of a ClassNotFound exception that
only occurs if it *isn't* the Worker thread (of an Executor) that tries to create the KryoRegistrator.
> A complete description of the problem and a project reproducing the problem can be found
> I have currently only tested this with Spark 1.0.0, but will try to test against 1.0.2.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message