spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sean Owen <so...@cloudera.com>
Subject Re: Spark 2.0.2, KyroSerializer, double[] is not registered.
Date Sun, 08 Jan 2017 10:03:02 GMT
Double[] is not of the same class as double[]. Kryo should already know how
to serialize double[], but I doubt Double[] is registered.

The error does seem to clearly indicate double[] though. That surprises
me.  Can you try manually registering it to see if that fixes it?
But then I'm not sure why tests wouldn't catch this.

On Sun, Jan 8, 2017 at 7:30 AM smartzjp <zjp_jdev@163.com> wrote:

> You can have a try the following code.
>
> ObjectArraySerializer serializer = new ObjectArraySerializer(kryo, Double
> [].class);
> kryo.register(Double[].class, serializer);
>
>
> -----------
>
> Hi, all.
> I enable kyro in spark with spark-defaults.conf:
>      spark.serializer
> org.apache.spark.serializer.KryoSerializer
>      spark.kryo.registrationRequired  true
>
> A KryoException is raised when a logistic regression algorithm is running:
>      Note: To register this class use: kryo.register(double[].class);
>      Serialization trace:
>      currL1 (org.apache.spark.mllib.stat.MultivariateOnlineSummarizer)
>             at
> com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:585)
>             at
> com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
>             at
> com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
>             at
> com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:36)
>             at
> com.twitter.chill.Tuple2Serializer.write(TupleSerializers.scala:33)
>            at
> com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:568)
>
> My question is:
> Doesn't double[].class be supported by default?
>
> Thanks.
>
>

Mime
View raw message