spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xiangrui Meng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SPARK-2200) breeze DenseVector not serializable with KryoSerializer
Date Thu, 19 Jun 2014 18:32:25 GMT

    [ https://issues.apache.org/jira/browse/SPARK-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037637#comment-14037637
] 

Xiangrui Meng commented on SPARK-2200:
--------------------------------------

[~neville] Do you know the root cause and how this is fixed in breeze 0.8.1? You disabled
reference tracking, which may be the reason.

> breeze DenseVector not serializable with KryoSerializer
> -------------------------------------------------------
>
>                 Key: SPARK-2200
>                 URL: https://issues.apache.org/jira/browse/SPARK-2200
>             Project: Spark
>          Issue Type: Bug
>          Components: MLlib
>    Affects Versions: 1.0.0
>            Reporter: Neville Li
>            Priority: Minor
>
> Spark 1.0.0 depends on breeze 0.7 and for some reason serializing DenseVector with KryoSerializer
throws the following stack trace. Looks like some recursive field in the object. Upgrading
to 0.8.1 solved this.
> {code}
> java.lang.StackOverflowError
> 	at java.lang.reflect.Field.getDeclaringClass(Field.java:154)
> 	at sun.reflect.UnsafeFieldAccessorImpl.ensureObj(UnsafeFieldAccessorImpl.java:54)
> 	at sun.reflect.UnsafeQualifiedObjectFieldAccessorImpl.get(UnsafeQualifiedObjectFieldAccessorImpl.java:38)
> 	at java.lang.reflect.Field.get(Field.java:379)
> 	at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:552)
> 	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
> 	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
> 	at com.esotericsoftware.kryo.serializers.FieldSerializer$ObjectField.write(FieldSerializer.java:564)
> 	at com.esotericsoftware.kryo.serializers.FieldSerializer.write(FieldSerializer.java:213)
> 	at com.esotericsoftware.kryo.Kryo.writeObject(Kryo.java:501)
> ...
> {code}
> Code to reproduce:
> {code}
> import breeze.linalg.DenseVector
> import org.apache.spark.SparkConf
> import org.apache.spark.serializer.KryoSerializer
> object SerializerTest {
>   def main(args: Array[String]) {
>     val conf = new SparkConf()
>       .set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
>       .set("spark.kryo.registrator", classOf[MyRegistrator].getName)
>       .set("spark.kryo.referenceTracking", "false")
>       .set("spark.kryoserializer.buffer.mb", "8")
>     val serializer = new KryoSerializer(conf).newInstance()
>     serializer.serialize(DenseVector.rand(10))
>   }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message