spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aureliano Buendia <buendia...@gmail.com>
Subject How to confirm serializer type on workers?
Date Wed, 08 Jan 2014 20:23:49 GMT
Hi,


I'm using kryo serializer, but getting java.io.EOFException error when
broadcasting an object. It seems one of the reasons for getting this error
is that the workers are not aware of the kryo serializer.

In the app code, before creating the context, I have:

System.setProperty("spark.serializer",
"org.apache.spark.serializer.KryoSerializer")
System.setProperty("spark.kryo.registrator", "myRegistrator")

Does this only work on the master and not on the workers?

I've also tried setting this in conf/spark-env.sh:

export
SPARK_JAVA_OPTS="-Dspark.serializer=org.apache.spark.serializer.KryoSerializer
-Dspark.kryo.registrator=my.Registrator"

But I'm still getting the eof error.

How is it possible to programmatically set the kryo serializer on the
workers side, the same as it is done on the master?

Mime
View raw message