spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrew Ash <and...@andrewash.com>
Subject Re: Issue using kryo serilization
Date Fri, 01 Aug 2014 05:45:52 GMT
Did you enable Kryo and have it use your registrator
using spark.serializer=org.apache.spark.serializer.KryoSerializer
and spark.kryo.registrator=mypackage.MyRegistrator ?  It looks like the
serializer being used is the default Java one

http://spark.apache.org/docs/latest/tuning.html#data-serialization


On Fri, Aug 1, 2014 at 1:23 AM, gpatcham <gpatcham@gmail.com> wrote:

> I'm new to spark programming and here I'm trying to use third party class
> in
> map with kryo serializer
>
>   val deviceApi = new DeviceApi()
>
>
> deviceApi.loadDataFromStream(this.getClass.getClassLoader.getResourceAsStream("20140730.json"))
>   val properties = uaRDD1.map(line => deviceApi.getProperties(line)
>
> Here is kryo register code
> import mobi.mtld.da.device.DeviceApi
>
> class MyRegistrator extends KryoRegistrator {
>   override def registerClasses(kryo: Kryo) {
>     kryo.register(classOf[DeviceApi])
>    }
> }
>
> I'm still seeing this error
>
> Caused by: org.apache.spark.SparkException: Job aborted: Task not
> serializable: java.io.NotSerializableException:
> mobi.mtld.da.device.DeviceApi
>
>
>
> Can anyone please let me know if something is wrong
>
> Thanks
> Giri
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Issue-using-kryo-serilization-tp11129.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>

Mime
View raw message