accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Elser <josh.el...@gmail.com>
Subject Re: Serialization error
Date Wed, 29 Apr 2015 14:10:17 GMT
Awesome. Thanks so much for sharing the solution! It's very appreciated.
On Apr 29, 2015 6:14 AM, "madhvi" <madhvi.gupta@orkash.com> wrote:

>  Hi Josh,
>
> The classes which are not serialized are automatically serialized by
> kryo.It is used as follows while making the sparkConf object:
> val conf = new SparkConf()
>
>       .setAppName(detail)
>
>       .set("spark.serializer",
> "org.apache.spark.serializer.KryoSerializer")
>
>       .set("spark.driver.maxResultSize", arguments.get("maxResultSize"
> ).get)
> .registerKryoClasses(Array(classOf[org.apache.accumulo.core.data.Key]))
>
> Thanks
> Madhvi
> On Tuesday 28 April 2015 11:32 PM, Josh Elser wrote:
>
> Thanks for the report back, Vaibhav.
>
> To clarify, did you have to write code to do the serialization with Kryo
> or was this something that Kryo could do automatically.
>
> If you could expand on the steps you took, that'd be awesome as I assume
> it will be extremely helpful in the mail archives for others who run into
> this problem.
>
> vaibhav thapliyal wrote:
>
> Hi josh,
>
> We solved it using the kryo serializer library to serialise the key class.
>
> Thanks
> vaibhav
>
> On 28-Apr-2015 11:14 pm, "Josh Elser" <josh.elser@gmail.com
> <mailto:josh.elser@gmail.com> <josh.elser@gmail.com>> wrote:
>
>     Hi Madhvi,
>
>     Thanks for posting this. I'm not super familiar, but my hunch is
>     that Spark requires objects that it works with to implement the Java
>     Serializable interface.
>
>     Accumulo deals with Key (and Value) through Hadoop's Writable
>     interface (technically WritableComparable, but still stems from
>     Writable). I'm not sure if there's a way that you can inform Spark
>     to use the Writable interface methods instead of Serializable.
>
>     If there isn't a way to do this, I don't see any reason why we
>     couldn't make Key (and Value) also implement Serializable for this
>     use case. Please open an issue on JIRA for this -- we can track the
>     investigation there.
>
>     Thanks!
>
>     - Josh
>
>     madhvi wrote:
>
>         Hi,
>
>         While connecting to accumulo through spark by making sparkRDD I am
>         getting the following error:
>         object not serializable (class: org.apache.accumulo.core.data.Key)
>
>         This is due to the 'key' class of accumulo which does not
> implement
>         serializable interface.How it can be solved and accumulo can be
> used
>         with spark
>
>         Thanks
>         Madhvi
>
>
>

Mime
View raw message