hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: custom writablecomparable with complex fields
Date Sat, 31 Aug 2013 20:52:10 GMT
The idea behind write(…) and readFields(…) is simply that of ordering.
You need to write your custom objects (i.e. a representation of them)
in order, and read them back in the same order.

An example way of serializing a list would be to first serialize the
length (so you know how many you'll be needed to read back), and then
serialize each item appropriately, using delimiters, or using
length-prefixes just like lists.

Mainly, you're required to tackle the serialization/deserialization on your own.

This is one of the reasons I highly recommend using a library like
Apache Avro instead. Its more powerful, faster, and yet simple to use:
http://avro.apache.org/docs/current/gettingstartedjava.html and
http://avro.apache.org/docs/current/mr.html. It is also popular and
carries first-grade support on several other hadoop-ecosystem
projects, such as Flume and Crunch.

On Sun, Sep 1, 2013 at 1:23 AM, Adeel Qureshi <adeelmahmood@gmail.com> wrote:
> I want to write a custom writablecomparable object with two List objects
> within it ..
> public class CompositeKey implements WritableComparable {
> private List<JsonKey> groupBy;
> private List<JsonKey> sortBy;
> ...
> }
> what I am not sure about is how to write
> readFields and write methods for this object. Any help would be appreciated.
> Thanks
> Adeel

Harsh J

View raw message