hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Hadoop Avro Question
Date Wed, 01 May 2013 05:05:27 GMT
Moving the question to Apache Avro's user@ lists. Please use the right
lists for the most relevant answers.

Avro is a different serialization technique that intends to replace
the Writable serialization defaults in Hadoop. MR accepts a list of
serializers it can use for its key/value structures and isn't limited
to Writable in any way. Look up the property "io.serializations" in
your Hadoop's core-default.xml for more information.

The Avro project also offers fast comparator classes that are used for
comparing the bytes/structures of Avro objects. This is mostly
auto-set for you when you use the MR framework as described at
http://avro.apache.org/docs/current/api/java/org/apache/avro/mapred/package-summary.html
(via AvroJob helper class).

On Tue, Apr 30, 2013 at 6:51 PM, Rahul Bhattacharjee
<rahul.rec.dgp@gmail.com> wrote:
> Hi,
>
> When dealing with Avro data files in MR jobs ,we use AvroMapper , I noticed
> that the output of K and V of AvroMapper isnt writable and neither the key
> is comparable (these are AvroKey and AvroValue). As the general
> serialization mechanism is writable , how is the K,V pairs in case of avro ,
> travel across nodes?
>
> Thanks,
> Rahul



-- 
Harsh J

Mime
View raw message