hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <ha...@cloudera.com>
Subject Re: Hadoop's Avro dependencies.
Date Wed, 22 Aug 2012 06:16:18 GMT
Hi,

By default, only the Writable serialization technique is used. If you
choose to use Avro in your job, only then Avro serialization is
utilized at the intermediate serialization step.

On Wed, Aug 22, 2012 at 11:42 AM, Rahul Bhattacharjee
<rahul.rec.dgp@gmail.com> wrote:
> Well , thanks a lot Harsh. I though avro was result of hadoop's
> serialization needs.
>
> If avro isn't used for serializing maps outputs and transfer it to other
> reducers then whats used for this , if not avro.
>
> Thanks,
> Rahul
>
> On Wed, Aug 22, 2012 at 11:22 AM, Harsh J <harsh@cloudera.com> wrote:
>>
>> Hi,
>>
>> Hadoop doesn't use Avro serialization on its own. However, Hadoop 2.x
>> does provide an AvroSerialization class you can use optionally to
>> serialize using Avro libraries, and the 2.x distribution does ship an
>> Avro jar along with it.
>>
>> On Wed, Aug 22, 2012 at 11:09 AM, Rahul Bhattacharjee
>> <rahul.rec.dgp@gmail.com> wrote:
>> > Hi,
>> >
>> > I was going through the Apache Hadoop's distribution dependencies (jars
>> > in
>> > lib folder) and I could not find avro-1.x.x.jar.
>> >
>> > I though hadoop internally uses avro as its serialization mechanism for
>> > intermediate data transmission (transporting maps output to reducers etc
>> > ),
>> > so hadoop distribution must have avro within it. But it doesn't !
>> >
>> > Can someone enlighten me on this?
>> >
>> > Thanks,
>> > Rahul
>> >
>>
>>
>>
>> --
>> Harsh J
>
>



-- 
Harsh J

Mime
View raw message