hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rahul Bhattacharjee <rahul.rec....@gmail.com>
Subject Re: Hadoop's Avro dependencies.
Date Wed, 22 Aug 2012 06:24:06 GMT
Thanks a lot Harsh! I should get the code instead of bothering people here
in the group.

Rgds,
Rahul

On Wed, Aug 22, 2012 at 11:46 AM, Harsh J <harsh@cloudera.com> wrote:

> Hi,
>
> By default, only the Writable serialization technique is used. If you
> choose to use Avro in your job, only then Avro serialization is
> utilized at the intermediate serialization step.
>
> On Wed, Aug 22, 2012 at 11:42 AM, Rahul Bhattacharjee
> <rahul.rec.dgp@gmail.com> wrote:
> > Well , thanks a lot Harsh. I though avro was result of hadoop's
> > serialization needs.
> >
> > If avro isn't used for serializing maps outputs and transfer it to other
> > reducers then whats used for this , if not avro.
> >
> > Thanks,
> > Rahul
> >
> > On Wed, Aug 22, 2012 at 11:22 AM, Harsh J <harsh@cloudera.com> wrote:
> >>
> >> Hi,
> >>
> >> Hadoop doesn't use Avro serialization on its own. However, Hadoop 2.x
> >> does provide an AvroSerialization class you can use optionally to
> >> serialize using Avro libraries, and the 2.x distribution does ship an
> >> Avro jar along with it.
> >>
> >> On Wed, Aug 22, 2012 at 11:09 AM, Rahul Bhattacharjee
> >> <rahul.rec.dgp@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > I was going through the Apache Hadoop's distribution dependencies
> (jars
> >> > in
> >> > lib folder) and I could not find avro-1.x.x.jar.
> >> >
> >> > I though hadoop internally uses avro as its serialization mechanism
> for
> >> > intermediate data transmission (transporting maps output to reducers
> etc
> >> > ),
> >> > so hadoop distribution must have avro within it. But it doesn't !
> >> >
> >> > Can someone enlighten me on this?
> >> >
> >> > Thanks,
> >> > Rahul
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>

Mime
View raw message