Return-Path: X-Original-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Delivered-To: apmail-hadoop-hdfs-user-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 9F9DFC263 for ; Wed, 22 Aug 2012 06:24:59 +0000 (UTC) Received: (qmail 22992 invoked by uid 500); 22 Aug 2012 06:24:54 -0000 Delivered-To: apmail-hadoop-hdfs-user-archive@hadoop.apache.org Received: (qmail 22753 invoked by uid 500); 22 Aug 2012 06:24:54 -0000 Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hadoop.apache.org Delivered-To: mailing list user@hadoop.apache.org Received: (qmail 22746 invoked by uid 99); 22 Aug 2012 06:24:54 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Aug 2012 06:24:54 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of rahul.rec.dgp@gmail.com designates 209.85.214.48 as permitted sender) Received: from [209.85.214.48] (HELO mail-bk0-f48.google.com) (209.85.214.48) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Aug 2012 06:24:48 +0000 Received: by bkty5 with SMTP id y5so183797bkt.35 for ; Tue, 21 Aug 2012 23:24:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :content-type; bh=GpM8b4RlgB6MIyQ1QTNwO4OYqEnle3fCABlxx6cL950=; b=pEybUvzhbFnQiFzpHfUE+3TwAjCzCkfRS+nzqvO8TEMHvndSkjWOUY6LLSbatb/xwV T1dExOqbX1Vt7jVbev0z+CNGc0ibVmWNiYCUi0rferO3nqhaaORRmMWn7hAwAEkTu5Bo Kd2JTMmVzLo13eZZuIxJ/APinQOrViFNnTfW3VXu8+KhVleZ4wgdCbzwfeIHMDXMusrq hkdlW4N9Dywh9rXdyOpXe+aJXuqsQiPRwJjipSWBmDTyObtbllKNdy3VZgP7Mr9dEHSF N/9WL+D37DsZwUyNcnCBnEdNfNY+5SYn48El8sFa/Z8gRa9XBXMNcplrqyHC7Q8FHZ4L YnXw== Received: by 10.205.136.3 with SMTP id ii3mr5999800bkc.101.1345616667208; Tue, 21 Aug 2012 23:24:27 -0700 (PDT) MIME-Version: 1.0 Received: by 10.204.224.209 with HTTP; Tue, 21 Aug 2012 23:24:06 -0700 (PDT) In-Reply-To: References: From: Rahul Bhattacharjee Date: Wed, 22 Aug 2012 11:54:06 +0530 Message-ID: Subject: Re: Hadoop's Avro dependencies. To: user@hadoop.apache.org Content-Type: multipart/alternative; boundary=000e0ce0d65e73ecb004c7d4cbdb X-Virus-Checked: Checked by ClamAV on apache.org --000e0ce0d65e73ecb004c7d4cbdb Content-Type: text/plain; charset=UTF-8 Thanks a lot Harsh! I should get the code instead of bothering people here in the group. Rgds, Rahul On Wed, Aug 22, 2012 at 11:46 AM, Harsh J wrote: > Hi, > > By default, only the Writable serialization technique is used. If you > choose to use Avro in your job, only then Avro serialization is > utilized at the intermediate serialization step. > > On Wed, Aug 22, 2012 at 11:42 AM, Rahul Bhattacharjee > wrote: > > Well , thanks a lot Harsh. I though avro was result of hadoop's > > serialization needs. > > > > If avro isn't used for serializing maps outputs and transfer it to other > > reducers then whats used for this , if not avro. > > > > Thanks, > > Rahul > > > > On Wed, Aug 22, 2012 at 11:22 AM, Harsh J wrote: > >> > >> Hi, > >> > >> Hadoop doesn't use Avro serialization on its own. However, Hadoop 2.x > >> does provide an AvroSerialization class you can use optionally to > >> serialize using Avro libraries, and the 2.x distribution does ship an > >> Avro jar along with it. > >> > >> On Wed, Aug 22, 2012 at 11:09 AM, Rahul Bhattacharjee > >> wrote: > >> > Hi, > >> > > >> > I was going through the Apache Hadoop's distribution dependencies > (jars > >> > in > >> > lib folder) and I could not find avro-1.x.x.jar. > >> > > >> > I though hadoop internally uses avro as its serialization mechanism > for > >> > intermediate data transmission (transporting maps output to reducers > etc > >> > ), > >> > so hadoop distribution must have avro within it. But it doesn't ! > >> > > >> > Can someone enlighten me on this? > >> > > >> > Thanks, > >> > Rahul > >> > > >> > >> > >> > >> -- > >> Harsh J > > > > > > > > -- > Harsh J > --000e0ce0d65e73ecb004c7d4cbdb Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Thanks a lot H= arsh! I should get the code instead of bothering people here in the group.<= br>
Rgds,
Rahul

On We= d, Aug 22, 2012 at 11:46 AM, Harsh J <harsh@cloudera.com> w= rote:
Hi,

By default, only the Writable serialization technique is used. If you
choose to use Avro in your job, only then Avro serialization is
utilized at the intermediate serialization step.

On Wed, Aug 22, 2012 at 11:42 AM, Rahul Bhattacharjee
<rahul.rec.dgp@gmail.com> wrote:
> Well , thanks a lot Harsh. I though avro was result of hadoop's > serialization needs.
>
> If avro isn't used for serializing maps outputs and transfer it to= other
> reducers then whats used for this , if not avro.
>
> Thanks,
> Rahul
>
> On Wed, Aug 22, 2012 at 11:22 AM, Harsh J <harsh@cloudera.com> wrote:
>>
>> Hi,
>>
>> Hadoop doesn't use Avro serialization on its own. However, Had= oop 2.x
>> does provide an AvroSerialization class you can use optionally to<= br> >> serialize using Avro libraries, and the 2.x distribution does ship= an
>> Avro jar along with it.
>>
>> On Wed, Aug 22, 2012 at 11:09 AM, Rahul Bhattacharjee
>> <rahul.rec.dgp@gmail= .com> wrote:
>> > Hi,
>> >
>> > I was going through the Apache Hadoop's distribution depe= ndencies (jars
>> > in
>> > lib folder) and I could not find avro-1.x.x.jar.
>> >
>> > I though hadoop internally uses avro as its serialization mec= hanism for
>> > intermediate data transmission (transporting maps output to r= educers etc
>> > ),
>> > so hadoop distribution must have avro within it. But it doesn= 't !
>> >
>> > Can someone enlighten me on this?
>> >
>> > Thanks,
>> > Rahul
>> >
>>
>>
>>
>> --
>> Harsh J
>
>



--
Harsh J

--000e0ce0d65e73ecb004c7d4cbdb--