Return-Path: Delivered-To: apmail-avro-user-archive@www.apache.org Received: (qmail 66556 invoked from network); 28 Jan 2011 21:54:04 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) by minotaur.apache.org with SMTP; 28 Jan 2011 21:54:04 -0000 Received: (qmail 59174 invoked by uid 500); 28 Jan 2011 21:54:04 -0000 Delivered-To: apmail-avro-user-archive@avro.apache.org Received: (qmail 58990 invoked by uid 500); 28 Jan 2011 21:54:03 -0000 Mailing-List: contact user-help@avro.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@avro.apache.org Delivered-To: mailing list user@avro.apache.org Received: (qmail 58982 invoked by uid 99); 28 Jan 2011 21:54:03 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Jan 2011 21:54:03 +0000 X-ASF-Spam-Status: No, hits=3.7 required=5.0 tests=FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of gre1600@gmail.com designates 209.85.216.43 as permitted sender) Received: from [209.85.216.43] (HELO mail-qw0-f43.google.com) (209.85.216.43) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 28 Jan 2011 21:53:59 +0000 Received: by qwk3 with SMTP id 3so3676594qwk.30 for ; Fri, 28 Jan 2011 13:53:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=gamma; h=domainkey-signature:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=Z8PMWnnkcrI6TM60hkdtSxhhP3JLNxaS6D4z0zSyXwE=; b=hsoW4ZFlYYyhyQIUNBMpsN5tUeTq81ZiNJKY1mUAvjBTJndVnDBgyTdkBaiI5IYfY9 GxQMIr6n6eHF/+GDFjAx0XO3tRqG5VusPzSogKqTdcrASoAx8I4dwIYRW9+RyNqzofRE 1ueYAqaa1feE/cx9/4xXamcVollpJcCiWSRkA= DomainKey-Signature: a=rsa-sha1; c=nofws; d=gmail.com; s=gamma; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; b=ig4zog0m1PEbJXyhF065oPvy7fr46Ek5fFpr4HsoqWvdcnXK50hhX3or5sLKsAjumv dO0VpQCFsvAd4wHjSSMhJS58cSjNfnj2MtQ2ExP0qN8+FQn8cevbUT7F1KNhnKXsHOEZ 6p3sFDBMaYIOzynQ/uxxJxPgfeWfzPMZ+cqQQ= MIME-Version: 1.0 Received: by 10.224.28.80 with SMTP id l16mr3338188qac.204.1296251618206; Fri, 28 Jan 2011 13:53:38 -0800 (PST) Received: by 10.220.76.136 with HTTP; Fri, 28 Jan 2011 13:53:38 -0800 (PST) In-Reply-To: <79A1E64A36D5D64A86E9A06BC828068001994D1A82@EXVMBX017-1.exch017.msoutlookonline.net> References: <79A1E64A36D5D64A86E9A06BC828068001994D1A82@EXVMBX017-1.exch017.msoutlookonline.net> Date: Fri, 28 Jan 2011 13:53:38 -0800 Message-ID: Subject: Re: How to get started with examples on avro From: felix gao To: user@avro.apache.org Content-Type: multipart/alternative; boundary=0015175cba7a3e2312049aef1913 --0015175cba7a3e2312049aef1913 Content-Type: text/plain; charset=ISO-8859-1 The goal to convert to avro is for us to use the splittable property so we can have some type of compressed data for huge log files so we can save some hdfs disk spaces. On Fri, Jan 28, 2011 at 1:02 PM, Anand Padmanaban wrote: > Meta question. I see avro is the means, what is the end goal? What do you > want to do with the data after converting it to avro? > > > -----Original Message----- > > From: felix gao [mailto:gre1600@gmail.com] > > Sent: Friday, January 28, 2011 12:30 PM > > To: user@avro.apache.org > > Subject: Re: How to get started with examples on avro > > > > Thanks for the quick reply. I am interested in doing this through the > java implementation and I would like to do it in parallel that > > utilizes the mapreduce framework. > > > > > > On Fri, Jan 28, 2011 at 12:22 PM, Harsh J > wrote: > > > > > > Based on the language you're targeting, have a look at its > test-cases > > available on the in the project's version control: > > http://svn.apache.org/repos/asf/avro/trunk/lang/ [You can check it > out > > via SVN, or via Git mirrors] > > > > Another good resource on the ends of Avro (Data and RPC) is by > phunt > > at http://github.com/phunt/avro-rpc-quickstart#readme > > > > I had written a python data-file centric snippet for Avro a while > ago > > at my blog; it may help if you're looking to get started with > Python > > (although it does not cover all aspects, which the functions in the > > available test cases for lang/python do): > > > http://www.harshj.com/2010/04/25/writing-and-reading-avro-data-files-using-python/ > > > > > > On Sat, Jan 29, 2011 at 1:34 AM, felix gao > wrote: > > > Hi all, > > > I am trying to convert a lot of our existing logs into avro > format in > > > hadoop. I am not sure if there are any examples to follow. > > > Thanks, > > > Felix > > > > > > > > > > -- > > Harsh J > > www.harshj.com > > > > > > --0015175cba7a3e2312049aef1913 Content-Type: text/html; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable The goal to convert to avro is for us to use the splittable property so we = can have some type of =A0compressed data for huge log files so we can save = some hdfs disk spaces. =A0

On Fri, Jan 28= , 2011 at 1:02 PM, Anand Padmanaban <apadmanaban@splunk.com> wrote:
Meta question. =A0I see avro is the means, = what is the end goal? =A0What do you want to do with the data after convert= ing it to avro?

> -----Original Message-----
> From: felix gao [mailto:gre1600@g= mail.com]
> Sent: Friday, January 28, 2011 12:30 PM
> To: user@avro.apache.org > Subject: Re: How to get started with examples on avro
>
> Thanks for the quick reply. =A0I am interested in doing this through t= he java implementation and I would like to do it in parallel that
> utilizes the mapreduce framework.
>
>
> On Fri, Jan 28, 2011 at 12:22 PM, Harsh J <qwertymaniac@gmail.com> wrote:
>
>
> =A0 =A0 =A0 Based on the language you're targeting, have a look at= its test-cases
> =A0 =A0 =A0 available on the in the project's version control:
> =A0 =A0 =A0 http://svn.apache.org/repos/asf/avro/trunk/lang/ [= You can check it out
> =A0 =A0 =A0 via SVN, or via Git mirrors]
>
> =A0 =A0 =A0 Another good resource on the ends of Avro (Data and RPC) i= s by phunt
> =A0 =A0 =A0 at http://github.com/phunt/avro-rpc-quickstart#readm= e
>
> =A0 =A0 =A0 I had written a python data-file centric snippet for Avro = a while ago
> =A0 =A0 =A0 at my blog; it may help if you're looking to get start= ed with Python
> =A0 =A0 =A0 (although it does not cover all aspects, which the functio= ns in the
> =A0 =A0 =A0 available test cases for lang/python do):
> =A0 =A0 =A0 http://www.harshj.co= m/2010/04/25/writing-and-reading-avro-data-files-using-python/
>
>
> =A0 =A0 =A0 On Sat, Jan 29, 2011 at 1:34 AM, felix gao <gre1600@gmail.com> wrote:
> =A0 =A0 =A0 > Hi all,
> =A0 =A0 =A0 > I am trying to convert a lot of our existing logs int= o avro format in
> =A0 =A0 =A0 > hadoop. =A0I am not sure if there are any examples to= follow.
> =A0 =A0 =A0 > Thanks,
> =A0 =A0 =A0 > Felix
>
>
>
>
> =A0 =A0 =A0 --
> =A0 =A0 =A0 Harsh J
> =A0 =A0 =A0 www.ha= rshj.com
>
>


--0015175cba7a3e2312049aef1913--