avro-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Harsh J <qwertyman...@gmail.com>
Subject Re: How to get started with examples on avro
Date Fri, 28 Jan 2011 21:31:14 GMT
On Sat, Jan 29, 2011 at 1:59 AM, felix gao <gre1600@gmail.com> wrote:
> Thanks for the quick reply.  I am interested in doing this through the java
> implementation and I would like to do it in parallel that utilizes the
> mapreduce framework.

That operation is pretty similar to writing a normal output data file.

You can use the MapReduce API of Avro (that provides an Input/Output
Format class to use, given a Schema) to do so, or write your own
custom record writing classes that do it by converting your input
format's record representation to Avro serialized records and writing
those out to an open DataFile for a given schema. Alternatively, you
can also write avro serialized data bytes into SequenceFiles.

I believe the Hadoop MapReduce trunk may have some good code on Avro
serialization classes and uses of that in MapReduce.

> On Fri, Jan 28, 2011 at 12:22 PM, Harsh J <qwertymaniac@gmail.com> wrote:
>> Based on the language you're targeting, have a look at its test-cases
>> available on the in the project's version control:
>> http://svn.apache.org/repos/asf/avro/trunk/lang/ [You can check it out
>> via SVN, or via Git mirrors]
>> Another good resource on the ends of Avro (Data and RPC) is by phunt
>> at http://github.com/phunt/avro-rpc-quickstart#readme
>> I had written a python data-file centric snippet for Avro a while ago
>> at my blog; it may help if you're looking to get started with Python
>> (although it does not cover all aspects, which the functions in the
>> available test cases for lang/python do):
>> http://www.harshj.com/2010/04/25/writing-and-reading-avro-data-files-using-python/
>> On Sat, Jan 29, 2011 at 1:34 AM, felix gao <gre1600@gmail.com> wrote:
>> > Hi all,
>> > I am trying to convert a lot of our existing logs into avro format in
>> > hadoop.  I am not sure if there are any examples to follow.
>> > Thanks,
>> > Felix
>> --
>> Harsh J
>> www.harshj.com

Harsh J

View raw message