Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Message-ID: <50AA3D41.5060705@crs4.it>
Date: Mon, 19 Nov 2012 15:08:01 +0100
From: Luca Pireddu <pireddu@crs4.it>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:16.0) Gecko/20121028 Thunderbird/16.0.2
MIME-Version: 1.0
To: user@hadoop.apache.org
Subject: Re: Pydoop 0.7.0-rc1 released
References: <50A246F5.5060904@crs4.it>
 <dadb8bc1be4fc04c921dacdc8a27fe3d@verwilst.be>
In-Reply-To: <dadb8bc1be4fc04c921dacdc8a27fe3d@verwilst.be>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 7bit

On 11/16/2012 10:02 PM, Bart Verwilst wrote:
> Hi Simone,
>
> I was wondering, is it possible to write AVRO files to hadoop straight
> from your lib ( mixed with avro libs ofcourse )? I'm currently trying to
> come up with a way to read from mysql ( but more complicated than sqoop
> can handle ) and write it out to avro files on HDFS. Is something like
> this feasible with this? How do you see it?
>
> Thanks!
>
> Bart

Hello,

you could use a record writer that uses the python-avro package 
(http://pypi.python.org/pypi/avro/1.7.2).  Unfortunately I've seen a few 
complaints about its speed.  For an example of a RecordWriter 
implemented in Python see wordcount-full in the Pydoop examples.

If that solution turns out  it's too slow for you, you may consider 
writing a Java record writer that uses the standard Avro implementation.

In either case, you'll have to get data to it from your reducers to the 
record writer.  Pydoop only supports emitting byte streams, so you'll 
have to serialize your data as a string of some sort, pass it to pydoop, 
receive it in the RecordWriter where you'll de-serialize it and then 
pass it to the Avro library.


-- 
Luca Pireddu
CRS4 - Distributed Computing Group
Loc. Pixina Manna Edificio 1
09010 Pula (CA), Italy
Tel: +39 0709250452