flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bart Verwilst <li...@verwilst.be>
Subject Re: Using Python and Flume to store avro data
Date Thu, 08 Nov 2012 18:57:26 GMT
 

Hi Hari, 

Are you talking about ipc.HTTPTransciever (
http://nullege.com/codes/search/avro.ipc.HTTPTransceiver )? This was the
class I tried before i noticed it wasn't supported by Flume-1.2 :) 

I
assume the http/json source will also allow for avro to be received?


Kind regards, 

Bart 

Hari Shreedharan schreef op 08.11.2012 19:51:


> The next release of Flume-1.3.0 adds support for an HTTP source,
which will allow you to send data to Flume via HTTP/JSON(the
representation of the data is pluggable - but a JSON representation is
default). You could use this to write data to Flume from Python, which I
believe has good http and json support. 
> 
> Thanks, 
> Hari 
> 
> --

> Hari Shreedharan 
> 
> On Thursday, November 8, 2012 at 10:45 AM,
Bart Verwilst wrote: 
> 
>> Hi, 
>> 
>> I've been spending quite a few
hours trying to push avro data to Flume 
>> so i can store it on HDFS,
this all with Python. 
>> It seems like something that is impossible for
now, since the only way 
>> to push avro data to Flume is by the use of
deprecated thrift binding 
>> that look pretty cumbersome to get
working. 
>> I would like to know what's the best way to import avro
data into Flume 
>> with Python? Maybe Flume isnt the right tool and I
should use something 
>> else? My goal is to have multiple python
workers pushing data to HDFS 
>> which ( by means of Flume in this case
) consolidates this all in 1 file 
>> there. 
>> 
>> Any thoughts? 
>>

>> Thanks! 
>> 
>> Bart
 
Mime
View raw message