flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Máté Gulyás <guly...@dmlab.hu>
Subject Re: Flume NG and S3
Date Tue, 01 Jul 2014 06:47:00 GMT
Please see my comments inline.

YIMEN YIMGA Gael wrote:
> Could you please communicate the link of the article you read please ?
https://gist.github.com/crowdmatt/5256881 and the last comment.

Sharninder wrote
> No reason to not use flume except for the fact that S3, since its over the wire, will
be a lot slower than a local hdfs cluster in which case you need a big enough channel to hold
events not yet processed out of the sink. If you have a fast enough pipe, you can very well
use flume for this sort of use-case.
I plan to aggregate 5-15GB data with Filechannel, as I want to flush
to S3 every hour on every node. As far as I know Flume can gzip it, so
the size would be about 500MB-1,5GB.

Thanks for the feedback, I will write If I have any results.

Mate Gulyas

On Tue, Jul 1, 2014 at 6:26 AM, Sharninder <sharninder@gmail.com> wrote:
> No reason to not use flume except for the fact that S3, since its over the
> wire, will be a lot slower than a local hdfs cluster in which case you need
> a big enough channel to hold events not yet processed out of the sink. If
> you have a fast enough pipe, you can very well use flume for this sort of
> use-case.
>
> The reason the author might have moved to kafka, and I'm just speculating
> here, is that kafka provides him better buffering support for exactly the
> case I've written above.
>
> HTH
> Sharninder
>
>
>
> On Mon, Jun 30, 2014 at 7:57 PM, Máté Gulyás <gulyasm@dmlab.hu> wrote:
>>
>> Hi!
>>
>> I would like to use flume to aggregate and send logs to an S3 bucket.
>> I did some research, but the last article I found on the topic was
>> more then a year old and the author abandoned Flume for Kafka. My
>> other concern is that most of the articles were written for Flume OG,
>> not NG.
>> Is there any reason why I should not use flume to sink messages to S3?
>>
>>
>> Thanks in advance.
>>
>> Mate Gulyas
>> Lead Developer at Dmlab
>
>

Mime
View raw message