Mailing-List: contact user-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flume.apache.org
Received-SPF: pass (nike.apache.org: domain of mgulyas86@gmail.com designates
 209.85.212.170 as permitted sender)
MIME-Version: 1.0
Sender: mgulyas86@gmail.com
In-Reply-To: 
 <CAACPw8A8VvnQjwtyv7kkZReBOn+LF-sBkweb1ebq7R0Hr6f8wg@mail.gmail.com>
References: 
 <CAHJ7p9h6ekzKiFRoWrVAXQ1gTJkwC-BqiqdoZqsnrLeHPT3y5Q@mail.gmail.com>
	<CAACPw8A8VvnQjwtyv7kkZReBOn+LF-sBkweb1ebq7R0Hr6f8wg@mail.gmail.com>
Date: Tue, 1 Jul 2014 08:47:00 +0200
Message-ID: 
 <CAHJ7p9i-Lmi4NXum+aJPdGgEetpu5V0--+aTdVtGy7D=q9=dHw@mail.gmail.com>
Subject: Re: Flume NG and S3
From: =?UTF-8?B?TcOhdMOpIEd1bHnDoXM=?= <gulyasm@dmlab.hu>
To: user@flume.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Please see my comments inline.

YIMEN YIMGA Gael wrote:
> Could you please communicate the link of the article you read please ?
https://gist.github.com/crowdmatt/5256881 and the last comment.

Sharninder wrote
> No reason to not use flume except for the fact that S3, since its over th=
e wire, will be a lot slower than a local hdfs cluster in which case you ne=
ed a big enough channel to hold events not yet processed out of the sink. I=
f you have a fast enough pipe, you can very well use flume for this sort of=
 use-case.
I plan to aggregate 5-15GB data with Filechannel, as I want to flush
to S3 every hour on every node. As far as I know Flume can gzip it, so
the size would be about 500MB-1,5GB.

Thanks for the feedback, I will write If I have any results.

Mate Gulyas

On Tue, Jul 1, 2014 at 6:26 AM, Sharninder <sharninder@gmail.com> wrote:
> No reason to not use flume except for the fact that S3, since its over th=
e
> wire, will be a lot slower than a local hdfs cluster in which case you ne=
ed
> a big enough channel to hold events not yet processed out of the sink. If
> you have a fast enough pipe, you can very well use flume for this sort of
> use-case.
>
> The reason the author might have moved to kafka, and I'm just speculating
> here, is that kafka provides him better buffering support for exactly the
> case I've written above.
>
> HTH
> Sharninder
>
>
>
> On Mon, Jun 30, 2014 at 7:57 PM, M=C3=A1t=C3=A9 Guly=C3=A1s <gulyasm@dmla=
b.hu> wrote:
>>
>> Hi!
>>
>> I would like to use flume to aggregate and send logs to an S3 bucket.
>> I did some research, but the last article I found on the topic was
>> more then a year old and the author abandoned Flume for Kafka. My
>> other concern is that most of the articles were written for Flume OG,
>> not NG.
>> Is there any reason why I should not use flume to sink messages to S3?
>>
>>
>> Thanks in advance.
>>
>> Mate Gulyas
>> Lead Developer at Dmlab
>
>