flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brock Noland <br...@cloudera.com>
Subject Re: Memory Channel
Date Wed, 16 Jan 2013 19:00:43 GMT
The HDFS Sink syncs at the end of each batch or when the file rolls.

On Wed, Jan 16, 2013 at 10:55 AM, Nitin Pawar <nitinpawar432@gmail.com> wrote:
> you can configure it as you nee
> number of events
> rollover by time
> and other ways as well
>
>
> On Thu, Jan 17, 2013 at 12:17 AM, Mohit Anchlia <mohitanchlia@gmail.com>
> wrote:
>>
>> Right. I was asking about sync to "sink". My sink is hdfs so does flume
>> sync to hdfs on every write operation?
>>
>>
>> On Wed, Jan 16, 2013 at 10:26 AM, Brock Noland <brock@cloudera.com> wrote:
>>>
>>> Memory Channel does not write to disk and as such never syncs to disk.
>>> File Channel does sync to disk for each batch put on or taken off the
>>> channel.
>>>
>>> On Wed, Jan 16, 2013 at 10:21 AM, Mohit Anchlia <mohitanchlia@gmail.com>
>>> wrote:
>>> > Thanks! What I am really trying to understand is when does flume sync
>>> > to the
>>> > sink. I am not using batch events.
>>> >
>>> >
>>> > On Wed, Jan 16, 2013 at 9:55 AM, Hari Shreedharan
>>> > <hshreedharan@cloudera.com> wrote:
>>> >>
>>> >> It means that the channel can store that many events. If it is full,
>>> >> then
>>> >> the put() calls (on the source side) will start throwing
>>> >> ChannelException.
>>> >> The put call will block only for keep-alive number of seconds, after
>>> >> which
>>> >> it will throw.
>>> >>
>>> >>
>>> >> Hari
>>> >>
>>> >> --
>>> >> Hari Shreedharan
>>> >>
>>> >> On Wednesday, January 16, 2013 at 9:46 AM, Mohit Anchlia wrote:
>>> >>
>>> >> Could someone help me understand capacity attribute of memoryChannel?
>>> >> Does
>>> >> it mean that memoryChannel flushes to sink only when this capacity is
>>> >> reached or does it mean that it's the max events stored in memory and
>>> >> call
>>> >> blocks until everything else gets freed?
>>> >>
>>> >>
>>> >> http://flume.apache.org/FlumeUserGuide.html#memory-channel
>>> >>
>>> >>
>>> >>
>>> >
>>>
>>>
>>>
>>> --
>>> Apache MRUnit - Unit testing MapReduce -
>>> http://incubator.apache.org/mrunit/
>>
>>
>
>
>
> --
> Nitin Pawar



-- 
Apache MRUnit - Unit testing MapReduce - http://incubator.apache.org/mrunit/

Mime
View raw message