flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hari Shreedharan" <hshreedha...@cloudera.com>
Subject Re: sink buffer event processing order
Date Fri, 31 Oct 2014 04:39:37 GMT
Ordering can get messed up if there are multiple sinks removing data and also if there are
failures. When multiple sinks remove data, the order depends on the ordering of the sink writes.
Same goes for multiple sources.


Thanks,
Hari

On Thu, Oct 30, 2014 at 9:27 PM, Ashish <paliwalashish@gmail.com> wrote:

> Channel is like a Queue, new data shall be appended at the end. So you
> shall receive data as 1,2,3,4,5,6,7,8 at upstream agent.
> On Fri, Oct 31, 2014 at 8:16 AM, terreyshih <terreyshih@gmail.com> wrote:
>> Hi,
>>
>> I would like to know what happens in the in following scenario:
>>
>> 1. down stream source agent is down for some reason.
>> 2. up stream agent buffers it in the sink channel while keeps receiving data from
the source
>> 3. down stream agent comes back up.
>> 4. does the upstream agent CLEAR the buffer before sending the new incoming data
from the source ?
>> 5. Is it possible that the buffer data and the newly incoming data INTERLEAVE each
other ???
>>
>> In other words,
>>
>> In the upstream agent:
>>
>> The buffer has 1,2,3,4,5
>> The new data comes 6,7,8
>>
>> The upstream agent sends the data in the order of 6,7,1,2,3,8,5, etc.
>>
>> Again, I know this would not happen if the down stream agent is never brought down.
 However, I am just wondering if it is possible for this to happen is the downstream is brought
down and then up again ?
>>
>> thanks,
>> -Gary
> -- 
> thanks
> ashish
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
Mime
View raw message