flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From terreyshih <terreys...@gmail.com>
Subject Re: sink buffer event processing order
Date Fri, 31 Oct 2014 15:30:27 GMT
Thanks for the response.

What we are experiencing is that the ordering gets messed up for a while and then after some
time, the ordering restores.

This seems to suggest that once the buffering clears, the ordering restores, and that is why
I was looking for some help on understanding this.

The problem is that I have got the source avro sink, agent one source/sink and agent two source/sink.
 We only found out the ordering is messed up at the agent sink when it’s hooked up to POJO
business entity.

In order to see where the ordering gets messed up, I would need to put a transaction id/timestamp
at the avro source and traces the transaction all the way through the topology to see where
the ordering actually gets out of order.

Can you suggest some other way to debug this ?

thx

> On Oct 30, 2014, at 9:39 PM, Hari Shreedharan <hshreedharan@cloudera.com> wrote:
> 
> Ordering can get messed up if there are multiple sinks removing data and also if there
are failures. When multiple sinks remove data, the order depends on the ordering of the sink
writes. Same goes for multiple sources.
> 
> Thanks,
> Hari
> 
> 
> On Thu, Oct 30, 2014 at 9:27 PM, Ashish <paliwalashish@gmail.com <mailto:paliwalashish@gmail.com>>
wrote:
> 
> Channel is like a Queue, new data shall be appended at the end. So you 
> shall receive data as 1,2,3,4,5,6,7,8 at upstream agent. 
> 
> On Fri, Oct 31, 2014 at 8:16 AM, terreyshih <terreyshih@gmail.com> wrote: 
> > Hi, 
> > 
> > I would like to know what happens in the in following scenario: 
> > 
> > 1. down stream source agent is down for some reason. 
> > 2. up stream agent buffers it in the sink channel while keeps receiving data from
the source 
> > 3. down stream agent comes back up. 
> > 4. does the upstream agent CLEAR the buffer before sending the new incoming data
from the source ? 
> > 5. Is it possible that the buffer data and the newly incoming data INTERLEAVE each
other ??? 
> > 
> > In other words, 
> > 
> > In the upstream agent: 
> > 
> > The buffer has 1,2,3,4,5 
> > The new data comes 6,7,8 
> > 
> > The upstream agent sends the data in the order of 6,7,1,2,3,8,5, etc. 
> > 
> > Again, I know this would not happen if the down stream agent is never brought down.
However, I am just wondering if it is possible for this to happen is the downstream is brought
down and then up again ? 
> > 
> > thanks, 
> > -Gary 
> 
> 
> 
> -- 
> thanks 
> ashish 
> 
> Blog: http://www.ashishpaliwal.com/blog 
> My Photo Galleries: http://www.pbase.com/ashishpaliwal 
> 
> 


Mime
View raw message