flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From terreyshih <terreys...@gmail.com>
Subject flume avro event overflow ?
Date Sun, 19 Oct 2014 06:42:16 GMT
Hi,

If I have a upstream agent duplicating the event to two channels and subsequently to two downstream
sinks.  If one of the downstream sinks for some reason can not process the events fast enough
and the event overflows back over the buffer limit of the upstream agent, would it cause any
issues for the other channel since the events are duplicated ?

In general, what happens if the downstream agent is not able to handle the speed of the upstream
agent sending the events ?  I have seen exceptions that says unable to send events or memory
channel commit issues.  

However, the flume agent does not seem to handle that gracefully and the whole agent just
stops processing the events for all channels, even for the channels that can keep up with
events.

What if I separate the two channels into two different agents, would that solve the problem.

Can upstream flume agent not just drop the events when one of the its sinks or downstream
agents cannot catch up  ?

thx
Mime
View raw message