flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juhani Connolly <juhani_conno...@cyberagent.co.jp>
Subject Re: Flume as event/alerts collector
Date Mon, 15 Oct 2012 10:19:34 GMT
Hi Karim

On 10/13/2012 08:03 AM, Zamani, Karim wrote:
>
> Hi,
>
> Is Flume (or Flume-NG) suitable for collecting events such as alters 
> generated by various nodes in a distributed architecture to a central 
> location?
>
> I read in the wiki FAQ that Flume was not designed to be a message 
> passing or eventing framework due to issues like message duplication, 
> ordering, etc.
>
Flume NG only guarrantees delivery(so long as you use durable channels 
on every hop). Avoiding duplication is not a hard objective, and it is 
possible. Same applies for message ordering.

That said, not all component will result in message order getting mixed 
or duplicates. If you deliver messages in order to flume and sink them 
without batching, along a single path, they should be in order and 
without duplicates. However this puts some severe restrictions on 
multiple delivery paths for redundancy and batching(not batching the 
sink at all is going to hurt performance), so I couldn't claim flume is 
ideal for a system where ordering/avoiding duplication is  necessary.

> Does Flume-NG have any enhancements to support message passing better?
>
> Thanks!
>
> Karim
>


Mime
View raw message