flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Juhani Connolly <juhani_conno...@cyberagent.co.jp>
Subject Re: Design problem while monitoring Flume
Date Thu, 29 Aug 2013 02:09:44 GMT
When I said process after writing I meant outside of flume with some 
kind of batch process. To be able to do this you'd have to of course use 
a serializer that writes out the headers as well as the event body.

If you want to do the processing in realtime, you would probably have to 
take the approach of replicating events and sending to a custom sink.

On 08/28/2013 09:59 PM, Anat Rozenzon wrote:
> Thank you for the quick answer.
> How can I process events after they have been written? is there any 
> post-write interceptor I can code?
> On Wed, Aug 28, 2013 at 11:45 AM, Juhani Connolly 
> <juhani_connolly@cyberagent.co.jp 
> <mailto:juhani_connolly@cyberagent.co.jp>> wrote:
>     The most common cause of resending events from the source would be
>     failure to write to the channel. Most of the time this would be
>     because the channel is full.
>     An approach to collecting statistics will vary on what exactly you
>     want to do, but perhaps you could write metadata to headers in the
>     interceptor and than batch process the serialized headers after
>     events have actually been written. Or if you need to be realtime
>     you can replicate events to an additional path which leads to a
>     custom sink that collects statistics. So long as the sink doesn't
>     "bounce" events(rollback transactions) it shouldn't get any events
>     resent.
>     One thing to keep in mind though is that flume in general only
>     guarantees delivery, it doesn't guarantee that stuff will only be
>     delivered once(though many components do only deliver once)
>     On 08/28/2013 04:09 PM, Anat Rozenzon wrote:
>         Hi,
>         I want to get some statistics out of Flume (For example, how
>         many records were collected, How many files etc.).
>         I've written my own interceptor that updates an MBean whenever
>         records arrive.
>         I've also written a MonitorServices that collects the data
>         from the MBean every X minutes and send it to a database.
>         My problem is that sometimes events are resent again from the
>         source, I saw that while debugging.
>         Not sure why... maybe because of a timeout while sending to
>         the sink?
>         Anyway, if this happens in production it will corrupt my
>         statistics.
>         Is there any way I can know that an event have failed reaching
>         the sink eventhough it passed the interceptor?
>         Is there a better place to collect such statistics than an
>         interceptor?
>         Thanks
>         Anat

View raw message