flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Roshan Naik <ros...@hortonworks.com>
Subject Re: Flume Event Ordering
Date Tue, 20 Sep 2016 17:20:43 GMT
You won¹t have *strict ordering* of events when you have more than 1 agent
or source or sink or channel in play due to the concurrency involved.
Ordering, if any, happens at the channel level.

Although Flume does not guarantee any ordering, in practice, there are a
few things worth noting for Memory & Spillable (and perhaps File channel

- Single source, channel and sink :  you should see ordered delivery
- Multiple sources, 1 sink on 1 channel : weaker ordering .. you should
see relative order of events from any specific source being preserved.
- Multiple sinks on same channel - here reordering can happen due 2
reasons...some sinks being faster, and failures in sink causing retries.

Not very sure about how things work out with Kafka channel.

If you want strict ordering you should be able to achieve it with single
MC/SC on 1 agent having 1 source and sinkŠ but it won¹t scale.

On 9/20/16, 3:45 AM, "Balazs Donat Bessenyei" <bessbd@cloudera.com> wrote:

>Hi Sreeram,
>I wanted to say almost the same as Denes, with the addition that you
>can add Timestamp Interceptors (
>https://flume.apache.org/FlumeUserGuide.html#timestamp-interceptor )
>to your config / setup.
>Basically, you would add extra headers to events that are monotonous,
>so you can reorder them when you are processing the ingested data.
>Best regards,
>On Tue, Sep 20, 2016 at 11:43 AM, Denes Arvay <denes@cloudera.com> wrote:
>> Hi Sreeram,
>> No, Flume doesn't guarantee the order of event delivery. Flume has at
>> once guarantee (when configured properly), so duplicated events and
>> out-of-order event delivery may occur.
>> As a start I'd really recommend Hari Shreedharan's book about Flume.
>> info regarding to the guarantees can be found in the "Flume¹s No Data
>> Guarantee, Channels, and Transactions" chapter:
>> If you have any other questions, feel free to ask, we are happy to help.
>> Cheers,
>> Denes
>> On Tue, Sep 20, 2016 at 11:23 AM Sreeram <sreeram.v@gmail.com> wrote:
>>> Hi,
>>>  I am a newbie to Flume and trying to understand more on the event
>>> ordering.
>>>  I am interested in knowing if Flume provides ordering guarantee for
>>> events from the source to destination.
>>>  Kindly let me know.
>>> Regards,
>>> Sreeram

View raw message