Mailing-List: contact user-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flume.apache.org
Received-SPF: pass (nike.apache.org: domain of brock@cloudera.com designates
 209.85.216.182 as permitted sender)
MIME-Version: 1.0
In-Reply-To: 
 <CAKhLvp10hQWMjH4iJQdRCYMTTx=b=GaQP8Zh4_cQmmzs_khkTw@mail.gmail.com>
References: 
 <CAGHc4uhUUabZ=Y60wTiSZqXL+F=WHS8JR+H77ah_h0PhxrEy7w@mail.gmail.com>
 <CAKhLvp10hQWMjH4iJQdRCYMTTx=b=GaQP8Zh4_cQmmzs_khkTw@mail.gmail.com>
From: Brock Noland <brock@cloudera.com>
Date: Wed, 12 Feb 2014 10:19:05 -0600
Message-ID: 
 <CAFukC=5HW3ez_2Rs_9smi2USK63mGozL1rfOnvhhY=qbbS_o1Q@mail.gmail.com>
Subject: Re: Ordering of messages in flume-ng
To: "user@flume.apache.org" <user@flume.apache.org>
Content-Type: multipart/alternative; boundary=001a11343cdce78a1d04f237f084

--001a11343cdce78a1d04f237f084
Content-Type: text/plain; charset=ISO-8859-1

Hi,

In the cast of no failures with a single source, single channel and single
sink you will see ordering. However, I believe when there is a failure file
channel will change ordering on rollback.

If strict ordering is required it's advisable to assign sequence numbers
upstream and then re-order the data with either a MR job or Impala query
once they land in MapReduce.

Brock


On Wed, Feb 12, 2014 at 12:02 AM, Christopher Shannon <cshannon108@gmail.com
> wrote:

> Interesting question.
>
> I can't answer it, but I would like to know what strategies others have
> pursued if they have had a need to order their data after it gets to the
> end of the Flume pipeline.
>
> - C.
>
>
> On Tue, Feb 11, 2014 at 11:52 PM, Chris Schneider <
> chris@christopher-schneider.com> wrote:
>
>> I've seen a fair number of resources on the web that describe the loose
>> ordering guarantees that flume offers for messages in the face of
>> degradation or failures.  But I can't tell what applies to flume-og, and
>> flume-ng.  Hopefully somebody can help clear up the situation.
>>
>> In the case of a single agent topology, (source -> FileSystem Channel ->
>> sink), can messages become out of order?  What situations cause that?
>>
>> In a multi agent topology, does that answer change?
>>
>> (Agent 1   Source -> FilesystemChannel -> Avro To Collector)
>> (Agent 2   Source -> FilesystemChannel -> Avro To Collector)
>> (Collector Avro from agents -> FilesystemChannel -> final Sink)
>>
>> And perhaps in an even more complicated setup, with multiple collectors,
>> does that answer change further?
>>
>>
>>
>>
>


-- 
Apache MRUnit - Unit testing MapReduce - http://mrunit.apache.org

--001a11343cdce78a1d04f237f084
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi,<div><br></div><div>In the cast of no failures with a s=
ingle source, single channel and single sink you will see ordering. However=
, I believe when there is a failure file channel will change ordering on ro=
llback.</div>

<div><br></div><div>If strict ordering is required it&#39;s advisable to as=
sign sequence numbers upstream and then re-order the data with either a MR =
job or Impala query once they land in MapReduce.<br><div><br></div><div>

Brock</div></div></div><div class=3D"gmail_extra"><br><br><div class=3D"gma=
il_quote">On Wed, Feb 12, 2014 at 12:02 AM, Christopher Shannon <span dir=
=3D"ltr">&lt;<a href=3D"mailto:cshannon108@gmail.com" target=3D"_blank">csh=
annon108@gmail.com</a>&gt;</span> wrote:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">Interesting question.<div><=
br></div><div>I can&#39;t answer it, but I would like to know what strategi=
es others have pursued if they have had a need to order their data after it=
 gets to the end of the Flume pipeline.</div>

<span class=3D"HOEnZb"><font color=3D"#888888">
<div><br></div><div>- C.</div></font></span></div><div class=3D"HOEnZb"><di=
v class=3D"h5"><div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote=
">On Tue, Feb 11, 2014 at 11:52 PM, Chris Schneider <span dir=3D"ltr">&lt;<=
a href=3D"mailto:chris@christopher-schneider.com" target=3D"_blank">chris@c=
hristopher-schneider.com</a>&gt;</span> wrote:<br>


<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">I&#39;ve seen a fair number=
 of resources on the web that describe the loose ordering guarantees that f=
lume offers for messages in the face of degradation or failures. =A0But I c=
an&#39;t tell what applies to flume-og, and flume-ng. =A0Hopefully somebody=
 can help clear up the situation.<div>


<br></div><div>In the case of a single agent topology, (source -&gt; FileSy=
stem Channel -&gt; sink), can messages become out of order? =A0What situati=
ons cause that?</div><div><br></div><div>In a multi agent topology, does th=
at answer change?</div>


<div><br></div><div>(Agent 1 =A0 Source -&gt; FilesystemChannel -&gt; Avro =
To Collector)</div><div>(Agent 2 =A0 Source -&gt; FilesystemChannel -&gt; A=
vro To Collector)</div><div>(Collector Avro from agents -&gt; FilesystemCha=
nnel -&gt; final Sink)</div>


<div><br></div><div>And perhaps in an even more complicated setup, with mul=
tiple collectors, does that answer change further?</div><div><br></div><div=
><br></div><div><br></div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear=3D"all"><div><br></div>-- <br>=
<div dir=3D"ltr">Apache MRUnit - Unit testing MapReduce - <a href=3D"http:/=
/mrunit.apache.org" target=3D"_blank">http://mrunit.apache.org</a></div>
</div>

--001a11343cdce78a1d04f237f084--