Mailing-List: contact user-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flume.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: 
 <CAJLbxRY+niaMPESQKAdkF2-vuSppzaFV5u3FCnLCCd9UehK-qA@mail.gmail.com>
References: 
 <CAK5Fx1hotSeoXsLaJHtQ8sSYcQpQqSJOov9Rsi+KbU5+awOEcQ@mail.gmail.com>
	<CAJLbxRY+niaMPESQKAdkF2-vuSppzaFV5u3FCnLCCd9UehK-qA@mail.gmail.com>
Date: Fri, 1 Feb 2013 10:44:22 -0500
Message-ID: 
 <CAK5Fx1iVxtaARCugko6W2dFuD1zYjmOR_hJvZKXahhnC9RS+Gw@mail.gmail.com>
Subject: Re: SpoolDir marks item as completed, when sink fails
From: Tzur Turkenitz <tzurt@vision.bi>
To: user@flume.apache.org
Content-Type: multipart/alternative; boundary=f46d0401fa5d01457204d4ab9e7c

--f46d0401fa5d01457204d4ab9e7c
Content-Type: text/plain; charset=ISO-8859-1

Mike, so when the data is committed to the channel, and the channel is of
type "File" then when the agent will be restarted the data will continue to
flow onto the sink?
And if only 20% of the data passed onto the sink before it crashed then a
"Replay" will be done to resend the whole data?

Just trying to grasp the basics....


On Fri, Feb 1, 2013 at 4:56 AM, Mike Percy <mpercy@apache.org> wrote:

> Tzur, that is expected, because the data is committed by the source onto
> the channel. Sources and sinks are decoupled, they only interact via the
> channel, which buffers the data and serves to mitigate impedance mismatches.
>
>
>
> On Thu, Jan 31, 2013 at 2:35 PM, Tzur Turkenitz <tzurt@vision.bi> wrote:
>
>> Hello all,
>>
>> I am running HDP 1.2 and Flume 1.3. I have a flume setup which includes a
>> (1) -  Load Balancer that uses SpoolDir adapter and sends events to Avro
>> sinks
>> (2) - Agents which consume the data using an avro source and writing to
>> hdfs.
>>
>> During testing I noticed that there's a dissonance between the Load
>> Balancer and the Consumers...
>> When a Load Balancer process a file it marks it as COMPLETED, even if the
>> consumer has crashed while writing to HDFS.
>>
>> A preferred behavior would be the Load Balancer to wait until the
>> consumer commits its transaction and reports it as successful before the
>> file is marked as COMPLETED. This does not allow me to verify which files
>> has been loaded successfully if an agent has crashed and recovery is in
>> process.
>>
>> Have I miss-configured my Agents or this is actually the desired behavior?
>>
>>
>> Kind Regards,
>> Tzur
>>
>
>


-- 
Regards,
Tzur Turkenitz
Vision.BI
http://www.vision.bi/

"*Facts are stubborn things, but statistics are more pliable*"
-Mark Twain

--f46d0401fa5d01457204d4ab9e7c
Content-Type: text/html; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Mike, so when the data is committed to the channel, and th=
e channel is of type &quot;File&quot; then when the agent will be restarted=
 the data will continue to flow onto the sink?<div>And if only 20% of the d=
ata passed onto the sink before it crashed then a &quot;Replay&quot; will b=
e done to resend the whole data?</div>
<div><br></div><div>Just trying to grasp the basics....<br><div><br></div><=
div><br></div></div></div><div class=3D"gmail_extra"><br><br><div class=3D"=
gmail_quote">On Fri, Feb 1, 2013 at 4:56 AM, Mike Percy <span dir=3D"ltr">&=
lt;<a href=3D"mailto:mpercy@apache.org" target=3D"_blank">mpercy@apache.org=
</a>&gt;</span> wrote:<br>
<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">Tzur, that is expected, bec=
ause the data is committed by the source onto the channel. Sources and sink=
s are decoupled, they only interact via the channel, which buffers the data=
 and serves to mitigate impedance mismatches.<div>


<br></div></div><div class=3D"HOEnZb"><div class=3D"h5"><div class=3D"gmail=
_extra"><br><br><div class=3D"gmail_quote">On Thu, Jan 31, 2013 at 2:35 PM,=
 Tzur Turkenitz <span dir=3D"ltr">&lt;<a href=3D"mailto:tzurt@vision.bi" ta=
rget=3D"_blank">tzurt@vision.bi</a>&gt;</span> wrote:<br>


<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div dir=3D"ltr">Hello all,<div><br></div><d=
iv>I am running HDP 1.2 and Flume 1.3. I have a flume setup which includes =
a</div>


<div>(1) - =A0Load Balancer that uses SpoolDir adapter and sends events to =
Avro sinks</div><div>(2) - Agents which consume the data using an avro sour=
ce and writing to hdfs.</div>
<div><br></div><div>During testing I noticed that there&#39;s a dissonance =
between the Load Balancer and the Consumers...</div><div>When a Load Balanc=
er process a file it marks it as COMPLETED, even if the consumer has crashe=
d while writing to HDFS.</div>


<div><br></div><div>A=A0preferred=A0behavior would be the Load Balancer to =
wait until the consumer commits its transaction and reports it as=A0success=
ful=A0before the file is marked as COMPLETED. This does not allow me to ver=
ify which files has been loaded successfully if an agent has crashed and re=
covery is in process.</div>


<div><br></div><div>Have I=A0miss-configured=A0my Agents or this is actuall=
y the desired behavior?</div><div><br></div><div><br></div><div><div>Kind R=
egards,</div><div>Tzur</div>
</div></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br><br clear=3D"all"><div><br></div>-- <br>=
<div dir=3D"ltr"><span style=3D"font-family:&#39;Times New Roman&#39;,Times=
,serif;line-height:19px"><div>Regards,</div><div>Tzur Turkenitz</div><div>V=
ision.BI=A0</div>
<div style=3D"font-size:17px"><a href=3D"http://www.vision.bi/" target=3D"_=
blank"><font color=3D"#3333FF">http://www.vision.bi/</font></a></div><div s=
tyle=3D"font-size:17px;color:rgb(255,102,102)"><font color=3D"#FF6666"><br>=
</font></div>
<div style=3D"font-size:17px;color:rgb(255,102,102)"><span style=3D"color:r=
gb(0,0,0);font-size:small"><font color=3D"#FF6666">&quot;</font><i><font co=
lor=3D"#330099">Facts are stubborn things, but statistics are more pliable<=
/font></i><font color=3D"#FF6666">&quot;</font></span></div>
</span><div><span style=3D"font-family:&#39;Times New Roman&#39;,Times,seri=
f;color:rgb(69,69,69);line-height:19px">-Mark Twain</span></div></div>
</div>

--f46d0401fa5d01457204d4ab9e7c--