Mailing-List: contact user-help@flume.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flume.apache.org
Received-SPF: pass (athena.apache.org: domain of hshreedharan@cloudera.com
 designates 209.85.213.169 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <7618E46C-5659-4394-99B8-DC97F86DDE19@gmail.com>
References: <5A16A7F7-9F49-413F-B1E0-F438B1B495A5@gmail.com>
 <CAHbPYVaDFJy3wWaOG9r+9vGXziirjF9y7ZCNG=BGXTRaY0L3+g@mail.gmail.com>
 <7618E46C-5659-4394-99B8-DC97F86DDE19@gmail.com>
From: Hari Shreedharan <hshreedharan@cloudera.com>
Date: Sun, 24 Aug 2014 22:20:16 -0700
Message-ID: 
 <CAHbPYVZXqaW-P9WAC_T3+ZyqqYkBSdu1wTj+4FRM5-6r56xfMQ@mail.gmail.com>
Subject: Re: starting a thread in the customized sink ?
To: "user@flume.apache.org" <user@flume.apache.org>
Content-Type: multipart/alternative; boundary=089e0111d1bed7b3e305016d588a

--089e0111d1bed7b3e305016d588a
Content-Type: text/plain; charset=UTF-8

You are probably ok. There are a bunch of sinks that already do this
(Asynchbase sink/Morphline Solr sink etc).


On Sun, Aug 24, 2014 at 10:17 PM, terreyshih <terreyshih@gmail.com> wrote:

> Thanks for answering my question.
>
> No, the transaction will not be shared by different threads.  The
> transaction is committed after I convert the events to some data structure.
>
> I would only have a Timer/Executor thread that fires off at specific
> intervals to send these converted data structures (from events) to
> customized storage/consumer for processing.
>
> Do you see an issue with this approach ?
>
> thanks,
> Terrey.
>
>
>
> On Aug 24, 2014, at 9:59 PM, Hari Shreedharan <hshreedharan@cloudera.com>
> wrote:
>
> If you create a large number of threads in a sink - that could affect the
> performance of the agent - simply because there are too many threads. Also,
> transactions are thread-local - so make sure each transaction is completely
> owned and run by the same thread.
>
>
> On Sun, Aug 24, 2014 at 7:28 PM, terreyshih <terreyshih@gmail.com> wrote:
>
>> Hi,
>>
>> If in my customized sink, I create some threads and schedule it to do
>> some queueing.  For example, I would like to drop the events off every 10
>> seconds.  If I start the thread or an executor in the process method of the
>> customized sink class, would have have some negative impact to the
>> threading model in Flume ?
>>
>> thanks,
>> -Terrey
>
>
>
>

--089e0111d1bed7b3e305016d588a
Content-Type: text/html; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">You are probably ok. There are a bunch of sinks that alrea=
dy do this (Asynchbase sink/Morphline Solr sink etc).</div><div class=3D"gm=
ail_extra"><br><br><div class=3D"gmail_quote">On Sun, Aug 24, 2014 at 10:17=
 PM, terreyshih <span dir=3D"ltr">&lt;<a href=3D"mailto:terreyshih@gmail.co=
m" target=3D"_blank">terreyshih@gmail.com</a>&gt;</span> wrote:<br>

<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex"><div style=3D"word-wrap:break-word">Thanks f=
or answering my question.<div><br></div><div>No, the transaction will not b=
e shared by different threads. =C2=A0The transaction is committed after I c=
onvert the events to some data structure.</div>

<div><br></div><div>I would only have a Timer/Executor thread that fires of=
f at specific intervals to send these converted data structures (from event=
s) to customized storage/consumer for processing.</div><div><br></div>
<div>
Do you see an issue with this approach ?</div><div><br></div><div>thanks,</=
div><div>Terrey.</div><div><div class=3D"h5"><div><br></div><div><br></div>=
<div><br><div><div>On Aug 24, 2014, at 9:59 PM, Hari Shreedharan &lt;<a hre=
f=3D"mailto:hshreedharan@cloudera.com" target=3D"_blank">hshreedharan@cloud=
era.com</a>&gt; wrote:</div>

<br><blockquote type=3D"cite"><div dir=3D"ltr">If you create a large number=
 of threads in a sink - that could affect the performance of the agent - si=
mply because there are too many threads. Also, transactions are thread-loca=
l - so make sure each transaction is completely owned and run by the same t=
hread.</div>


<div class=3D"gmail_extra"><br><br><div class=3D"gmail_quote">On Sun, Aug 2=
4, 2014 at 7:28 PM, terreyshih <span dir=3D"ltr">&lt;<a href=3D"mailto:terr=
eyshih@gmail.com" target=3D"_blank">terreyshih@gmail.com</a>&gt;</span> wro=
te:<br>


<blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1p=
x #ccc solid;padding-left:1ex">Hi,<br>
<br>
If in my customized sink, I create some threads and schedule it to do some =
queueing.=C2=A0 For example, I would like to drop the events off every 10 s=
econds.=C2=A0 If I start the thread or an executor in the process method of=
 the customized sink class, would have have some negative impact to the thr=
eading model in Flume ?<br>


<br>
thanks,<br>
-Terrey</blockquote></div><br></div>
</blockquote></div><br></div></div></div></div></blockquote></div><br></div=
>

--089e0111d1bed7b3e305016d588a--