flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: starting a thread in the customized sink ?
Date Mon, 25 Aug 2014 05:20:16 GMT
You are probably ok. There are a bunch of sinks that already do this
(Asynchbase sink/Morphline Solr sink etc).


On Sun, Aug 24, 2014 at 10:17 PM, terreyshih <terreyshih@gmail.com> wrote:

> Thanks for answering my question.
>
> No, the transaction will not be shared by different threads.  The
> transaction is committed after I convert the events to some data structure.
>
> I would only have a Timer/Executor thread that fires off at specific
> intervals to send these converted data structures (from events) to
> customized storage/consumer for processing.
>
> Do you see an issue with this approach ?
>
> thanks,
> Terrey.
>
>
>
> On Aug 24, 2014, at 9:59 PM, Hari Shreedharan <hshreedharan@cloudera.com>
> wrote:
>
> If you create a large number of threads in a sink - that could affect the
> performance of the agent - simply because there are too many threads. Also,
> transactions are thread-local - so make sure each transaction is completely
> owned and run by the same thread.
>
>
> On Sun, Aug 24, 2014 at 7:28 PM, terreyshih <terreyshih@gmail.com> wrote:
>
>> Hi,
>>
>> If in my customized sink, I create some threads and schedule it to do
>> some queueing.  For example, I would like to drop the events off every 10
>> seconds.  If I start the thread or an executor in the process method of the
>> customized sink class, would have have some negative impact to the
>> threading model in Flume ?
>>
>> thanks,
>> -Terrey
>
>
>
>

Mime
View raw message