flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From terreyshih <terreys...@gmail.com>
Subject Re: starting a thread in the customized sink ?
Date Mon, 25 Aug 2014 05:17:28 GMT
Thanks for answering my question.

No, the transaction will not be shared by different threads.  The transaction is committed
after I convert the events to some data structure.

I would only have a Timer/Executor thread that fires off at specific intervals to send these
converted data structures (from events) to customized storage/consumer for processing.

Do you see an issue with this approach ?


On Aug 24, 2014, at 9:59 PM, Hari Shreedharan <hshreedharan@cloudera.com> wrote:

> If you create a large number of threads in a sink - that could affect the performance
of the agent - simply because there are too many threads. Also, transactions are thread-local
- so make sure each transaction is completely owned and run by the same thread.
> On Sun, Aug 24, 2014 at 7:28 PM, terreyshih <terreyshih@gmail.com> wrote:
> Hi,
> If in my customized sink, I create some threads and schedule it to do some queueing.
 For example, I would like to drop the events off every 10 seconds.  If I start the thread
or an executor in the process method of the customized sink class, would have have some negative
impact to the threading model in Flume ?
> thanks,
> -Terrey

View raw message