flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From terreyshih <terreys...@gmail.com>
Subject Re: starting a thread in the customized sink ?
Date Mon, 25 Aug 2014 08:50:35 GMT
thanks
On Aug 24, 2014, at 10:20 PM, Hari Shreedharan <hshreedharan@cloudera.com> wrote:

> You are probably ok. There are a bunch of sinks that already do this (Asynchbase sink/Morphline
Solr sink etc).
> 
> 
> On Sun, Aug 24, 2014 at 10:17 PM, terreyshih <terreyshih@gmail.com> wrote:
> Thanks for answering my question.
> 
> No, the transaction will not be shared by different threads.  The transaction is committed
after I convert the events to some data structure.
> 
> I would only have a Timer/Executor thread that fires off at specific intervals to send
these converted data structures (from events) to customized storage/consumer for processing.
> 
> Do you see an issue with this approach ?
> 
> thanks,
> Terrey.
> 
> 
> 
> On Aug 24, 2014, at 9:59 PM, Hari Shreedharan <hshreedharan@cloudera.com> wrote:
> 
>> If you create a large number of threads in a sink - that could affect the performance
of the agent - simply because there are too many threads. Also, transactions are thread-local
- so make sure each transaction is completely owned and run by the same thread.
>> 
>> 
>> On Sun, Aug 24, 2014 at 7:28 PM, terreyshih <terreyshih@gmail.com> wrote:
>> Hi,
>> 
>> If in my customized sink, I create some threads and schedule it to do some queueing.
 For example, I would like to drop the events off every 10 seconds.  If I start the thread
or an executor in the process method of the customized sink class, would have have some negative
impact to the threading model in Flume ?
>> 
>> thanks,
>> -Terrey
>> 
> 
> 


Mime
View raw message