accumulo-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Keith Massey <keith.mas...@digitalreasoning.com>
Subject Re: ScannerIterator thread use
Date Tue, 01 Nov 2011 22:12:06 GMT
On 11/1/11 3:31 PM, Keith Turner wrote:
> On Tue, Nov 1, 2011 at 4:00 PM, Keith Massey
> <keith.massey@digitalreasoning.com>  wrote:
>> We're querying accumulo through a web application. After it had been hit
>> with one of our test scripts for a few minutes with the debugger attached I
>> noticed that there were hundreds and hundreds of threads being garbage
>> collected. Eventually it crashes my IDE and the server becomes unresponsive.
>> The server recovers eventually. After looking through the code a little bit,
>> it appears that these threads are coming from
>> org.apache.accumulo.core.client.impl.ScannerIterator.initiateReadAhead(). We
>> actually get many threads per iterator. Is there any reason that it can't
>> use a thread pool instead of creating a new thread for every call to that
>> method?
>> Thanks.
>>
>> Keith
>>
> The reason I did not use a thread pool is because the scanner does not
> have a close method.  I suppose we could use a thread pool where the
> threads timeout when not used.  This could still lead to a lot of
> threads depending on the timeout and how many scanner iterators are
> created.
>
> The BatchScanner and BatchWriter interfaces use thread pools and have
> close methods.
>
> Do you think this issue needs a ticket?

I'm not incredibly familiar with this code, but it could be a static 
thread pool right? And just let all ScannerIterators share some 
configurable thread pool? The thread would just be returned to the pool 
when the Reader completed.

Mime
View raw message