lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chuck Williams <ch...@manawiz.com>
Subject Re: ParallelMultiSearcher reimplementation
Date Sun, 05 Nov 2006 18:53:37 GMT
Doug Cutting wrote on 11/03/2006 12:18 PM:
> Chuck Williams wrote:
>> Why would a thread pool be more controversial?  Dynamically creating and
>> garbaging threads has many downsides.
>
> The JVM already pools native threads, so mostly what's saved by thread
> pools is the allocation & initialization of new Thread instances. 
> There are also downsides to thread pools.  They alter ThreadLocal
> semantics and generally add complexity that may not be warranted.
>
> Like most optimizations, use of thread pools should be motivated by
> benchmarks.

I followed this same logic in ParallelWriter and got burned.  My first
implementation (still the version submitted as a patch in jira) used
dynamic threads to add the subdocuments to the parallel subindexes
simultaneously.  This hit a problem with abnormal native heap OOM's in
the jvm.  At first I thought it was simply a thread stack size / java
heap size configuration issue, but adjusting these did not resolve the
issue.  This was on linux.  ps -L showed large numbers of defunct
threads.  jconsole showed enormous growing total-ever-allocated thread
counts.  I switched to a thread pool and the issue went away with the
same config settings.

So, I'm not convinced the jvm does such a good job a pooling OS native
threads.

Re. ThreadLocals, I agree the semantics are different, but arguably they
are most useful with thread pools.  With dynamic threads, you get a
reallocation every time, while with thread pools you avoid constant
reallocations.

Chuck


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message