lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Doug Cutting <>
Subject Re: ParallelMultiSearcher reimplementation
Date Mon, 13 Nov 2006 20:50:28 GMT
Chuck Williams wrote:
> I followed this same logic in ParallelWriter and got burned.  My first
> implementation (still the version submitted as a patch in jira) used
> dynamic threads to add the subdocuments to the parallel subindexes
> simultaneously.  This hit a problem with abnormal native heap OOM's in
> the jvm.  At first I thought it was simply a thread stack size / java
> heap size configuration issue, but adjusting these did not resolve the
> issue.  This was on linux.  ps -L showed large numbers of defunct
> threads.  jconsole showed enormous growing total-ever-allocated thread
> counts.  I switched to a thread pool and the issue went away with the
> same config settings.

Can you demonstrate the problem with a standalone program?

Way back in the 90's I implemented a system at Excite that spawned one 
or more Java threads per request, and it ran for days on end, handling 
20 or more requests per second.  The thread spawning overhead was 
insignificant.  That was JDK 1.2 on Solaris.  Have things gotten that 
much worse in the interim?  Today Hadoop's RPC allocates a thread per 
connection, and we see good performance.  So I certainly have 


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message