lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Benson Margulies <bimargul...@gmail.com>
Subject Concurrency and multiple merge threads
Date Sun, 19 Feb 2012 02:21:08 GMT
Using Lucene 3.5.0, on a 32-core machine, I have coded something shaped like:

make a writer on a RAMDirectory.

start:

  Create a near-real-time searcher from it.

  farm work out to multiple threads, each of which performs a search
and retrieves some docs.

  When all are done, write some new docs.

back to start.

The returns of adding threads diminish faster than I would like.
According to YourKit, a major contribution when I try 16 is conflict
on the RAMFile monitor.

The conflict shows five Lucene Merge Threads holding the monitor, plus
my own threads. I'm not sure that I'm interpreting this correctly;
perhaps there were five different occasions when a merge thread
blocked my threads.

In any case, I'm fairly stumped as to how my threads manage to
materially block each other, since the synchronized methods used on
the search side in RAMFile are pretty tiny.

YourKit claims that the problem is in RAMFile.numBuffers, but I have
not been able to catch this being called in a search.

I did spot the following backtrace.

In any case, I'd be grateful if anyone could tell me if this is a
familiar story or one for which there's a solution.


	RAMFile.getBuffer(int) line: 75	
	RAMInputStream.switchCurrentBuffer(boolean) line: 107	
	RAMInputStream.seek(long) line: 144	
	SegmentNorms.bytes() line: 163	
	SegmentNorms.bytes() line: 143	
	ReadOnlySegmentReader(SegmentReader).norms(String) line: 599	
	TermQuery$TermWeight.scorer(IndexReader, boolean, boolean) line: 107	
	BooleanQuery$BooleanWeight.scorer(IndexReader, boolean, boolean) line: 298	
	IndexSearcher.search(Weight, Filter, Collector) line: 577	
	IndexSearcher.search(Weight, Filter, int, Sort, boolean) line: 517	
	IndexSearcher.search(Weight, Filter, int, Sort) line: 487	
	IndexSearcher.search(Query, Filter, int, Sort) line: 400

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message