lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mike McCandless <>
Subject Re: Concurrency and multiple merge threads
Date Sun, 19 Feb 2012 11:09:50 GMT
Sounds like a nice machine!

It's frustrating that RAMFile even has any sync'd methods... Lucene is write once, so once
a RAMFile is written we don't need any sync to read it.  Maybe on creating a RAMInputStream
we could make a new ReadOnlyRAMFile, holding the same buffers without sync.

That said the ops inside the sync are tiny so it's strange if this really is the cause of
the contention... It could just be a profiling ghost and something else is the real bottleneck...


On Feb 18, 2012, at 9:21 PM, Benson Margulies <> wrote:

> Using Lucene 3.5.0, on a 32-core machine, I have coded something shaped like:
> make a writer on a RAMDirectory.
> start:
>  Create a near-real-time searcher from it.
>  farm work out to multiple threads, each of which performs a search
> and retrieves some docs.
>  When all are done, write some new docs.
> back to start.
> The returns of adding threads diminish faster than I would like.
> According to YourKit, a major contribution when I try 16 is conflict
> on the RAMFile monitor.
> The conflict shows five Lucene Merge Threads holding the monitor, plus
> my own threads. I'm not sure that I'm interpreting this correctly;
> perhaps there were five different occasions when a merge thread
> blocked my threads.
> In any case, I'm fairly stumped as to how my threads manage to
> materially block each other, since the synchronized methods used on
> the search side in RAMFile are pretty tiny.
> YourKit claims that the problem is in RAMFile.numBuffers, but I have
> not been able to catch this being called in a search.
> I did spot the following backtrace.
> In any case, I'd be grateful if anyone could tell me if this is a
> familiar story or one for which there's a solution.
>    RAMFile.getBuffer(int) line: 75    
>    RAMInputStream.switchCurrentBuffer(boolean) line: 107    
> line: 144    
>    SegmentNorms.bytes() line: 163    
>    SegmentNorms.bytes() line: 143    
>    ReadOnlySegmentReader(SegmentReader).norms(String) line: 599    
>    TermQuery$TermWeight.scorer(IndexReader, boolean, boolean) line: 107    
>    BooleanQuery$BooleanWeight.scorer(IndexReader, boolean, boolean) line: 298    
>, Filter, Collector) line: 577    
>, Filter, int, Sort, boolean) line: 517    
>, Filter, int, Sort) line: 487    
>, Filter, int, Sort) line: 400
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message