lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Robert Engels" <reng...@ix.netcom.com>
Subject RE: FieldsReader synchronized access vs. ThreadLocal ?
Date Wed, 17 May 2006 18:45:01 GMT
If you run a concurrent searches over a million documents, returning only
the matching 500 of each, you will still encounter significant blocking if
there are not many matches between the documents sets - the blocking is even
worse due to the isDeleted() synchronization, as query performance will be
severely impacted the longer each document() call takes.

Using the ThreadLocal demonstrates excellent performance improvements for
multi-threaded queries. Not sure why it should not be used, especially since
it also fixed the synchronization problem between isDeleted() and
document().


-----Original Message-----
From: Yonik Seeley [mailto:yseeley@gmail.com]
Sent: Wednesday, May 17, 2006 1:36 PM
To: java-dev@lucene.apache.org
Subject: Re: FieldsReader synchronized access vs. ThreadLocal ?


On 5/17/06, Robert Engels <rengels@ix.netcom.com> wrote:
> Since reading a document is a relatively expensive operation

Expensive relative to searching operations that cover 1 million
documents (i.e. you don't want to call doc() a million times)

Solr has a document cache, and I've found that it doesn't help max
throughput that much (I just needed more concurrent searchers to reach
the max), but it does help latency of individual requests.

>[...] Since the
> isDeleted() method uses the same synchronized lock as document(), all
query
> scorers that filter out deleted documents will also be impacted, as they
> will block while the document is being read.

Interesting observation... that lock need not be shared.
Using a different lock would mean aquiring two locks per doc() call,
but it may be worth it to unblock the scorers waiting on isDeleted()

It might also be worth it to make a ReadOnlyIndexReader that didn't
have to deal with issues of synchronizing access to the deleted docs
vector.

If only Sun had done their APIs correctly, we could easily make
non-synchronizing implementations of IndexInput and friends w/o
resorting to ThreadLocals... Ah well.

-Yonik
http://incubator.apache.org/solr Solr, the open-source Lucene search server

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message