lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Yu ...@AI.SRI.COM>
Subject Re: thread safe shared IndexSearcher
Date Mon, 24 Sep 2007 22:47:32 GMT
Mark,

Great effort getting the original lucene index accessor package in this 
shape. I am sure this will benefit a lot of people using Lucene in a 
multithread env.
I have a quick question to ask you:
Do you have to use the core Lucene 2.3-dev in order to use the accessor?

I will take a look at your codes to see if I could help. I used a 
slightly modified version of the original package in my project but it 
breaks some of my tests. I hope your version works better.

Thanks a lot!

Jay


Mark Miller wrote:
> I have sat down and rewrote IndexAccessor from scratch. I copied in the 
> same reference counting logic, pruned some things, and tried to make the 
> whole package a bit simpler to use. I have a few things to do, but its 
> pretty solid already. The only major thing I'd still like to do is add 
> an option to warm searchers before putting them in the Searcher cache. 
> Id like to writer some more tests as well. Any help greatly appreciated 
> if your interested in using the thing.
> 
> 
> http://myhardshadow.com/indexaccessor/trunk/src/test/com/mhs/indexaccessor/SimpleSearchServer.java

> 
> 
> Here is a an example of a class that can be instantiated in one of 
> multiple threads and read /modify a single index without worrying about 
> what any
> of the other threads are doing to the index at any given time. This is a 
> very simple example of how to use the IndexAccessor and not necessarily an
> example of best practices. The main idea is that you get your Writer, 
> Searcher, or Reader, and then be sure to release it as soon as your done 
> with it
> in a finally block. For loading, you will want to load many docs with a 
> Writer (batch them) before releasing it, but remember that Readers will 
> not get a new view
> of the index until you release all of the Writers. So beware hogging a 
> Writer unless you thats what your intending.
> 
> JavaDoc:
> http://myhardshadow.com/indexaccessorapi/
> 
> Code:
> http://myhardshadow.com/indexaccessor/trunk/
> 
> Jar:
> http://myhardshadow.com/indexaccessorreleases/indexaccessor.jar
> 
> 
> Your synchronized block concerns:
> 
> The synchronized blocks that control accesss to the IndexAccessor do not 
> have a huge impact on performance. Keep in mind that all of the work is 
> not done in a synchonrized block, just the retrieval of the Searcher, 
> Writer, Reader. Even if the synchronization makes the method twice as 
> expensive, it is still overpowered by the cost of parsing queries and 
> searching the index. This applies with or without contention. I wrote a 
> simple test and included the output below. You might use the IBM Lock 
> Analyzer for Java to further analyze these costs. Trust me, this thing 
> is speedy. Its many times better than using IndexModifier.
> 
> Without Contention
> Just retrieve and release Searcher 100000 times
> ----
> avg time:6.3E-4 ms
> total time:63 ms
> 
> Parse query and search on 1 doc 100000 times
> ----
> avg time:0.03107 ms
> total time:3107 ms
> 
> 
> With Contention (40 other threads running 80000 searches)
> Just retrieve and release Searcher 100000 times
> ----
> avg time:0.04643 ms
> total time:4643 ms
> 
> Parse query and search on 1 doc 100000 times
> ----
> avg time:0.64337 ms
> total time:64337 ms
> 
> 
> - Mark
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message