lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: thread safe shared IndexSearcher
Date Mon, 24 Sep 2007 23:00:46 GMT
One final note....if you are using the IndexAccessor and you are only 
accessing the index from one JVM, you can use the NoLockFactory and save 
some sync cost there.

Jay Yu wrote:
> Mark,
>
> Great effort getting the original lucene index accessor package in 
> this shape. I am sure this will benefit a lot of people using Lucene 
> in a multithread env.
> I have a quick question to ask you:
> Do you have to use the core Lucene 2.3-dev in order to use the accessor?
>
> I will take a look at your codes to see if I could help. I used a 
> slightly modified version of the original package in my project but it 
> breaks some of my tests. I hope your version works better.
>
> Thanks a lot!
>
> Jay
>
>
> Mark Miller wrote:
>> I have sat down and rewrote IndexAccessor from scratch. I copied in 
>> the same reference counting logic, pruned some things, and tried to 
>> make the whole package a bit simpler to use. I have a few things to 
>> do, but its pretty solid already. The only major thing I'd still like 
>> to do is add an option to warm searchers before putting them in the 
>> Searcher cache. Id like to writer some more tests as well. Any help 
>> greatly appreciated if your interested in using the thing.
>>
>>
>> http://myhardshadow.com/indexaccessor/trunk/src/test/com/mhs/indexaccessor/SimpleSearchServer.java

>>
>>
>> Here is a an example of a class that can be instantiated in one of 
>> multiple threads and read /modify a single index without worrying 
>> about what any
>> of the other threads are doing to the index at any given time. This 
>> is a very simple example of how to use the IndexAccessor and not 
>> necessarily an
>> example of best practices. The main idea is that you get your Writer, 
>> Searcher, or Reader, and then be sure to release it as soon as your 
>> done with it
>> in a finally block. For loading, you will want to load many docs with 
>> a Writer (batch them) before releasing it, but remember that Readers 
>> will not get a new view
>> of the index until you release all of the Writers. So beware hogging 
>> a Writer unless you thats what your intending.
>>
>> JavaDoc:
>> http://myhardshadow.com/indexaccessorapi/
>>
>> Code:
>> http://myhardshadow.com/indexaccessor/trunk/
>>
>> Jar:
>> http://myhardshadow.com/indexaccessorreleases/indexaccessor.jar
>>
>>
>> Your synchronized block concerns:
>>
>> The synchronized blocks that control accesss to the IndexAccessor do 
>> not have a huge impact on performance. Keep in mind that all of the 
>> work is not done in a synchonrized block, just the retrieval of the 
>> Searcher, Writer, Reader. Even if the synchronization makes the 
>> method twice as expensive, it is still overpowered by the cost of 
>> parsing queries and searching the index. This applies with or without 
>> contention. I wrote a simple test and included the output below. You 
>> might use the IBM Lock Analyzer for Java to further analyze these 
>> costs. Trust me, this thing is speedy. Its many times better than 
>> using IndexModifier.
>>
>> Without Contention
>> Just retrieve and release Searcher 100000 times
>> ----
>> avg time:6.3E-4 ms
>> total time:63 ms
>>
>> Parse query and search on 1 doc 100000 times
>> ----
>> avg time:0.03107 ms
>> total time:3107 ms
>>
>>
>> With Contention (40 other threads running 80000 searches)
>> Just retrieve and release Searcher 100000 times
>> ----
>> avg time:0.04643 ms
>> total time:4643 ms
>>
>> Parse query and search on 1 doc 100000 times
>> ----
>> avg time:0.64337 ms
>> total time:64337 ms
>>
>>
>> - Mark
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message