lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Yu ...@AI.SRI.COM>
Subject Re: thread safe shared IndexSearcher
Date Mon, 24 Sep 2007 23:10:00 GMT
Thanks for the tip.
One small improvement on the IndexAccessorFactory might be to allow user 
to specify the Analyzer instead of using a default KeywordAnalyzer, 
which of course will make your static init of the cached accessors 
difficult unless you add more interfaces to the accessor to allow reset 
analyzer/Dir as in my own version.




Jay

Mark Miller wrote:
> One final note....if you are using the IndexAccessor and you are only 
> accessing the index from one JVM, you can use the NoLockFactory and save 
> some sync cost there.
> 
> Jay Yu wrote:
>> Mark,
>>
>> Great effort getting the original lucene index accessor package in 
>> this shape. I am sure this will benefit a lot of people using Lucene 
>> in a multithread env.
>> I have a quick question to ask you:
>> Do you have to use the core Lucene 2.3-dev in order to use the accessor?
>>
>> I will take a look at your codes to see if I could help. I used a 
>> slightly modified version of the original package in my project but it 
>> breaks some of my tests. I hope your version works better.
>>
>> Thanks a lot!
>>
>> Jay
>>
>>
>> Mark Miller wrote:
>>> I have sat down and rewrote IndexAccessor from scratch. I copied in 
>>> the same reference counting logic, pruned some things, and tried to 
>>> make the whole package a bit simpler to use. I have a few things to 
>>> do, but its pretty solid already. The only major thing I'd still like 
>>> to do is add an option to warm searchers before putting them in the 
>>> Searcher cache. Id like to writer some more tests as well. Any help 
>>> greatly appreciated if your interested in using the thing.
>>>
>>>
>>> http://myhardshadow.com/indexaccessor/trunk/src/test/com/mhs/indexaccessor/SimpleSearchServer.java

>>>
>>>
>>> Here is a an example of a class that can be instantiated in one of 
>>> multiple threads and read /modify a single index without worrying 
>>> about what any
>>> of the other threads are doing to the index at any given time. This 
>>> is a very simple example of how to use the IndexAccessor and not 
>>> necessarily an
>>> example of best practices. The main idea is that you get your Writer, 
>>> Searcher, or Reader, and then be sure to release it as soon as your 
>>> done with it
>>> in a finally block. For loading, you will want to load many docs with 
>>> a Writer (batch them) before releasing it, but remember that Readers 
>>> will not get a new view
>>> of the index until you release all of the Writers. So beware hogging 
>>> a Writer unless you thats what your intending.
>>>
>>> JavaDoc:
>>> http://myhardshadow.com/indexaccessorapi/
>>>
>>> Code:
>>> http://myhardshadow.com/indexaccessor/trunk/
>>>
>>> Jar:
>>> http://myhardshadow.com/indexaccessorreleases/indexaccessor.jar
>>>
>>>
>>> Your synchronized block concerns:
>>>
>>> The synchronized blocks that control accesss to the IndexAccessor do 
>>> not have a huge impact on performance. Keep in mind that all of the 
>>> work is not done in a synchonrized block, just the retrieval of the 
>>> Searcher, Writer, Reader. Even if the synchronization makes the 
>>> method twice as expensive, it is still overpowered by the cost of 
>>> parsing queries and searching the index. This applies with or without 
>>> contention. I wrote a simple test and included the output below. You 
>>> might use the IBM Lock Analyzer for Java to further analyze these 
>>> costs. Trust me, this thing is speedy. Its many times better than 
>>> using IndexModifier.
>>>
>>> Without Contention
>>> Just retrieve and release Searcher 100000 times
>>> ----
>>> avg time:6.3E-4 ms
>>> total time:63 ms
>>>
>>> Parse query and search on 1 doc 100000 times
>>> ----
>>> avg time:0.03107 ms
>>> total time:3107 ms
>>>
>>>
>>> With Contention (40 other threads running 80000 searches)
>>> Just retrieve and release Searcher 100000 times
>>> ----
>>> avg time:0.04643 ms
>>> total time:4643 ms
>>>
>>> Parse query and search on 1 doc 100000 times
>>> ----
>>> avg time:0.64337 ms
>>> total time:64337 ms
>>>
>>>
>>> - Mark
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message