lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Mark Miller <markrmil...@gmail.com>
Subject Re: thread safe shared IndexSearcher
Date Mon, 24 Sep 2007 22:52:35 GMT
2.3-dev is not required. I'd have to look and see what is...I'd guess 
around 2.1. I did program for java 1.5 though, but converting to 1.4 
should be minimal work.

Jay Yu wrote:
> Mark,
>
> Great effort getting the original lucene index accessor package in 
> this shape. I am sure this will benefit a lot of people using Lucene 
> in a multithread env.
> I have a quick question to ask you:
> Do you have to use the core Lucene 2.3-dev in order to use the accessor?
>
> I will take a look at your codes to see if I could help. I used a 
> slightly modified version of the original package in my project but it 
> breaks some of my tests. I hope your version works better.
>
> Thanks a lot!
>
> Jay
>
>
> Mark Miller wrote:
>> I have sat down and rewrote IndexAccessor from scratch. I copied in 
>> the same reference counting logic, pruned some things, and tried to 
>> make the whole package a bit simpler to use. I have a few things to 
>> do, but its pretty solid already. The only major thing I'd still like 
>> to do is add an option to warm searchers before putting them in the 
>> Searcher cache. Id like to writer some more tests as well. Any help 
>> greatly appreciated if your interested in using the thing.
>>
>>
>> http://myhardshadow.com/indexaccessor/trunk/src/test/com/mhs/indexaccessor/SimpleSearchServer.java

>>
>>
>> Here is a an example of a class that can be instantiated in one of 
>> multiple threads and read /modify a single index without worrying 
>> about what any
>> of the other threads are doing to the index at any given time. This 
>> is a very simple example of how to use the IndexAccessor and not 
>> necessarily an
>> example of best practices. The main idea is that you get your Writer, 
>> Searcher, or Reader, and then be sure to release it as soon as your 
>> done with it
>> in a finally block. For loading, you will want to load many docs with 
>> a Writer (batch them) before releasing it, but remember that Readers 
>> will not get a new view
>> of the index until you release all of the Writers. So beware hogging 
>> a Writer unless you thats what your intending.
>>
>> JavaDoc:
>> http://myhardshadow.com/indexaccessorapi/
>>
>> Code:
>> http://myhardshadow.com/indexaccessor/trunk/
>>
>> Jar:
>> http://myhardshadow.com/indexaccessorreleases/indexaccessor.jar
>>
>>
>> Your synchronized block concerns:
>>
>> The synchronized blocks that control accesss to the IndexAccessor do 
>> not have a huge impact on performance. Keep in mind that all of the 
>> work is not done in a synchonrized block, just the retrieval of the 
>> Searcher, Writer, Reader. Even if the synchronization makes the 
>> method twice as expensive, it is still overpowered by the cost of 
>> parsing queries and searching the index. This applies with or without 
>> contention. I wrote a simple test and included the output below. You 
>> might use the IBM Lock Analyzer for Java to further analyze these 
>> costs. Trust me, this thing is speedy. Its many times better than 
>> using IndexModifier.
>>
>> Without Contention
>> Just retrieve and release Searcher 100000 times
>> ----
>> avg time:6.3E-4 ms
>> total time:63 ms
>>
>> Parse query and search on 1 doc 100000 times
>> ----
>> avg time:0.03107 ms
>> total time:3107 ms
>>
>>
>> With Contention (40 other threads running 80000 searches)
>> Just retrieve and release Searcher 100000 times
>> ----
>> avg time:0.04643 ms
>> total time:4643 ms
>>
>> Parse query and search on 1 doc 100000 times
>> ----
>> avg time:0.64337 ms
>> total time:64337 ms
>>
>>
>> - Mark
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message