lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bruce Ritchie <br...@jivesoftware.com>
Subject Re: Caching filter wrapper (was Re: RE : DateFilter.Before/After)
Date Tue, 16 Sep 2003 04:24:16 GMT
Doug Cutting wrote:
>> I think that using a pool of cloned inputStreams would be the best 
>> solution. I've implemented such a solution locally using two pools of 
>> 3 readers each (configurable via system properties) and will post the 
>> diff after I do some testing to confirm accuracy and speed improvements.
> 
> 
> Could you also benchmark this against a version that clones new streams 
> for each call?  That sounds extravagant, but it removes a configuration 
> parameter, always a good thing.

After doing some testing tonight I've come up with the following numbers after running some
tests on 
my personal workstation.

Dell 2 ghz with single HD, Win XP SP1, 768 Mb ram
Sun JDK 1.4.2
Resin 2.1.0 (-DdisableLuceneLocks=true -J-server)
Lucene index with 474128 documents, not completely optimized (most content is in 1 segment)
First run after startup discarded
Tested with 5 simultaneous threads


Unmodified CVS source

Run  Count Time (ms) Queries/Second
  1   1001  542050    1.85
  2   1001  508458    1.97
  3   1001  524396    1.93

CVS source using suggested clone solution for FieldsReader and removing synchronized from

SegmentReader.document(i)

Run  Count Time (ms) Queries/Second
  1   1008  674123    1.495
  2   1017  675363    1.51
  3   1005  655551    1.53

CVS source using pool of 3 input streams for the previous fieldsStream and indexStream variables
in 
FieldsReader and removing synchronized from SegmentReader.document(i)

Run  Count Time (ms) Queries/Second
  1   1009  392536    2.57
  2   999   364783    2.74
  3   995   386501    2.57



The times shown above is only the time taken to call the following code (numResults is a max
of 1500 
or hits.length(), whichever is smaller):

for (int i = 0; i < numResults; i++) {
    ids[i] = Long.parseLong((hits.doc(i)).get("messageID"));
}


I've uploaded my testing app with 3 prebuilt lucene libs (unomodified, clone and pool) and
my source 
modifications to FieldsReader to http://www.jivesoftware.com/~bruce/lucene/lucene-test.zip
(624K) if 
anyone else wants to run the tests on their hardware. You'll have to edit the 
lucenetest.LuceneTestThread class as it has the location of the search directory hardcoded,
but it 
should be pretty easy to understand what is going on.


Regards,

Bruce Ritchie

Mime
View raw message