lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: 2.4 Performance
Date Wed, 19 Nov 2008 10:15:06 GMT

Can you describe the queries in more detail?  Can you narrow down  
exactly which operations / types of queries are substantially slower?

Also, I'm assuming both of you are NOT on Windows?  NIOFSDirectory has  
poor performance on Windows due to this bug in Sun's JVM:

     http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6265734

Mike

Eric Bowman wrote:

> lucene@triplehelix.org wrote:
>> On an index of around 20 gigs I've been seeing a performance drop of
>> around 35% after upgrading to 2.4 (measured on ~10000 requests
>> identical requests, executed in parallel against a threaded lucene /
>> apache setup, after a roughly 10000 query warmup). The principal
>> changes I've made so far are just to switch to NIOFSDirectories and
>> use read-only index readers.
>>
>> Our design is roughly as follows: we have some pre-query filters,
>> queries typically involving around 25 clauses, and some
>> post-processing of hits. We collect counts and filter post query  
>> using
>> a hit collector, which uses the (now deprecated) bits() method of
>> Filters.
>>
>> I looked at converting us to use the new DocIdSet infrastructure (to
>> gain the supposed 30% speed bump), but this seems to be somewhat
>> problematic as there is no guarantee for whether we will get back a
>> set we can do binary operations on (for example, if we get back a
>> SortedVIntList, we're pretty much out of luck - the cardinality of  
>> the set
>> is large (as it's a sortedvintlist), so we can't coerce it into
>> another type, and it doesn't have the set operations we need to use  
>> it
>> directly.
>>
>> Has anyone else seen this? Is there anything else
>> we should be changing in the upgrade to 2.4?
>>
>
> We are seeing something very similar to this.  We had an IndexReader  
> per
> core, and performance was ok.  Since upgrading to 2.4, performance has
> gone down, in some cases quite dramatically.  We switched to a single
> IndexReader, and things got better.
>
> We are now also looking at NIOFSDirectory after noticing a lot of
> contention in IndexReader, but that didn't help as much as we had  
> hoped.
>
> To be honest I'm considering trying to downgrade back to 2.3 for now,
> this is looking like a step backwards for us.
>
> Thanks for any suggestions,
> Eric
>
> -- 
> Eric Bowman
> Boboco Ltd
> ebowman@boboco.ie
> http://www.boboco.ie/ebowman/pubkey.pgp
> +35318394189/+353872801532
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message