lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eric Bowman <ebow...@boboco.ie>
Subject Re: 2.4 Performance
Date Wed, 19 Nov 2008 09:01:06 GMT
lucene@triplehelix.org wrote:
> On an index of around 20 gigs I've been seeing a performance drop of
> around 35% after upgrading to 2.4 (measured on ~10000 requests
> identical requests, executed in parallel against a threaded lucene /
> apache setup, after a roughly 10000 query warmup). The principal
> changes I've made so far are just to switch to NIOFSDirectories and
> use read-only index readers.
>
> Our design is roughly as follows: we have some pre-query filters,
> queries typically involving around 25 clauses, and some
> post-processing of hits. We collect counts and filter post query using
> a hit collector, which uses the (now deprecated) bits() method of
> Filters.
>
> I looked at converting us to use the new DocIdSet infrastructure (to
> gain the supposed 30% speed bump), but this seems to be somewhat
> problematic as there is no guarantee for whether we will get back a
> set we can do binary operations on (for example, if we get back a
> SortedVIntList, we're pretty much out of luck - the cardinality of the set
> is large (as it's a sortedvintlist), so we can't coerce it into
> another type, and it doesn't have the set operations we need to use it
> directly.
>
> Has anyone else seen this? Is there anything else
> we should be changing in the upgrade to 2.4?
>   

We are seeing something very similar to this.  We had an IndexReader per
core, and performance was ok.  Since upgrading to 2.4, performance has
gone down, in some cases quite dramatically.  We switched to a single
IndexReader, and things got better.

We are now also looking at NIOFSDirectory after noticing a lot of
contention in IndexReader, but that didn't help as much as we had hoped.

To be honest I'm considering trying to downgrade back to 2.3 for now,
this is looking like a step backwards for us.

Thanks for any suggestions,
Eric

-- 
Eric Bowman
Boboco Ltd
ebowman@boboco.ie
http://www.boboco.ie/ebowman/pubkey.pgp
+35318394189/+353872801532


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message