lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aviran" <amo...@infosciences.com>
Subject RE: Sort: 1.4-rc3 vs. 1.4-final
Date Wed, 21 Jul 2004 17:45:40 GMT
I just saw this post, I guess we both came to the same conclusion. 
The only problem is that the cached object never gets released, and a new
one will get created every time you open a new IndexReader

Aviran

-----Original Message-----
From: Greg Gershman [mailto:greggersh@yahoo.com] 
Sent: Wednesday, July 21, 2004 13:30 PM
To: Lucene Users List
Subject: RE: Sort: 1.4-rc3 vs. 1.4-final


I switched the Comparators and FieldCache classes to
use java.util.HashMap instead of
java.util.WeakHashMap, and got the performance boost I
was looking for (test index of 100K documents; initial
search took 991 ms, all subsequent searchs took <
90ms.  Before, I was seeing initial query of ~1sec,
subsequent queries between 500 and 700 ms, with
comparator and field lookup table computed each time).

I guess the question is why use a WeakHashMap here as
opposed to a HashMap?

Greg

--- Greg Gershman <greggersh@yahoo.com> wrote:
> I've done a bit more snooping around; it seems that
> in
> FieldSortedHitQueue.getCachedComparator(line 153),
> calls to lookup a stored comparator in the cache
> always return null.  This occurs even for the
> built-in
> sort types (I tested it on integers and my code for
> longs).  The comparators don't even appear to be
> being
> stored in the HashMap to begin with.
> 
> Any ideas?
> 
> Greg
> 
>  
> 
> --- Aviran <amordo@infosciences.com> wrote:
> > Since I had to implement sorting in lucene 1.2 I
> had
> > to write my own sorting
> > using something similar to a lucene's contribution
> > called SortField.
> > Yesterday I did some tests, trying to use lucene
> 1.4
> > Sort objects and I
> > realized that my old implementation works 40%
> faster
> > then Lucene's
> > implementation. My guess is that you are right and
> > there is a problem with
> > the cache although I couldn't find what that is
> yet.
> > 
> > Aviran
> > 
> > -----Original Message-----
> > From: Greg Gershman [mailto:greggersh@yahoo.com]
> > Sent: Wednesday, July 21, 2004 9:22 AM
> > To: lucene-user@jakarta.apache.org
> > Subject: Sort: 1.4-rc3 vs. 1.4-final
> > 
> > 
> > When rc3 came out, I modified the classes used for
> > Sorting to, in addition to Integer, Float and
> > String-based sort keys, use Long values.  All I
> did
> > was add extra statements in 2 classes (SortField
> and
> > FieldSortedHitQueue) that made a special case for
> > longs, and created a LongSortedHitQueue identical
> to
> > the IntegerSortedHitQueue, only using longs.
> > 
> > This worked as expected; Long values converted to
> > strings and stored in Field.Keyword type fields
> > would
> > be sorted according to Long order.  The initial
> > query
> > would take a while, to build the sorted array, but subsequent 
> > queries would take little to no time at all.
> > 
> > I went back to look at 1.4 final, and noticed the
> > Sort implementation has
> > changed quite a bit.  I tried the same type of modifications to the 
> > existing source files, but was unable to achieve similiar
> > results. 
> > Each subsequent query seems to take a significant
> > amount of time, as if the Sorted array is being
> > rebuilt each time.  Also, I tried sorting on an
> > Integer fields and got similar results, which
> leads
> > me
> > to believe there might be a caching problem
> > somewhere.
> > 
> > Has anyone else seen this in 1.4-final?  Also, I
> > would
> > like it if Long sorted fields could become a part
> of
> > the API; it makes sorting by date a breeze.
> > 
> > Thanks!
> > 
> > Greg Gershman
> > 
> > 
> > 		
> > __________________________________
> > Do you Yahoo!?
> > New and Improved Yahoo! Mail - Send 10MB messages! 
> > http://promotions.yahoo.com/new_mail
> > 
> >
>
---------------------------------------------------------------------
> > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> > lucene-user-help@jakarta.apache.org
> > 
> > 
> > 
> > 
> >
>
---------------------------------------------------------------------
> > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> > lucene-user-help@jakarta.apache.org
> > 
> > 
> 
> 
> 
> 	
> 		
> __________________________________
> Do you Yahoo!?
> Vote for the stars of Yahoo!'s next ad campaign!
>
http://advision.webevents.yahoo.com/yahoo/votelifeengine/
> 
>
---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> 
> 



	
		
__________________________________
Do you Yahoo!?
Vote for the stars of Yahoo!'s next ad campaign!
http://advision.webevents.yahoo.com/yahoo/votelifeengine/

---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org




---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org


Mime
View raw message