lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From t...@simpleobjects.com
Subject some optimization and other hacks
Date Wed, 02 Mar 2005 19:44:23 GMT
I have a few changes that have really paid off for me.  They are hacks,
but I thought I would pass this on in case anyone else might be interested:
http://simpleobjects.com/LuceneOptimizationHacks.zip

(my apologies for the crude packaging)


changes:
1.  jdk 1.4 file locking mechanism implementation (this could be
implemented using reflection in case you want to leave in support for jdk
1.3).
2.  compact bit set implementation (ztree) for rangefilters that are very
sparse.  When I filter on dates, it turns out, most of the doc ids (in
general) are found near each other.  This saves me huge amounts of memory.
3.  a hack to remove some code that stores the field values for fields
on which you are sorting (this appears to only be required when using
multi-indexes (which I am not)).  This also saves me a lot of memory.

other ideas i have found helpful (more for the users list)...

1.  when you need to filter and sort by dates, store multiple copies
(fields) of the date.  1 copy for the sorting (this can have millisecond
resolution) and another for the precision you want to filter by.  I have
found minute resolution very useful and also very fast for filtering to
ranges even spanning months with huge numbers of docs.

Tony Schwartz

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message