lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Elschot <>
Subject Re: CVS Lucene 2.0
Date Mon, 02 May 2005 07:29:02 GMT
On Monday 02 May 2005 06:01, Yonik Seeley wrote:
> I can't say whats actually ready, but I am very interested in sparse
> filter representations.  I'm working on a project that needs to
> dynamic categorization of search results, and this requires caching
> thousands of filters.

The need for many filters is what started me off on the boolean scorer
changes in the first place. Caching many filters implies filters must be
sparse (ie. take little memory). Sparse filters implies that they must
be used with iterator access (no more random access as on BitSet).
Iterator access implies all scoring should be done in
order of document numbers, which is not provided by the 1.4.3 BooleanScorer.

> It does currently require Java 1.4, so I guess it needs to wait for Lucene 

It is the need to apply skipTo on the BitSet that introduces the need
for java 1.4 . In case one would only use SortedVIntList, (the sparse
filter), the java 1.4 dependency would not be there.

Btw. this also shows why both filters only work in RAM: they have
no underlying support for skipTo in their data structure. Java has
bolted this on in 1.4 over the BitSet (probably by a linear search
over the underlying array), and for the SkipFilter skipTo
is implemented by using the iterator, ie. a linear search.
> And I'm not sure if I'm reading it correctly, but it looks like
> BooleanScorer1 could be a replacement for both BooleanScorer and
> BooleanScorer2.

It's not stated there,  but BooleanScorer2 currently depends on
DisjunctionScorer in svn.
Paul Elschot

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message