lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Paul Elschot (JIRA)" <>
Subject [jira] Commented: (LUCENE-584) Decouple Filter from BitSet
Date Thu, 26 Jul 2007 08:47:34 GMT


Paul Elschot commented on LUCENE-584:


The exhausted flag is only in the iterator/Matcher, not in the underlying set data structure.
One can use as many iterators as necessary, for example one per thread, and then there is
never a threadsafety problem. (See BitSetMatcher.getMatcher() which uses a local class for
the resulting Matcher.)

You wrote: I use BooleanFilter a lot for security where many large sets are cached and combined
on the fly - caching all the possible combinations as single bitsets would lead to too many
possible combinations.

That can still be done, but one needs to get to the BitSets for example by caching them outside
the Filters and constructing the resulting BitSetMatcher for the combined Filter on the fly.

An alternative would be to have a BooleanQuery.add(Matcher, Occur), where the occurrence can
only be required or prohibited. Then there is no need to construct any resulting filter because
the boolean logic will be executed during the search.  This might even be more efficient than
combining the full BitSets ahead of the search.

And with many large BitSets cache memory savings from more compact implementations can also
be helpful.

> Decouple Filter from BitSet
> ---------------------------
>                 Key: LUCENE-584
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Search
>    Affects Versions: 2.0.1
>            Reporter: Peter Schäfer
>            Priority: Minor
>         Attachments: bench-diff.txt, bench-diff.txt, Matcher-core20070725.patch, Matcher-default20070725.patch,
Matcher-ground20070725.patch, Some
> {code}
> package;
> public abstract class Filter implements 
> {
>   public abstract AbstractBitSet bits(IndexReader reader) throws IOException;
> }
> public interface AbstractBitSet 
> {
>   public boolean get(int index);
> }
> {code}
> It would be useful if the method =Filter.bits()= returned an abstract interface, instead
of =java.util.BitSet=.
> Use case: there is a very large index, and, depending on the user's privileges, only
a small portion of the index is actually visible.
> Sparsely populated =java.util.BitSet=s are not efficient and waste lots of memory. It
would be desirable to have an alternative BitSet implementation with smaller memory footprint.
> Though it _is_ possibly to derive classes from =java.util.BitSet=, it was obviously not
designed for that purpose.
> That's why I propose to use an interface instead. The default implementation could still
delegate to =java.util.BitSet=.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message