lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doron Cohen (JIRA)" <j...@apache.org>
Subject [jira] Commented: (LUCENE-584) Decouple Filter from BitSet
Date Sun, 08 Apr 2007 20:20:32 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12487456
] 

Doron Cohen commented on LUCENE-584:
------------------------------------

...right, your diff-txt had the Match tasks - I missed that - checked it, it is exactly what
I did, so we're ok here. 

When you rerun, you may want to use my alg - to compare the two approaches in one run. You
can run this by something like:
     ant run-task -Dtask.mem=256M -Dtask.alg=conf\matcher-vs-bitset.alg

Also, to get cleaner results, add the line:
     ResetSystemSoft
just in the beginning of the "search round" - this resets the (query) inputs and also calls
GC.

I tried like this twice, and got inconsistent results:

When the bitset searches preceded the match searches:
     [java] Operation           round   runCnt   recsPerRun        rec/s  elapsedSec    avgUsedMem
   avgTotalMem
     [java] SrchBitsSamRdr_5000     -       10         5000        706.4       70.78     7,511,219
    16,573,645
     [java] SrchMtchSamRdr_5000 -   - -  -  10 -  -  - 5000 -  -   689.6 -  -  72.50 -   8,223,005
-   11,926,323
     [java] SrchBitsNewRdr_500      -       10          500        152.5       32.80    14,360,618
    16,962,356
     [java] SrchMtchNewRdr_500 -  - - -  -  10 -  -  -  500 -  -   171.3 -  -  29.19 -  15,150,797
-   17,395,712

When the match searches preceded the bitset searches:
     [java] Operation           round   runCnt   recsPerRun        rec/s  elapsedSec    avgUsedMem
   avgTotalMem
     [java] SrchMtchSamRdr_5000     -       10         5000        763.5       65.49     9,563,243
    17,128,244
     [java] SrchBitsSamRdr_5000 -   - -  -  10 -  -  - 5000 -  -   729.3 -  -  68.56 -  10,003,775
-   13,001,114
     [java] SrchMtchNewRdr_500      -       10          500        175.7       28.46    12,068,559
    17,524,326
     [java] SrchBitsNewRdr_500 -  - - -  -  10 -  -  -  500 -  -   183.7 -  -  27.22 -  15,098,480
-   17,974,476

My conclusion from this is that the speed-up, if exists, is minor, at least for the setup
of this test. 

There are only 15 unique queries in this test - also printed in the log - are these the queries
you would expect to save in? 

I didn't follow this issue very closely so I don't know where the saving is expected here.
Both SearchTask and MatchTask now do nothing in collect, so no difference at the actual collect()
call.

Also, Scorer.score(HitCollector) and Matcher.match(MatchCollector) are very similar:
  public void score(HitCollector hc) throws IOException {
    while (next()) {
      hc.collect(doc(), score());
    }
  }
  public void match(MatchCollector mc) throws IOException {
    while (next()) {
      mc.collect(doc());
    }
  }
Especially for the case that the collect() method is doing nothing, as in this test.

I think there is a potential gain for large boolean OR queries, because score() would have
to call next() on all TermScorers and collect/sum their scores, while match() could use skipTo(last+1)
because any match encountered is a match and there is no need to sum the individual scores
for the same doc by other scorers. However as far as I can tell, current match() implementation
does not take advantage of this, but I may be overlooking something?

> Decouple Filter from BitSet
> ---------------------------
>
>                 Key: LUCENE-584
>                 URL: https://issues.apache.org/jira/browse/LUCENE-584
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: Search
>    Affects Versions: 2.0.1
>            Reporter: Peter Schäfer
>            Priority: Minor
>         Attachments: bench-diff.txt, bench-diff.txt, BitsMatcher.java, Filter-20060628.patch,
HitCollector-20060628.patch, IndexSearcher-20060628.patch, MatchCollector.java, Matcher.java,
Matcher20070226.patch, Scorer-20060628.patch, Searchable-20060628.patch, Searcher-20060628.patch,
Some Matchers.zip, SortedVIntList.java, TestSortedVIntList.java
>
>
> {code}
> package org.apache.lucene.search;
> public abstract class Filter implements java.io.Serializable 
> {
>   public abstract AbstractBitSet bits(IndexReader reader) throws IOException;
> }
> public interface AbstractBitSet 
> {
>   public boolean get(int index);
> }
> {code}
> It would be useful if the method =Filter.bits()= returned an abstract interface, instead
of =java.util.BitSet=.
> Use case: there is a very large index, and, depending on the user's privileges, only
a small portion of the index is actually visible.
> Sparsely populated =java.util.BitSet=s are not efficient and waste lots of memory. It
would be desirable to have an alternative BitSet implementation with smaller memory footprint.
> Though it _is_ possibly to derive classes from =java.util.BitSet=, it was obviously not
designed for that purpose.
> That's why I propose to use an interface instead. The default implementation could still
delegate to =java.util.BitSet=.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message