lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-1536) if a filter can support random access API, we should use it
Date Mon, 10 Oct 2011 11:02:30 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123998#comment-13123998
] 

Michael McCandless commented on LUCENE-1536:
--------------------------------------------

On the diff that luceneutil hits, it looks like there's an float iota
difference:

On trunk we get these results:

{noformat}
TASK: cat=Fuzzy1F90.0 q=body:changer~1.0 s=null f=CachingWrapperFilter(PreComputedRandomFilter(pctAccept=90.0))
group=null hits=198715
  32.160243 msec
  thread 5
  doc=6199951 score=40.27584
  doc=6199960 score=40.27584
  doc=6200023 score=40.27584
  doc=7580697 score=40.27584
  doc=7995191 score=33.34529
  doc=8684145 score=31.100195
  doc=6260043 score=31.100193
  doc=7320778 score=31.100193
  doc=7454704 score=31.100193
  doc=7979518 score=26.333052
  50 expanded terms
{noformat}

With the patch we get this:

{noformat}
TASK: cat=Fuzzy1F90.0 q=body:changer~1.0 s=null f=CachingWrapperFilter(PreComputedRandomFilter(pctAccept=90.0))
group=null hits=198715
  19.300811 msec
  thread 4
  doc=6199951 score=40.27584
  doc=6199960 score=40.27584
  doc=6200023 score=40.27584
  doc=7580697 score=40.27584
  doc=7995191 score=33.34529
  doc=6260043 score=31.100195
  doc=7454704 score=31.100195
  doc=7320778 score=31.100193
  doc=8684145 score=31.100193
  doc=7979518 score=26.333052
  50 expanded terms
{noformat}

several of the docs with score 31.100193 or 31.100195 flipped around.

                
> if a filter can support random access API, we should use it
> -----------------------------------------------------------
>
>                 Key: LUCENE-1536
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1536
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/search
>    Affects Versions: 2.4
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>            Priority: Minor
>              Labels: gsoc2011, lucene-gsoc-11, mentor
>             Fix For: 4.0
>
>         Attachments: CachedFilterIndexReader.java, LUCENE-1536-rewrite.patch, LUCENE-1536-rewrite.patch,
LUCENE-1536-rewrite.patch, LUCENE-1536-rewrite.patch, LUCENE-1536-rewrite.patch, LUCENE-1536-rewrite.patch,
LUCENE-1536-rewrite.patch, LUCENE-1536-rewrite.patch, LUCENE-1536.patch, LUCENE-1536.patch,
LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch,
LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch,
LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch,
LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch,
LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, LUCENE-1536.patch, changes-yonik-uwe.patch,
luceneutil.patch
>
>
> I ran some performance tests, comparing applying a filter via
> random-access API instead of current trunk's iterator API.
> This was inspired by LUCENE-1476, where we realized deletions should
> really be implemented just like a filter, but then in testing found
> that switching deletions to iterator was a very sizable performance
> hit.
> Some notes on the test:
>   * Index is first 2M docs of Wikipedia.  Test machine is Mac OS X
>     10.5.6, quad core Intel CPU, 6 GB RAM, java 1.6.0_07-b06-153.
>   * I test across multiple queries.  1-X means an OR query, eg 1-4
>     means 1 OR 2 OR 3 OR 4, whereas +1-4 is an AND query, ie 1 AND 2
>     AND 3 AND 4.  "u s" means "united states" (phrase search).
>   * I test with multiple filter densities (0, 1, 2, 5, 10, 25, 75, 90,
>     95, 98, 99, 99.99999 (filter is non-null but all bits are set),
>     100 (filter=null, control)).
>   * Method high means I use random-access filter API in
>     IndexSearcher's main loop.  Method low means I use random-access
>     filter API down in SegmentTermDocs (just like deleted docs
>     today).
>   * Baseline (QPS) is current trunk, where filter is applied as iterator up
>     "high" (ie in IndexSearcher's search loop).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message