lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Willnauer (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (LUCENE-4069) Segment-level Bloom filters for a 2 x speed up on rare term searches
Date Thu, 31 May 2012 20:58:23 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286926#comment-13286926
] 

Simon Willnauer edited comment on LUCENE-4069 at 5/31/12 8:57 PM:
------------------------------------------------------------------

bq. This seemed relevant given the earlier comments about Solr's use of non-compound files:
We can't make "wrong" decisions just because higher level apps make "wrong" decisions. The
dependency goes Solr -> Lucene not the other way around. We provide fine grained control
when to use CFS ie for smallish segments etc. If you have hundreds of fields all using different
PF etc. you have to deal with tons of files but that is to be honest not very likely to be
the common case.

bq. Create a PerFieldPF implementation that reuses wrapped PFs using some generic means of
discovering recyclable wrapped PFs (i.e go further than what 4093 currently proposes in adding
.equals support)

I think we should investigate this further. Lets keep this issue simple and remove the field
handling and fix this on a higher level!
                
      was (Author: simonw):
    bq. Create a PerFieldPF implementation that reuses wrapped PFs using some generic means
of discovering recyclable wrapped PFs (i.e go further than what 4093 currently proposes in
adding .equals support)

I think we should investigate this further. Lets keep this issue simple and remove the field
handling and fix this on a higher level!

                  
> Segment-level Bloom filters for a 2 x speed up on rare term searches
> --------------------------------------------------------------------
>
>                 Key: LUCENE-4069
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4069
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/index
>    Affects Versions: 3.6, 4.0
>            Reporter: Mark Harwood
>            Priority: Minor
>             Fix For: 4.0, 3.6.1
>
>         Attachments: BloomFilterPostings40.patch, MHBloomFilterOn3.6Branch.patch, PrimaryKey40PerformanceTestSrc.zip
>
>
> An addition to each segment which stores a Bloom filter for selected fields in order
to give fast-fail to term searches, helping avoid wasted disk access.
> Best suited for low-frequency fields e.g. primary keys on big indexes with many segments
but also speeds up general searching in my tests.
> Overview slideshow here: http://www.slideshare.net/MarkHarwood/lucene-bloomfilteredsegments
> Benchmarks based on Wikipedia content here: http://goo.gl/X7QqU
> Patch based on 3.6 codebase attached.
> There are no 3.6 API changes currently - to play just add a field with "_blm" on the
end of the name to invoke special indexing/querying capability. Clearly a new Field or schema
declaration(!) would need adding to APIs to configure the service properly.
> Also, a patch for Lucene4.0 codebase introducing a new PostingsFormat

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message