lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Harwood (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-4069) Segment-level Bloom filters for a 2 x speed up on rare term searches
Date Thu, 31 May 2012 17:15:24 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13286754#comment-13286754
] 

Mark Harwood commented on LUCENE-4069:
--------------------------------------

Its true to say that Bloom is a different case to Pulsing - Bloom does not interfere in any
with the normal recording of content in the wrapped delegate whereas Pulsing does.
It may prove useful for us to mark a formal distinction between these mutating/non mutating
types so we can treat them differently and provide optimisations?


bq. And separately, you can always contain the number of files even today by using only unique
instances yourself when writing

Contained but not optimal - roughly double the number of required files if I want the common
case of a primary key indexed with Bloom. I can't see a way of indexing with Bloom-plus-Lucene40
on field "A" and indexing with just Lucene40 on fields B,C and D and winding up with only
one Lucene40 set of files with a common segment suffix. The way I did find of achieving this
was to add a "bloomFilteredFields" set into my single Bloom+Lucene40 instance used for all
fields. Is there any other option here currently? 

Looking to the future, 4093 may have more capabilities at optimising if it understands the
distinction between mutating wrappers and non-mutating ones and how they are composed?


                
> Segment-level Bloom filters for a 2 x speed up on rare term searches
> --------------------------------------------------------------------
>
>                 Key: LUCENE-4069
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4069
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/index
>    Affects Versions: 3.6, 4.0
>            Reporter: Mark Harwood
>            Priority: Minor
>             Fix For: 4.0, 3.6.1
>
>         Attachments: BloomFilterPostings40.patch, MHBloomFilterOn3.6Branch.patch, PrimaryKey40PerformanceTestSrc.zip
>
>
> An addition to each segment which stores a Bloom filter for selected fields in order
to give fast-fail to term searches, helping avoid wasted disk access.
> Best suited for low-frequency fields e.g. primary keys on big indexes with many segments
but also speeds up general searching in my tests.
> Overview slideshow here: http://www.slideshare.net/MarkHarwood/lucene-bloomfilteredsegments
> Benchmarks based on Wikipedia content here: http://goo.gl/X7QqU
> Patch based on 3.6 codebase attached.
> There are no 3.6 API changes currently - to play just add a field with "_blm" on the
end of the name to invoke special indexing/querying capability. Clearly a new Field or schema
declaration(!) would need adding to APIs to configure the service properly.
> Also, a patch for Lucene4.0 codebase introducing a new PostingsFormat

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message