lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Willnauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-4069) Segment-level Bloom filters for a 2 x speed up on rare term searches
Date Mon, 28 May 2012 19:10:23 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284520#comment-13284520
] 

Simon Willnauer commented on LUCENE-4069:
-----------------------------------------

hey mark, this looks very interesting. I wonder if that also helps indexing in terms of applying
deletes. did you test that at all or is that something are looking into as well? For deletes
this could be very handy since bloom filters should help a lot when a given key is likely
to only in one segment.

Regarding the code, I think BloomFilter should be like Pulsing and only be a PostingsFormat.
You should make the BloomFilterPostingsFormat abstract and pass in the "wrapped" postings
format and delegate independent of the field. PostingsFormat is per field and is resolved
by the codec. An actual implementation should subclass the abstract BloomFilterPostingsFormat
that way you have a fixed PostingsFormat per "delegate" and don't need to deal with "which
field got which delegate" bla bla. 

We should not have a Bloomfilter codec but rather swap in the bloomfilter postings format
in test via random codec. I also wonder if we can extract a "bloomfilter" class into utils
that builds an BloomFilter this could be useful elsewhere. 


                
> Segment-level Bloom filters for a 2 x speed up on rare term searches
> --------------------------------------------------------------------
>
>                 Key: LUCENE-4069
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4069
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/index
>    Affects Versions: 3.6
>            Reporter: Mark Harwood
>            Priority: Minor
>             Fix For: 3.6.1
>
>         Attachments: BloomFilterCodec40.patch, MHBloomFilterOn3.6Branch.patch, PrimaryKey40PerformanceTestSrc.zip
>
>
> An addition to each segment which stores a Bloom filter for selected fields in order
to give fast-fail to term searches, helping avoid wasted disk access.
> Best suited for low-frequency fields e.g. primary keys on big indexes with many segments
but also speeds up general searching in my tests.
> Overview slideshow here: http://www.slideshare.net/MarkHarwood/lucene-bloomfilteredsegments
> Benchmarks based on Wikipedia content here: http://goo.gl/X7QqU
> Patch based on 3.6 codebase attached.
> There are no API changes currently - to play just add a field with "_blm" on the end
of the name to invoke special indexing/querying capability. Clearly a new Field or schema
declaration(!) would need adding to APIs to configure the service properly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message