lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Harwood (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-4069) Segment-level Bloom filters for a 2 x speed up on rare term searches
Date Tue, 17 Jul 2012 17:33:34 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13416383#comment-13416383
] 

Mark Harwood commented on LUCENE-4069:
--------------------------------------

A quick benchmark looks like the new right-sized bitset as opposed to the old worst-case-scenario-sized
bitset is buying us a small performance improvement.

bq. I also don't think this PF should be per-field

There was a lengthy discussion earlier on this topic. The approach presented here seems reasonable.
For the average user you have the DefaultBloomFilterFactory default which now has reasonable
sizing for all fields passed its way (assuming a heuristic based on numDocs=numKeys to anticipate).
For expert users you can provide a BloomFilterFactory with a custom choice of sizing heuristic
per-field and can also simply return "null" for non-bloomed fields.

Having a single, carefully configured BloomPF wrapper is preferable because you can channel
appropriately configured bloom settings to a common PF delegate and avoid creating multiple
.tii, .tis files etc because the PerFieldPF isn't smart enough to figure out that these Bloom-ing
choices do not require different physical files for all the delegated tii etc structures.

You don't *have* to use the Per-field stuff in BloomPF but there are benefits to be had in
doing so which can't otherwise be achieved.


bq. Can you add @lucene.experimental to all the new APIs?

Done.
                
> Segment-level Bloom filters for a 2 x speed up on rare term searches
> --------------------------------------------------------------------
>
>                 Key: LUCENE-4069
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4069
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/index
>    Affects Versions: 3.6, 4.0-ALPHA
>            Reporter: Mark Harwood
>            Priority: Minor
>             Fix For: 4.0
>
>         Attachments: BloomFilterPostingsBranch4x.patch, LUCENE-4069-tryDeleteDocument.patch,
LUCENE-4203.patch, MHBloomFilterOn3.6Branch.patch, PKLookupUpdatePerfTest.java, PKLookupUpdatePerfTest.java,
PKLookupUpdatePerfTest.java, PKLookupUpdatePerfTest.java, PrimaryKeyPerfTest40.java
>
>
> An addition to each segment which stores a Bloom filter for selected fields in order
to give fast-fail to term searches, helping avoid wasted disk access.
> Best suited for low-frequency fields e.g. primary keys on big indexes with many segments
but also speeds up general searching in my tests.
> Overview slideshow here: http://www.slideshare.net/MarkHarwood/lucene-bloomfilteredsegments
> Benchmarks based on Wikipedia content here: http://goo.gl/X7QqU
> Patch based on 3.6 codebase attached.
> There are no 3.6 API changes currently - to play just add a field with "_blm" on the
end of the name to invoke special indexing/querying capability. Clearly a new Field or schema
declaration(!) would need adding to APIs to configure the service properly.
> Also, a patch for Lucene4.0 codebase introducing a new PostingsFormat

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message