lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Harwood (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (LUCENE-4069) Segment-level Bloom filters for a 2 x speed up on rare term searches
Date Thu, 31 May 2012 13:20:23 GMT

     [ https://issues.apache.org/jira/browse/LUCENE-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Mark Harwood updated LUCENE-4069:
---------------------------------

    Attachment: BloomFilterPostings40.patch

This is looking more promising.

Running "ant test-core -Dtests.postingsformat=TestBloomFilteredLucene40Postings" now passes
all tests but causes OOM exception on 3 tests:
* TestConsistentFieldNumbers.testManyFields
* TestIndexableField.testArbitraryFields
* TestIndexWriter.testManyFields

Any pointers on how to annotate or otherwise avoid the BloomFilter class for "many-field"
tests would be welcome. These are not realistic tests for this class (we don't expect indexes
with 100s of primary-key like fields).

In this patch I've
* added an SPI lookup mechanism for pluggable hash algos.
* documented the file format
* fixed issues with TermVector tests
* changed the API


To use:
BloomFilteringPostingFormat now takes a delegate PostingsFormat and a set of field names that
are to have bloom-filters created.
Fields that are not listed in the filter set can be safely indexed as per normal and doing
so is beneficial because it allows filtered and non filtered field data to co-exist in the
same physical files created by the delegate PostingsFormat.

                
> Segment-level Bloom filters for a 2 x speed up on rare term searches
> --------------------------------------------------------------------
>
>                 Key: LUCENE-4069
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4069
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/index
>    Affects Versions: 3.6, 4.0
>            Reporter: Mark Harwood
>            Priority: Minor
>             Fix For: 4.0, 3.6.1
>
>         Attachments: BloomFilterCodec40.patch, BloomFilterPostings40.patch, MHBloomFilterOn3.6Branch.patch,
PrimaryKey40PerformanceTestSrc.zip
>
>
> An addition to each segment which stores a Bloom filter for selected fields in order
to give fast-fail to term searches, helping avoid wasted disk access.
> Best suited for low-frequency fields e.g. primary keys on big indexes with many segments
but also speeds up general searching in my tests.
> Overview slideshow here: http://www.slideshare.net/MarkHarwood/lucene-bloomfilteredsegments
> Benchmarks based on Wikipedia content here: http://goo.gl/X7QqU
> Patch based on 3.6 codebase attached.
> There are no 3.6 API changes currently - to play just add a field with "_blm" on the
end of the name to invoke special indexing/querying capability. Clearly a new Field or schema
declaration(!) would need adding to APIs to configure the service properly.
> Also, a patch for Lucene4.0 codebase introducing a new PostingsFormat

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message