lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Harwood (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-4069) Segment-level Bloom filters for a 2 x speed up on rare term searches
Date Wed, 30 May 2012 08:59:24 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285518#comment-13285518
] 

Mark Harwood commented on LUCENE-4069:
--------------------------------------

Thanks for the comment, Rob.
While the choice of Codec can be an anonymous inner class, resolving the choice of PostingsFormat
is trickier.
BloomFilterPostingsFormat is now intended to wrap any another choice of PostingsFormat and
Simon has suggested leaving the Bloom support purely abstract.
However, as an end user if I want to use Bloom support on the standard Lucene codec I would
then have to write one of these:
{code:title=MyBloomFilteredLucene40Postings.java}
public class MyBloomFilteredLucene40Postingsextends BloomFilteringPostingsFormatBase {

  public MyBloomFilteredLucene40Postings() {
    //declare my choice of PostingsFormat to be wrapped and provide a unique name for this
combo of Bloom-plus-delegate
    super("myBL40", new Lucene40PostingsFormat());
  }
}
{code}
The resulting index files are then named [segname]_myBL40.[filetypesuffix].
At read-time the "myBL40" bit of the filename is used to lookup via Service Provider registrations
the decoding class so "com.xx.MyBloomFilteredLucene40Postings" would need adding to a o.a.l.codecs.PostingsFormat
file for the registration to work.

I imagine Bloom-plus-Lucene40Postings would be a common combo and if both are in core it would
be annoying to have to code support for this in each app or for things like Luke to have to
have classpaths redefined to access the app-specific class that was created purely to bind
this combo of core components.

I think a better option might be to change the Bloom filtering base class to record the choice
of delegate PostingsFormat in it's own "blm" file at write-time and instantiate the appropriate
delegate instance at read-time using the recorded name. The BloomFilteringBaseClass would
need changing to a final class rather than an abstract so that core Lucene could load it as
the handler for [segname]_BloomPosting.xxx files and it would then have to examine the [segname].blm
file to discover and instantiate the choice of delegate PostingsFormat using the standard
service registration mechanism. At write-time clients would need to instantiate the BloomFilterPostingsFormat,
passing a choice of PostingsFormat delegate to the constructor. At read-time Lucene core would
invoke a zero-arg constructor.
I'll look into this as an approach.






 
                
> Segment-level Bloom filters for a 2 x speed up on rare term searches
> --------------------------------------------------------------------
>
>                 Key: LUCENE-4069
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4069
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/index
>    Affects Versions: 3.6, 4.0
>            Reporter: Mark Harwood
>            Priority: Minor
>             Fix For: 4.0, 3.6.1
>
>         Attachments: BloomFilterCodec40.patch, MHBloomFilterOn3.6Branch.patch, PrimaryKey40PerformanceTestSrc.zip
>
>
> An addition to each segment which stores a Bloom filter for selected fields in order
to give fast-fail to term searches, helping avoid wasted disk access.
> Best suited for low-frequency fields e.g. primary keys on big indexes with many segments
but also speeds up general searching in my tests.
> Overview slideshow here: http://www.slideshare.net/MarkHarwood/lucene-bloomfilteredsegments
> Benchmarks based on Wikipedia content here: http://goo.gl/X7QqU
> Patch based on 3.6 codebase attached.
> There are no 3.6 API changes currently - to play just add a field with "_blm" on the
end of the name to invoke special indexing/querying capability. Clearly a new Field or schema
declaration(!) would need adding to APIs to configure the service properly.
> Also, a patch for Lucene4.0 codebase introducing a new PostingsFormat

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message