lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Willnauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-4069) Segment-level Bloom filters for a 2 x speed up on rare term searches
Date Mon, 28 May 2012 20:49:23 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13284545#comment-13284545
] 

Simon Willnauer commented on LUCENE-4069:
-----------------------------------------

bq. Thanks for the tips for making the patch more generic. I'll get on it tomorrow and changed
to FixedBitSet while I'm at it.
YW! A couple of things regarding your patch. I think you should write a header for the bloomfilter
file using CodecUtils.writeHeader() to be consistent. Once you are ready you should also look
at the other *PostingsFormat impls how we document the file formats there. We try to have
versioned file formats instead of the monolithic one on the website. I saw some minor things
like 

{code}
finally {
  if (output != null)  {
    output.close();
  }
}
{code}

you can use IOUtils.close*() for that which handles some exception cases etc.

I briefly looked at your reuse code for TermsEnum and it seems it has the same issues as pulsing
has (sovled). When you get a TermsEnum that is infact a Bloomfilter wrapper but wraps another
codec underneath you keep on switching back an forth creating new delegated TermsEnum instances.
You can look at PulsingPostingsReader and in particular at PulsingEnumAttributeImpl how we
solved that in Pulsing. 
                
> Segment-level Bloom filters for a 2 x speed up on rare term searches
> --------------------------------------------------------------------
>
>                 Key: LUCENE-4069
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4069
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/index
>    Affects Versions: 3.6
>            Reporter: Mark Harwood
>            Priority: Minor
>             Fix For: 3.6.1
>
>         Attachments: BloomFilterCodec40.patch, MHBloomFilterOn3.6Branch.patch, PrimaryKey40PerformanceTestSrc.zip
>
>
> An addition to each segment which stores a Bloom filter for selected fields in order
to give fast-fail to term searches, helping avoid wasted disk access.
> Best suited for low-frequency fields e.g. primary keys on big indexes with many segments
but also speeds up general searching in my tests.
> Overview slideshow here: http://www.slideshare.net/MarkHarwood/lucene-bloomfilteredsegments
> Benchmarks based on Wikipedia content here: http://goo.gl/X7QqU
> Patch based on 3.6 codebase attached.
> There are no API changes currently - to play just add a field with "_blm" on the end
of the name to invoke special indexing/querying capability. Clearly a new Field or schema
declaration(!) would need adding to APIs to configure the service properly.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message