lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Willnauer (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (LUCENE-4069) Segment-level Bloom filters for a 2 x speed up on rare term searches
Date Tue, 29 May 2012 19:46:24 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285054#comment-13285054
] 

Simon Willnauer edited comment on LUCENE-4069 at 5/29/12 7:45 PM:
------------------------------------------------------------------

hey mark, 

I think you should really not provide any field handling at all. this should be done in user
code ie. a general codes/postings format per field. What would be very useful here is integration
into RandomCodec so we can randomly swap it in. Same is true for the hash algos & memory
settings, I think they should be part of the abstract class ctor and if a user wants to use
its own algo they should write their own codec ie. subclass. this stuff is very expert and
apps like solr can handle this on top. 

I am still worried about the TermsEnum reuse code, are you planning to look into this?

you should also add license headers to the files you are adding, note that we don't do javadoc
license headers anymore so they should just be block comments.

                
      was (Author: simonw):
    hey mark, 

I think you should really not provide any field handling at all. this should be done in user
code ie. a general codes/postings format per field. What would be very useful here is integration
into RandomCodec so we can randomly swap it in. Same is true for the hash algos & memory
settings, I think they should be part of the abstract class ctor and if a user wants to use
its own algo they should write their own codec ie. subclass. this stuff is very expert and
apps like solr can handle this on top. 

I am still worried about the TermsEnum reuse code, are you planning to look into this?


                  
> Segment-level Bloom filters for a 2 x speed up on rare term searches
> --------------------------------------------------------------------
>
>                 Key: LUCENE-4069
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4069
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/index
>    Affects Versions: 3.6, 4.0
>            Reporter: Mark Harwood
>            Priority: Minor
>             Fix For: 4.0, 3.6.1
>
>         Attachments: BloomFilterCodec40.patch, MHBloomFilterOn3.6Branch.patch, PrimaryKey40PerformanceTestSrc.zip
>
>
> An addition to each segment which stores a Bloom filter for selected fields in order
to give fast-fail to term searches, helping avoid wasted disk access.
> Best suited for low-frequency fields e.g. primary keys on big indexes with many segments
but also speeds up general searching in my tests.
> Overview slideshow here: http://www.slideshare.net/MarkHarwood/lucene-bloomfilteredsegments
> Benchmarks based on Wikipedia content here: http://goo.gl/X7QqU
> Patch based on 3.6 codebase attached.
> There are no 3.6 API changes currently - to play just add a field with "_blm" on the
end of the name to invoke special indexing/querying capability. Clearly a new Field or schema
declaration(!) would need adding to APIs to configure the service properly.
> Also, a patch for Lucene4.0 codebase introducing a new PostingsFormat

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message