Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Tue, 29 May 2012 20:46:25 +0000 (UTC)
From: "Mark Harwood (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: <1960194682.12858.1338324385245.JavaMail.jiratomcat@issues-vm>
In-Reply-To: 
 <779773358.14423.1337355913561.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (LUCENE-4069) Segment-level Bloom filters for a
 2 x speed up on rare term searches
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/LUCENE-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13285100#comment-13285100 ] 

Mark Harwood commented on LUCENE-4069:
--------------------------------------

bq. I think you should really not provide any field handling at all.

By that I think you mean keep the abstract BloomFilteringPostingsFormatBase and dispense with the BloomFilteringLucene40Codec (and BloomFilteredLucene40PostingsFormat?). I was trying to limit the extensions apps would have to write to use this service (1 custom postings format subclass, 1 custom Codec subclass and 1 custom SPI config file) but I can see that equally we shouldn't offer implementations for all the many different service permutations.

I'll look at adding something to RandomCodec for Bloom-plus-random delegate PostingsFormat.

bq. I am still worried about the TermsEnum reuse code, are you planning to look into this?
bq.  you keep on switching back an forth creating new delegated TermsEnum instances

I'm not sure what you mean in creating new delegated TermsEnum instances?
In my impl of"iterator(TermsEnum reuse)" I take care to unwrap my wrapper for TermsEnum to find the original delegate's TermsEnum and then call the delegateTerms iterator method with this object as the reuse parameter. At this stage shouldn't the delegate Terms just recycle that unwrapped TermsEnum as per the normal reuse contract when no wrapping has been done? 

bq. you should also add license headers to the files you are adding

Will do.
 
                
> Segment-level Bloom filters for a 2 x speed up on rare term searches
> --------------------------------------------------------------------
>
>                 Key: LUCENE-4069
>                 URL: https://issues.apache.org/jira/browse/LUCENE-4069
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/index
>    Affects Versions: 3.6, 4.0
>            Reporter: Mark Harwood
>            Priority: Minor
>             Fix For: 4.0, 3.6.1
>
>         Attachments: BloomFilterCodec40.patch, MHBloomFilterOn3.6Branch.patch, PrimaryKey40PerformanceTestSrc.zip
>
>
> An addition to each segment which stores a Bloom filter for selected fields in order to give fast-fail to term searches, helping avoid wasted disk access.
> Best suited for low-frequency fields e.g. primary keys on big indexes with many segments but also speeds up general searching in my tests.
> Overview slideshow here: http://www.slideshare.net/MarkHarwood/lucene-bloomfilteredsegments
> Benchmarks based on Wikipedia content here: http://goo.gl/X7QqU
> Patch based on 3.6 codebase attached.
> There are no 3.6 API changes currently - to play just add a field with "_blm" on the end of the name to invoke special indexing/querying capability. Clearly a new Field or schema declaration(!) would need adding to APIs to configure the service properly.
> Also, a patch for Lucene4.0 codebase introducing a new PostingsFormat

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org