lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] [Commented] (LUCENE-4069) Segment-level Bloom filters for a 2 x speed up on rare term searches
Date Fri, 25 May 2012 16:21:23 GMT


Michael McCandless commented on LUCENE-4069:

bq. I see a 35% improvement over standard Codecs on random lookups on a warmed index.

Impressive!  This is for primary key lookups?

It looks like the primary keys are GUID-like right?  (Ie randomly generated).  I wonder if
they had some structure instead (eg '%09d' % (id++)) how the results would look...

bq. I also notice that the PulsingCodec is no longer faster than standard Codec - is this
news to people as I thought it was supposed to be the way forward?

That's baffling to me: it should only save seeks vs Lucene40 codec, so on a cold index you
should see substantial gains, and on a warm index I'd still expect some gains.  Not sure what's

bq. I can open a seperate JIRA issue for this 4.0 version of the code if that makes more sense.

I think it's fine to do it here?  Really 3.6.x is only for bug fixes now ... so I think we
should commit this to trunk.

I wonder if you can wrap any other PostingsFormat (ie instead of hard-coding to Lucene40PostingsFormat)?
 This way users can wrap any PF they have w/ the bloom filter...

Can you use FixedBitSet instead of OpenBitSet?  Or is there a reason to use OpenBitSet here...?
> Segment-level Bloom filters for a 2 x speed up on rare term searches
> --------------------------------------------------------------------
>                 Key: LUCENE-4069
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/index
>    Affects Versions: 3.6
>            Reporter: Mark Harwood
>            Priority: Minor
>             Fix For: 3.6.1
>         Attachments: BloomFilterCodec40.patch, MHBloomFilterOn3.6Branch.patch,
> An addition to each segment which stores a Bloom filter for selected fields in order
to give fast-fail to term searches, helping avoid wasted disk access.
> Best suited for low-frequency fields e.g. primary keys on big indexes with many segments
but also speeds up general searching in my tests.
> Overview slideshow here:
> Benchmarks based on Wikipedia content here:
> Patch based on 3.6 codebase attached.
> There are no API changes currently - to play just add a field with "_blm" on the end
of the name to invoke special indexing/querying capability. Clearly a new Field or schema
declaration(!) would need adding to APIs to configure the service properly.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message