lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mark Harwood (JIRA)" <>
Subject [jira] [Commented] (LUCENE-4069) Segment-level Bloom filters for a 2 x speed up on rare term searches
Date Fri, 25 May 2012 16:34:23 GMT


Mark Harwood commented on LUCENE-4069:

My current focus is speeding up primary key lookups but this may have applications outside
of that (Zipf tells us there is a lot of low frequency stuff in free text).

Following the principle of the best IO is no IO the Bloom Filter helps us quickly understand
which segments to even bother looking in. That has to be a win overall.

I started trying to write this Codec as a wrapper for any other Codec (it simply listens to
a stream of terms and stores a bitset of recorded hashes in a ".blm" file). However that was
trickier than I expected - I'd need to encode a special entry in my blm files just to know
the name of the delegated codec I needed to instantiate at read-time because Lucene's normal
Codec-instantiation logic would be looking for "BloomCodec" and I'd have to discover the delegate
that was used to write all of the non-blm files.

Not looked at FixedBitSet but I imagine that could be used instead.
> Segment-level Bloom filters for a 2 x speed up on rare term searches
> --------------------------------------------------------------------
>                 Key: LUCENE-4069
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/index
>    Affects Versions: 3.6
>            Reporter: Mark Harwood
>            Priority: Minor
>             Fix For: 3.6.1
>         Attachments: BloomFilterCodec40.patch, MHBloomFilterOn3.6Branch.patch,
> An addition to each segment which stores a Bloom filter for selected fields in order
to give fast-fail to term searches, helping avoid wasted disk access.
> Best suited for low-frequency fields e.g. primary keys on big indexes with many segments
but also speeds up general searching in my tests.
> Overview slideshow here:
> Benchmarks based on Wikipedia content here:
> Patch based on 3.6 codebase attached.
> There are no API changes currently - to play just add a field with "_blm" on the end
of the name to invoke special indexing/querying capability. Clearly a new Field or schema
declaration(!) would need adding to APIs to configure the service properly.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message