lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] Created: (LUCENE-2723) Speed up Lucene's low level bulk postings read API
Date Mon, 25 Oct 2010 19:14:19 GMT
Speed up Lucene's low level bulk postings read API

                 Key: LUCENE-2723
             Project: Lucene - Java
          Issue Type: Improvement
          Components: Index
            Reporter: Michael McCandless
            Assignee: Michael McCandless
             Fix For: 4.0

Spinoff from LUCENE-1410.

The flex DocsEnum has a simple bulk-read API that reads the next chunk
of docs/freqs.  But it's a poor fit for intblock codecs like FOR/PFOR
(from LUCENE-1410).  This is not unlike sucking coffee through those
tiny plastic coffee stirrers they hand out airplanes that,
surprisingly, also happen to function as a straw.

As a result we see no perf gain from using FOR/PFOR.

I had hacked up a fix for this, described at in my blog post at

I'm opening this issue to get that work to a committable point.

So... I've worked out a new bulk-read API to address performance
bottleneck.  It has some big changes over the current bulk-read API:

  * You can now also bulk-read positions (but not payloads), but, I
     have yet to cutover positional queries.

  * The buffer contains doc deltas, not absolute values, for docIDs
    and positions (freqs are absolute).

  * Deleted docs are not filtered out.

  * The doc & freq buffers need not be "aligned".  For fixed intblock
    codecs (FOR/PFOR) they will be, but for varint codecs (Simple9/16,
    Group varint, etc.) they won't be.

It's still a work in progress...

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message