lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Adrien Grand (JIRA)" <>
Subject [jira] [Commented] (LUCENE-3892) Add a useful intblock postings format (eg, FOR, PFOR, PFORDelta, Simple9/16/64, etc.)
Date Thu, 09 Aug 2012 12:22:19 GMT


Adrien Grand commented on LUCENE-3892:


I prefixed with "MIN" because it is the minimum size the encoded buffer size must have to
be able to handle all cases. But I think you are right, "MAX" or "REQUIRED" would be clearer.

bq. I think readVIntBlock shouldn't be in ForUtil?

I'll move it back to BlockPackedPostingsReader.

{quote} Do we really need to write/write the 32 format.getId(), numBits into
the postings file header? I guess it's either that or ... store the float
acceptableOverheadRatio (eg using Float.floatToIntBits I guess) and
have some back-compat enforced in the logic in
PackedInts.fastestFormatAndBits... hmm.{quote}

I hesitated between these two approaches but I think writing all cases to the header is less
error-prone? Moreover it would allow us to change the logic of {{fastestFormatAndBits}} without
having to bump the version number.

{quote} Hmm ... MIN_DATA_SIZE is 147 (PACKED_SINGLE_BLOCK, bpv=3), but
BLOCK_SIZE is 128 ... so I guess this means if we ever pick that
format (because acceptableOverheadRatio allowed us to), we're
encoding/decoding those extra 19 unused ints right? (I was just
trying to understand why we alloc all the int[] to MIN_DATA_SIZE not

Exactly. The other problem is that we are also storing these unnecessary 19 values (but it
is not easy to fix since PACKED_SINGLE_BLOCK writes values in the low-order long bits first
(little endian)). Maybe we should make PACKED_SINGLE_BLOCK write values in the high-order
bits first and split byte encoders and decoders from the long ones (so that they have a lower

bq. ForUtil.getMinRequiredBufferSize seems like dead code?

I'll remove it.
> Add a useful intblock postings format (eg, FOR, PFOR, PFORDelta, Simple9/16/64, etc.)
> -------------------------------------------------------------------------------------
>                 Key: LUCENE-3892
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Michael McCandless
>              Labels: gsoc2012, lucene-gsoc-12
>             Fix For: 4.1
>         Attachments: LUCENE-3892-BlockTermScorer.patch, LUCENE-3892-blockFor&hardcode(base).patch,
LUCENE-3892-blockFor&packedecoder(comp).patch, LUCENE-3892-blockFor-with-packedints-decoder.patch,
LUCENE-3892-blockFor-with-packedints-decoder.patch, LUCENE-3892-blockFor-with-packedints.patch,
LUCENE-3892-bulkVInt.patch, LUCENE-3892-direct-IntBuffer.patch, LUCENE-3892-for&pfor-with-javadoc.patch,
LUCENE-3892-handle_open_files.patch, LUCENE-3892-non-specialized.patch, LUCENE-3892-pfor-compress-iterate-numbits.patch,
LUCENE-3892-pfor-compress-slow-estimate.patch, LUCENE-3892_for_byte[].patch, LUCENE-3892_for_int[].patch,
LUCENE-3892_for_unfold_method.patch, LUCENE-3892_pfor_unfold_method.patch, LUCENE-3892_pulsing_support.patch,
LUCENE-3892_settings.patch, LUCENE-3892_settings.patch
> On the flex branch we explored a number of possible intblock
> encodings, but for whatever reason never brought them to completion.
> There are still a number of issues opened with patches in different
> states.
> Initial results (based on prototype) were excellent (see
> ).
> I think this would make a good GSoC project.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message