lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Renaud Delbru (JIRA)" <>
Subject [jira] Commented: (LUCENE-1410) PFOR implementation
Date Tue, 23 Feb 2010 13:08:28 GMT


Renaud Delbru commented on LUCENE-1410:

I am reporting here some experiments I performed over the past weeks while I was trying to
improve the FOR implementation. 
I re-implement the FOR algorithms by getting rid of the IntBuffer and working directly with
the byte array. I have implemented multiple variants, such as one that directly performs transformation
over bytes to create the uncompressed data (what I call byte-level in the next), and one that
first convert bytes into integers, and then performs transformation on integers to create
the uncompress data (what I call integer-level in the next). The last one is very similar
to your original FOR implementation, but without the IntBuffer.

I think these results can be of interest for you, especially to optimise certain cases (byte
level manipulation for certain cases such as bit frame of 2, 4 or 8 seems more suitable).
I have attached a file containing a summary of the results for space consideration. I can
provide you the raw results, and the code if you would like to test it on your side.
However, I get very different results if I perform the benchmarks on a 64 bits OS or 32 Bits
OS (on a same computer, IBM T61, the only difference is that on one computer Ubuntu 9.10 32
bits is installed, on the other one it is Ubuntu 9.10 64 bits).

I am a little bit puzzled by these results. I thought that removing the IntBuffer and working
directly with the byte array will be faster (as I noticed in other compression algorithms,
such as GroupVarInt). The IntBuffer you are currently using is a view on a byte buffer. It
therefore does the conversion between byte to int, plus it does several checks (if the index
is in the range of the buffer) and function calls.
But it seems that with FOR this does not make too much a difference on large integers (>
8 bits). Moreover, I observe a decrease of performance on 64 bits OS.
Maybe, you have an idea about the difference in behavior.

> PFOR implementation
> -------------------
>                 Key: LUCENE-1410
>                 URL:
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Other
>            Reporter: Paul Elschot
>            Priority: Minor
>         Attachments: autogen.tgz, LUCENE-1410-codecs.tar.bz2, LUCENE-1410b.patch, LUCENE-1410c.patch,
LUCENE-1410d.patch, LUCENE-1410e.patch, TermQueryTests.tgz,,,
>   Original Estimate: 21840h
>  Remaining Estimate: 21840h
> Implementation of Patched Frame of Reference.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message