Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Received-SPF: neutral (athena.apache.org: local policy)
MIME-Version: 1.0
In-Reply-To: <AANLkTimO7zaaExR5acoXid3Ndh4s0c5hXuSnAO0uRZaY@mail.gmail.com>
References: <AANLkTikbunS95faEOy2d1ref_tboJrqDTvotupUXjqQz@mail.gmail.com>
	<AANLkTinUubgGH_4nNAQehq4eBw7CYfL0V6y+HS61Q4s1@mail.gmail.com>
	<AANLkTi=7q+F7a-1r6fxKqSH0Du8F_AY81UVRws7-39cg@mail.gmail.com>
	<AANLkTikrQkmWdsgrBBiM2oauANHaAkum0DBpoU0H0h2x@mail.gmail.com>
	<AANLkTi=ad84f8LGCP+O_z8kKs98tGjamrUwujx7qWPPr@mail.gmail.com>
	<AANLkTi=zQW0Gr_c6eMzTNEVhj=+VhV7bSWki3xT1irBS@mail.gmail.com>
	<AANLkTim2BPa9Eb9+f4ngt_yiDvutmqWRsEE49sigJBsC@mail.gmail.com>
	<AANLkTinfN-xFWZg_q_EB1Avtudq1gmHFTaR8nHQxSi8X@mail.gmail.com>
	<AANLkTimO7zaaExR5acoXid3Ndh4s0c5hXuSnAO0uRZaY@mail.gmail.com>
Date: Mon, 20 Dec 2010 10:33:11 -0500
Message-ID: <AANLkTimGh1z2NBxCChmA2WRxnyCztwtGZgWPbRoeuik7@mail.gmail.com>
Subject: Re: strange problem of PForDelta decoder
From: Michael McCandless <lucene@mikemccandless.com>
To: dev@lucene.apache.org
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

On Mon, Dec 20, 2010 at 5:49 AM, Li Li <fancyerii@gmail.com> wrote:
> =A0 I think random test is not sufficient.
> =A0 for normal situation, some branches are not executed. I tested
> http://code.google.com/p/integer-array-compress-kit/ with many random
> int arrays and it works. But when I use it in real indexing, when in
> optimize stage, it corrupted.
> =A0Because PForDelta will choose best numFrameBits and some bit such as
> 31 is hardly generated by random arrays. So I "force" the encoder to
> choose all possible numFrameBits to test all the decode1 ...decode32
> and find some bugs in it.

Good point -- we need to make sure we cover all numFrameBits.  And a
series of 128 random ints in a row will heavily bias for the high num
bits cases.  Maybe if we doing a better job w/ the random source to
try to target all numBits, w/ varying numbers of exceptions, etc...
I'll put a nocommit for this.

> =A0 =A0what's pfor2? using s9/s16 to encode exception and offset?

Yeah I just committed pfor2 this morning on the bulk branch.  You can
check it out from
https://svn.apache.org/repos/asf/lucene/dev/branches/bulkpostings

pfor2 came from the patch attached on
https://issues.apache.org/jira/browse/LUCENE-1410 by Hao Yan
(thanks!). It uses s16 for the exceptions (though, there's a bug
somewhere, because it fails the random test), and it takes a different
approachy for encoding exceptions.

> =A0 =A0In http://code.google.com/p/integer-array-compress-kit/ it's s9
> for NewPForDelta also have many bugs and also need test each branch to
> ensure it works well.

OK we should have a look at that one still.  We need to converge on a
good default codec for 4.0.  Fortunately it's trivial to take any int
block encoder (fixed or variable block) and make a Lucene codec out of
it!

Mike

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org