lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Simon Willnauer (Updated) (JIRA)" <>
Subject [jira] [Updated] (LUCENE-3186) DocValues type should be recored in FNX file to early fail if user specifies incompatible type
Date Sat, 08 Oct 2011 20:51:30 GMT


Simon Willnauer updated LUCENE-3186:

    Attachment: LUCENE-3186.patch

attaching current state. I updated the latest patch to trunk and fixed several issues...

bq. You can remove the TODO in DocValuesConsumer.merge 


bq.Make sure we mark TypePromoter as @lucene.internal.


{quote}f we hit exc inside FixedStraightBytesImpl.merge, we are still
setting merge=true; is finish then called, up above? (At which
point we are going to try to write to the closed file).{quote}
no, this merge only is taken into account if the merge is successful. in the case of an exception
we don't call finish. I renamed it to hasMerged.

{quote}The jdocs for DocValuesConsumer.merge say "Merging segments out of
order is not supported", but you just mean segments must arrive in
sorted order right? (Ie, TieredMP merges non-sequential segments,
which we have been calling out-of-order merge).{quote}

this is bogus (leftover from branch) I removed it

{quote}Since DocValuesConsumer.merge ignores the incoming IndexDocValues
(a MultiIndexDocValues), do we even need to pass that in...?{quote} 
I fixed this too. I don't create this MIDV anymore and use the readers directly which I need
anyway for type promotion.

It looks like we don't handle the case of merging segs when a
given field is always FixedStraightBytes but the size had changed
from one seg to another? (We throw IAE in
FixedStraightBytesImpl.merge). Are there other cases that will
lead to "late binding" exc during merge?{quote}

FixedStraightBytes with different sizes are promoted to VarStraightBytes so this is not an

{quote}With this patch, does this mean we can type-promote on the fly?
Ie, if I make a SlowMRWrapper, and pull its perDocValues, we will
present the promoted type (across all segments) correctly? And
when the user looks up the value for a certain docID, we promote
it as needed?{quote}

actually yes! I implemented this in the patch but need to test it though.

the current patch has one problem, if you add three incompatible types A,B,C and Segs with
A & B get merged (and dropped) but segs containing C are not in that merge those the type
will be C eventually. but this seems ok to me though.

still some nocommits left but I think we are getting close 
> DocValues type should be recored in FNX file to early fail if user specifies incompatible
> ----------------------------------------------------------------------------------------------
>                 Key: LUCENE-3186
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/index
>    Affects Versions: 4.0
>            Reporter: Simon Willnauer
>            Assignee: Simon Willnauer
>             Fix For: 4.0
>         Attachments: LUCENE-3186.patch, LUCENE-3186.patch
> Currently segment merger fails if the docvalues type is not compatible across segments.
We already catch this problem if somebody changes the values type for a field within one segment
but not across segments. in order to do that we should record the type in the fnx fiel alone
with the field numbers.
> I marked this 4.0 since it should not block the landing on trunk

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message