lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LUCENE-3186) DocValues type should be recored in FNX file to early fail if user specifies incompatible type
Date Thu, 09 Jun 2011 15:06:59 GMT

    [ https://issues.apache.org/jira/browse/LUCENE-3186?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046602#comment-13046602
] 

Michael McCandless commented on LUCENE-3186:
--------------------------------------------

I think there are a few separate questions here...

Today, on doc values branch, if you mix up your doc values, ie a field
"foo" is at first indexed as a FLOAT_32 and then later you change your
mind and later docs are index field "foo" as BYTES_FIXED_STRAIGHT,
then this is bad news right now because everything will index fine,
you can close your IW, etc., but at some later time merges will hit
unrecoverable exceptions.  You'll have no choice but to fully rebuild
the index, which is rather awful.

However, this is true even for cases you would expect to work, eg say
"foo" was BYTES_FIXED_STRAIGHT but then later you decided you will
want to sort on this field and so you use BYTES_FIXED_SORTED.  (Simon:
this also results in exception I think...?).  Ideally we should do the
right thing here and "upgrade" the BYTES_FIXED_STRAIGHT to
BYTES_FIXED_SORTED (I think) -- Simon is there an issue open for this?

So, I think the first question here is: which cases should be merged
"properly" and which should be considered "an error"?  Probably we have
to work out the full matrix...

Then the second question is, for the "error" cases (if any!),
can/should we detect this up front, as you're indexing?

Then third question is, if we want to detect up front, do we do that
w/ fnx file or do we do that on init of IW (= no index change).


> DocValues type should be recored in FNX file to early fail if user specifies incompatible
type
> ----------------------------------------------------------------------------------------------
>
>                 Key: LUCENE-3186
>                 URL: https://issues.apache.org/jira/browse/LUCENE-3186
>             Project: Lucene - Java
>          Issue Type: Improvement
>          Components: core/index
>    Affects Versions: 4.0
>            Reporter: Simon Willnauer
>            Assignee: Simon Willnauer
>             Fix For: 4.0
>
>
> Currently segment merger fails if the docvalues type is not compatible across segments.
We already catch this problem if somebody changes the values type for a field within one segment
but not across segments. in order to do that we should record the type in the fnx fiel alone
with the field numbers.
> I marked this 4.0 since it should not block the landing on trunk

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message