lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shai Erera (JIRA)" <>
Subject [jira] Commented: (LUCENE-1585) Allow to control how payloads are merged
Date Fri, 14 May 2010 03:48:42 GMT


Shai Erera commented on LUCENE-1585:

bq. Could we also use this for the term bytes itsself?

I think you'd want to use the same approach, yes. But I'm not sure I want to reuse the same
classes for that purpose, for several reasons:
* The classes have the word Payload all over the place - javadocs, names etc. And for a good
reason IMO - that's what they do.
* One is expected to include different PP for different Directory and Term, but to convert
the NumericField terms, I don't think one will use different PPs at all? I.e. a single TermsConverter
/ NumericFieldsTermConverter will be good for whatever Directory + Term?
* The sort of operation you suggest (converting terms) seems to be a one time op -- when I
migrate my indexes? PPP on the other hand (at least in my case) will be used whenever I call
addIndexes* so that I can process and rewrite the payloads of the incoming indexes.

So while both will do byte[] conversion, I think those are two separate tools. Your should
probably exist in a o.a.l.migration package or something, because it will be relevant to index
migration only? Or did I misunderstood you?

> Allow to control how payloads are merged
> ----------------------------------------
>                 Key: LUCENE-1585
>                 URL:
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index
>            Reporter: Michael Busch
>            Assignee: Shai Erera
>            Priority: Minor
>             Fix For: 3.1, 4.0
>         Attachments: LUCENE-1585_3x.patch, LUCENE-1585_3x.patch, LUCENE-1585_3x.patch,
LUCENE-1585_3x.patch, LUCENE-1585_3x.patch, LUCENE-1585_trunk.patch, LUCENE-1585_trunk.patch,
> Lucene handles backwards-compatibility of its data structures by
> converting them from the old into the new formats during segment
> merging. 
> Payloads are simply byte arrays in which users can store arbitrary
> data. Applications that use payloads might want to convert the format
> of their payloads in a similar fashion. Otherwise it's not easily
> possible to ever change the encoding of a payload without reindexing.
> So I propose to introduce a PayloadMerger class that the SegmentMerger
> invokes to merge the payloads from multiple segments. Users can then
> implement their own PayloadMerger to convert payloads from an old into
> a new format.
> In the future we need this kind of flexibility also for column-stride
> fields (LUCENE-1231) and flexible indexing codecs.
> In addition to that it would be nice if users could store version
> information in the segments file. E.g. they could store "in segment _2
> the term a:b uses payloads of format x.y".

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message