lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tim Smith (JIRA)" <>
Subject [jira] [Commented] (LUCENE-4560) Support Filtering Segments During Merge
Date Mon, 19 Nov 2012 02:54:57 GMT


Tim Smith commented on LUCENE-4560:

My base requirement here is that this be an online process.
As such, the add indexes approach is really not useful as i see it, especially as it requires
2x disk space, as well as completely new index directories, it does not play well with upgrading
a user's existing index.

what i see as needed is the ability to gradually migrate indexes such that any individual
segment is itself consistent.
currently, merging of indexes can result in loss of indexed data or otherwise break consistency,
as in LUCENE-4557

it is 100% ok if all segments have not been processed as i can identify each segment's settings
at index open/search time, and optionally filter/search/read segments differently.

It is true that once you start using this SegmentMergeFilter, you pretty much have to keep
using it forever.
I don't see this as an issue as when dealing with supporting old indexes, you constantly have
to support migration of data that was indexed using old code. For instance, as time goes on,
my "MergeSegmentFilter" will do more, supporting migrating more and more old index formats/config
settings to the latest indexing format/settings.

At quick glance, FilteringCodec looks like it applies to writing new content, not reading
existing indexes?
Doesn't seem quite like that would do the trick here. I would need some way to have the index
writer wrap the codec for existing segments in order to inject my custom filtering that would
apply during merging. That would be logically identical to the patch provided, however would
potentially result in a much more complex patch.

> Support Filtering Segments During Merge
> ---------------------------------------
>                 Key: LUCENE-4560
>                 URL:
>             Project: Lucene - Core
>          Issue Type: Improvement
>            Reporter: Tim Smith
>         Attachments: LUCENE-4560.patch
> Spun off from LUCENE-4557
> It is desirable to be able to filter segments during merge.
> Most often, full reindex of content is not possible.
> Merging segments can sometimes have negative consequences when fields are have different
options (most restrictive option is forced during merge)
> Being able to filter segments during merges will allow gradually migrating indexed data
to new index settings, support pruning/enhancing existing data gradually
> Use Cases:
> * Migrate IndexOptions for fields (See LUCENE-4557)
> * Gradually Remove index fields no longer used
> * Migrate indexed sort fields to DocValues
> * Support converting data types for indexed data
> * and so on
> patch will be forthcoming

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message