lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Babak Farhang <>
Subject Re: Incremental Field Updates
Date Sat, 03 Apr 2010 05:25:27 GMT
> I think they get merged in by the merger, ideally in the background.

That sounds sensible. (In other words, we wont concern ourselves with
roll backs--something possible while a "layer" is still around.)

I've been thinking about this problem also. One approach discussed
earlier in these mailing lists has been to somehow maintain a parallel
index of the update-able of the fields in such a way that the docIds
of the parallel index remain in sync with the "master" index. Mike
McCandless and I were discussing some variants of this approach a few
months back:
 That approach involved the concept of mapping (chaining, if you will)
internal docIds to view ids.  That docid mapping concept sounds
analogous to this layer concept we are discussing now.

I now think the parallel index approach may not be such a great idea,
after all: it simply pushes the problem to the edge--the slave index.
If we can solve update problem in the slave index, I reason, then
shouldn't we also be able to solve the same update problem in the
master index (and thereby remove the necessity of maintaining a
(user-level) parallel index in the first place)?

Which seems to align with the approach being discussed here..

I imagine the "layers" being discussed here are somehow threaded by
docId. That is, given a docId, you can quickly find it's "layers."  If
so, then the docId mapping idea may be one way to thread these layers.
(A logical document would be constructed by a chain of docIds, each
overriding the previous for each field it defines (or deletes).  Such
a construction would have to be "merge-aware" (perhaps using machinery
similar to that used in LUCENE-1879) in order that it may maintain the
docId chain.

What do you think?

On Fri, Apr 2, 2010 at 4:56 AM, Grant Ingersoll <> wrote:
> On Apr 2, 2010, at 2:50 AM, Babak Farhang wrote:
>> [Late to this party, but thought I'd chime in]
>> I think this "layer" concept is right on.  But I'm wondering about the
>> life cycle of these layers.  Do layers live forever? Or do they
>> collapse at some point? (Like, as I think was already pointed out,
>> deletes are when segments are merged today.)
> I think they get merged in by the merger, ideally in the background.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message