lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Babak Farhang <farh...@gmail.com>
Subject Re: Incremental Field Updates
Date Sat, 03 Apr 2010 05:25:27 GMT
> I think they get merged in by the merger, ideally in the background.

That sounds sensible. (In other words, we wont concern ourselves with
roll backs--something possible while a "layer" is still around.)

I've been thinking about this problem also. One approach discussed
earlier in these mailing lists has been to somehow maintain a parallel
index of the update-able of the fields in such a way that the docIds
of the parallel index remain in sync with the "master" index. Mike
McCandless and I were discussing some variants of this approach a few
months back: http://markmail.org/message/uifz5v37k6qxxhvz?q=%22incremental+document+field+update%22+site:markmail%2Eorg&page=1&refer=ipebtbf24y7rleps
 That approach involved the concept of mapping (chaining, if you will)
internal docIds to view ids.  That docid mapping concept sounds
analogous to this layer concept we are discussing now.

I now think the parallel index approach may not be such a great idea,
after all: it simply pushes the problem to the edge--the slave index.
If we can solve update problem in the slave index, I reason, then
shouldn't we also be able to solve the same update problem in the
master index (and thereby remove the necessity of maintaining a
(user-level) parallel index in the first place)?

Which seems to align with the approach being discussed here..

I imagine the "layers" being discussed here are somehow threaded by
docId. That is, given a docId, you can quickly find it's "layers."  If
so, then the docId mapping idea may be one way to thread these layers.
(A logical document would be constructed by a chain of docIds, each
overriding the previous for each field it defines (or deletes).  Such
a construction would have to be "merge-aware" (perhaps using machinery
similar to that used in LUCENE-1879) in order that it may maintain the
docId chain.

What do you think?


On Fri, Apr 2, 2010 at 4:56 AM, Grant Ingersoll <gsingers@apache.org> wrote:
>
> On Apr 2, 2010, at 2:50 AM, Babak Farhang wrote:
>
>> [Late to this party, but thought I'd chime in]
>>
>> I think this "layer" concept is right on.  But I'm wondering about the
>> life cycle of these layers.  Do layers live forever? Or do they
>> collapse at some point? (Like, as I think was already pointed out,
>> deletes are when segments are merged today.)
>
> I think they get merged in by the merger, ideally in the background.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-dev-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message