lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Babak Farhang <farh...@gmail.com>
Subject Re: Incremental Field Updates
Date Sat, 03 Apr 2010 05:49:56 GMT
Grant,

Reading your post a 3rd time, I see my "suggestion" is in fact the
approach you describe.  Sorry for being redundant.

-Babak

On Fri, Apr 2, 2010 at 11:25 PM, Babak Farhang <farhang@gmail.com> wrote:
>> I think they get merged in by the merger, ideally in the background.
>
> That sounds sensible. (In other words, we wont concern ourselves with
> roll backs--something possible while a "layer" is still around.)
>
> I've been thinking about this problem also. One approach discussed
> earlier in these mailing lists has been to somehow maintain a parallel
> index of the update-able of the fields in such a way that the docIds
> of the parallel index remain in sync with the "master" index. Mike
> McCandless and I were discussing some variants of this approach a few
> months back: http://markmail.org/message/uifz5v37k6qxxhvz?q=%22incremental+document+field+update%22+site:markmail%2Eorg&page=1&refer=ipebtbf24y7rleps
>  That approach involved the concept of mapping (chaining, if you will)
> internal docIds to view ids.  That docid mapping concept sounds
> analogous to this layer concept we are discussing now.
>
> I now think the parallel index approach may not be such a great idea,
> after all: it simply pushes the problem to the edge--the slave index.
> If we can solve update problem in the slave index, I reason, then
> shouldn't we also be able to solve the same update problem in the
> master index (and thereby remove the necessity of maintaining a
> (user-level) parallel index in the first place)?
>
> Which seems to align with the approach being discussed here..
>
> I imagine the "layers" being discussed here are somehow threaded by
> docId. That is, given a docId, you can quickly find it's "layers."  If
> so, then the docId mapping idea may be one way to thread these layers.
> (A logical document would be constructed by a chain of docIds, each
> overriding the previous for each field it defines (or deletes).  Such
> a construction would have to be "merge-aware" (perhaps using machinery
> similar to that used in LUCENE-1879) in order that it may maintain the
> docId chain.
>
> What do you think?
>
>
> On Fri, Apr 2, 2010 at 4:56 AM, Grant Ingersoll <gsingers@apache.org> wrote:
>>
>> On Apr 2, 2010, at 2:50 AM, Babak Farhang wrote:
>>
>>> [Late to this party, but thought I'd chime in]
>>>
>>> I think this "layer" concept is right on.  But I'm wondering about the
>>> life cycle of these layers.  Do layers live forever? Or do they
>>> collapse at some point? (Like, as I think was already pointed out,
>>> deletes are when segments are merged today.)
>>
>> I think they get merged in by the merger, ideally in the background.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-dev-help@lucene.apache.org
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message