incubator-lucy-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marvin Humphrey <>
Subject Re: real time updates
Date Sun, 15 Mar 2009 22:14:56 GMT
On Sun, Mar 15, 2009 at 05:44:43PM -0400, Michael McCandless wrote:

> So eg you could merge 2 sets of segments at once (like Lucene)?


>> These processes are forbidden from merging any files that pre-date the
>> establishment of "consolidate.lock".
> Why?  It seems like it needs to merge segments created before it acquired
> that lock (that's why it was launched).

I was unclear.  By "these processes", I meant write processes launched while
the consolidation process is active.  

>> It finishes that task, commits, releases "write.lock", releases
>> "consolidate.lock",then exits.

> That, and update the master "segments" file to actually record the merge,
> and incRef/decRef to delete files.

By "commits", I was referring to updating the master file; I probably should
have used more precise language.  You're right that I'd left off file

> But, what if while a large merge is happening, and enough segments have
> been written to warrant a small merge to kick off & finish?

That should work OK.  Write processes launched during consolidation will be
allowed to performing any mods or merges they like on segments that were
written *after* the the consolidator grabbed consolidate.lock.  They just
can't touch stuff that the consolidator might be operating on.

The only thing I'm concerned about is the time that it takes to carry recent
deletions against merged segments forward so that they apply against the new,
consolidated segment.  The consolidator process has to obtain the write.lock
for that, blocking any new mods to the index.  Hopefully that doesn't cause
too big a blip.

Marvin Humphrey

View raw message