incubator-lucy-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <>
Subject Re: real time updates
Date Sun, 15 Mar 2009 22:22:42 GMT

Marvin Humphrey wrote:

>>> These processes are forbidden from merging any files that pre-date  
>>> the
>>> establishment of "consolidate.lock".
>> Why?  It seems like it needs to merge segments created before it  
>> acquired
>> that lock (that's why it was launched).
> I was unclear.  By "these processes", I meant write processes  
> launched while
> the consolidation process is active.

OK makes sense.  You basically split the index segments into 2 sets;
the first set, only consolidator can change; the 2nd set, only your  
primary writer
can change.

>>> It finishes that task, commits, releases "write.lock", releases
>>> "consolidate.lock",then exits.
>> That, and update the master "segments" file to actually record the  
>> merge,
>> and incRef/decRef to delete files.
> By "commits", I was referring to updating the master file; I  
> probably should
> have used more precise language.  You're right that I'd left off file
> deletion.


>> But, what if while a large merge is happening, and enough segments  
>> have
>> been written to warrant a small merge to kick off & finish?
> That should work OK.  Write processes launched during consolidation  
> will be
> allowed to performing any mods or merges they like on segments that  
> were
> written *after* the the consolidator grabbed consolidate.lock.  They  
> just
> can't touch stuff that the consolidator might be operating on.

OK, so writer can choose to do merging if it wants (I was under the  
only consolidator could).

> The only thing I'm concerned about is the time that it takes to  
> carry recent
> deletions against merged segments forward so that they apply against  
> the new,
> consolidated segment.  The consolidator process has to obtain the  
> write.lock
> for that, blocking any new mods to the index.  Hopefully that  
> doesn't cause
> too big a blip.

Ahh, right.  Lucene also has a blip, but it's different because Lucene
will still accept added/deleted documents; but, one cannot reopen
a new realtime (LUCENE-1516) reader during the blip.

Hang on: does your writer process hold onto the write lock the whole
time it's open?  Or it only grabs it when it needs to commit a change?


View raw message