Mailing-List: contact lucy-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: lucy-dev@lucene.apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Message-Id: <C96A899C-AC10-44C0-890A-78DCB8AF3276@mikemccandless.com>
From: Michael McCandless <lucene@mikemccandless.com>
To: lucy-dev@lucene.apache.org
In-Reply-To: <20090315221456.GA23339@rectangular.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v930.3)
Subject: Re: real time updates
Date: Sun, 15 Mar 2009 18:22:42 -0400
References: <65d3176c0903131248h4666e832k3910ad11135452fc@mail.gmail.com>
 <20090313230326.GA4020@rectangular.com>
 <69448C25-4E9A-4401-B455-B6A273BC0B01@mikemccandless.com>
 <20090315160825.GA22702@rectangular.com>
 <4B6E853E-B74B-4F3D-A3FD-CF47F2CF84FE@mikemccandless.com>
 <20090315221456.GA23339@rectangular.com>


Marvin Humphrey wrote:

>>> These processes are forbidden from merging any files that pre-date  
>>> the
>>> establishment of "consolidate.lock".
>>
>> Why?  It seems like it needs to merge segments created before it  
>> acquired
>> that lock (that's why it was launched).
>
> I was unclear.  By "these processes", I meant write processes  
> launched while
> the consolidation process is active.

OK makes sense.  You basically split the index segments into 2 sets;
the first set, only consolidator can change; the 2nd set, only your  
primary writer
can change.

>>> It finishes that task, commits, releases "write.lock", releases
>>> "consolidate.lock",then exits.
>
>> That, and update the master "segments" file to actually record the  
>> merge,
>> and incRef/decRef to delete files.
>
> By "commits", I was referring to updating the master file; I  
> probably should
> have used more precise language.  You're right that I'd left off file
> deletion.

OK

>> But, what if while a large merge is happening, and enough segments  
>> have
>> been written to warrant a small merge to kick off & finish?
>
> That should work OK.  Write processes launched during consolidation  
> will be
> allowed to performing any mods or merges they like on segments that  
> were
> written *after* the the consolidator grabbed consolidate.lock.  They  
> just
> can't touch stuff that the consolidator might be operating on.

OK, so writer can choose to do merging if it wants (I was under the  
impression
only consolidator could).

> The only thing I'm concerned about is the time that it takes to  
> carry recent
> deletions against merged segments forward so that they apply against  
> the new,
> consolidated segment.  The consolidator process has to obtain the  
> write.lock
> for that, blocking any new mods to the index.  Hopefully that  
> doesn't cause
> too big a blip.

Ahh, right.  Lucene also has a blip, but it's different because Lucene
will still accept added/deleted documents; but, one cannot reopen
a new realtime (LUCENE-1516) reader during the blip.

Hang on: does your writer process hold onto the write lock the whole
time it's open?  Or it only grabs it when it needs to commit a change?

Mike