lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mark harwood <markharw...@yahoo.co.uk>
Subject Re: Merge and commit behaviour - changed between 2.4 and 2.9?
Date Wed, 06 Oct 2010 15:24:41 GMT
> OK. I'll double check the reports.

The issue I'm tackling in this app is that multiple servers are applying 
equal-size volumes of updates in a distributed transaction and one means of node 
failure detection is based on detecting a server's failure to prepare a batch of 
updates in a timescale proportional to other nodes. Due to recent timeout 
failures after upgrading Lucene, the suggestion was that the way merge 
operations is handled changed from 2.4 to 2.9 and now some servers took longer 
to apply updates and reach prepareToCommit() stage based on some shift to 
foreground rather than background merges in some stages.That looks not to be the 
case.

The theory I'm working on now is that a call to IW.updateDocuments() is blocking 
on a server because ConcurrentMergeScheduler has exhausted maxMergeCount due to 
some ongoing large segment merges.It looks like this blocking logic is not new 
between 2.4 and 2.9 so maybe this has just appeared now simply as a result of 
the index reaching a particular size/state.

Cheers
Mark





----- Original Message ----
From: Michael McCandless <lucene@mikemccandless.com>
To: java-user@lucene.apache.org
Sent: Wed, 6 October, 2010 0:18:20
Subject: Re: Merge and commit behaviour - changed between 2.4 and 2.9?

Well, the merges continue running and altering the segments files, but
the changes done by the merges are not committed until the app calls
commit (or closes the IW).  Ie, no new segments_N file is written due
to merging after the last commit.

Mike

On Tue, Oct 5, 2010 at 6:45 PM, Mark Harwood <markharw00d@yahoo.co.uk> wrote:
> OK. I'll double check the reports.
> So presumably when merges occur outside of transaction control (post commit) 
>the post-merge update of the segments_N file is managed safely somehow?
> I can see the logic in continuing file changes outside of transaction control 
>because the document updates are safely committed (flushed as new segments) and 
>the merges are essentially just a background tidying of committed state that can 
>fail with no detrimental effect on the previous transaction.
>
>
> On 5 Oct 2010, at 23:31, Michael McCandless wrote:
>
>> Hmm that shouldn't be the case, I think?
>>
>> In both 2.4 and 2.9.x (and all later versions), neither .prepareCommit
>> nor .commit wait for merges.
>>
>> That said, if a merge happens to complete before you call those
>> methods, then it is in fact committed.
>>
>> Mike
>>
>> On Tue, Oct 5, 2010 at 1:13 PM, Mark Harwood <markharw00d@yahoo.co.uk> wrote:
>>> Having upgraded a live system from 2.4 to 2.9.3 the client is reporting a 
>>>change in merge behaviour that is causing some issues with their update 
>>>monitoring logic.
>>>
>>> The suggestion is that any merge operations now complete as part of the 
>>>IW.prepareCommit() call rather than previously when they ran in the background

>>>after IW.commit calls.
>>> This seems to make sense to me but I couldn't see any direct reference to this

>>>change in behaviour in changes.txt.
>>>
>>> Can anyone confirm this change between versions?
>>>
>>> Cheers,
>>> Mark
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


      

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message