lucene-solr-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "jayson.minard" <jayson.min...@gmail.com>
Subject Re: DirectUpdateHandler2 threads pile up behind scheduleCommitWithin
Date Tue, 12 May 2009 13:14:38 GMT

The change to make commit consistent is possible during the call-back, I was
trying to avoid that level of lock if possible but if it is the contract of
that callback then we have to maintain it.  

Would be nice to be able to do the snapshot first, then release the lock and
call the callback on the snapshot instead of holding a lock.  But it really
isn't built that way currently with only one callback event.


Mike Klaas wrote:
> 
> Hi Jayson,
> 
> Thanks, I'll take a look in the next few days.  The current patch  
> doesn't guarantee index consistency during post-commit callback hooks,  
> right?  This could be a problem for index replication.  (Incidentally,  
> I'm rather unfamiliar with the new java-based replication design.   
> Anyone care to comment on the implications?)
> 
> cheers,
> -MIke
> 
> On 10-May-09, at 10:54 AM, jayson.minard wrote:
> 
>>
>> Mike,
>>
>> I revamped the DirectUpdateHandler2 into DirectUpdateHandler3 in  
>> SOLR-1155,
>> probably ready enough for your review to see if locking makes sense  
>> for
>> current Lucene behavior.
>>
>> https://issues.apache.org/jira/browse/SOLR-1155
>>
>> --j
>>
>>
>> Mike Klaas wrote:
>>>
>>> On 7-May-09, at 10:36 AM, jayson.minard wrote:
>>>
>>>>
>>>> Does every thread really need to notify the update handler of the
>>>> commit
>>>> interval/threshold being reached, or really just the first thread  
>>>> that
>>>> notices should send the signal, or better yet a background commit
>>>> watching
>>>> thread so that no "foreground" thread has to pay attention at all.
>>>> That is
>>>> assuming they wouldn't need to block like they are now for a reason
>>>> I'm
>>>> likely unaware of...
>>>
>>> This is due to the way Lucene was designed (although recent
>>> improvements in Lucene mean we can do better here).  See the recent
>>> thread "Autocommit blocking adds?" on solr-user for a related
>>> discussion.
>>>
>>> As the person who first wrote the multi-threaded-ness of DUH2, I'd be
>>> very happy to promptly review any improvements made to it.
>>>
>>> -Mike
>>>
>>>
>>
>> -- 
>> View this message in context:
>> http://www.nabble.com/DirectUpdateHandler2-threads-pile-up-behind-scheduleCommitWithin-tp23431691p23472391.html
>> Sent from the Solr - Dev mailing list archive at Nabble.com.
>>
> 
> 
> 

-- 
View this message in context: http://www.nabble.com/DirectUpdateHandler2-threads-pile-up-behind-scheduleCommitWithin-tp23431691p23502368.html
Sent from the Solr - Dev mailing list archive at Nabble.com.


Mime
View raw message