activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gary Tully (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-4122) Lease Database Locker failover broken
Date Fri, 22 Feb 2013 19:50:14 GMT

    [ https://issues.apache.org/jira/browse/AMQ-4122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13584596#comment-13584596
] 

Gary Tully commented on AMQ-4122:
---------------------------------

@SouNayi - thanks for the slq log.
>From looking at https://issues.apache.org/jira/secure/attachment/12570319/mysql.log, it
looks like a configuration problem.
node-h03-ap21 is obtaining a 5s lease that it renews every 10s. So there is a s period when
the lease is available to others.

It needs to obtain a 10 second lease and update it every 5 seconds. So that a second (slave)
broker always sees time > now when it attempts an update as part of an acquire.

You need:
{code}
<jdbcPersistenceAdapter ... lockKeepAlivePeriod="5000">
   ..
   <locker>
     <lease-database-locker lockAcquireSleepInterval="10000"/>
   </locker>
{code}
lockAcquireSleepInterval is the lease duration, lockKeepAlivePeriod is the lease renew period.
On a renew, the lease is extended by the lockAcquireSleepInterval (lease duration). So a master
is always (lockAcquireSleepInterval - lockKeepAlivePeriod) ahead with its lease.
In short, ensure: lockAcquireSleepInterval > lockKeepAlivePeriod.

Can you verify this.
I think it may also makes sense to add lease related attributes to this locker. leaseDuration,
leaseRenewPeriod so that it is a little more intuitive and obvious that the leaseDuration
> leaseRenewPeriod

                
> Lease Database Locker failover broken
> -------------------------------------
>
>                 Key: AMQ-4122
>                 URL: https://issues.apache.org/jira/browse/AMQ-4122
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.7.0
>         Environment: Java 7u9, SUSE 11, Mysql
>            Reporter: st.h
>            Assignee: Gary Tully
>             Fix For: 5.8.0
>
>         Attachments: activemq-kyle.xml, activemq.xml, activemq.xml, AMQ4122.patch, mysql.log
>
>
> We are using ActiveMQ 5.7.0 together with a mysql database and could not observe correct
failover behavior with lease database locker.
> It seems that there is a race condition, which prevents the correct failover procedure.
> We noticed that when starting up two instances, both instance are becoming master.
> We did several test, including the following and could not observe intended functionality:
> - shutdown all instances
> - manipulate database lock that one node has lock and set expiry time in distance future
> - start up both instances. both instances are unable to acquire lock, as the lock hasn't
expired, which should be correct behavior.
> - update the expiry time in database, so that the lock is expired.
> - first instance notices expired lock and becomes master
> - when second instance checks for lock, it also updates the database and becomes master.
> To my understanding the second instance should not be able to update the lock, as it
is held by the first instance and should not be able to become master.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message