activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Mueller (JIRA)" <>
Subject [jira] Updated: (AMQ-2317) Duplicate messages with transacted persistent messages during JDBC Master/Slave failover
Date Sun, 05 Jul 2009 14:06:07 GMT


Daniel Mueller updated AMQ-2317:

    Attachment: FailoverTransactionalTest.patch

testcase in patch form against activemq-core that demonstrates the problem.
Adds a new Testcase org.apache.activemq.transport.failover.FailoverTransactionalTest and patches
the pom with new dependencies (DBCP & MySQL JDBC Connector).
Was taken against rev. 791238

> Duplicate messages with transacted persistent messages during JDBC Master/Slave failover
> ----------------------------------------------------------------------------------------
>                 Key: AMQ-2317
>                 URL:
>             Project: ActiveMQ
>          Issue Type: Bug
>    Affects Versions: 5.3.0
>         Environment: OS: MacOS X  10.5.7 MacBook Core 2 Duo 2 Ghz
> DBMS: MySQL 5.0.83 (through macports), SQLServer 2005 (in VMWare), other suspected but
not thouroughly tested (including HSQL)
> All observations are against trunk: rev 790957 (2009-07-03 23:07:04 +0700 (Fri, 03 Jul
2009)) (fuse progress and ActiveMQ 5.2.0 seem to have the same problem though)
>            Reporter: Daniel Mueller
>            Priority: Critical
>         Attachments: FailoverTransactionalTest.patch
> There is a race condition somewhere in the transaction/replay code involving failovers
of JDBC only Master/Slave configurations.
> Observed problems:
> If messages are sent to a master broker in one transaction, and during the time of the
transaction the master fails over to the slave, then the messages seem to be replayed twice
(both database holds duplicates (see query at the end) and the broker answer with message
count containing duplicates).
> Severity: 
> If the clients are connected to the new master and start consuming, the broker will not
deliver dups. The dups will be delivered though, if there is another failover (a common case
for system upgrades). It seems like a single consumer will not get duplicates, even if it
fails over again to new broker, but if the consumer is restarted, it loses it's state as well,
and subsequently gets the duplicates delivered.
> Attached is a testcase that demonstrates the problem. It shows that with a single producer
doing commits after each send, it creates on additional message in the broker with a duplicate
MSGID_SEQ. If everything is committed in one transaction, then every single message in the
transaction is duplicated (and not only the ones before the failover occurred).
> The testcase uses an external MySQL instance though, and needs the DBCP and the MySQL
JDBC connector on the classpath (the pom is patched in the attached file to resolve that automatically).
> Out of the 6 tests, the following almost always fail on my machine:
> testProducer_MasterFailoverByShutdown_AtRandomTimes_CommitPerMessage  (expected <6000>,
but was <6001>)
> testProducer_MasterFailoverByShutdown_AtRandomTimes_OneCommit  (expected <6000>,
but was <12000>)
> Rarely (3-5% of the cases) this one also fails:
> testProducer_MasterFailoverByShutdown_SingleMsgCommit_AfterCommit  (expected <500>,
but was <501>)
> Other observations made:
> 1) The problem seems to be a race condition because while trying to find the cause through
debugging, the problem disapeared when setting a break point in TransactionInfo.visit(line:100).
The race condition is met on my machine (specs above) basically all the time without interaction
(from maven, on the shell with a build, inside eclipse debugged and normal).
> 2) It seems that TransactionBroker.commitTransaction(line:100) is called once with duplicated
synchronizations (2x size). On the other hand MemoryTransactionStore$Tx(line:109) is called
twice with the correct amount first, and later a doubled amount.
> 3) The problem is not reproducible with Kaha, the problem is related to JDBC.
> 4) It might be possible to have the testcase fail reliably with one of Derby/HSQL/H2,
but I didn't investigate.
> 5) The testcase is not exactly very pretty, but it does show the problem ;)
> 6) The attached testcase is a patch against activemq-core.
> 7) The tests can be executed directly (in bash) with:
> env MAVEN_OPTS="$MAVEN_OPTS -Xmx800M" mvn -Dtest=org.apache.activemq.transport.failover.FailoverTransactionalTest
> 8) For MySQL the following should work: 
>       MSGID_PROD
>      ,MSGID_SEQ
>   FROM activemq_msgs
> 9) if you need the my.cnf for the database, I can attach that as well.
> 10) The tables are correctly created as InnoDB
> I think that's it...

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message