activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Fran├žois Guillemette (JIRA) <j...@apache.org>
Subject [jira] Created: (AMQ-1658) Messages are sometimes skipped when using JDBC master/slave
Date Wed, 09 Apr 2008 17:38:33 GMT
Messages are sometimes skipped when  using JDBC master/slave
------------------------------------------------------------

                 Key: AMQ-1658
                 URL: https://issues.apache.org/activemq/browse/AMQ-1658
             Project: ActiveMQ
          Issue Type: Bug
          Components: Message Store
    Affects Versions: 5.0.0
         Environment: Windows Xp sp2, Java 6, MySql 5.0.51a (InnoDb table)
            Reporter: Fran├žois Guillemette
             Fix For: 5.1.0


Sometime, a (or some) message(s) hang in the queue while no consumer eat it. It happen more
often after failover.

Scenario:
2 brokers (jdbc master/slave), 2 consumers (with prefetch set to 1), 2 producers

Producers :
  ant producer -Durl="failover:(tcp://localhost:61618,tcp://localhost:61619)" -Ddurable=true
-Dmax=500
Consumer 1:
  ant consumer -Durl="failover:(tcp://localhost:61618,tcp://localhost:61619)" -Dmax=10000
-DclientId=c1
Consumer 2:
  ant consumer -Durl="failover:(tcp://localhost:61618,tcp://localhost:61619)" -Dmax=10000
-DclientId=c2

1 - Start the two brokers (one will be master, the other will be slave)
2 - Start the producers, consumers
3 - Wait a little,
4 - Kill the master -> slave become master
5 - Producers continue producing, consumers continue consuming
6 - After all producers finish their task, the consumer will finish consuming, and sometimes
there still messages left in the queue (in the database, and using JMX to see the state of
the queue).
7 - Restart a new broker, kill the master
8 - The messages will be consumed 

There is a race condition between the time the message is set with the broker sequence number
(RegionBroker.java in send method), and the time it is actually put in the database (DefaultJDBCAdapter.java
in doAddMessage method).

I have seen that sometimes message with higher sequence number are put in database before
a lower sequence number.  For example: 386 is put in the database before 385. If it is happening
when JDBCMessageStore is recovering the next message (lastMessageId is 384), then 386 will
be fetched and the lastMessageId will change to be 386. 385 is then put in the db but never
retrieved (stopping and restarting the broker will allow to retrieve the message because at
start the lastMessageId is -1).

I have synchronized the code inside the RegionBroker.send, and I don't have gaps anymore.
This is a workaround for us since we don't process a lot of message. But maybe a more elegant
solution is to set the brokerSequenceId in doAddMessage of JDBCAdapter (I may be wrong, I
didn't check if the brokerSequenceId is used elsewhere).


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message