activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From uromahn <ulr...@ulrichromahn.net>
Subject Re: Potential Bug in Master-Slave with Replicated LevelDB Store
Date Mon, 20 Oct 2014 13:36:21 GMT
Ok, looks like the issue is back again.

The network issues have been fixed.
It is *not* a slow network - pings between VMs are less than 1ms.

I have not investigated the different throughput but wanted to focus on the
reliability of the replicated message store.

I made some configuration changes to the network connectors: I defined five
connectors per node (amq1-3).

Here is what I observed:
* When I launch one producer connecting to amq1 and one consumer connecting
to amq4 and send 100,000 messages, everything works fine
* When I launch five producer connecting to amq1 and five consumer
connecting to amq4 and send 100,000 messages, still fine
* When I launch 10 producer connecting to amq1 and 10 consumer connecting to
amq4 and send 100,000 messages, I can see the following:
  1. number of pending messages in the queue on amq1 is slowly but steadily
increasing, consumer on amq4 is still reading messages
  2. after about 70,000 to 80,000 messages amq1 suddenly stops working and
amq2 gets promoted to master. amq4 is still reading messages
  3. From that time on, the log of amq4 is filling up with the following
exceptions: 2014-10-20 13:11:43,227 | ERROR | Exception:
org.apache.activemq.transport.InactivityIOException: Cannot send, channel
has already failed: null on duplex forward of: ActiveMQTextMessage ... <dump
of message comes here>

Here is an excerpt of the log from amq1 at the time it got "demoted" to
slave:
2014-10-20 12:56:44,007 | INFO  | Slave has now caught up:
2607dbe5-e42a-44bf-8f90-6edf8caa8d87 |
org.apache.activemq.leveldb.replicated.MasterLevelDBStore |
hawtdispatch-DEFAULT-1
2014-10-20 13:11:42,535 | INFO  | Client session timed out, have not heard
from server in 2763ms for sessionid 0x2492d8210c30003, closing socket
connection and attempting reconnect | org.apache.zookeeper.ClientCnxn |
main-SendThread(uromahn-zk2-9775:2181)
2014-10-20 13:11:42,639 | INFO  | Demoted to slave |
org.apache.activemq.leveldb.replicated.MasterElector | ZooKeeper state
change dispatcher thread

(NOTE: 12:56 was the time the broker cluster was started. Between that time
and 13:11, I was running the various tests)

After that I can see a ton of exceptions and error messages saying that the
replicated store has stopped and similar. After some time, it looks the
broker amq1 has re-stabilized itself and reporting to have been started as
slave.

I don't know what exactly is going on, but it appears that something is
wrong with the replicated LevelDB which needs more investigation.



--
View this message in context: http://activemq.2283324.n4.nabble.com/Potential-Bug-in-Master-Slave-with-Replicated-LevelDB-Store-tp4686450p4686548.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message