activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pwalker <pwal...@navinet.net>
Subject Any way to protect from corruption being replicated in LevelDB?
Date Mon, 09 Dec 2013 16:14:40 GMT
Hey,

Quite new to ActiveMQ. We noticed an issue with replicated LevelDB where the
corrupted leveldb was being replicated between machines.

I had hoped that I would be able to use the configuration parameters for
verifyChecksums or paranoidChecks to help identify this scenario. Am I
missing something here?

To test this we did the following in our cluster of 3 brokers.

Cleared the level db directory on 2 brokers leaving them clean.
Left the corrupt DB (corrupt in that it's causing the exception below when
trying to expire messages) on 1 broker

I set both config parameters above to true but I see the broker failing over
with an IO Exception as below

INFO   | jvm 1    | 2013/12/06 05:12:20 |  INFO | Stopping
BrokerService[localhost] due to exception, java.io.IOException
INFO   | jvm 1    | 2013/12/06 05:12:20 | java.io.IOException
INFO   | jvm 1    | 2013/12/06 05:12:20 |       at
org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:39)
INFO   | jvm 1    | 2013/12/06 05:12:20 |       at
org.apache.activemq.leveldb.LevelDBClient.might_fail(LevelDBClient.scala:543)
INFO   | jvm 1    | 2013/12/06 05:12:20 |       at
org.apache.activemq.leveldb.LevelDBClient.might_fail_using_index(LevelDBClient.scala:974)
INFO   | jvm 1    | 2013/12/06 05:12:20 |       at
org.apache.activemq.leveldb.LevelDBClient.collectionCursor(LevelDBClient.scala:1270)
INFO   | jvm 1    | 2013/12/06 05:12:20 |       at
org.apache.activemq.leveldb.LevelDBClient.queueCursor(LevelDBClient.scala:1194)
INFO   | jvm 1    | 2013/12/06 05:12:20 |       at
org.apache.activemq.leveldb.DBManager.cursorMessages(DBManager.scala:708)
INFO   | jvm 1    | 2013/12/06 05:12:20 |       at
org.apache.activemq.leveldb.LevelDBStore$LevelDBMessageStore.recoverNextMessages(LevelDBStore.scala:741)

This happens on both servers as the corrupted db has been replicated over.

Having a quick look it doesn't seem like ReadOptions are used for this
function so no verifyChecksums flag is passed in right?

Was hoping that on initialization of the masterLevelDBClient we would be
able to validate the datastore at that point and if it was invalid fall
over?

Is that the expected behavior of those parameters or is it completely
distinct from the replication process as they are used in the non-replicated
leveldb adapter as well?



--
View this message in context: http://activemq.2283324.n4.nabble.com/Any-way-to-protect-from-corruption-being-replicated-in-LevelDB-tp4675294.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message