activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From oliverd <oliver.deck...@hotmail.com>
Subject Re: LeaseDatabaseLocker and parallel masters
Date Thu, 17 Apr 2014 15:23:51 GMT
The low memory situation in these tests is mainly caused by a high number of
client connections leading to a high number of transport threads (not using
NIO). I noticed that the marshalling cache used with openwire eats up quite
a lot of memory. This cache can be reduced. So, the test is running under
non-optimal conditions. You can call it a "bad day" test. But that's
typically good for finding edge cases. 

But anyhow, a certain overload and resulting memory shortage cannot be fully
prevented. Of course, one tries to size a system accordingly to reduce the
risk of running into resource shortages or low memory. But still, by
Murphy's law, you can end up in a low memory situation where long(er) GCs
may happen. When using a low value for lockAcquireSleepInterval then this
situation may happen even earlier (10s GC runs will cause this with the
default lockAcquireSleepInterval setting).

Coming back to the sequence id. You mentioned the broker reads its sequence
id from the store when it has the lock. That probably means the new master
reads it upon initialization when it takes over. It seems the sequence
generator is initialized with the value from store when the RegionBroker is
instantiated. 

Does the old master re-read its sequence id in the situation when it looses
the lock? Probably not.
Then, both masters could generate duplicate sequence ids. They might face a
duplicate key on insert to MSGS table, but only if the message with this
sequence id was not already consumed in the meantime (in case the old master
is on hold due to GC, it will probably generate outdated sequence ids).
What is the effect of generating duplicate sequence ids? Could this cause
follow up issues with clients?

For the JDBC store this means the effect of a parallel master situation is
that masters may generate duplicate sequence ids. Probably, it would help if
a master detects such a duplicate key situation and does either stop (if not
already triggered) or tries to resync the sequence generator.

I will try to think about other effects of having a parallel master, e.g.:
- "loss" of non-persistent messages (in case producer/consumer are connected
to different masters)
- stale persistent messages that the old master still accepted and did not
deliver before stopping
- not sure what duplicate sequence ids could mean for clients (does this
affect message replays, ack or tx handling in any way)
- duplicate delivery of persistent inflight messages (as you already
mentioned)




--
View this message in context: http://activemq.2283324.n4.nabble.com/LeaseDatabaseLocker-and-parallel-masters-tp4680368p4680391.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message