activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From neon18 <>
Subject Re: Async Writer Thread Shutdown error adding a message on 5.15.2 broker
Date Fri, 12 Jan 2018 23:12:17 GMT
I checked with IT guys and the disk I/O is not the issue, it can handle the
data rate just fine.

I did not have time in finding a way to slow the producers.

Building from 5.15.x/3 branch from source is not an option at this time. I
also believe [AMQ-6815 KahaDB checkpoint needs to fail fast in the event of
journal write failure] is not the culprit, althrough it might explain some
of the other errors that I see in the activemq.log

Does ActiveMQ baseline have any load test that maybe we can simulate this
scenario: e.g. send decent amount of messages to a non-persistent queue
(with a relatively small JVM max heap) to test this under ActiveMQ 5.14.x vs
5.15.x? That would be better to have some type of regression test for this
scenario inside ActiveMQ source code.

I have more info from another experiment I did today. I changed
systemUsage.memoryUsage to 1%. I see an INFO Usage Manager Memory Limit ..
reached on queue://... Producers will be throttled to the rate at which
messages are removed from this destination to prevent flooding it... a
couple times. Then ~200ms afterward, the tmp_storage initialized INFO log.
On the web console, I see Memory percent used to be 99% and 100% and Temp
percent used 3%. Eventually, the broker starting having errors in the Async
Writer Thread Shutdown. 

I did a similar test under ActiveMQ 5.14.5 broker (memoryUsage is at default
70%), and here the Memory used says under 70%, of particular note is that
the 5.14.5 broker Memory percent used drops alot, from 70% to 25% when it
flushes non-persistent message to the Temp storage (I can see Temp used
increase). I recall a similar scenario running under ActiveMQ 5.12.2, where
the Memory used decreases dramatically when it flushes non-persistent
message to the Temp storage on disk.



Sent from:

View raw message