activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From neon18 <neo...@nngo.net>
Subject Re: Async Writer Thread Shutdown error adding a message on 5.15.2 broker
Date Wed, 10 Jan 2018 19:42:22 GMT
We run the broker with max heap of 4G and initial of 1G (-Xmx4G -Xms1G).
We use non-persistent messages on these particular queues (3 of them in this
test).
The number of messages sent to the broker in my last "flood gate" test was
around 40,000 (40k) in 5 minutes or about 8K msgs/min. After this flood of
messages, the producers send messages at a much much lower rate. I have
pretty much the factory default activemq.xml with
systemUsage/memoryUsage/percentOfJvmHeap=70 and queuePrefetch=20 on these 3
queues.

So I have seen two different scenarios when lots of non-persistent messages
are put on queue.
1. Async Writer Thread Shutdown errors (with no prior warnings/errors), then
OutOfMemoryErrors
2. INFO PListStore: ...tmp_storage initialized, ~10 seconds later WARN:
IOException: OutOfMemoryError: GC overhead limit exceeded ... ActiveMQ
Transport: tcp:..., then that repeats and other errors follow. Also there is
no warnings/errors prior to the tmp_storage init info log msg. FYI: the web
console was responsive until I saw the tmp_storage initialized (KahaDB) INFO
msg (~4.5 minutes into my test), then it stops responding. The last count of
messages on queues via web console is ~30K msgs under ActiveMQ 5.15.2
broker. Under the 5.14.5 broker, I was able to see the flood of ~40K msgs
added to the 3 queues in ~6 minutes.

In more controlled testing in the past 2 days, where I clear the AMQ_DATA
dir before each test run, I have not seen issue #1 (Async Writer Thread
Shutdown / OutOfMemoryError). I see issue #2 (OutOfMemoryError) a few
seconds after KahaDB tmp_storage is initialized, then the web console stop
responding and lots of OoMerrors and other errors in the activemq.log.

Running with ActiveMQ 5.14.5 and 5.12.2 brokers, we do not have any
OutOfMemoryErrors with this same load or higher load vs running under
ActiveMQ 5.15.2. Running with 5.15.2 broker it seems like there might be an
issue with throttling the producers of the queue when the JVM hit's the
configured memoryUsage (default of 70%).

Running on that thought, I did another test with
systemUsage/memoryUsage/percentOfJvmHeap=50 %, but same thing (except that
the OoM error occurs about 20 seconds after the tmp_storage init info log.

So, I ran the test again with systemUsage/memoryUsage to 20%, same thing,
except the OoM error occurs about 40 seconds after the tmp_storage init info
log. This time, I also monitored the memory percent used and temp memory
percent used via the web console. right after I see the tmp_storage init
info log, I can see memUsed=39% tempUsed=1%, ~10 seconds later memUsed=56%
tempUsed=2%, ~10 seconds later memUsed=69% tempUsed=2%, next refresh failed
and of course in the activemq.log I see the OutOfMemoryErrors and other
warnings and errors appearing in the log.

Also, I grepped my old logs for "Journal failed" and did see some results,
but they happen after the a few OutOfMemoryErrors, so I did not include them
in this thread.

I can pretty reliably recreate the problem in about 6 minutes (with a clean
amq_data_dir) running ActiveMQ 5.15.2 broker and no issues under 5.14.5 or
5.12.2 brokers.

Regards,

Neon



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Mime
View raw message