activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: Async Writer Thread Shutdown error adding a message on 5.15.2 broker
Date Mon, 08 Jan 2018 22:56:28 GMT
OK, this is a good start, thanks for working to narrow down the variables a
bit.

Do you get the "thread shut down" message on the first message enqueued, or
only after a certain number of messages are sent?

Have you set any options in your activemq.xml that might change the
broker's behavior regarding the temp store? If you can, please post your
configuration.

On Jan 8, 2018 7:57 AM, "neon18" <neon18@nngo.net> wrote:

I have tried out a few things to reduce the variables, one step at a time.

1. I upgrade the ActiveMQ client libraries to 5.15.2 to match the server.
This results in same issue (OutOfMemoryError on ActiveMQ 5.15.2 server).

2. I change the memoryUsage percentOfJvmHeap from 70% to 50%, restarted the
server but same. Issue.

3. I downgraded to ActiveMQ 5.14.5 server, and all is good (also when I
downgraded to ActiveMQ 5.12.2 - all is good)

There are not other errors in activemq.log.

I am using a non-persistent queue, but when memory hits the default the
memory usage percent of 70%, , the server *should* flush to the temp
storage. So I believe ActiveMQ 5.15 has a bug when flushing in-memory
(non-persistent) messages to the temp storage under
<amq_data_dir>/localhost/tmp_storage. I am guessing the temp storage is
using KahaDB underneath).


The temp store does indeed use KahaDB as its store type. Do you see KahaDB
data files (with a .log extension) in that directory? Are they recreated if
you stop the broker, delete them, and start the broker? And are any bytes
written to them after the broker's initial startup?

It seems as though either the thread that dumps messages out of memory and
into the temp store isn't started at all, or is dying at some point, and it
would be useful to know which it is. Can you attach JConsole to the broker
process as soon as possible after it starts (before sending any messages)
and look for the Async Writer Thread in the list of threads? If it's there,
then try to figure out when it dies (e.g. after the first message is sent).

With no messages (or with a few messages before JVM max memory), ActiveMQ
5.15.2 server is fine, it responds on the Web console, producer and consumer
clients can add to queue and dequeue messages fine. After the
OutOfMemoryError, the server (broker) no longer responds to any client and
the web console is NOT responsive.


All bets are off after an OOM in a Java process; the only safe option is to
restart the process. So that doesn't surprise me.

Regards,

Neon



--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message