activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From hakanj <hakan.johans...@jeppesen.com>
Subject Need help investigating committed messages resurfacing after restart.
Date Mon, 23 Oct 2017 12:00:36 GMT
We have a very strange problem with our ActiveMQ broker that happens when we
restart it after
running it for several months. We have not been able to reproduce it
in-house. It only happens in
production at our customer's site. At this time we have seen this issue at
least three times.

What happens is that some very old (~1 month) messages come back from the
dead when we restart the
broker. The last time this happened we saw ~1600 resurrected messages. We
can see in our logs that
these messages have already been processed. There have been more messages
written to the same
queue, but these did not come back to life.

At the time of broker shutdown the problematic queue was empty with no
consumers or producers
connected (they were stopped before the broker was stopped).

We tried deleting the kahadb directory when this happened the last time, but
that didn't help.

The log does not contain any errors during startup. We get a couple of rows
similar to
"Recovery replayed 170629 operations" in the log, but this is normal as far
as I know. If I am
wrong about this being normal, please let me know.

---

We have a mix of applications that communicate using an ActiveMQ broker.
The Java applications, like Wildfly, use the OpenWire protocol.
The C++ applications use the AMQP protocol, with "transport.transformer=jms"
setting in
"activemq.xml".

The C++ applications use transactional sessions to consume messages from the
broker. The
problematic queues are consumed by one of these C++ applications. The
messages are produced by one
of the Java/Wildfly applications.

Most, but not all, messages that are sent through the broker are
non-persistent. The messages on
the problem queues are persistent.

We use queues for everything. Topics are not used.

There is only a single ActiveMQ broker instance.

ActiveMQ version: 5.14.5.

---

We have tried to trigger this error in-house, but without any success. We
have only seen it in our
customer's production environment.

Our own hypothesis is that the kahadb data file that contains the acks is
deleted, but not the file
containing the actual messages. After restart it then looks like the
messages were never sent. Even
if this is true we have no idea how it can happen. All our manual tests show
that ActiveMQ does the
right thing. There must be some special case that we cannot find.

Does anyone have an idea what could be the cause of this issue?
Has anyone seen anything like this before?
Any ideas on how get a reproducible case?




--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Mime
View raw message