activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: Need help with a message broker configuration
Date Thu, 24 Dec 2015 14:35:45 GMT
Queue browsing just involves HTTP interactions with the Jetty web server,
and can be scripted if the volume of queues makes manual browsing
impossible.

After you browsed those queues, did any of the very old KahaDB data files
get deleted?  That's the second question you're trying to answer by
browsing the queues: are there any messages that are past their expiration
date but haven't yet been expired by the broker, such that the pure act of
attempting to browse them triggers their deletion?  (And note that you
can't answer that question until you've browsed all queues that have
unconsumed messages, because a KahaDB data file can't be deleted until
every single message in it is deleted, so browsing "a lot of queues" might
trigger a file deletion or might not, but it doesn't indicate anything
about whether that's our problem or not.)

I recommend you make a copy of your KahaDB data files and put them
somewhere you can keep coming back them, then make a copy of those files
and use the second copy to stand up a new broker in a dev/test environment,
with no consumers and no producers, so you can do some experiments without
affecting your production broker.  Then purge every queue (again, this can
be scripted, with a little work) and see if all of your KahaDB files
(except the current one) get deleted.  (Keep in mind that the deletion
thread runs periodically, so there may be a small delay, depending on your
configuration.)  I expect that everything will get deleted once there are
no messages left in any queue, which would mean that KahaDB itself is
working correctly, and your problem is that you have old messages that
aren't being deleted.

Next, use your archived copy of the KahaDB data files to restore the test
broker to a state where all of those messages are present again, and browse
every queue.  As you go, note whether any message has either no
JMSExpiration header set, or a JMSExpiration header that's more than your
intended timeout (3 weeks?) after the creation date (JMSExpiration -
JMSTimestamp > 3 weeks), which will tell you whether any of your
assumptions about the behavior of your producers are invalid and that your
perceived problem may in fact be caused by invalid expectations.  When
you're done, see if any of the earliest KahaDB data files get deleted; if
it did, then your problem is that the broker isn't deleting expired
messages when there is no consumer for them.

Running those experiments should help to zero in on what's causing the
behavior you're seeing, and whether it's a bug in ActiveMQ or problems in
your software.

Also, keep in mind that the raw number of files is irrelevant; what should
be constant (after a warm-up period equal to the expiration interval all
producers are using) is the number of files *created* since the oldest live
file, of which some may have been deleted since then.  KahaDB can delete
any file it's done with, and there's a lot of randomness about what
messages a given file contains and therefore whether it can be deleted, so
expecting a constant rate of file deletions isn't realistic.  But since the
filenames have a one-up number in them, you can tell easily how many files
were created between the oldest file and the newest (current) file.  All of
that assumes a constant rate of message production and consumption; in a
real-world application such as your own, there may be variability of data
file creation rates due to the variability of message
production/consumption rates, so the "constant" rate of file creation is
very much an average, not a fixed value.

The other thing that should be constant is that the oldest data file should
never be substantially older than your largest expiration interval (if your
producers don't all use the same value).  All messages in the file should
be older than the last-modified timestamp of the file, so they should all
be candidates for expiration when the file passes the expiration interval.
If your oldest data file is not older than the max expiration interval,
then you don't have a problem.

What is the modification date/time of your oldest file?  Please make sure
you answer this question now, before you go running the experiments I
recommended, since your answer could eliminate the need for you to run
those experiments.

Tim

On Wed, Dec 23, 2015 at 1:59 PM, Shine <activemq@club.webhop.org> wrote:

> Hi Tim,
>
> there are about 2500 queues and it is not so easy to browse all queues and
> unconsumed messages.
>
> I took a look to a lot of queues with unconsumed messages and all messages
> has an (valid = future date) expiration date.
>
>
> Now 561 files / 17,9 GB
>
> regards
> Shine
>
>
>
> --
> View this message in context:
> http://activemq.2283324.n4.nabble.com/Need-help-with-a-message-broker-configuration-tp4705074p4705414.html
> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message