activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tim Bain <tb...@alumni.duke.edu>
Subject Re: Several hundred messages occupies almost 30G storage
Date Sat, 28 Feb 2015 16:37:04 GMT
KahaDB can only delete a data file when every message in it can be
deleted.  If a single message is still needed, the whole file must be kept,
and it doesn't perform any sort of compaction.  And if the last message in
a file that must be kept (because of some other message) has an ack in the
next file, that next file must be kept itself.  This can theoretically
repeat forever if the messages happen to lay out just right in the data
files, so a single old unprocessed message can theoretically prevent KahaDB
from deleting any of its data files.  There was a recently-fixed bug where
the file with the ack was being improperly deleted, resulting in redelivery
of the acked messages on broker restart; see
https://issues.apache.org/jira/browse/AMQ-5542, which is fixed in 5.11.1
and 5.12.0.  So the version you're running won't recognize the chain of
files (if any) that need to be kept; with that fix, I'd expect you to hit
your limit even faster.

So your DLQish messages are in fact keeping alive any data files in which
they exist.  If they all came in as a batch, that would be just one file,
but since they're spread out over time, that's probably a decent number of
files.

So you could do as Tim suggested and make a separate KahaDB store for these
long-lived messages; that would solve this problem, but it's ultimately a
workaround.  Shrinking the size of each data file would help right now, but
once you upgrade to 5.11.1 or 5.12.0 it wouldn't be able to guarantee that
you didn't have to keep all the files; I'd focus on other options.

So the real question is, why are you keeping your DLQ-like messages for 5
days?  (This is probably the point where Art will chime in with "ActiveMQ
is not a database.")  You should be doing something with those messages
quickly, not keeping them around for ages.  If the messages get consumed
immediately, the KahaDB files won't stick around long, and your problem is
solved.  So figure out how to change your application logic so you don't
rely on messages staying on the broker for days; anything else is just a
workaround for this flaw in your application logic.

Tim
One more question: will the same thing happen if I switch to leveldb?

2015-02-28 22:53 GMT+08:00 Rural Hunter <ruralhunter@gmail.com>:

> I'm sorry I made a mistake. My storage is kahadb. We switched from leveldb
> to kahadb a while ago and I forgot that.
> Thanks for the links. Now understand what happened!
>
> 2015-02-28 19:03 GMT+08:00 Tim Robbins <tim.robbins@outlook.com>:
>
>> Hi,
>>
>> Two suggestions for you:
>>
>> 1. Try decreasing the logSize parameter for LevelDB. You’ve have a
>> greater number of smaller log files, and a greater chance of each log
file
>> being garbage-collected.
>> 2. With KahaDB, it’s possible to configure multiple KahaDB stores, and to
>> put your dead-letter type messages into a different store than everything
>> else to reduce overhead:
>> http://blog.garytully.com/2011/11/activemq-multiple-kahadb-instances.html
>> <
>> http://blog.garytully.com/2011/11/activemq-multiple-kahadb-instances.html
>> >
>> Unfortunately it doesn’t appear that this applies to LevelDB yet!
>>
>> Regards,
>>
>> Tim
>>
>> > On 28 Feb 2015, at 7:27 pm, Rural Hunter <ruralhunter@gmail.com> wrote:
>> >
>> > Hi,
>> >
>> > Activemq version 5.10.2, storage: leveldb.
>> >
>> > I have a queue which serves similiar function as dead letter queue. My
>> > application process messages from another queue and if the processing
>> > fails, it put the message into this queue. The messages are persistent
>> and
>> > average several KBs in size. My application processes many messages but
>> the
>> > failed message count is very small, less than 100 a day. I noticed
after
>> > the application running for several days, my activemq storage becomes
>> > almost full. I configured the storage to 30G. I checked the normal
>> queues
>> > and topics and there is no queue with large count of message. Most of
>> them
>> > are empty and some have only several messages. Only the failure message
>> > queue I metioned above has a few hundred messages(about 500) which are
>> > accumulated in several days.
>> >
>> > I have no idea what takes so much storage. I checked the storage files
>> and
>> > found there are many db-xxxx.log with timestamp almost through the
>> several
>> > days. They are not consequent though. Some of the db-xxx.log files are
>> not
>> > there. So the file list is like this:
>> > db-1000.log
>> > db-1001.log
>> > db-1003.log
>> > db-1004.log
>> > db-1005.log
>> > db-1008.log
>> > ...
>> > I suspect the failed messages are in those db-xxx.log files so I just
>> tried
>> > to clear the failed message queue. Right after that I found those old
>> > db-xxx.log disappeared and the storage usage went back to 2%. So it
>> seems
>> > clear that the about 500 failed messages took around 30G storage. But
>> how
>> > can it be? Those messages are very small in size and the total size of
>> the
>> > messsage should be no more than a few MBs.
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message