Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (athena.apache.org: domain of tbain98@gmail.com designates
 209.85.223.182 as permitted sender)
MIME-Version: 1.0
Sender: tbain98@gmail.com
In-Reply-To: 
 <CAOe1oo-=BcOB7hyt+muaw0So0vOEFP7O57AQczzA1h0TYanTAA@mail.gmail.com>
References: 
 <CAOe1oo8eEyh_zwmFX+XEvJEQtDH9u+aVupkgPqVCq+KbavP1oA@mail.gmail.com>
	<BLU436-SMTP22159950481EE38E1159740EB120@phx.gbl>
	<CAOe1oo8TRNKFDVB77q0Efjmz2shWGnCZW83beMmsRG1q-7uhCw@mail.gmail.com>
	<CAOe1oo-=BcOB7hyt+muaw0So0vOEFP7O57AQczzA1h0TYanTAA@mail.gmail.com>
Date: Sat, 28 Feb 2015 09:37:04 -0700
Message-ID: 
 <CAPVVMa_357D_fyKar8BSJQsXAWn1caKF1cDCM4PCqTAJsGigCA@mail.gmail.com>
Subject: Re: Several hundred messages occupies almost 30G storage
From: Tim Bain <tbain@alumni.duke.edu>
To: ActiveMQ Users <users@activemq.apache.org>
Content-Type: multipart/alternative; boundary=047d7bdc123c5f56aa0510289881

--047d7bdc123c5f56aa0510289881
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

KahaDB can only delete a data file when every message in it can be
deleted.  If a single message is still needed, the whole file must be kept,
and it doesn't perform any sort of compaction.  And if the last message in
a file that must be kept (because of some other message) has an ack in the
next file, that next file must be kept itself.  This can theoretically
repeat forever if the messages happen to lay out just right in the data
files, so a single old unprocessed message can theoretically prevent KahaDB
from deleting any of its data files.  There was a recently-fixed bug where
the file with the ack was being improperly deleted, resulting in redelivery
of the acked messages on broker restart; see
https://issues.apache.org/jira/browse/AMQ-5542, which is fixed in 5.11.1
and 5.12.0.  So the version you're running won't recognize the chain of
files (if any) that need to be kept; with that fix, I'd expect you to hit
your limit even faster.

So your DLQish messages are in fact keeping alive any data files in which
they exist.  If they all came in as a batch, that would be just one file,
but since they're spread out over time, that's probably a decent number of
files.

So you could do as Tim suggested and make a separate KahaDB store for these
long-lived messages; that would solve this problem, but it's ultimately a
workaround.  Shrinking the size of each data file would help right now, but
once you upgrade to 5.11.1 or 5.12.0 it wouldn't be able to guarantee that
you didn't have to keep all the files; I'd focus on other options.

So the real question is, why are you keeping your DLQ-like messages for 5
days?  (This is probably the point where Art will chime in with "ActiveMQ
is not a database.")  You should be doing something with those messages
quickly, not keeping them around for ages.  If the messages get consumed
immediately, the KahaDB files won't stick around long, and your problem is
solved.  So figure out how to change your application logic so you don't
rely on messages staying on the broker for days; anything else is just a
workaround for this flaw in your application logic.

Tim
One more question: will the same thing happen if I switch to leveldb?

2015-02-28 22:53 GMT+08:00 Rural Hunter <ruralhunter@gmail.com>:

> I'm sorry I made a mistake. My storage is kahadb. We switched from leveld=
b
> to kahadb a while ago and I forgot that.
> Thanks for the links. Now understand what happened!
>
> 2015-02-28 19:03 GMT+08:00 Tim Robbins <tim.robbins@outlook.com>:
>
>> Hi,
>>
>> Two suggestions for you:
>>
>> 1. Try decreasing the logSize parameter for LevelDB. You=E2=80=99ve have=
 a
>> greater number of smaller log files, and a greater chance of each log
file
>> being garbage-collected.
>> 2. With KahaDB, it=E2=80=99s possible to configure multiple KahaDB store=
s, and to
>> put your dead-letter type messages into a different store than everythin=
g
>> else to reduce overhead:
>> http://blog.garytully.com/2011/11/activemq-multiple-kahadb-instances.htm=
l
>> <
>> http://blog.garytully.com/2011/11/activemq-multiple-kahadb-instances.htm=
l
>> >
>> Unfortunately it doesn=E2=80=99t appear that this applies to LevelDB yet=
!
>>
>> Regards,
>>
>> Tim
>>
>> > On 28 Feb 2015, at 7:27 pm, Rural Hunter <ruralhunter@gmail.com> wrote=
:
>> >
>> > Hi,
>> >
>> > Activemq version 5.10.2, storage: leveldb.
>> >
>> > I have a queue which serves similiar function as dead letter queue. My
>> > application process messages from another queue and if the processing
>> > fails, it put the message into this queue. The messages are persistent
>> and
>> > average several KBs in size. My application processes many messages bu=
t
>> the
>> > failed message count is very small, less than 100 a day. I noticed
after
>> > the application running for several days, my activemq storage becomes
>> > almost full. I configured the storage to 30G. I checked the normal
>> queues
>> > and topics and there is no queue with large count of message. Most of
>> them
>> > are empty and some have only several messages. Only the failure messag=
e
>> > queue I metioned above has a few hundred messages(about 500) which are
>> > accumulated in several days.
>> >
>> > I have no idea what takes so much storage. I checked the storage files
>> and
>> > found there are many db-xxxx.log with timestamp almost through the
>> several
>> > days. They are not consequent though. Some of the db-xxx.log files are
>> not
>> > there. So the file list is like this:
>> > db-1000.log
>> > db-1001.log
>> > db-1003.log
>> > db-1004.log
>> > db-1005.log
>> > db-1008.log
>> > ...
>> > I suspect the failed messages are in those db-xxx.log files so I just
>> tried
>> > to clear the failed message queue. Right after that I found those old
>> > db-xxx.log disappeared and the storage usage went back to 2%. So it
>> seems
>> > clear that the about 500 failed messages took around 30G storage. But
>> how
>> > can it be? Those messages are very small in size and the total size of
>> the
>> > messsage should be no more than a few MBs.
>>
>>
>

--047d7bdc123c5f56aa0510289881--