activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lichtin <lich...@yahoo.com>
Subject Re: KAHADB clean up old log files
Date Fri, 23 Sep 2016 08:05:23 GMT
https://issues.apache.org/jira/browse/AMQ-6440


Tim Bain wrote
> I was going to write a JIRA enhancement request for that when I had some
> free time, but if you have time to write it before I get to it, that would
> be fine.  Please put the link here if you do.
> 
> On Sep 22, 2016 9:00 AM, "lichtin" &lt;

> lichtin@

> &gt; wrote:
> 
>> Changing trace levels and using the debugger is not always an option in
>> production.
>> I'm wondering why we could not revive the "KahaDBJournalReader" tool
>> from https://issues.jboss.org/browse/MB-756 and ship it with ActiveMQ.
>>
>>
>> Tim Bain wrote
>> > Alternatively, you all could run the KahaDB file analysis tool (
>> > http://activemq.apache.org/why-do-kahadb-log-files-
>> remain-after-cleanup.html)
>> > with a debugger attached to try to see what destination has the
>> message(s)
>> > that are keeping 980 and 1438 alive.  (You say that 1328 shouldn't be
>> > there, but the log shows that it's there because it contains acks for
>> > messages in 980, so it should be there, at least until 980 is deleted.)
>> >
>> > Tim
>> >
>> > On Sep 22, 2016 6:39 AM, "Steve Hill" <
>>
>> > shill@.co
>>
>> > > wrote:
>> >
>> > After the comment about the TOPIC I decided to take a quick look at
>> that
>> > in
>> > the web interface (just to make sure nothing had gone awry)  In deed
>> one
>> > of
>> > our "queues" had been created as a topic, not sure why as we use the
>> same
>> > code to connect to the queues however once removed most of the data
>> files
>> > vanished.
>> >
>> > Under the topics there are several activemq.advisory.xyz topics. Could
>> > these be the source of the remaining kahadb log files that still
>> remain?
>> >
>> > I am now down to three data files that should not be there. (db-980,
>> > db-1329, and db-1438)
>> >
>> >
>> > -rw-r--r--. 1 activemq activemq 33554432 Aug  8 14:00 db-980.log
>> > -rw-r--r--. 1 activemq activemq   527645 Sep 19 19:11 db-1329.log
>> > drwxr-xr-x. 5 activemq activemq     4096 Sep 20 14:57 ..
>> > -rw-r--r--. 1 activemq activemq        8 Sep 21 13:17 lock
>> > -rw-r--r--. 1 activemq activemq 33553518 Sep 21 21:52 db-1438.log
>> > drwxr-xr-x. 2 activemq activemq    12288 Sep 22 08:20 .
>> > -rw-r--r--. 1 activemq activemq  3287296 Sep 22 08:34 db.redo
>> > -rw-r--r--. 1 activemq activemq 12357632 Sep 22 08:34 db.data
>> > -rw-r--r--. 1 activemq activemq 33554432 Sep 22 08:34 db-1456.log
>> >
>> > From kahadb logs
>> >
>> >  2016-09-22 08:31:07,503 [eckpoint Worker] TRACE MessageDatabase
>> >     - gc candidates: [1329]
>> >  2016-09-22 08:31:07,504 [eckpoint Worker] TRACE MessageDatabase
>> >     - ackMessageFileMap: {1329=[980], 1438=[], 1456=[]}
>> >  2016-09-22 08:31:07,504 [eckpoint Worker] TRACE MessageDatabase
>> >     - not removing data file: 1329 as contained ack(s) refer to
>> referenced
>> > file: [980]
>> >  2016-09-22 08:31:07,505 [eckpoint Worker] TRACE MessageDatabase
>> >     - Not yet time to check for compaction: 7 of 10 cycles
>> >  2016-09-22 08:31:07,505 [eckpoint Worker] DEBUG MessageDatabase
>> >     - Checkpoint done.
>> >
>> > Thanks!
>> > Steve.
>> >
>> >
>> > On 9/22/16 8:14 AM, Steve Hill wrote:
>> >
>> >> Chris/Tim:
>> >>
>> >> Thanks for the ideas.  Unfortunately neither are the case in our
>> >> situation.  The web client shows 0 messages in any of our queues and
>> we
>> >> are
>> >> not using topics.  We did set up the remote debug port and downloaded
>> the
>> >> source code if there is anything useful to further troubleshoot that
>> >> could
>> >> be gleaned from there?
>> >>
>> >> Thanks!
>> >> Steve.
>> >>
>> >>
>> >> The most likely place for you to have old messages laying around is in
>> >> the
>> >> DLQ.  The next most likely place is on a topic for an offline (and
>> never
>> >> coming back) durable subscriber.
>> >>
>> >> Tim
>> >>
>> >> On Sep 20, 2016 2:02 PM, "Christopher Shannon" <
>> >>
>>
>> > christopher.l.shannon@
>>
>> >> wrote:
>> >>
>> >> Do you have old messages laying around?  KahaDB can't clean up old
>> >> journal
>> >> files if there are messages scattered throughout the logs.  Even a
>> single
>> >> unacked message means that a journal file (and the future files) can't
>> be
>> >> cleaned up. This situation tends to happen either when there are old
>> >> messages not being acked quick enough or there are different
>> destinations
>> >> with different speeds.  Ie if one queue has a very slow message rate
>> and
>> >> another queue has a high message rate.
>> >>
>> >> Assuming there isn't some sort of bug with the GC process then one way
>> to
>> >> help lower the usage would be to split up destinations into different
>> >> stores using multiKahaDB.  Take a look at Gary's blog post on this:
>> >> http://blog.garytully.com/2011/11/activemq-multiple-
>> kahadb-instances.html
>> >>
>> >>
>> >> On Tue, Sep 20, 2016 at 1:04 PM, Hill, Steve <
>>
>> > SHill@
>>
>> > > wrote:
>> >>
>> >> Hi fellow ActiveMQ users!
>> >>>
>> >>> I have searched all over the internet and can not find a solution to
>> a
>> >>> problem we are experiencing where old log files are not removed from
>> >>> KAHADB.  Needless to say we uncovered this problem when we exceeded
>> the
>> >>> size of our persistent store in production :-)
>> >>>
>> >>> We are using client side acknowledgement for all reads from the Queue
>> >>> and
>> >>> have restarted all clients and also ActiveMQ numerous times, yet the
>> >>> messages do not get removed.  We are also using the network connector
>> to
>> >>> move messages to the queue that has an active consumer.  We were
>> using
>> >>> 5.13.0 but moved to 5.14.0 on the hope it might address this problem.
>> >>>
>> >>> Follows are some more detailed messages; does anyone have an idea how
>> we
>> >>> can resolve this problem?
>> >>>
>> >>> KAHADB logging is enabled and we see the following messages in the
>> log
>> >>> file.
>> >>>
>> >>>
>> >>>  2016-09-20 08:59:10,207 [eckpoint Worker] TRACE MessageDatabase
>> >>>       - not removing data file: 1278 as contained ack(s) refer to
>> >>> referenced file: [1278, 1277]
>> >>>
>> >>>  2016-09-20 08:59:10,207 [eckpoint Worker] TRACE MessageDatabase
>> >>>       - not removing data file: 1279 as contained ack(s) refer to
>> >>> referenced file: [674]
>> >>>
>> >>>  2016-09-20 08:59:10,208 [eckpoint Worker] TRACE MessageDatabase
>> >>>       - not removing data file: 1280 as contained ack(s) refer to
>> >>> referenced file: [691]
>> >>>
>> >>>  2016-09-20 08:59:10,208 [eckpoint Worker] TRACE MessageDatabase
>> >>>       - not removing data file: 1281 as contained ack(s) refer to
>> >>> referenced file: [699]
>> >>>
>> >>>  2016-09-20 08:59:10,209 [eckpoint Worker] TRACE MessageDatabase
>> >>>       - not removing data file: 1282 as contained ack(s) refer to
>> >>> referenced file: [699]
>> >>>
>> >>>  2016-09-20 08:59:10,210 [eckpoint Worker] TRACE MessageDatabase
>> >>>       - not removing data file: 1283 as contained ack(s) refer to
>> >>> referenced file: [705]
>> >>>
>> >>>  2016-09-20 08:59:10,210 [eckpoint Worker] TRACE MessageDatabase
>> >>>       - not removing data file: 1284 as contained ack(s) refer to
>> >>> referenced file: [711]
>> >>>
>> >>>  2016-09-20 08:59:10,211 [eckpoint Worker] TRACE MessageDatabase
>> >>>       - not removing data file: 1288 as contained ack(s) refer to
>> >>> referenced file: [674]
>> >>>
>> >>>  2016-09-20 08:59:10,211 [eckpoint Worker] TRACE MessageDatabase
>> >>>       - not removing data file: 1289 as contained ack(s) refer to
>> >>> referenced file: [790]
>> >>>
>> >>>  2016-09-20 08:59:10,212 [eckpoint Worker] TRACE MessageDatabase
>> >>>       - not removing data file: 1290 as contained ack(s) refer to
>> >>> referenced file: [1290, 1278]
>> >>>
>> >>>  2016-09-20 08:59:10,213 [eckpoint Worker] TRACE MessageDatabase
>> >>>       - not removing data file: 1291 as contained ack(s) refer to
>> >>> referenced file: [790]
>> >>>
>> >>> The files at this point are months old.
>> >>>
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 14 21:23 db-674.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 16 01:57 db-691.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 17 02:10 db-699.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 18 03:55 db-705.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 19 01:32 db-711.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 25 15:15 db-790.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 26 10:32 db-858.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 27 00:19 db-865.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 27 03:03 db-866.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 28 01:45 db-877.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 29 01:18 db-888.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 30 02:10 db-899.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Jul 31 01:11 db-904.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug  1 00:19 db-909.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug  1 04:42 db-910.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug  2 01:07 db-918.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug  3 01:38 db-930.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug  4 00:19 db-939.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug  4 03:16 db-940.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug  5 00:23 db-949.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug  6 02:42 db-960.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug  7 02:18 db-968.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug  8 00:44 db-975.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug  8 14:00 db-980.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug  8 15:41 db-981.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug  9 01:43 db-1051.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 11 00:20 db-1066.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 11 03:01 db-1067.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 12 00:19 db-1074.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 12 03:05 db-1075.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 13 00:19 db-1082.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 13 03:04 db-1083.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 14 01:06 db-1093.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 15 01:18 db-1111.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 16 02:38 db-1126.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 17 00:19 db-1137.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 17 02:32 db-1138.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 18 00:21 db-1155.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Aug 19 00:51 db-1167.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq 33554432 Sep 19 13:42 db-1277.log
>> >>>
>> >>> -rw-r--r--. 1 activemq activemq   597139 Sep 19 14:59 db-1279.log
>> >>>
>> >>> Thanks!
>> >>>
>> >>> Steve Hill
>> >>>
>> >>> ARMADA
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> The information contained in this e-mail and any attachments is
>> >>> confidential and
>> >>> intended only for the recipient. If you are not the intended
>> recipient,
>> >>>
>> >> the
>> >>
>> >>> information contained in this message may not be used, copied, or
>> >>> forwarded to
>> >>> third parties or otherwise distributed for any other purpose. Please
>> >>> notify the
>> >>> sender if you received this e-mail in error and delete the e-mail and
>> >>> its
>> >>> attachments promptly.  Nothing in this e-mail may be used or deemed
>> to
>> >>> form the
>> >>> basis of a contractual or any other legally binding obligation unless
>> >>> separately
>> >>> confirmed in writing by an authorized representative of ARMADA.
>> >>>
>> >>>
>> >>
>> >>
>> >>
>>
>>
>>
>>
>>
>> --
>> View this message in context: http://activemq.2283324.n4.
>> nabble.com/KAHADB-clean-up-old-log-files-tp4716661p4716804.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>





--
View this message in context: http://activemq.2283324.n4.nabble.com/KAHADB-clean-up-old-log-files-tp4716661p4716824.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.

Mime
View raw message