activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Timothy Bish <tabish...@gmail.com>
Subject Re: KahaDB log files not getting cleaned up -- Why?
Date Fri, 13 Dec 2013 18:45:04 GMT
On 12/13/2013 01:02 PM, Andy Boothe [WCG] wrote:
> Timothy,
>
> Thank you for the quick response!
>
> I had read that, but didn't see anything useful in it. Having re-read it,
> I realize that my DLQs (which contain 100k messages) were a likely suspect
> in keeping the log files "live." I cleared the DLQs, and ActiveMQ
> diligently deleted most of the log files a couple minutes later:
>
>      $ cd apache-activemq-5.8.0/data/kahadb/
>      $ ls -lht | head
>      $ ls -lht | head
>      total 4.9G
>      -rw-rw-r-- 1 ubuntu ubuntu  32M Dec 13 18:00 db-1564.log
>      -rw-rw-r-- 1 ubuntu ubuntu  36M Dec 13 18:00 db.data
>      -rw-rw-r-- 1 ubuntu ubuntu 2.6M Dec 13 18:00 db.redo
>      -rw-rw-r-- 1 ubuntu ubuntu  33M Dec 13 17:44 db-1563.log
>      -rw-rw-r-- 1 ubuntu ubuntu  33M Dec 13 16:14 db-1562.log
>      -rw-rw-r-- 1 ubuntu ubuntu  33M Dec 13 14:47 db-1561.log
>      -rw-rw-r-- 1 ubuntu ubuntu  33M Dec 13 13:28 db-1560.log
>      -rw-rw-r-- 1 ubuntu ubuntu  33M Dec 13 12:09 db-1559.log
>      -rw-rw-r-- 1 ubuntu ubuntu  33M Dec 13 10:55 db-1558.log
>
> Thank you for the hint, Timothy!
>
>
> Andy Boothe
> Director of Data Engineering, Analytics
>
> WCG
> 60 Francisco St, San Francisco, 94133
> direct 512-961-3993	cell 979-574-1089
> twitter @sigpwned <http://twitter.com/sigpwned>	linkedin aboothe
> <http://www.linkedin.com/in/aboothe>
> github sigpwned <https://github.com/sigpwned>		stack overflow sigpwned
> <http://stackoverflow.com/users/2103602/sigpwned>
>
> € € € € € € € Go. Ahead.
> ŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠŠ
>
>
>
>
>
> On 12/13/13 10:59 AM, "Timothy Bish" <tabish121@gmail.com> wrote:
>
>> On 12/13/2013 11:27 AM, Andy Boothe [WCG] wrote:
>>> Hello,
>>>
>>> I have an ActiveMQ instance that has been running nicely for a couple
>>> of months, but doesn't seem to clean up its KahaDB logs like I expect
>>> it to. (More specifically, I expect to have like 1 log file, but in
>>> reality I have ~1500 log files with 49G of data.)
>>>
>>> ActiveMQ has processed about 12M messages over the past couple of
>>> months, and has roughly 106K messages currently pending (in DLQs):
>>>
>>>
>>> Given the number of messages currently pending, I would not expect the
>>> KahaDB logs to be very big. In reality, they're about 49G:
>>>
>>>      $ cd apache-activemq-5.8.0/data/kahadb/
>>>      $ ls -lht | head
>>>      total 49G
>>>      -rw-rw-r-- 1 ubuntu ubuntu  32M Dec 13 16:00 db-1562.log
>>>      -rw-rw-r-- 1 ubuntu ubuntu  36M Dec 13 16:00 db.data
>>>      -rw-rw-r-- 1 ubuntu ubuntu 173K Dec 13 16:00 db.redo
>>>      -rw-rw-r-- 1 ubuntu ubuntu  33M Dec 13 14:47 db-1561.log
>>>      -rw-rw-r-- 1 ubuntu ubuntu  33M Dec 13 13:28 db-1560.log
>>>      -rw-rw-r-- 1 ubuntu ubuntu  33M Dec 13 12:09 db-1559.log
>>>      -rw-rw-r-- 1 ubuntu ubuntu  33M Dec 13 10:55 db-1558.log
>>>      -rw-rw-r-- 1 ubuntu ubuntu  33M Dec 13 09:48 db-1557.log
>>>      -rw-rw-r-- 1 ubuntu ubuntu  33M Dec 13 08:40 db-1556.log
>>>
>>> Based on the maximum log ID, almost none of the log files appear to
>>> have been cleaned up:
>>>
>>>      $ ls db-*.log | wc
>>>         1540    1540 17421
>>>
>>> I'm using ActiveMQ 5.8.0. The only change I've made to the vanilla
>>> configuration is Java heap size, which shouldn't affect whether or not
>>> DB log files get cleaned up (AFAIK). I'm "connecting" to ActiveMQ with
>>> Camel. Obviously from the number of queues, I have several different
>>> workflows in flight, but here's a representative example of how I'm
>>> using ActiveMQ in my Java code:
>>>
>>>                  from("activemq:statuslinks")
>>>
>>>                      .bean(new LinkExtract())
>>>
>>>                      .split(body(String.class).tokenize("\n"))
>>>
>>>                      .filter(new LinkHandled(jdbcConnectionString))
>>>
>>>                      .to("activemq:linkunwind");
>>>
>>>
>>> The only problems I'm seeing in my activemq.log file appear to be
>>> "normal." I get rashes of transport exceptions every few weeks, which
>>> I have associated (rightly or wrongly) with hitting the web admin
>>> interface. Given the infrequency and nature of the errors, I don't
>>> think they explain why the logs wouldn't be deleted. In case it's
>>> useful, though, here are a couple of representative lines from the log
>>> file:
>>>
>>>      2013-12-13 09:46:52,269 | WARN  | Transport Connection to:
>>>      tcp://127.0.0.1:39604 failed:
>>>      org.apache.activemq.transport.InactivityIOException: Channel was
>>>      inactive for too (>30000) long: tcp://127.0.0.1:39604 |
>>>      org.apache.activemq.broker.TransportConnection.Transport |
>>>      ActiveMQ InactivityMonitor Worker
>>>      2013-12-13 14:43:41,215 | WARN  | Transport Connection to:
>>>      tcp://127.0.0.1:35711 failed: java.io.EOFException |
>>>      org.apache.activemq.broker.TransportConnection.Transport |
>>>      ActiveMQ Transport: tcp:///127.0.0.1:35711@61616
>>>
>>>
>>> Given the use case ‹ very simple message processing using Camel ‹ and
>>> ActiveMQ's vanilla configuration, I would expect the log files to be
>>> tiny. Clearly, though they're not. Does anyone have any idea why those
>>> files aren't getting deleted? I'm sure I'm doing (or not doing)
>>> something very silly, but after pouring over the logs and
>>> documentation, I just can't figure out what I'm doing wrong. Does
>>> anyone have an idea why my logs wouldn't be getting cleaned up?
>>>
>>> Thanks everyone in advance for your attention and help! And thank you
>>> ActiveMQ devs for such a wonderful product! :)
>>>
>>> *Andy Boothe*
>>>
>>> ****
>> Have you referenced this documentation:
>> http://cp.mcafee.com/d/FZsS93gs839J5BVMQsLInjhdTdFEIzDxRQTxNJd5AsYZtZCVJd5
>> AsYeKCYYrjhp7ff8T7c6Z8lHcDUacm8CaNfgVsSoIhclyuxOVJxd1AxGr_nVdBVVBPHTbETpd7
>> fc9L9CzBXBHFShhlLtzBgY-F6lK1FJ4SMrLOtXTLuZXTdTdw0ZJehDRhrJmSNf-00Uo9ByW5iV
>> niuE42VZi9tSh_QbNZnpgbWXfUBX7_I6RK1NJKDNEnzYSUMbIzCnXBZ8lHcDVNyN4Nm9W7bwHr
>> jApZkmXlJIj_w09Is--yr1vF6y02J9W739EwfGhd43Iz_Eq88lCq82fPDa14Qgmf_ouwq86Rzh
>> 0nbCy01ipoArKruq0ctEi81xx
>>
>>
>> -- 
>> Tim Bish
>> Sr Software Engineer | RedHat Inc.
>> tim.bish@redhat.com |
>> http://cp.mcafee.com/d/k-Kr6xEp3xESyOYUqenSbFECXCQQmhPMWWrMUSCyOeuuK-PsSCy
>> Oeu7njuudFEIzDDArzC3uAaRCjY56b4j5oDEsKrcm8CaNfgVsSMCwOgRd_HYCOYYOVRXBQrICz
>> DC4TAPhOZORQX8EGTKNOEuvkzaT0QSCrodTVeZXTLuZXCXCODeJQ-BmUS_BPrtfzgL7VJNwnp7
>> cLTbWgHmpfPz5y9yIjQen1mSD8PWEJSHroD_00joVZZ4S2_id405qjQe6jh0vkyq87p7_gQggH
>> cQg4vDek29EwIv-MZ0QgdH6y0Knd402AON8TsS5aMcvwY |
>> http://cp.mcafee.com/d/2DRPoAd6QmnD3hO-Ntd4TsSCyOeu7nju76QQmhPPRTSrCQQmhPM
>> WWrPNJd5AsYYzssMrQxmIOvwENoyoH4Z3BPpyN4Nm9W7bCS4Q6i6FLZvASnDCneLsKztAQsYMC
>> YCqenKmKDp55mZSel3PWApmU6CSjr1K_9TLuZXTLsTsSkWfE4ZDOVJKDNEnzYSUMbIzCnXBZ8l
>> HcDVNyN4Nm9W7bwHrjApZkmXlJIj_w09Is--yr1vF6y02J9W739EwfGhd43Iz_Eq88lCq82fPD
>> a14Qgmf_ouwq86Rzh0nbCy01ipoArKrZzCB4t6l
>> skype: tabish121 | twitter: @tabish121
>> blog:
>> http://cp.mcafee.com/d/FZsS73hJ5BVMQsLInjhdTdFEIzDxRQTxNJd5AsYZtZCVJd5AsYe
>> KCYYrjhp7ff8T7c6Z8lHcDUacm8CaNfgVsSoIhclyuxOVJxd1AxGr_nVdBVVBPHTbETpd7fc9L
>> 9CzBXBHFShhlLtzBgY-F6lK1FJwSMrLOtXTLuZXTdTdw0VOl8iEjUwvkzhqI-nMSTjUQbN-rso
>> 5ShPbZO-AaRCjYUNoyoH4Z3BMlJFOc-GbtGSS9_M04SevvhdwLQzh01mAZ3xAQg7R8Cy1Sh_Qd
>> 44aPd417VPB0yq8b7_Ifgd43qNEwbBPh00FcIidTdKLPH3zP8
>>
>
Without seeing the log messages from that article all we can say is, you 
still have some messages in there.

-- 
Tim Bish
Sr Software Engineer | RedHat Inc.
tim.bish@redhat.com | www.fusesource.com | www.redhat.com
skype: tabish121 | twitter: @tabish121
blog: http://timbish.blogspot.com/


Mime
View raw message