Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (athena.apache.org: domain of rajdavies@gmail.com
 designates 64.233.182.189 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:from:to:in-reply-to:content-type:content-transfer-encoding:mime-version:subject:date:references:x-mailer;
        b=pv5nsoNEZBoPqoTrbKgBde9rT9n9qqaCpDNEILFDvhV7kK3wyFCgXr9PTwSu8haUPfjinHgXlwc/q6bdEUftXlXdD4MHJJ31kILm5IzAtLbeuFhe83d7QrXapKE9lFGNJaz5eUTtne4wrCx1ZyC9KSPfP+zsiETJf0gQ1DJoCAQ=
Message-Id: <D2C69F4D-FA5E-49DE-AA2C-35C3905CACB5@gmail.com>
From: Rob Davies <rajdavies@gmail.com>
To: users@activemq.apache.org
In-Reply-To: 
 <5A211522579EBD4A83155ED7FA153063034A0846@corpatsmail1.corp.sensis.com>
Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes
Content-Transfer-Encoding: 7bit
Mime-Version: 1.0 (Apple Message framework v924)
Subject: Re: Journal files don't get cleaned up
Date: Sat, 31 May 2008 07:32:04 +0100
References: 
 <5A211522579EBD4A83155ED7FA153063034A0846@corpatsmail1.corp.sensis.com>


On 31 May 2008, at 02:15, Desai, Brian wrote:

> I'm running ActiveMQ 5.1.0 with the AMQ persistence adapter, and it
> appears that not all of the journal files get cleaned up.  My setup  
> is a
> little abnormal, as I'm trying to test out ActiveMQ's ability to  
> handle
> queue messaging with consumers that may become inactive for periods of
> time.
>
> So for this test, I have a single publisher pushing messages out to 21
> queues.  These are persistent messages with an expiration time.  I  
> have
> a message listener reading from all queues (reading from '>').  So, as
> soon as the message is sent to the queues, it's read by the message
> listener, taking it off the queue.  So far, so good.
>
> I have a 2 MB max file length set on the AMQ persistence adapter.   
> So, I
> would expect to see for the journal, 2 MB files that get cleaned up
> after the file rolls over.  However, the journal files don't always  
> get
> cleaned up, as shown in the file listing below.  Out of 181 rollovers,
> 30 of the files did not get cleaned up.  The message listener showed  
> no
> errors, and as far as I can tell, it didn't drop any messages.
>
> -rw-r--r-- 1 root root  2096753 2008-05-30 20:30 data/journal/data-13
> -rw-r--r-- 1 root root  2096967 2008-05-30 20:31 data/journal/data-14
> -rw-r--r-- 1 root root  2096899 2008-05-30 20:45 data/journal/data-25
> -rw-r--r-- 1 root root  2097057 2008-05-30 21:20 data/journal/data-52
> -rw-r--r-- 1 root root  2096916 2008-05-30 21:22 data/journal/data-54
> -rw-r--r-- 1 root root  2096536 2008-05-30 21:45 data/journal/data-72
> -rw-r--r-- 1 root root  2096894 2008-05-30 21:47 data/journal/data-73
> -rw-r--r-- 1 root root  2097129 2008-05-30 21:49 data/journal/data-75
> -rw-r--r-- 1 root root  2097101 2008-05-30 21:58 data/journal/data-82
> -rw-r--r-- 1 root root  2097026 2008-05-30 21:59 data/journal/data-83
> -rw-r--r-- 1 root root  2096906 2008-05-30 22:02 data/journal/data-85
> -rw-r--r-- 1 root root  2096973 2008-05-30 22:13 data/journal/data-94
> -rw-r--r-- 1 root root  2097105 2008-05-30 22:24 data/journal/data-102
> -rw-r--r-- 1 root root  2097033 2008-05-30 22:41 data/journal/data-113
> -rw-r--r-- 1 root root  2096730 2008-05-30 22:42 data/journal/data-114
> -rw-r--r-- 1 root root  2096569 2008-05-30 22:45 data/journal/data-116
> -rw-r--r-- 1 root root  2096870 2008-05-30 22:50 data/journal/data-118
> -rw-r--r-- 1 root root  2096567 2008-05-30 22:52 data/journal/data-119
> -rw-r--r-- 1 root root  2096766 2008-05-30 23:03 data/journal/data-128
> -rw-r--r-- 1 root root  2096877 2008-05-30 23:06 data/journal/data-130
> -rw-r--r-- 1 root root  2096888 2008-05-30 23:18 data/journal/data-140
> -rw-r--r-- 1 root root  2096699 2008-05-30 23:20 data/journal/data-141
> -rw-r--r-- 1 root root  2096973 2008-05-30 23:22 data/journal/data-143
> -rw-r--r-- 1 root root  2096924 2008-05-30 23:31 data/journal/data-150
> -rw-r--r-- 1 root root  2096936 2008-05-30 23:45 data/journal/data-161
> -rw-r--r-- 1 root root  2096527 2008-05-30 23:57 data/journal/data-170
> -rw-r--r-- 1 root root  2097151 2008-05-30 23:58 data/journal/data-171
> -rw-r--r-- 1 root root  2096972 2008-05-31 00:11 data/journal/data-179
> -rw-r--r-- 1 root root  2096703 2008-05-31 00:13 data/journal/data-180
> -rw-r--r-- 1 root root  2097069 2008-05-31 00:14 data/journal/data-181
>
> I haven't even gotten to the test yet where the listener is not  
> running.
> So, in this "normal" operation, all messages are consumed.  Yet, not  
> all
> journal files get cleaned up.  These left-over files don't ever get
> cleaned up.  They will eventually start filling the hard drive.  I can
> understand files being left behind when there's no consumer, but there
> is a consumer the whole time.
>
> Does anyone know why this happens, or how I can fix it?
>
> What I'm basically looking for is a persistence layer for messaging to
> multiple clients, so that consumers can get messages retroactively  
> when
> they start up.  I could try to use topics with durable clients, but I
> thought the queues would be easier to setup, as messages in queues are
> persisted by default.  However, I don't want the consumer to process
> "stale" messages, which is why I set an expiration time.  So, I would
> think that, with a constant rate of messages, the persistent disk  
> store
> utilization would eventually level out as the messages started to
> expire.  I realize that if there's no consumer for a queue, expired
> messages won't get cleaned up (am currently trying to figure out a
> work-around for that - periodically checking the queues with a
> QueueBrowser seems to trigger the removal of expired messages).
> However, even when all consumers are active, the journal keeps  
> growing,
> as it's not always cleaning up it's files!
>
> Here's my configuration.
>
> <beans
>  xmlns="http://www.springframework.org/schema/beans"
>  xmlns:amq="http://activemq.apache.org/schema/core"
>  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>  xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>  http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd
>  http://activemq.apache.org/camel/schema/spring
> http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">
>
>    <!-- Allows us to use system properties as variables in this
> configuration file -->
>    <bean
> class 
> ="org.springframework.beans.factory.config.PropertyPlaceholderConfi
> gurer"/>
>
>    <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="localhost" dataDirectory="${activemq.base}/data"
> advisorySupport="false">
>
>        <!-- Destination specific policies using destination names or
> wildcards -->
>        <destinationPolicy>
>            <policyMap>
>                <policyEntries>
>                    <policyEntry queue=">" memoryLimit="5mb">
>                        <deadLetterStrategy>
>                            <sharedDeadLetterStrategy
> processExpired="false"/>
>                        </deadLetterStrategy>
>                    </policyEntry>
>                    <policyEntry topic=">" memoryLimit="5mb">
>                        <dispatchPolicy>
>                            <strictOrderDispatchPolicy/>
>                        </dispatchPolicy>
>                        <subscriptionRecoveryPolicy>
>                            <lastImageSubscriptionRecoveryPolicy/>
>                        </subscriptionRecoveryPolicy>
>                    </policyEntry>
>                </policyEntries>
>            </policyMap>
>        </destinationPolicy>
>
>        <!-- Use the following to configure how ActiveMQ is exposed in
> JMX -->
>        <managementContext>
>            <managementContext createConnector="true"/>
>        </managementContext>
>
>        <persistenceAdapter>
>            <amqPersistenceAdapter archiveDataLogs="false"
>                                   syncOnWrite="false"
>                                   directory="${activemq.base}/data"
>                                   maxFileLength="2 mb"/>
>        </persistenceAdapter>
>
>        <!--  The maximum about of space the broker will use before
> slowing down producers -->
>        <systemUsage>
>            <systemUsage>
>                <memoryUsage>
>                    <memoryUsage limit="20 mb"/>
>                </memoryUsage>
>                <storeUsage>
>                    <!--<storeUsage limit="1 gb" name="foo"/>-->
>                    <storeUsage limit="5 mb" name="foo"/>
>                </storeUsage>
>                <tempUsage>
>                    <!--<tempUsage limit="100 mb"/>-->
>                    <tempUsage limit="5 mb"/>
>                </tempUsage>
>            </systemUsage>
>        </systemUsage>
>
>        <!-- The transport connectors ActiveMQ will listen to -->
>        <transportConnectors>
>            <transportConnector name="openwire"
> uri="tcp://localhost:61606" discoveryUri="multicast://default"/>
>            <transportConnector name="ssl" uri="ssl://localhost: 
> 61617"/>
>            <transportConnector name="stomp"
> uri="stomp://localhost:61613"/>
>            <transportConnector name="xmpp"
> uri="xmpp://localhost:61222"/>
>        </transportConnectors>
>
>    </broker>
>
> </beans>
>
>
>
> Any help would be greatly appreciated!
>
> Thanks,
> Brian
>

Hi Brian,

thanks for the very detailed email! This does look like a bug - and   
its probably related to the use of the wildcard  '> ' for the Queue  
listener.
I wonder if you've looked ay virtual topics - see http://activemq.apache.org/virtual-destinations.html 
  ?

Could you raise an issue for us - so we don't forget to track?


cheers,

Rob

http://open.iona.com/products/enterprise-activemq
http://rajdavies.blogspot.com/