activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Desai, Brian" <brian.de...@sensis.com>
Subject Journal files don't get cleaned up
Date Sat, 31 May 2008 01:15:54 GMT
I'm running ActiveMQ 5.1.0 with the AMQ persistence adapter, and it
appears that not all of the journal files get cleaned up.  My setup is a
little abnormal, as I'm trying to test out ActiveMQ's ability to handle
queue messaging with consumers that may become inactive for periods of
time.
 
So for this test, I have a single publisher pushing messages out to 21
queues.  These are persistent messages with an expiration time.  I have
a message listener reading from all queues (reading from '>').  So, as
soon as the message is sent to the queues, it's read by the message
listener, taking it off the queue.  So far, so good.
 
I have a 2 MB max file length set on the AMQ persistence adapter.  So, I
would expect to see for the journal, 2 MB files that get cleaned up
after the file rolls over.  However, the journal files don't always get
cleaned up, as shown in the file listing below.  Out of 181 rollovers,
30 of the files did not get cleaned up.  The message listener showed no
errors, and as far as I can tell, it didn't drop any messages.
 
-rw-r--r-- 1 root root  2096753 2008-05-30 20:30 data/journal/data-13
-rw-r--r-- 1 root root  2096967 2008-05-30 20:31 data/journal/data-14
-rw-r--r-- 1 root root  2096899 2008-05-30 20:45 data/journal/data-25
-rw-r--r-- 1 root root  2097057 2008-05-30 21:20 data/journal/data-52
-rw-r--r-- 1 root root  2096916 2008-05-30 21:22 data/journal/data-54
-rw-r--r-- 1 root root  2096536 2008-05-30 21:45 data/journal/data-72
-rw-r--r-- 1 root root  2096894 2008-05-30 21:47 data/journal/data-73
-rw-r--r-- 1 root root  2097129 2008-05-30 21:49 data/journal/data-75
-rw-r--r-- 1 root root  2097101 2008-05-30 21:58 data/journal/data-82
-rw-r--r-- 1 root root  2097026 2008-05-30 21:59 data/journal/data-83
-rw-r--r-- 1 root root  2096906 2008-05-30 22:02 data/journal/data-85
-rw-r--r-- 1 root root  2096973 2008-05-30 22:13 data/journal/data-94
-rw-r--r-- 1 root root  2097105 2008-05-30 22:24 data/journal/data-102
-rw-r--r-- 1 root root  2097033 2008-05-30 22:41 data/journal/data-113
-rw-r--r-- 1 root root  2096730 2008-05-30 22:42 data/journal/data-114
-rw-r--r-- 1 root root  2096569 2008-05-30 22:45 data/journal/data-116
-rw-r--r-- 1 root root  2096870 2008-05-30 22:50 data/journal/data-118
-rw-r--r-- 1 root root  2096567 2008-05-30 22:52 data/journal/data-119
-rw-r--r-- 1 root root  2096766 2008-05-30 23:03 data/journal/data-128
-rw-r--r-- 1 root root  2096877 2008-05-30 23:06 data/journal/data-130
-rw-r--r-- 1 root root  2096888 2008-05-30 23:18 data/journal/data-140
-rw-r--r-- 1 root root  2096699 2008-05-30 23:20 data/journal/data-141
-rw-r--r-- 1 root root  2096973 2008-05-30 23:22 data/journal/data-143
-rw-r--r-- 1 root root  2096924 2008-05-30 23:31 data/journal/data-150
-rw-r--r-- 1 root root  2096936 2008-05-30 23:45 data/journal/data-161
-rw-r--r-- 1 root root  2096527 2008-05-30 23:57 data/journal/data-170
-rw-r--r-- 1 root root  2097151 2008-05-30 23:58 data/journal/data-171
-rw-r--r-- 1 root root  2096972 2008-05-31 00:11 data/journal/data-179
-rw-r--r-- 1 root root  2096703 2008-05-31 00:13 data/journal/data-180
-rw-r--r-- 1 root root  2097069 2008-05-31 00:14 data/journal/data-181

I haven't even gotten to the test yet where the listener is not running.
So, in this "normal" operation, all messages are consumed.  Yet, not all
journal files get cleaned up.  These left-over files don't ever get
cleaned up.  They will eventually start filling the hard drive.  I can
understand files being left behind when there's no consumer, but there
is a consumer the whole time.
 
Does anyone know why this happens, or how I can fix it?
 
What I'm basically looking for is a persistence layer for messaging to
multiple clients, so that consumers can get messages retroactively when
they start up.  I could try to use topics with durable clients, but I
thought the queues would be easier to setup, as messages in queues are
persisted by default.  However, I don't want the consumer to process
"stale" messages, which is why I set an expiration time.  So, I would
think that, with a constant rate of messages, the persistent disk store
utilization would eventually level out as the messages started to
expire.  I realize that if there's no consumer for a queue, expired
messages won't get cleaned up (am currently trying to figure out a
work-around for that - periodically checking the queues with a
QueueBrowser seems to trigger the removal of expired messages).
However, even when all consumers are active, the journal keeps growing,
as it's not always cleaning up it's files!
 
Here's my configuration.
 
<beans
  xmlns="http://www.springframework.org/schema/beans"
  xmlns:amq="http://activemq.apache.org/schema/core"
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
  xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
  http://activemq.apache.org/schema/core
http://activemq.apache.org/schema/core/activemq-core.xsd
  http://activemq.apache.org/camel/schema/spring
http://activemq.apache.org/camel/schema/spring/camel-spring.xsd">
 
    <!-- Allows us to use system properties as variables in this
configuration file -->
    <bean
class="org.springframework.beans.factory.config.PropertyPlaceholderConfi
gurer"/>
 
    <broker xmlns="http://activemq.apache.org/schema/core"
brokerName="localhost" dataDirectory="${activemq.base}/data"
advisorySupport="false">
 
        <!-- Destination specific policies using destination names or
wildcards -->
        <destinationPolicy>
            <policyMap>
                <policyEntries>
                    <policyEntry queue=">" memoryLimit="5mb">
                        <deadLetterStrategy>
                            <sharedDeadLetterStrategy
processExpired="false"/>
                        </deadLetterStrategy>
                    </policyEntry>
                    <policyEntry topic=">" memoryLimit="5mb">
                        <dispatchPolicy>
                            <strictOrderDispatchPolicy/>
                        </dispatchPolicy>
                        <subscriptionRecoveryPolicy>
                            <lastImageSubscriptionRecoveryPolicy/>
                        </subscriptionRecoveryPolicy>
                    </policyEntry>
                </policyEntries>
            </policyMap>
        </destinationPolicy>
 
        <!-- Use the following to configure how ActiveMQ is exposed in
JMX -->
        <managementContext>
            <managementContext createConnector="true"/>
        </managementContext>

        <persistenceAdapter>
            <amqPersistenceAdapter archiveDataLogs="false"
                                   syncOnWrite="false"
                                   directory="${activemq.base}/data"
                                   maxFileLength="2 mb"/>
        </persistenceAdapter>
 
        <!--  The maximum about of space the broker will use before
slowing down producers -->
        <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage limit="20 mb"/>
                </memoryUsage>
                <storeUsage>
                    <!--<storeUsage limit="1 gb" name="foo"/>-->
                    <storeUsage limit="5 mb" name="foo"/>
                </storeUsage>
                <tempUsage>
                    <!--<tempUsage limit="100 mb"/>-->
                    <tempUsage limit="5 mb"/>
                </tempUsage>
            </systemUsage>
        </systemUsage>

        <!-- The transport connectors ActiveMQ will listen to -->
        <transportConnectors>
            <transportConnector name="openwire"
uri="tcp://localhost:61606" discoveryUri="multicast://default"/>
            <transportConnector name="ssl" uri="ssl://localhost:61617"/>
            <transportConnector name="stomp"
uri="stomp://localhost:61613"/>
            <transportConnector name="xmpp"
uri="xmpp://localhost:61222"/>
        </transportConnectors>
 
    </broker>

</beans>

 
Any help would be greatly appreciated!
 
Thanks,
Brian
 

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message