Hi, thanks for the response.
Some questions on these points from the troubleshooting.
1. *It contains a pending message for a destination or durable topic
subscription*
This seems a little flawed, if a consumer who I have little control of is
mis-behaving then my ActiveMQ can end up shutting down and unrecoverable.
Is there some way of timing this out or similar?
*2. It contains an ack for a message which is in an in-use data file - the
ack cannot be removed as a recovery would then mark the message for
redelivery*
Same comment as 1.
*3. The journal references a pending transaction*
I'm not using transactions, but are there transactions under the hood?
*4. It is a journal file, and there may be a pending write to it*
Why would this be the case?
I'll see if I can change the logging settings, since the first occurrence
the number of log files does not seem to have been an issue. I have it
configured to keep messages for 7 days so regardless of the above
conditions I would have thought that at that expiry the log would be
cleaned up so we don't end up in such a situation where the system stops
and cannot restart.
On 14 September 2017 at 00:02, Tim Bain <tbain@alumni.duke.edu> wrote:
> Have you run the troubleshooting procedures in
> http://activemq.apache.org/why-do-kahadb-log-files-
> remain-after-cleanup.html?
> What did they indicate?
>
> On Sep 12, 2017 8:27 PM, "Lionel van den Berg" <lionelv@gmail.com> wrote:
>
> Hi all,
> ActiveMQ 5.14.4 running on Suse enterprise 11 sp4
>
> I'm having trouble with the log files (like db-1.log) not being cleaned up
> and after about 6 weeks of running ActiveMQ shuts down because there are
> too many files in the directory (I think it was 3000), it cannot be
> restarted we think due to a JVM limitation. We use a mix of topics and
> queues, we can't guarantee that the consumer from the third party won't be
> shutdown for an extended period. We are storing messages for 7 days, or at
> least we think we are, I've copied the config further down. Don't assume I
> know everything that we are doing with all of the config :). I've read
> various posts but I'm not sure still whether there might be a parameter
> that I can set that will trigger the cleanup.
>
> It looks similar to this https://issues.apache.org/jira/browse/AMQ-5695
> but
> the last comment there is that it is resolved in 5.14.3, I'm running 5.14.4
> on the server side at least, client libraries we did not change.
>
> Does anyone have any ideas?
>
>
>
> <!--
> Licensed to the Apache Software Foundation (ASF) under one or more
> contributor license agreements. See the NOTICE file distributed with
> this work for additional information regarding copyright ownership.
> The ASF licenses this file to You under the Apache License, Version 2.0
> (the "License"); you may not use this file except in compliance with
> the License. You may obtain a copy of the License at
>
> http://www.apache.org/licenses/LICENSE-2.0
>
> Unless required by applicable law or agreed to in writing, software
> distributed under the License is distributed on an "AS IS" BASIS,
> WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
> implied.
> See the License for the specific language governing permissions and
> limitations under the License.
> -->
> <beans
> xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd
> http://activemq.apache.org/schema/core
> http://activemq.apache.org/schema/core/activemq-core.xsd">
>
> <!-- Allows us to use system properties as variables in this
> configuration file -->
> <bean
> class="org.springframework.beans.factory.config.
> PropertyPlaceholderConfigurer">
> <property name="locations">
> <value>file:${activemq.conf}/credentials.properties</value>
> </property>
> </bean>
>
> <!-- Allows accessing the server log -->
> <bean id="logQuery" class="io.fabric8.insight.log.log4j.Log4jLogQuery"
> lazy-init="false" scope="singleton"
> init-method="start" destroy-method="stop">
> </bean>
>
> <!-- The <broker> element is used to configure the ActiveMQ broker. -->
> <broker xmlns="http://activemq.apache.org/schema/core"
> brokerName="AutomationServerThree" dataDirectory="${activemq.data}">
>
> <destinationPolicy>
> <policyMap>
> <policyEntries>
> <!-- policy for all topics -->
> <policyEntry topic=">" producerFlowControl="true"
> useCache="true" expireMessagesPeriod="86400000" alwaysRetroactive="true">
> <pendingMessageLimitStrategy>
> <constantPendingMessageLimitStrategy limit="1000"/>
> </pendingMessageLimitStrategy>
> <!-- discard expired messages -->
> <deadLetterStrategy>
> <sharedDeadLetterStrategy processExpired="false"/>
> </deadLetterStrategy>
> </policyEntry>
> <!-- policy for all queues -->
> <policyEntry queue=">" producerFlowControl="true"
> useCache="true" expireMessagesPeriod="86400000">
> <pendingMessageLimitStrategy>
> <constantPendingMessageLimitStrategy limit="1000"/>
> </pendingMessageLimitStrategy>
> <!-- discard expired messages -->
> <deadLetterStrategy>
> <sharedDeadLetterStrategy processExpired="false"/>
> </deadLetterStrategy>
> </policyEntry>
> </policyEntries>
> </policyMap>
> </destinationPolicy>
>
>
> <!--
> The managementContext is used to configure how ActiveMQ is
> exposed in
> JMX. By default, ActiveMQ uses the MBean server that is started
> by
> the JVM. For more information, see:
>
> http://activemq.apache.org/jmx.html
> -->
> <managementContext>
> <managementContext createConnector="true"/>
> </managementContext>
>
> <!--
> Configure message persistence for the broker. The default
> persistence
> mechanism is the KahaDB store (identified by the kahaDB tag).
> For more information, see:
>
> http://activemq.apache.org/persistence.html
> -->
> <persistenceAdapter>
> <kahaDB directory="/media/SharedStorage/Binaries/activemq"
> indexCacheSize="40000"
> checkForCorruptJournalFiles="true">
> <locker>
> <shared-native-file-locker lockAcquireSleepInterval="5000"
> />
> </locker>
> </kahaDB>
> </persistenceAdapter>
>
>
> <!--
> The systemUsage controls the maximum amount of space the broker
> will
> use before disabling caching and/or slowing down producers. For
> more information, see:
> http://activemq.apache.org/producer-flow-control.html
> -->
> <systemUsage>
> <systemUsage>
> <memoryUsage>
> <memoryUsage percentOfJvmHeap="70" />
> </memoryUsage>
> <storeUsage>
> <storeUsage limit="100 gb"/>
> </storeUsage>
> <tempUsage>
> <tempUsage limit="50 gb"/>
> </tempUsage>
> </systemUsage>
> </systemUsage>
>
> <!--
> The transport connectors expose ActiveMQ over a given protocol
> to
> clients and other brokers. For more information, see:
>
> http://activemq.apache.org/configuring-transports.html
> -->
> <transportConnectors>
> <!-- DOS protection, limit concurrent connections to 1000 and
> frame size to 100MB -->
> <transportConnector name="openwire" uri="tcp://
> 0.0.0.0:61616?maximumConnections=1000&wireFormat.maxFrameSize=
> 104857600
> "/>
> <transportConnector name="amqp" uri="amqp://
> 0.0.0.0:5672?maximumConnections=1000&wireFormat.maxFrameSize=104857600
> "/>
> <transportConnector name="stomp" uri="stomp://
> 0.0.0.0:61613?maximumConnections=1000&wireFormat.maxFrameSize=
> 104857600
> "/>
> <transportConnector name="mqtt" uri="mqtt://
> 0.0.0.0:1883?maximumConnections=1000&wireFormat.maxFrameSize=104857600
> "/>
> <transportConnector name="ws" uri="ws://
> 0.0.0.0:61614?maximumConnections=1000&wireFormat.maxFrameSize=
> 104857600
> "/>
> </transportConnectors>
>
> <!-- destroy the spring context on shutdown to stop jetty -->
> <shutdownHooks>
> <bean xmlns="http://www.springframework.org/schema/beans"
> class="org.apache.activemq.hooks.SpringContextHook" />
> </shutdownHooks>
>
> </broker>
>
> <!--
> Enable web consoles, REST and Ajax APIs and demos
> The web consoles requires by default login, you can disable this in
> the jetty.xml file
>
> Take a look at ${ACTIVEMQ_HOME}/conf/jetty.xml for more details
> -->
> <import resource="jetty.xml"/>
>
> </beans>
>
|