Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (athena.apache.org: domain of mhixson@gmail.com designates
 209.85.212.171 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=oRtFJkFyIZrCihja357z7nrq+gs2CSWkozRtVxVDUxFiSnL0Pc3jPA7jI5lJbxWL/w
         lC7UIic0U4aeX/ah5LBJkenPIiQ7e1ENWXVGjVj6OPB8cK8GKXUgqdxd1NRF2r6U3JWK
         /cvgR0H4JA5N+GxklWANEZvWzYkIY8gA6VqRM=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1081)
Subject: Re: 5.4.0 stops accepting messages randomly
From: Matthew Hixson <mhixson@gmail.com>
In-Reply-To: <AANLkTimKAu6T0hnvpQRNzq-xgv6Zvb9HhXazyqvVbkKw@mail.gmail.com>
Date: Sat, 11 Sep 2010 15:39:12 -0700
Content-Transfer-Encoding: quoted-printable
Message-Id: <B81CAA2D-9DC8-4816-BAFB-789557372D66@gmail.com>
References: <1E9D6014-0412-4F24-A7E1-8A58FA71A07E@gmail.com>
 <AANLkTimKAu6T0hnvpQRNzq-xgv6Zvb9HhXazyqvVbkKw@mail.gmail.com>
To: users@activemq.apache.org

We did find this in the logs.

activemq.log:2010-09-08 10:55:02,555 | INFO  | Usage Manager Memory =
Limit is full. Producer (ID:isp01-55443-1283902621643-0:0:1:1) stopped =
to prevent
 flooding queue://mpqueue. See =
http://activemq.apache.org/producer-flow-control.html for more info =
(blocking for: 1s) | org.apache.activemq.broker.re
gion.Queue | ActiveMQ Transport: tcp:///10.13.85.14:52682

We have since increased the memory limit and we'll see if this happens =
again.  If so I'll get a full thread dump.
 Thanks,
  -M@


On Sep 10, 2010, at 12:51 AM, Dejan Bosanac wrote:

> Can you turn on debug logging and post logs and thread dumps when the
> broker hangs. Also, do you see messages in logs about producer flow
> control being used?
>=20
>=20
> Cheers
> --
> Dejan Bosanac - http://twitter.com/dejanb
>=20
> Open Source Integration - http://fusesource.com/
> ActiveMQ in Action - http://www.manning.com/snyder/
> Blog - http://www.nighttale.net
>=20
>=20
>=20
> On Thu, Sep 9, 2010 at 11:33 PM, Matthew Hixson <mhixson@gmail.com> =
wrote:
>> We have been running 5.4.0 in production for a few weeks and it can =
easily handle tens of thousands of messages a day, but then for no =
apparent reason it will cease accepting new messages from our producers. =
 The consumers will drain the queue empty, but then nothing new gets =
accepted into Active MQ.  Restarting Active MQ and then restarting all =
the producers and consumers allows things to proceed as normal.
>>  Below is our activemq.xml.
>>  Thanks for any help.
>>   -M@
>>=20
>> <!--
>>    Licensed to the Apache Software Foundation (ASF) under one or more
>>    contributor license agreements.  See the NOTICE file distributed =
with
>>    this work for additional information regarding copyright =
ownership.
>>    The ASF licenses this file to You under the Apache License, =
Version 2.0
>>    (the "License"); you may not use this file except in compliance =
with
>>    the License.  You may obtain a copy of the License at
>>=20
>>    http://www.apache.org/licenses/LICENSE-2.0
>>=20
>>    Unless required by applicable law or agreed to in writing, =
software
>>    distributed under the License is distributed on an "AS IS" BASIS,
>>    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or =
implied.
>>    See the License for the specific language governing permissions =
and
>>    limitations under the License.
>> -->
>> <beans
>>  xmlns=3D"http://www.springframework.org/schema/beans"
>>  xmlns:amq=3D"http://activemq.apache.org/schema/core"
>>  xmlns:xsi=3D"http://www.w3.org/2001/XMLSchema-instance"
>>  xsi:schemaLocation=3D"http://www.springframework.org/schema/beans =
http://www.springframework.org/schema/beans/spring-beans-2.0.xsd
>>  http://activemq.apache.org/schema/core =
http://activemq.apache.org/schema/core/activemq-core.xsd">
>>=20
>>    <!-- Allows us to use system properties as variables in this =
configuration file -->
>>    <bean =
class=3D"org.springframework.beans.factory.config.PropertyPlaceholderConfi=
gurer">
>>        <property name=3D"locations">
>>            =
<value>file:${activemq.base}/conf/credentials.properties</value>
>>        </property>
>>    </bean>
>>=20
>>    <!--
>>        The <broker> element is used to configure the ActiveMQ broker.
>>    -->
>>    <broker xmlns=3D"http://activemq.apache.org/schema/core" =
brokerName=3D"localhost" dataDirectory=3D"${activemq.base}/data" =
destroyApplicationContextOnStop=3D"true">
>>=20
>>        <!--
>>                        For better performances use VM cursor and =
small memory limit.
>>                        For more information, see:
>>=20
>>            http://activemq.apache.org/message-cursors.html
>>=20
>>            Also, if your producer is "hanging", it's probably due to =
producer flow control.
>>            For more information, see:
>>            http://activemq.apache.org/producer-flow-control.html
>>        -->
>>=20
>>        <destinationPolicy>
>>            <policyMap>
>>              <policyEntries>
>>                <policyEntry topic=3D">" producerFlowControl=3D"false" =
memoryLimit=3D"100mb">
>>                  <pendingSubscriberPolicy>
>>                    <vmCursor />
>>                  </pendingSubscriberPolicy>
>>                </policyEntry>
>>                <policyEntry queue=3D">" producerFlowControl=3D"true" =
memoryLimit=3D"1mb">
>>                  <!-- Use VM cursor for better latency
>>                       For more information, see:
>>=20
>>                       http://activemq.apache.org/message-cursors.html
>>=20
>>                  <pendingQueuePolicy>
>>                    <vmQueueCursor/>
>>                  </pendingQueuePolicy>
>>                  -->
>>                </policyEntry>
>>              </policyEntries>
>>            </policyMap>
>>        </destinationPolicy>
>>=20
>>=20
>>        <!--
>>            The managementContext is used to configure how ActiveMQ is =
exposed in
>>            JMX. By default, ActiveMQ uses the MBean server that is =
started by
>>            the JVM. For more information, see:
>>=20
>>            http://activemq.apache.org/jmx.html
>>        -->
>>        <managementContext>
>>            <managementContext createConnector=3D"false"/>
>>        </managementContext>
>>=20
>>        <!--
>>            Configure message persistence for the broker. The default =
persistence
>>            mechanism is the KahaDB store (identified by the kahaDB =
tag).
>>            For more information, see:
>>=20
>>            http://activemq.apache.org/persistence.html
>>        -->
>>        <persistenceAdapter>
>>            <kahaDB directory=3D"${activemq.base}/data/kahadb"/>
>>        </persistenceAdapter>
>>=20
>>=20
>>          <!--
>>            The systemUsage controls the maximum amount of space the =
broker will
>>            use before slowing down producers. For more information, =
see:
>>=20
>>            http://activemq.apache.org/producer-flow-control.html
>>=20
>>        <systemUsage>
>>            <systemUsage>
>>                <memoryUsage>
>>                    <memoryUsage limit=3D"100 mb"/>
>>                </memoryUsage>
>>                <storeUsage>
>>                    <storeUsage limit=3D"10 gb"/>
>>                </storeUsage>
>>                <tempUsage>
>>                    <tempUsage limit=3D"100 mb"/>
>>                </tempUsage>
>>            </systemUsage>
>>        </systemUsage>
>>                -->
>>=20
>>        <!--
>>            The transport connectors expose ActiveMQ over a given =
protocol to
>>            clients and other brokers. For more information, see:
>>=20
>>            http://activemq.apache.org/configuring-transports.html
>>        -->
>>        <transportConnectors>
>>            <transportConnector name=3D"openwire" =
uri=3D"tcp://0.0.0.0:61616"/>
>>        </transportConnectors>
>>=20
>>    </broker>
>>=20
>>    <!--
>>        Uncomment to enable Camel
>>        Take a look at activemq-camel.xml for more details
>>=20
>>    <import resource=3D"camel.xml"/>
>>    -->
>>=20
>>    <!--
>>        Enable web consoles, REST and Ajax APIs and demos
>>        Take a look at activemq-jetty.xml for more details
>>    -->
>>    <import resource=3D"jetty.xml"/>
>>=20
>> </beans>