From users-return-22734-apmail-activemq-users-archive=activemq.apache.org@activemq.apache.org Fri Mar 05 15:01:53 2010
Return-Path: <users-return-22734-apmail-activemq-users-archive=activemq.apache.org@activemq.apache.org>
Delivered-To: apmail-activemq-users-archive@www.apache.org
Received: (qmail 69952 invoked from network); 5 Mar 2010 15:01:53 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 5 Mar 2010 15:01:53 -0000
Received: (qmail 28023 invoked by uid 500); 5 Mar 2010 15:01:39 -0000
Delivered-To: apmail-activemq-users-archive@activemq.apache.org
Received: (qmail 27997 invoked by uid 500); 5 Mar 2010 15:01:38 -0000
Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:users-help@activemq.apache.org>
List-Unsubscribe: <mailto:users-unsubscribe@activemq.apache.org>
List-Post: <mailto:users@activemq.apache.org>
List-Id: <users.activemq.apache.org>
Reply-To: users@activemq.apache.org
Delivered-To: mailing list users@activemq.apache.org
Received: (qmail 27989 invoked by uid 99); 5 Mar 2010 15:01:38 -0000
Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Mar 2010 15:01:38 +0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=FREEMAIL_FROM,SPF_HELO_PASS,SPF_PASS,T_TO_NO_BRKTS_FREEMAIL
X-Spam-Check-By: apache.org
Received-SPF: pass (athena.apache.org: domain of lists@nabble.com designates 216.139.236.158 as permitted sender)
Received: from [216.139.236.158] (HELO kuber.nabble.com) (216.139.236.158)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 05 Mar 2010 15:01:32 +0000
Received: from isper.nabble.com ([192.168.236.156])
	by kuber.nabble.com with esmtp (Exim 4.63)
	(envelope-from <lists@nabble.com>)
	id 1NnZ1c-0001gh-Be
	for users@activemq.apache.org; Fri, 05 Mar 2010 07:01:12 -0800
Message-ID: <27795010.post@talk.nabble.com>
Date: Fri, 5 Mar 2010 07:01:12 -0800 (PST)
From: lucious <marcin.giedz@arise.pl>
To: users@activemq.apache.org
Subject: Re: AMQ (5.3.0) with CMS-3.01 -> consumes all memory and stops
  clients and producers
In-Reply-To: <7c982ec31003031230g15e1b604r771347f77ab5d94f@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable
X-Nabble-From: marcin.giedz@arise.pl
References: <27771356.post@talk.nabble.com> <7c982ec31003031230g15e1b604r771347f77ab5d94f@mail.gmail.com>


Hi Mats,

Doesn't work neither :(

1) running AMQ version SNAPSHOT from march 03=20
2) config file as follows:
    <broker xmlns=3D"http://activemq.apache.org/schema/core"
brokerName=3D"localhost" useJmx=3D"true" dataDirectory=3D"${activemq.base}/=
data">
       =20
        <managementContext>
            <managementContext createConnector=3D"false"/>
        </managementContext>

        <persistenceAdapter>
            <kahaDB directory=3D"${activemq.base}/data/kahadb"/>
        </persistenceAdapter>
       =20
        <destinationPolicy>
            <policyMap>
              <policyEntries>
                <policyEntry topic=3D">" producerFlowControl=3D"false"
memoryLimit=3D"100mb">

                  <dispatchPolicy>
              =09=09<strictOrderDispatchPolicy />
            =09  </dispatchPolicy>

=09=09  <pendingSubscriberPolicy>
                  =09<vmCursor />
                  </pendingSubscriberPolicy>

=09=09  <pendingDurableSubscriberPolicy>
                  =09<vmDurableCursor/>
                  </pendingDurableSubscriberPolicy>
                 =20
                </policyEntry>
                <policyEntry queue=3D">" producerFlowControl=3D"false"
memoryLimit=3D"100mb" optimizedDispatch=3D"true">
                 =20
=09=09  <dispatchPolicy>
              =09  =09<strictOrderDispatchPolicy />
            =09  </dispatchPolicy>
    =20
                  <pendingQueuePolicy>
                  =09<vmQueueCursor/>
                  </pendingQueuePolicy>
                 =20
                 =20
                </policyEntry>
              </policyEntries>
            </policyMap>
        </destinationPolicy>=20
=20
        <systemUsage>
            <systemUsage>
                <memoryUsage>
                    <memoryUsage limit=3D"1024 mb"/>
                </memoryUsage>
                <storeUsage>
                    <storeUsage limit=3D"15 gb" name=3D"store"/>
                </storeUsage>
<!--
                <tempUsage>
                    <tempUsage limit=3D"10 gb"/>
                </tempUsage>
-->
            </systemUsage>
        </systemUsage>
        <transportConnectors>
            <transportConnector name=3D"openwire"
uri=3D"tcp://0.0.0.0:61616?wireFormat.maxInactivityDuration=3D0"/>
        </transportConnectors>

    </broker>

3) this was in log when MemoryPrecentageUsage reached 100% :=20
2010-03-05 15:35:56,090 | INFO  | Usage Manager Memory Limit reached on
temp-queue://e84351f4-d345-2d5c-44e8-35dc2523aea4:2. Producers will be
throttled to the rate at which messages are removed from this destination t=
o
prevent flooding it. See
http://activemq.apache.org/producer-flow-control.html for more info |
org.apache.activemq.broker.region.Queue | ActiveMQ Transport:
tcp:///127.0.0.1:57096

again temp-queue


Maybe I should remove VMCursor or .... I really don't know what to do with
this ... reopen the same BUG?

Thanks,
Marcin



Mats Henrikson wrote:
>=20
> Hi Marcin,
>=20
> This looks very similar to a bug that I have logged:
>=20
> https://issues.apache.org/activemq/browse/AMQ-2616
>=20
> Supposed to be fixed in 5.3.1 and later.
>=20
> Mats
>=20
>=20
>=20
>=20
> On 4 March 2010 06:50, lucious <marcin.giedz@arise.pl> wrote:
>>
>> Hello here is what we have:
>>
>> 1) AMQ -5 3.0 with the following configuration:
>> <broker xmlns=3D"http://activemq.apache.org/schema/core"
>> brokerName=3D"localhost" useJmx=3D"true"
>> dataDirectory=3D"${activemq.base}/data">
>>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0<managementContext>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<managementContext createConnec=
tor=3D"false"/>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0</managementContext>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0<persistenceAdapter>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<kahaDB directory=3D"${activemq=
.base}/data/kahadb"/>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0</persistenceAdapter>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0<destinationPolicy>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<policyMap>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<policyEntries>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<policyEntry topi=
c=3D">" producerFlowControl=3D"false"
>> memoryLimit=3D"100mb">
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<dispatchP=
olicy>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0<strictOrderDispatchPolicy />
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0</dispatch=
Policy>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<pendingSu=
bscriberPolicy>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0<vmCursor />
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0</pendingS=
ubscriberPolicy>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 <pendingDurableS=
ubscriberPolicy>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0<vmDurableCursor/>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0</pendingD=
urableSubscriberPolicy>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0</policyEntry>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<policyEntry queu=
e=3D">" producerFlowControl=3D"false"
>> memoryLimit=3D"100mb">
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<dispatchP=
olicy>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0<strictOrderDispatchPolicy />
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0</dispatch=
Policy>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<pendingQu=
euePolicy>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0<vmQueueCursor/>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0</pendingQ=
ueuePolicy>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0</policyEntry>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0</policyEntries>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0</policyMap>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0</destinationPolicy>
>>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0<systemUsage>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<systemUsage>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<memoryUsage>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<me=
moryUsage limit=3D"1024 mb"/>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0</memoryUsage>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<storeUsage>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<st=
oreUsage limit=3D"10 gb" name=3D"foo"/>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0</storeUsage>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<tempUsage>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<te=
mpUsage limit=3D"10 gb"/>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0</tempUsage>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0</systemUsage>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0</systemUsage>
>>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0<transportConnectors>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<transportConnector name=3D"ope=
nwire"
>> uri=3D"tcp://0.0.0.0:61616?wireFormat.maxInactivityDuration=3D0"/>
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0</transportConnectors>
>>
>> System is being run with Xm=3D2048M
>>
>> 2) Producers and clients use CMS - 3.0.1 - we don't have any features
>> added
>> to CreateFactory - all defaults are used.
>>
>> Results:
>> 1) during the day watch MQ via JMX I have: totalEnqueueCount=3D1193470,
>> totaldequeuecount=3D628130,totalmessagecount=3D1052834 - this I think is
>> wrong
>> ... shouldn't be like that: totalmessage+totaldequeue =3D totalenqueue?
>>
>> 2) memorypercentusage=3D73 and it's counting ... once it gets to 100% al=
l
>> is
>> frozen and in log file I get messages like this: INFO | Usage Manager
>> memory
>> limit reached on temp-queue://42455840-0bb5-0101-49b1-304ecf24dbbd:2.
>> Producers will be throttled to the rate at which messages are removed
>> from
>> this destination to prevent flooding it. See
>> http://activemq.apache.org/producer-flow-control.html for more info
>>
>> quite strange as flow-control is disabled.
>>
>> 3) watching Heap memory Usage I can see "saw chart" but it's going up an=
d
>> up
>> to 1GB and then is saturated (AMQ is frozen at that time).
>>
>> 4) the only thing I can do is to stop AMQ and restart ... however on
>> production is not a solution.
>>
>> Maybe I have something wrong with configuration file? I'm also a little
>> bit
>> confused about memoryLimit in topic/queue policy and memoryUsage. What
>> exactly these options do? I thought that if memoryUsage is 1GB and
>> percentUsage is 70% all messages are dropped into "file" and memory is
>> clean. Maybe I mixed this all up and it's not as I think it's?
>>
>> Please help,
>> thx,
>> M.
>>
>> --
>> View this message in context:
>> http://old.nabble.com/AMQ-%285.3.0%29-with-CMS-3.01--%3E-consumes-all-me=
mory-and-stops-clients-and-producers-tp27771356p27771356.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>=20
>=20

--=20
View this message in context: http://old.nabble.com/AMQ-%285.3.0%29-with-CM=
S-3.01--%3E-consumes-all-memory-and-stops-clients-and-producers-tp27771356p=
27795010.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.