activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rob Davies <rajdav...@gmail.com>
Subject Re: why is activeMQ stalled?
Date Fri, 15 Jan 2010 10:50:07 GMT
David - can you upgrade 5.3 ?

On 15 Jan 2010, at 10:40, David Delbecq wrote:

> Le 15/01/10 11:25, Gary Tully a écrit :
>> A thread dump of the broker jvm, when the clients are stalled,  
>> would help
>> diagnose what is going on. btw, what activemq version are you using?
>>
> BrokerService                  - ActiveMQ 5.2.0 JMS Message Broker
> (localhost) is starting
>
> I'll come back with broker dump when there is a problem again. This  
> can
> be in several hours or within a few weeks. However i'll try to force a
> stalling next week...
>
>
>> 2010/1/15 David Delbecq <delbd@oma.be>
>>
>>
>>> Le 15/01/10 10:24, Dejan Bosanac a écrit :
>>>
>>>> Hi David,
>>>>
>>>> you're probably blocked by "producer flow control". Take a look  
>>>> here
>>>> for more info: http://activemq.apache.org/producer-flow- 
>>>> control.html
>>>>
>>>> Cheers
>>>>
>>> I already took a look at it, read it three time to be sure i  
>>> understand.
>>> It's quite technical but beyond that, i don't think this apply for  
>>> me.
>>> My queues are all empty!!! (they stalled when there were 50 messages
>>> pending but where finally emptied), connections are not shared by
>>> several queues and am not sure how to implement the cursor part of
>>> documentation. There is no "slow consumer" that could influence it  
>>> at
>>> that right moment (customers are running an waiting for new  
>>> messages as
>>> you can see in capture). Moreover my messages are of about 1k  
>>> payload
>>> (about 10 small text entries in a MapMessage), and my activeMQ  
>>> config is
>>> like this
>>>                   <policyEntry queue=">" memoryLimit="500mb"/>
>>>                   <policyEntry topic=">" memoryLimit="500mb">
>>> ....
>>>       <systemUsage>
>>>           <systemUsage>
>>>               <memoryUsage>
>>>                   <memoryUsage limit="20 mb"/>
>>>               </memoryUsage>
>>>               <storeUsage>
>>>                   <storeUsage limit="1 gb" name="foo"/>
>>>               </storeUsage>
>>>               <tempUsage>
>>>                   <tempUsage limit="100 mb"/>
>>>               </tempUsage>
>>>           </systemUsage>
>>>       </systemUsage>
>>>
>>>
>>> There is no way a bunch of 10 pending messages could fill a 1gb  
>>> store,
>>> isn't it? What config part of activemq do i need to change to ensure
>>> that it can store at least about 10000 persistent messages (that  
>>> makes
>>> only 10Mb of datas!!) before failing? Currently it fails and  is  
>>> never
>>> recovering without a restart of broker when there are only 50 small
>>> pending messages in the queue. As a workaround i have set
>>> "sendFailIfNoSpace" to true, so my client will fail instead of  
>>> stalling,
>>> i'll see when this happen more quickly like that....
>>>
>>>
>>> Here is broker part of my config file, can someone help me fix this?
>>> Thank you:
>>>
>>>
>>>   <broker xmlns="http://activemq.apache.org/schema/core"
>>> brokerName="localhost" dataDirectory="${activemq.base}/data"
>>> persistent="true">
>>>       <plugins>
>>>       <simpleAuthenticationPlugin>
>>>         <users>
>>>               <authenticationUser username="admin" password="******"
>>>                       groups="users,admins"/>
>>>               <authenticationUser username="sharkTool"  
>>> password="********"
>>>                       groups="serverTools"/>
>>>               <authenticationUser username="adminConsole"
>>> password="*********"
>>>                       groups="serverTools"/>
>>>         </users>
>>>       </simpleAuthenticationPlugin>
>>>     <authorizationPlugin>
>>>       <map>
>>>         <authorizationMap>
>>>           <authorizationEntries>
>>>             <authorizationEntry queue=">" read="admins"  
>>> write="admins"
>>> admin="admins" />
>>>             <authorizationEntry queue="WF.>"  
>>> read="users,serverTools"
>>> write="serverTools" admin="admins,serverTools" />
>>>
>>>             <authorizationEntry topic=">" read="admins"  
>>> write="admins"
>>> admin="admins" />
>>>             <authorizationEntry topic="WF.>"  
>>> read="users,serverTools"
>>> write="serverTools" admin="admins,serverTools" />
>>>
>>>             <authorizationEntry topic="ActiveMQ.Advisory.>"
>>> read="guests,users,serverTools" write="guests,users,serverTools"
>>> admin="guests,users,serverTools"/>
>>>           </authorizationEntries>
>>>
>>>           <!-- let's assign roles to temporary destinations. comment
>>> this entry if we don't want any roles assigned to temp destinations
>>>           <tempDestinationAuthorizationEntry>
>>>             <tempDestinationAuthorizationEntry
>>> read="tempDestinationAdmins" write="tempDestinationAdmins"
>>> admin="tempDestinationAdmins"/>
>>>          </tempDestinationAuthorizationEntry>          -->
>>>         </authorizationMap>
>>>       </map>
>>>     </authorizationPlugin>
>>>
>>>
>>>       </plugins>
>>>       <!-- Destination specific policies using destination names or
>>> wildcards -->
>>>       <destinationPolicy>
>>>           <policyMap>
>>>               <policyEntries>
>>>                   <policyEntry queue=">" memoryLimit="500mb"/>
>>>                   <policyEntry topic=">" memoryLimit="500mb">
>>>                     <!-- you can add other policies too such as  
>>> these
>>>                       <dispatchPolicy>
>>>                           <strictOrderDispatchPolicy/>
>>>                       </dispatchPolicy>
>>>                       <subscriptionRecoveryPolicy>
>>>                           <lastImageSubscriptionRecoveryPolicy/>
>>>                       </subscriptionRecoveryPolicy>
>>>                     -->
>>>                   </policyEntry>
>>>               </policyEntries>
>>>           </policyMap>
>>>       </destinationPolicy>
>>>
>>>       <!-- Use the following to configure how ActiveMQ is exposed in
>>> JMX -->
>>>       <managementContext>
>>>           <managementContext createConnector="false"/>
>>>       </managementContext>
>>>
>>>       <!-- The store and forward broker networks ActiveMQ will  
>>> listen
>>> to -->
>>>       <networkConnectors>
>>>           <!-- by default just auto discover the other brokers -->
>>>           <!-- <networkConnector name="default-nc"
>>> uri="multicast://default"/>-->
>>>           <!-- Example of a static configuration:
>>>           <networkConnector name="host1 and host2"
>>> uri="static://(tcp://host1:61616,tcp://host2:61616)"/>
>>>           -->
>>>       </networkConnectors>
>>>
>>>       <persistenceAdapter>
>>>           <amqPersistenceAdapter syncOnWrite="false"
>>> directory="${activemq.base}/data" maxFileLength="20 mb"/>
>>>       </persistenceAdapter>
>>>
>>>
>>>
>>>       <sslContext>
>>>           <sslContext keyStore="file:${activemq.base}/conf/ 
>>> broker.ks"
>>> keyStorePassword="password"
>>> trustStore="file:${activemq.base}/conf/broker.ts"
>>> trustStorePassword="password"/>
>>>       </sslContext>
>>>
>>>       <!--  The maximum about of space the broker will use before
>>> slowing down producers -->
>>>       <systemUsage>
>>>           <systemUsage>
>>>               <memoryUsage>
>>>                   <memoryUsage limit="20 mb"/>
>>>               </memoryUsage>
>>>               <storeUsage>
>>>                   <storeUsage limit="1 gb" name="foo"/>
>>>               </storeUsage>
>>>               <tempUsage>
>>>                   <tempUsage limit="100 mb"/>
>>>               </tempUsage>
>>>           </systemUsage>
>>>       </systemUsage>
>>>
>>>
>>>       <!-- The transport connectors ActiveMQ will listen to -->
>>>       <transportConnectors>
>>>           <transportConnector name="openwire"
>>> uri="tcp://localhost:61616"/>
>>>           <!-- <transportConnector name="ssl"
>>> uri="ssl://localhost:61617"/>
>>>           <transportConnector name="stomp" uri="stomp://localhost: 
>>> 61613"/>
>>>           <transportConnector name="xmpp"
>>> uri="xmpp://localhost:61222"/> -->
>>>       </transportConnectors>
>>>       <destinations>
>>>           <queue physicalName="WF.Regraph" />
>>>             <queue physicalName="WF.Notify"/>
>>>             <topic physicalName="WF.Events"/>
>>>       </destinations>
>>>           <destinationInterceptors>
>>>     <virtualDestinationInterceptor>
>>>       <virtualDestinations>
>>>         <compositeQueue name="WF.IncomingEvents">
>>>           <forwardTo>
>>>             <queue physicalName="WF.Regraph"/>
>>>             <queue physicalName="WF.Notify"/>
>>>             <topic physicalName="WF.Events" />
>>>           </forwardTo>
>>>         </compositeQueue>
>>>       </virtualDestinations>
>>>     </virtualDestinationInterceptor>
>>>   </destinationInterceptors>
>>>
>>>
>>>   </broker>
>>>
>>>
>>>
>>>> --
>>>>
>>>>
>>>> On Fri, Jan 15, 2010 at 10:24 AM, David Delbecq <delbd@oma.be
>>>> <mailto:delbd@oma.be>> wrote:
>>>>
>>>>    Hello,
>>>>
>>>>    can someone explain me why my activeMQ client is stalled in such
>>>>    situation (i have 10 threads blocked like this having as
>>>>    consequence a complete blocking of my client application):
>>>>
>>>>    "http-8081-Processor6" daemon prio=10 tid=00841c38 nid=512
>>>>    lwp_id=2426611 waiting on condition [236f3000..236f153c]
>>>>       java.lang.Thread.State: WAITING (parking)
>>>>            at sun.misc.Unsafe.park(Native Method)
>>>>            - parking to wait for  <40ae1828> (a
>>>>
>>>>
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer 
>>> $ConditionObject)
>>>
>>>>            at
>>>>    java.util.concurrent.locks.LockSupport.park(LockSupport.java: 
>>>> 158)
>>>>            at
>>>>
>>>>
>>> java.util.concurrent.locks.AbstractQueuedSynchronizer 
>>> $ConditionObject.await(AbstractQueuedSynchronizer.java:1889)
>>>
>>>>            at
>>>>
>>>>
>>> java 
>>> .util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java: 
>>> 317)
>>>
>>>>            at
>>>>
>>>>
>>> org 
>>> .apache 
>>> .activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
>>>
>>>>            at
>>>>
>>>>
>>> org 
>>> .apache 
>>> .activemq 
>>> .transport.ResponseCorrelator.request(ResponseCorrelator.java:80)
>>>
>>>>            at
>>>>
>>>>
>>> org 
>>> .apache 
>>> .activemq 
>>> .ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1233)
>>>
>>>>            at
>>>>    org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java: 
>>>> 1659)
>>>>            - locked <40ae01e8> (a java.lang.Object)
>>>>            at
>>>>
>>>>
>>> org 
>>> .apache 
>>> .activemq 
>>> .ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:227)
>>>
>>>>            at
>>>>
>>>>
>>> org 
>>> .apache 
>>> .activemq 
>>> .ActiveMQMessageProducerSupport 
>>> .send(ActiveMQMessageProducerSupport.java:241)
>>>
>>>>            at
>>>>    be.rmi.shark.JMSEventManager.sendEvent(JMSEventManager.java:146)
>>>>
>>>>
>>>>    considering that the activeMQ lists size, as referenced by
>>>>    activeMQ admin interface are as follow:
>>>>    Name      Number Of Pending Messages      Number Of Consumers
>>>>    Messages Sent     Messages Received       Views            
>>>> Operations
>>>>    WF.Regraph        0       1       1714    1720    Browse  Send  
>>>> To
>>>>
>>> Purge Delete
>>>
>>>>    WF.Notify         0       1       1696    1742    Browse  Send  
>>>> To
>>>>
>>> Purge Delete
>>>
>>>>    WF.IncomingEvents         0       0       0       0       Browse
>>>>
>>> Send To Purge Delete
>>>
>>>>    Is this to be considered normal that i must restart activeMQ
>>>>    producers on a regular basis because of those this stalling?  
>>>> Where
>>>>    can i find clear documentation on proper configuration of  
>>>> activeMQ
>>>>    to prevent this stalling? Thank you.
>>>>
>>>>    --
>>>>    David Delbecq
>>>>    ICT
>>>>    Institut Royal Météorologique
>>>>    Ext:557
>>>>
>>>>
>>>>
>>>>
>>> --
>>> David Delbecq
>>> ICT
>>> Institut Royal Météorologique
>>> Ext:557
>>>
>>>
>>>
>>
>>
>>
>
>
> -- 
> David Delbecq
> ICT
> Institut Royal Météorologique
> Ext:557
>

Rob Davies
http://twitter.com/rajdavies
I work here: http://fusesource.com
My Blog: http://rajdavies.blogspot.com/
I'm writing this: http://www.manning.com/snyder/






Mime
View raw message