activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Delbecq <de...@oma.be>
Subject Re: why is activeMQ stalled?
Date Fri, 15 Jan 2010 11:04:19 GMT
Le 15/01/10 11:50, Rob Davies a écrit :
> David - can you upgrade 5.3 ?
Not immediatly, but worth a try
>
> On 15 Jan 2010, at 10:40, David Delbecq wrote:
>
>> Le 15/01/10 11:25, Gary Tully a écrit :
>>> A thread dump of the broker jvm, when the clients are stalled, would
>>> help
>>> diagnose what is going on. btw, what activemq version are you using?
>>>
>> BrokerService                  - ActiveMQ 5.2.0 JMS Message Broker
>> (localhost) is starting
>>
>> I'll come back with broker dump when there is a problem again. This can
>> be in several hours or within a few weeks. However i'll try to force a
>> stalling next week...
>>
>>
>>> 2010/1/15 David Delbecq <delbd@oma.be>
>>>
>>>
>>>> Le 15/01/10 10:24, Dejan Bosanac a écrit :
>>>>
>>>>> Hi David,
>>>>>
>>>>> you're probably blocked by "producer flow control". Take a look here
>>>>> for more info: http://activemq.apache.org/producer-flow-control.html
>>>>>
>>>>> Cheers
>>>>>
>>>> I already took a look at it, read it three time to be sure i
>>>> understand.
>>>> It's quite technical but beyond that, i don't think this apply for me.
>>>> My queues are all empty!!! (they stalled when there were 50 messages
>>>> pending but where finally emptied), connections are not shared by
>>>> several queues and am not sure how to implement the cursor part of
>>>> documentation. There is no "slow consumer" that could influence it at
>>>> that right moment (customers are running an waiting for new
>>>> messages as
>>>> you can see in capture). Moreover my messages are of about 1k payload
>>>> (about 10 small text entries in a MapMessage), and my activeMQ
>>>> config is
>>>> like this
>>>>                   <policyEntry queue=">" memoryLimit="500mb"/>
>>>>                   <policyEntry topic=">" memoryLimit="500mb">
>>>> ....
>>>>       <systemUsage>
>>>>           <systemUsage>
>>>>               <memoryUsage>
>>>>                   <memoryUsage limit="20 mb"/>
>>>>               </memoryUsage>
>>>>               <storeUsage>
>>>>                   <storeUsage limit="1 gb" name="foo"/>
>>>>               </storeUsage>
>>>>               <tempUsage>
>>>>                   <tempUsage limit="100 mb"/>
>>>>               </tempUsage>
>>>>           </systemUsage>
>>>>       </systemUsage>
>>>>
>>>>
>>>> There is no way a bunch of 10 pending messages could fill a 1gb store,
>>>> isn't it? What config part of activemq do i need to change to ensure
>>>> that it can store at least about 10000 persistent messages (that makes
>>>> only 10Mb of datas!!) before failing? Currently it fails and  is never
>>>> recovering without a restart of broker when there are only 50 small
>>>> pending messages in the queue. As a workaround i have set
>>>> "sendFailIfNoSpace" to true, so my client will fail instead of
>>>> stalling,
>>>> i'll see when this happen more quickly like that....
>>>>
>>>>
>>>> Here is broker part of my config file, can someone help me fix this?
>>>> Thank you:
>>>>
>>>>
>>>>   <broker xmlns="http://activemq.apache.org/schema/core"
>>>> brokerName="localhost" dataDirectory="${activemq.base}/data"
>>>> persistent="true">
>>>>       <plugins>
>>>>       <simpleAuthenticationPlugin>
>>>>         <users>
>>>>               <authenticationUser username="admin" password="******"
>>>>                       groups="users,admins"/>
>>>>               <authenticationUser username="sharkTool"
>>>> password="********"
>>>>                       groups="serverTools"/>
>>>>               <authenticationUser username="adminConsole"
>>>> password="*********"
>>>>                       groups="serverTools"/>
>>>>         </users>
>>>>       </simpleAuthenticationPlugin>
>>>>     <authorizationPlugin>
>>>>       <map>
>>>>         <authorizationMap>
>>>>           <authorizationEntries>
>>>>             <authorizationEntry queue=">" read="admins" write="admins"
>>>> admin="admins" />
>>>>             <authorizationEntry queue="WF.>" read="users,serverTools"
>>>> write="serverTools" admin="admins,serverTools" />
>>>>
>>>>             <authorizationEntry topic=">" read="admins" write="admins"
>>>> admin="admins" />
>>>>             <authorizationEntry topic="WF.>" read="users,serverTools"
>>>> write="serverTools" admin="admins,serverTools" />
>>>>
>>>>             <authorizationEntry topic="ActiveMQ.Advisory.>"
>>>> read="guests,users,serverTools" write="guests,users,serverTools"
>>>> admin="guests,users,serverTools"/>
>>>>           </authorizationEntries>
>>>>
>>>>           <!-- let's assign roles to temporary destinations. comment
>>>> this entry if we don't want any roles assigned to temp destinations
>>>>           <tempDestinationAuthorizationEntry>
>>>>             <tempDestinationAuthorizationEntry
>>>> read="tempDestinationAdmins" write="tempDestinationAdmins"
>>>> admin="tempDestinationAdmins"/>
>>>>          </tempDestinationAuthorizationEntry>          -->
>>>>         </authorizationMap>
>>>>       </map>
>>>>     </authorizationPlugin>
>>>>
>>>>
>>>>       </plugins>
>>>>       <!-- Destination specific policies using destination names or
>>>> wildcards -->
>>>>       <destinationPolicy>
>>>>           <policyMap>
>>>>               <policyEntries>
>>>>                   <policyEntry queue=">" memoryLimit="500mb"/>
>>>>                   <policyEntry topic=">" memoryLimit="500mb">
>>>>                     <!-- you can add other policies too such as these
>>>>                       <dispatchPolicy>
>>>>                           <strictOrderDispatchPolicy/>
>>>>                       </dispatchPolicy>
>>>>                       <subscriptionRecoveryPolicy>
>>>>                           <lastImageSubscriptionRecoveryPolicy/>
>>>>                       </subscriptionRecoveryPolicy>
>>>>                     -->
>>>>                   </policyEntry>
>>>>               </policyEntries>
>>>>           </policyMap>
>>>>       </destinationPolicy>
>>>>
>>>>       <!-- Use the following to configure how ActiveMQ is exposed in
>>>> JMX -->
>>>>       <managementContext>
>>>>           <managementContext createConnector="false"/>
>>>>       </managementContext>
>>>>
>>>>       <!-- The store and forward broker networks ActiveMQ will listen
>>>> to -->
>>>>       <networkConnectors>
>>>>           <!-- by default just auto discover the other brokers -->
>>>>           <!-- <networkConnector name="default-nc"
>>>> uri="multicast://default"/>-->
>>>>           <!-- Example of a static configuration:
>>>>           <networkConnector name="host1 and host2"
>>>> uri="static://(tcp://host1:61616,tcp://host2:61616)"/>
>>>>           -->
>>>>       </networkConnectors>
>>>>
>>>>       <persistenceAdapter>
>>>>           <amqPersistenceAdapter syncOnWrite="false"
>>>> directory="${activemq.base}/data" maxFileLength="20 mb"/>
>>>>       </persistenceAdapter>
>>>>
>>>>
>>>>
>>>>       <sslContext>
>>>>           <sslContext keyStore="file:${activemq.base}/conf/broker.ks"
>>>> keyStorePassword="password"
>>>> trustStore="file:${activemq.base}/conf/broker.ts"
>>>> trustStorePassword="password"/>
>>>>       </sslContext>
>>>>
>>>>       <!--  The maximum about of space the broker will use before
>>>> slowing down producers -->
>>>>       <systemUsage>
>>>>           <systemUsage>
>>>>               <memoryUsage>
>>>>                   <memoryUsage limit="20 mb"/>
>>>>               </memoryUsage>
>>>>               <storeUsage>
>>>>                   <storeUsage limit="1 gb" name="foo"/>
>>>>               </storeUsage>
>>>>               <tempUsage>
>>>>                   <tempUsage limit="100 mb"/>
>>>>               </tempUsage>
>>>>           </systemUsage>
>>>>       </systemUsage>
>>>>
>>>>
>>>>       <!-- The transport connectors ActiveMQ will listen to -->
>>>>       <transportConnectors>
>>>>           <transportConnector name="openwire"
>>>> uri="tcp://localhost:61616"/>
>>>>           <!-- <transportConnector name="ssl"
>>>> uri="ssl://localhost:61617"/>
>>>>           <transportConnector name="stomp"
>>>> uri="stomp://localhost:61613"/>
>>>>           <transportConnector name="xmpp"
>>>> uri="xmpp://localhost:61222"/> -->
>>>>       </transportConnectors>
>>>>       <destinations>
>>>>           <queue physicalName="WF.Regraph" />
>>>>             <queue physicalName="WF.Notify"/>
>>>>             <topic physicalName="WF.Events"/>
>>>>       </destinations>
>>>>           <destinationInterceptors>
>>>>     <virtualDestinationInterceptor>
>>>>       <virtualDestinations>
>>>>         <compositeQueue name="WF.IncomingEvents">
>>>>           <forwardTo>
>>>>             <queue physicalName="WF.Regraph"/>
>>>>             <queue physicalName="WF.Notify"/>
>>>>             <topic physicalName="WF.Events" />
>>>>           </forwardTo>
>>>>         </compositeQueue>
>>>>       </virtualDestinations>
>>>>     </virtualDestinationInterceptor>
>>>>   </destinationInterceptors>
>>>>
>>>>
>>>>   </broker>
>>>>
>>>>
>>>>
>>>>> -- 
>>>>>
>>>>>
>>>>> On Fri, Jan 15, 2010 at 10:24 AM, David Delbecq <delbd@oma.be
>>>>> <mailto:delbd@oma.be>> wrote:
>>>>>
>>>>>    Hello,
>>>>>
>>>>>    can someone explain me why my activeMQ client is stalled in such
>>>>>    situation (i have 10 threads blocked like this having as
>>>>>    consequence a complete blocking of my client application):
>>>>>
>>>>>    "http-8081-Processor6" daemon prio=10 tid=00841c38 nid=512
>>>>>    lwp_id=2426611 waiting on condition [236f3000..236f153c]
>>>>>       java.lang.Thread.State: WAITING (parking)
>>>>>            at sun.misc.Unsafe.park(Native Method)
>>>>>            - parking to wait for  <40ae1828> (a
>>>>>
>>>>>
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>>>
>>>>>            at
>>>>>    java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>>>            at
>>>>>
>>>>>
>>>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1889)
>>>>
>>>>
>>>>>            at
>>>>>
>>>>>
>>>> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
>>>>
>>>>
>>>>>            at
>>>>>
>>>>>
>>>> org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
>>>>
>>>>
>>>>>            at
>>>>>
>>>>>
>>>> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:80)
>>>>
>>>>
>>>>>            at
>>>>>
>>>>>
>>>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1233)
>>>>
>>>>
>>>>>            at
>>>>>   
>>>>> org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1659)
>>>>>            - locked <40ae01e8> (a java.lang.Object)
>>>>>            at
>>>>>
>>>>>
>>>> org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:227)
>>>>
>>>>
>>>>>            at
>>>>>
>>>>>
>>>> org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:241)
>>>>
>>>>
>>>>>            at
>>>>>    be.rmi.shark.JMSEventManager.sendEvent(JMSEventManager.java:146)
>>>>>
>>>>>
>>>>>    considering that the activeMQ lists size, as referenced by
>>>>>    activeMQ admin interface are as follow:
>>>>>    Name      Number Of Pending Messages      Number Of Consumers
>>>>>    Messages Sent     Messages Received       Views          
>>>>> Operations
>>>>>    WF.Regraph        0       1       1714    1720    Browse  Send To
>>>>>
>>>> Purge Delete
>>>>
>>>>>    WF.Notify         0       1       1696    1742    Browse  Send To
>>>>>
>>>> Purge Delete
>>>>
>>>>>    WF.IncomingEvents         0       0       0       0       Browse
>>>>>
>>>> Send To Purge Delete
>>>>
>>>>>    Is this to be considered normal that i must restart activeMQ
>>>>>    producers on a regular basis because of those this stalling? Where
>>>>>    can i find clear documentation on proper configuration of activeMQ
>>>>>    to prevent this stalling? Thank you.
>>>>>
>>>>>    --
>>>>>    David Delbecq
>>>>>    ICT
>>>>>    Institut Royal Météorologique
>>>>>    Ext:557
>>>>>
>>>>>
>>>>>
>>>>>
>>>> -- 
>>>> David Delbecq
>>>> ICT
>>>> Institut Royal Météorologique
>>>> Ext:557
>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>> -- 
>> David Delbecq
>> ICT
>> Institut Royal Météorologique
>> Ext:557
>>
>
> Rob Davies
> http://twitter.com/rajdavies
> I work here: http://fusesource.com
> My Blog: http://rajdavies.blogspot.com/
> I'm writing this: http://www.manning.com/snyder/
>
>
>
>
>


-- 
David Delbecq
ICT
Institut Royal Météorologique
Ext:557


Mime
View raw message