activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Delbecq <de...@oma.be>
Subject Re: why is activeMQ stalled?
Date Fri, 15 Jan 2010 10:40:55 GMT
Le 15/01/10 11:25, Gary Tully a écrit :
> A thread dump of the broker jvm, when the clients are stalled, would help
> diagnose what is going on. btw, what activemq version are you using?
>   
BrokerService                  - ActiveMQ 5.2.0 JMS Message Broker
(localhost) is starting

I'll come back with broker dump when there is a problem again. This can
be in several hours or within a few weeks. However i'll try to force a
stalling next week...


> 2010/1/15 David Delbecq <delbd@oma.be>
>
>   
>> Le 15/01/10 10:24, Dejan Bosanac a écrit :
>>     
>>> Hi David,
>>>
>>> you're probably blocked by "producer flow control". Take a look here
>>> for more info: http://activemq.apache.org/producer-flow-control.html
>>>
>>> Cheers
>>>       
>> I already took a look at it, read it three time to be sure i understand.
>> It's quite technical but beyond that, i don't think this apply for me.
>> My queues are all empty!!! (they stalled when there were 50 messages
>> pending but where finally emptied), connections are not shared by
>> several queues and am not sure how to implement the cursor part of
>> documentation. There is no "slow consumer" that could influence it at
>> that right moment (customers are running an waiting for new messages as
>> you can see in capture). Moreover my messages are of about 1k payload
>> (about 10 small text entries in a MapMessage), and my activeMQ config is
>> like this
>>                    <policyEntry queue=">" memoryLimit="500mb"/>
>>                    <policyEntry topic=">" memoryLimit="500mb">
>> ....
>>        <systemUsage>
>>            <systemUsage>
>>                <memoryUsage>
>>                    <memoryUsage limit="20 mb"/>
>>                </memoryUsage>
>>                <storeUsage>
>>                    <storeUsage limit="1 gb" name="foo"/>
>>                </storeUsage>
>>                <tempUsage>
>>                    <tempUsage limit="100 mb"/>
>>                </tempUsage>
>>            </systemUsage>
>>        </systemUsage>
>>
>>
>> There is no way a bunch of 10 pending messages could fill a 1gb store,
>> isn't it? What config part of activemq do i need to change to ensure
>> that it can store at least about 10000 persistent messages (that makes
>> only 10Mb of datas!!) before failing? Currently it fails and  is never
>> recovering without a restart of broker when there are only 50 small
>> pending messages in the queue. As a workaround i have set
>> "sendFailIfNoSpace" to true, so my client will fail instead of stalling,
>> i'll see when this happen more quickly like that....
>>
>>
>> Here is broker part of my config file, can someone help me fix this?
>> Thank you:
>>
>>
>>    <broker xmlns="http://activemq.apache.org/schema/core"
>> brokerName="localhost" dataDirectory="${activemq.base}/data"
>> persistent="true">
>>        <plugins>
>>        <simpleAuthenticationPlugin>
>>          <users>
>>                <authenticationUser username="admin" password="******"
>>                        groups="users,admins"/>
>>                <authenticationUser username="sharkTool" password="********"
>>                        groups="serverTools"/>
>>                <authenticationUser username="adminConsole"
>> password="*********"
>>                        groups="serverTools"/>
>>          </users>
>>        </simpleAuthenticationPlugin>
>>      <authorizationPlugin>
>>        <map>
>>          <authorizationMap>
>>            <authorizationEntries>
>>              <authorizationEntry queue=">" read="admins" write="admins"
>> admin="admins" />
>>              <authorizationEntry queue="WF.>" read="users,serverTools"
>> write="serverTools" admin="admins,serverTools" />
>>
>>              <authorizationEntry topic=">" read="admins" write="admins"
>> admin="admins" />
>>              <authorizationEntry topic="WF.>" read="users,serverTools"
>> write="serverTools" admin="admins,serverTools" />
>>
>>              <authorizationEntry topic="ActiveMQ.Advisory.>"
>> read="guests,users,serverTools" write="guests,users,serverTools"
>> admin="guests,users,serverTools"/>
>>            </authorizationEntries>
>>
>>            <!-- let's assign roles to temporary destinations. comment
>> this entry if we don't want any roles assigned to temp destinations
>>            <tempDestinationAuthorizationEntry>
>>              <tempDestinationAuthorizationEntry
>> read="tempDestinationAdmins" write="tempDestinationAdmins"
>> admin="tempDestinationAdmins"/>
>>           </tempDestinationAuthorizationEntry>          -->
>>          </authorizationMap>
>>        </map>
>>      </authorizationPlugin>
>>
>>
>>        </plugins>
>>        <!-- Destination specific policies using destination names or
>> wildcards -->
>>        <destinationPolicy>
>>            <policyMap>
>>                <policyEntries>
>>                    <policyEntry queue=">" memoryLimit="500mb"/>
>>                    <policyEntry topic=">" memoryLimit="500mb">
>>                      <!-- you can add other policies too such as these
>>                        <dispatchPolicy>
>>                            <strictOrderDispatchPolicy/>
>>                        </dispatchPolicy>
>>                        <subscriptionRecoveryPolicy>
>>                            <lastImageSubscriptionRecoveryPolicy/>
>>                        </subscriptionRecoveryPolicy>
>>                      -->
>>                    </policyEntry>
>>                </policyEntries>
>>            </policyMap>
>>        </destinationPolicy>
>>
>>        <!-- Use the following to configure how ActiveMQ is exposed in
>> JMX -->
>>        <managementContext>
>>            <managementContext createConnector="false"/>
>>        </managementContext>
>>
>>        <!-- The store and forward broker networks ActiveMQ will listen
>> to -->
>>        <networkConnectors>
>>            <!-- by default just auto discover the other brokers -->
>>            <!-- <networkConnector name="default-nc"
>> uri="multicast://default"/>-->
>>            <!-- Example of a static configuration:
>>            <networkConnector name="host1 and host2"
>> uri="static://(tcp://host1:61616,tcp://host2:61616)"/>
>>            -->
>>        </networkConnectors>
>>
>>        <persistenceAdapter>
>>            <amqPersistenceAdapter syncOnWrite="false"
>> directory="${activemq.base}/data" maxFileLength="20 mb"/>
>>        </persistenceAdapter>
>>
>>
>>
>>        <sslContext>
>>            <sslContext keyStore="file:${activemq.base}/conf/broker.ks"
>> keyStorePassword="password"
>> trustStore="file:${activemq.base}/conf/broker.ts"
>> trustStorePassword="password"/>
>>        </sslContext>
>>
>>        <!--  The maximum about of space the broker will use before
>> slowing down producers -->
>>        <systemUsage>
>>            <systemUsage>
>>                <memoryUsage>
>>                    <memoryUsage limit="20 mb"/>
>>                </memoryUsage>
>>                <storeUsage>
>>                    <storeUsage limit="1 gb" name="foo"/>
>>                </storeUsage>
>>                <tempUsage>
>>                    <tempUsage limit="100 mb"/>
>>                </tempUsage>
>>            </systemUsage>
>>        </systemUsage>
>>
>>
>>        <!-- The transport connectors ActiveMQ will listen to -->
>>        <transportConnectors>
>>            <transportConnector name="openwire"
>> uri="tcp://localhost:61616"/>
>>            <!-- <transportConnector name="ssl"
>> uri="ssl://localhost:61617"/>
>>            <transportConnector name="stomp" uri="stomp://localhost:61613"/>
>>            <transportConnector name="xmpp"
>> uri="xmpp://localhost:61222"/> -->
>>        </transportConnectors>
>>        <destinations>
>>            <queue physicalName="WF.Regraph" />
>>              <queue physicalName="WF.Notify"/>
>>              <topic physicalName="WF.Events"/>
>>        </destinations>
>>            <destinationInterceptors>
>>      <virtualDestinationInterceptor>
>>        <virtualDestinations>
>>          <compositeQueue name="WF.IncomingEvents">
>>            <forwardTo>
>>              <queue physicalName="WF.Regraph"/>
>>              <queue physicalName="WF.Notify"/>
>>              <topic physicalName="WF.Events" />
>>            </forwardTo>
>>          </compositeQueue>
>>        </virtualDestinations>
>>      </virtualDestinationInterceptor>
>>    </destinationInterceptors>
>>
>>
>>    </broker>
>>
>>
>>     
>>> --
>>>
>>>
>>> On Fri, Jan 15, 2010 at 10:24 AM, David Delbecq <delbd@oma.be
>>> <mailto:delbd@oma.be>> wrote:
>>>
>>>     Hello,
>>>
>>>     can someone explain me why my activeMQ client is stalled in such
>>>     situation (i have 10 threads blocked like this having as
>>>     consequence a complete blocking of my client application):
>>>
>>>     "http-8081-Processor6" daemon prio=10 tid=00841c38 nid=512
>>>     lwp_id=2426611 waiting on condition [236f3000..236f153c]
>>>        java.lang.Thread.State: WAITING (parking)
>>>             at sun.misc.Unsafe.park(Native Method)
>>>             - parking to wait for  <40ae1828> (a
>>>
>>>       
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
>>     
>>>             at
>>>     java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
>>>             at
>>>
>>>       
>> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1889)
>>     
>>>             at
>>>
>>>       
>> java.util.concurrent.ArrayBlockingQueue.take(ArrayBlockingQueue.java:317)
>>     
>>>             at
>>>
>>>       
>> org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:40)
>>     
>>>             at
>>>
>>>       
>> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:80)
>>     
>>>             at
>>>
>>>       
>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1233)
>>     
>>>             at
>>>     org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1659)
>>>             - locked <40ae01e8> (a java.lang.Object)
>>>             at
>>>
>>>       
>> org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:227)
>>     
>>>             at
>>>
>>>       
>> org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:241)
>>     
>>>             at
>>>     be.rmi.shark.JMSEventManager.sendEvent(JMSEventManager.java:146)
>>>
>>>
>>>     considering that the activeMQ lists size, as referenced by
>>>     activeMQ admin interface are as follow:
>>>     Name      Number Of Pending Messages      Number Of Consumers
>>>     Messages Sent     Messages Received       Views           Operations
>>>     WF.Regraph        0       1       1714    1720    Browse  Send To
>>>       
>> Purge Delete
>>     
>>>     WF.Notify         0       1       1696    1742    Browse  Send To
>>>       
>> Purge Delete
>>     
>>>     WF.IncomingEvents         0       0       0       0       Browse
>>>       
>>  Send To Purge Delete
>>     
>>>     Is this to be considered normal that i must restart activeMQ
>>>     producers on a regular basis because of those this stalling? Where
>>>     can i find clear documentation on proper configuration of activeMQ
>>>     to prevent this stalling? Thank you.
>>>
>>>     --
>>>     David Delbecq
>>>     ICT
>>>     Institut Royal Météorologique
>>>     Ext:557
>>>
>>>
>>>
>>>       
>> --
>> David Delbecq
>> ICT
>> Institut Royal Météorologique
>> Ext:557
>>
>>
>>     
>
>
>   


-- 
David Delbecq
ICT
Institut Royal Météorologique
Ext:557


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message