activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dbrondy <dbro...@yahoo.fr>
Subject Re: Strange behavior using failover and network of broker
Date Mon, 12 Jul 2010 07:55:44 GMT

Hi Clark,

If you do reproduce the problem, it would be great.
For now, the only actions I can see to try fixing / targeting the problem
are the followings :
- try running our plateform using only one node (on the 4 availables) and
see what occurs
- try to upgrade to ACTIVEMQ 5.3.2.
- try changing the topology of our NoB
- collect the maximum of information... which can be huge enough ;-) (the
problem occurs after 3 days...)

Attached to this email, is a periodic thread dump of our application. I put
some comments on it. This file reflet the problem we are facing.

Concerning the wireFormat.maxInactivityDuration, what do you recommend ?
Increasing it ? Do we setup a too small value ?
Thanks for you help
denis

-> the file attached
http://old.nabble.com/file/p29136727/dm_debug_2010_07_02_08_28_19_light.txt
periodic_thread_dump.txt 




cobrien wrote:
> 
> denis,
> Your assumptions are correct when everything is functioning as expected.
> It appears to me that some external anomalous behavior causes a
> wireFormat.maxInactivityDuration to time out or perhaps an issue with the
> Subscriber.  I am going to try to reproduce the issue on my servers. 
> 
>  -Clark 
> 
> 
> www.ttmsolutions.com 
> ActiveMQ reference guide at 
> http://bit.ly/AMQRefGuide  
> 
> 
> 
> 
> dbrondy wrote:
>> 
>> 
>> Hi Cobrien, and thanks for you answer,
>> 
>> In my understanding, I set networkTTL equal to 1 because we need only one
>> hop to forward one produced message to a consumer, connected anywhere.
>> Let say's PubA sends on TEST topic using BROKER1, if Sub1, Sub2, Sub3 are
>> respectively connected to TEST Topic using BROKER2, BROKER3, BROKER4, the
>> 3 Subscribers will received it in only one hop.
>> Perhaps I'm wrong...
>> 
>> I see the option dynamicOnly=true but I didn't test it for now. In fact,
>> to be precise, this is our application which turn into OOM after 3 or 4
>> days of normal run.
>> 
>> In the first analysis, I thought our application was misbehaving but I
>> couldn't find any clue which could target the problem.
>> 
>> The fact is that prior to the crash, the ActiveMQTransport of my
>> application get reconnected periodically to other nodes... This is a
>> strange behavior.. And the ActiveMQ node logbook don't tell they faced a
>> network problem..
>> 
>> Could it be possible to get some important back flood messages while
>> switching the transport from one element to another one ? In such case,
>> our application would received an important flow of message...
>> 
>> Thanks for your feedback
>> denis
>> 
>> 
>> 
>> cobrien wrote:
>>> 
>>> denis, 
>>> if networkTTL="1",  without a consumer consuming messages on  BROKERn 
>>> messages will just accumulate.  I would try after  setting
>>> dynamicOnly=true on the network connectors. 
>>> 
>>>  -Clark 
>>> PS
>>> Which broker received the OOM error?
>>> 
>>> www.ttmsolutions.com 
>>> ActiveMQ reference guide at 
>>> http://bit.ly/AMQRefGuide  
>>> 
>>> 
>>> 
>>> 
>>> dbrondy wrote:
>>>> 
>>>> Hi everybody,
>>>> 
>>>> We are currently using ActiveMQ 5.2 application in our project and we
>>>> are glad to use this great app.
>>>> 
>>>> One of our java application is misbehaving while receiving and
>>>> producing message and I don't have a lot of clue to troubleshoot the
>>>> problem.
>>>> 
>>>> In fact, we are using 4 computers to run ActiveMQ broker. The following
>>>> configuration has been implemented :
>>>> 
>>>> BROKER1 : 
>>>> <broker useJmx="true" persistent="false" dataDirectory="data"
>>>> brokerName="activemq" xmlns="http://activemq.apache.org/schema/core">
>>>> <networkConnectors>
>>>> <networkConnector name="NoB" networkTTL="1"
>>>> uri="static://(tcp://BROKER2:61616,tcp://BROKER3:61616,tcp://BROKER4:61616)"/>
>>>> </networkConnectors>
>>>> <transportConnectors>
>>>> <transportConnector uri="tcp://BROKER1:61616"/>
>>>> </transportConnectors>
>>>> <managementContext>
>>>> <managementContext connectorPort="1399"
>>>> jmxDomainName="org.apache.activemq"/>
>>>> </managementContext>
>>>> </broker>
>>>> 
>>>> BROKER2 : 
>>>> <broker useJmx="true" persistent="false" dataDirectory="data"
>>>> brokerName="activemq" xmlns="http://activemq.apache.org/schema/core">
>>>> <networkConnectors>
>>>> <networkConnector name="NoB" networkTTL="1"
>>>> uri="static://(tcp://BROKER1:61616,tcp://BROKER3:61616,tcp://BROKER4:61616)"/>
>>>> </networkConnectors>
>>>> <transportConnectors>
>>>> <transportConnector uri="tcp://BROKER2:61616"/>
>>>> </transportConnectors>
>>>> <managementContext>
>>>> <managementContext connectorPort="1399"
>>>> jmxDomainName="org.apache.activemq"/>
>>>> </managementContext>
>>>> </broker>
>>>> 
>>>> BROKER3 : 
>>>> <broker useJmx="true" persistent="false" dataDirectory="data"
>>>> brokerName="activemq" xmlns="http://activemq.apache.org/schema/core">
>>>> <networkConnectors>
>>>> <networkConnector name="NoB" networkTTL="1"
>>>> uri="static://(tcp://BROKER1:61616,tcp://BROKER2:61616,tcp://BROKER4:61616)"/>
>>>> </networkConnectors>
>>>> <transportConnectors>
>>>> <transportConnector uri="tcp://BROKER3:61616"/>
>>>> </transportConnectors>
>>>> <managementContext>
>>>> <managementContext connectorPort="1399"
>>>> jmxDomainName="org.apache.activemq"/>
>>>> </managementContext>
>>>> </broker>
>>>> 
>>>> BROKER4 : 
>>>> <broker useJmx="true" persistent="false" dataDirectory="data"
>>>> brokerName="activemq" xmlns="http://activemq.apache.org/schema/core">
>>>> <networkConnectors>
>>>> <networkConnector name="NoB" networkTTL="1"
>>>> uri="static://(tcp://BROKER1:61616,tcp://BROKER2:61616,tcp://BROKER3:61616)"/>
>>>> </networkConnectors>
>>>> <transportConnectors>
>>>> <transportConnector uri="tcp://BROKER4:61616"/>
>>>> </transportConnectors>
>>>> <managementContext>
>>>> <managementContext connectorPort="1399"
>>>> jmxDomainName="org.apache.activemq"/>
>>>> </managementContext>
>>>> </broker>
>>>> 
>>>> Following is the illustrated topology :
>>>>  http://old.nabble.com/file/p29107245/topology.jpg 
>>>> 
>>>> All our applications use Topic. They publish and subscribe messages
>>>> using a TopicConnectionFactory defined as follow :
>>>> failover:(tcp://BROKER1:61616?connectionTimeout=2000&soTimeout=2000&wireFormat.maxInactivityDuration=2000,tcp://BROKER2:61616?connectionTimeout=2000&soTimeout=2000&wireFormat.maxInactivityDuration=2000,tcp://BROKER3:61616?connectionTimeout=2000&soTimeout=2000&wireFormat.maxInactivityDuration=2000,tcp://BROKER4:61616?connectionTimeout=2000&soTimeout=2000&wireFormat.maxInactivityDuration=2000)?jms.useAsyncSend=true&maxReconnectDelay=2000&backup=false&useExponentialBackOff=false&maxReconnectAttempts=2
>>>> 
>>>> The application causing troubles subscribes bytes messages on Topic A,
>>>> performs a dedicated processing internally and publishes object
>>>> messages on Topic B. The bytes messages posted on Topic A are created
>>>> by 6 or 7 publishers. Object messages published on Topic B are also
>>>> received by different consumers. All messages used for now are NON
>>>> PERSISTENT message and all subscritions are non durable with
>>>> AUTO-ACKNOWLEDGE mode.
>>>> 
>>>> After 3 or 4 days of normal work, the transport thread called "ActiveMQ
>>>> Transport: tcp://BROKER1/10.160.14.31:61616" starts been recreated and
>>>> connected on another broker element. More precisely, during 3 days, the
>>>> application used the BROKER1 for publish/subscribe and at a given time,
>>>> the transport thread get recreated passing randomly from one element to
>>>> all other BROKER element every 5 minutes (more or less). Remark : it
>>>> never came back connected on BROKER1. After couple of switching, I see
>>>> our application unable retaining a large amount of incoming messages
>>>> which cannot be treated in a timely maner. If we don't do anything, the
>>>> application will fails in JavaHeapSpace.
>>>> 
>>>> Could it be possible to get a kind of duplicated message flooding at
>>>> the time the transport get reconnected ? Is our BROKER configuration
>>>> suitable (topology and network connector definition) ? Does someone see
>>>> this problem already ?
>>>> 
>>>> I will really appreciate any clue, ideas or recommandation.
>>>> Tks in advance and thanks for all the great job you do.
>>>> denis
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 

-- 
View this message in context: http://old.nabble.com/Strange-behavior-using-failover-and-network-of-broker-tp29107245p29136727.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message