activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From dbrondy <dbro...@yahoo.fr>
Subject Re: Strange behavior using failover and network of broker
Date Thu, 15 Jul 2010 07:27:44 GMT


Hi all,

We faced again a crash...
Our application get its transport switched from one element to another in
the last 3 hours...

Any other clue ?
I think it's time for us to perform other tests like exposed previously
denis



dbrondy wrote:
> 
> Hi Clark,
> 
> If you do reproduce the problem, it would be great.
> For now, the only actions I can see to try fixing / targeting the problem
> are the followings :
> - try running our plateform using only one node (on the 4 availables) and
> see what occurs
> - try to upgrade to ACTIVEMQ 5.3.2.
> - try changing the topology of our NoB
> - collect the maximum of information... which can be huge enough ;-) (the
> problem occurs after 3 days...)
> 
> Attached to this email, is a periodic thread dump of our application. I
> put some comments on it. This file reflet the problem we are facing.
> 
> Concerning the wireFormat.maxInactivityDuration, what do you recommend ?
> Increasing it ? Do we setup a too small value ?
> Thanks for you help
> denis
> 
> -> the file attached
> 
> http://old.nabble.com/file/p29136727/dm_debug_2010_07_02_08_28_19_light.txt
> periodic_thread_dump.txt 
> 
> 
> 
> 
> cobrien wrote:
>> 
>> denis,
>> Your assumptions are correct when everything is functioning as expected.
>> It appears to me that some external anomalous behavior causes a
>> wireFormat.maxInactivityDuration to time out or perhaps an issue with the
>> Subscriber.  I am going to try to reproduce the issue on my servers. 
>> 
>>  -Clark 
>> 
>> 
>> www.ttmsolutions.com 
>> ActiveMQ reference guide at 
>> http://bit.ly/AMQRefGuide  
>> 
>> 
>> 
>> 
>> dbrondy wrote:
>>> 
>>> 
>>> Hi Cobrien, and thanks for you answer,
>>> 
>>> In my understanding, I set networkTTL equal to 1 because we need only
>>> one hop to forward one produced message to a consumer, connected
>>> anywhere.
>>> Let say's PubA sends on TEST topic using BROKER1, if Sub1, Sub2, Sub3
>>> are respectively connected to TEST Topic using BROKER2, BROKER3,
>>> BROKER4, the 3 Subscribers will received it in only one hop.
>>> Perhaps I'm wrong...
>>> 
>>> I see the option dynamicOnly=true but I didn't test it for now. In fact,
>>> to be precise, this is our application which turn into OOM after 3 or 4
>>> days of normal run.
>>> 
>>> In the first analysis, I thought our application was misbehaving but I
>>> couldn't find any clue which could target the problem.
>>> 
>>> The fact is that prior to the crash, the ActiveMQTransport of my
>>> application get reconnected periodically to other nodes... This is a
>>> strange behavior.. And the ActiveMQ node logbook don't tell they faced a
>>> network problem..
>>> 
>>> Could it be possible to get some important back flood messages while
>>> switching the transport from one element to another one ? In such case,
>>> our application would received an important flow of message...
>>> 
>>> Thanks for your feedback
>>> denis
>>> 
>>> 
>>> 
>>> cobrien wrote:
>>>> 
>>>> denis, 
>>>> if networkTTL="1",  without a consumer consuming messages on  BROKERn 
>>>> messages will just accumulate.  I would try after  setting
>>>> dynamicOnly=true on the network connectors. 
>>>> 
>>>>  -Clark 
>>>> PS
>>>> Which broker received the OOM error?
>>>> 
>>>> www.ttmsolutions.com 
>>>> ActiveMQ reference guide at 
>>>> http://bit.ly/AMQRefGuide  
>>>> 
>>>> 
>>>> 
>>>> 
>>>> dbrondy wrote:
>>>>> 
>>>>> Hi everybody,
>>>>> 
>>>>> We are currently using ActiveMQ 5.2 application in our project and we
>>>>> are glad to use this great app.
>>>>> 
>>>>> One of our java application is misbehaving while receiving and
>>>>> producing message and I don't have a lot of clue to troubleshoot the
>>>>> problem.
>>>>> 
>>>>> In fact, we are using 4 computers to run ActiveMQ broker. The
>>>>> following configuration has been implemented :
>>>>> 
>>>>> BROKER1 : 
>>>>> <broker useJmx="true" persistent="false" dataDirectory="data"
>>>>> brokerName="activemq" xmlns="http://activemq.apache.org/schema/core">
>>>>> <networkConnectors>
>>>>> <networkConnector name="NoB" networkTTL="1"
>>>>> uri="static://(tcp://BROKER2:61616,tcp://BROKER3:61616,tcp://BROKER4:61616)"/>
>>>>> </networkConnectors>
>>>>> <transportConnectors>
>>>>> <transportConnector uri="tcp://BROKER1:61616"/>
>>>>> </transportConnectors>
>>>>> <managementContext>
>>>>> <managementContext connectorPort="1399"
>>>>> jmxDomainName="org.apache.activemq"/>
>>>>> </managementContext>
>>>>> </broker>
>>>>> 
>>>>> BROKER2 : 
>>>>> <broker useJmx="true" persistent="false" dataDirectory="data"
>>>>> brokerName="activemq" xmlns="http://activemq.apache.org/schema/core">
>>>>> <networkConnectors>
>>>>> <networkConnector name="NoB" networkTTL="1"
>>>>> uri="static://(tcp://BROKER1:61616,tcp://BROKER3:61616,tcp://BROKER4:61616)"/>
>>>>> </networkConnectors>
>>>>> <transportConnectors>
>>>>> <transportConnector uri="tcp://BROKER2:61616"/>
>>>>> </transportConnectors>
>>>>> <managementContext>
>>>>> <managementContext connectorPort="1399"
>>>>> jmxDomainName="org.apache.activemq"/>
>>>>> </managementContext>
>>>>> </broker>
>>>>> 
>>>>> BROKER3 : 
>>>>> <broker useJmx="true" persistent="false" dataDirectory="data"
>>>>> brokerName="activemq" xmlns="http://activemq.apache.org/schema/core">
>>>>> <networkConnectors>
>>>>> <networkConnector name="NoB" networkTTL="1"
>>>>> uri="static://(tcp://BROKER1:61616,tcp://BROKER2:61616,tcp://BROKER4:61616)"/>
>>>>> </networkConnectors>
>>>>> <transportConnectors>
>>>>> <transportConnector uri="tcp://BROKER3:61616"/>
>>>>> </transportConnectors>
>>>>> <managementContext>
>>>>> <managementContext connectorPort="1399"
>>>>> jmxDomainName="org.apache.activemq"/>
>>>>> </managementContext>
>>>>> </broker>
>>>>> 
>>>>> BROKER4 : 
>>>>> <broker useJmx="true" persistent="false" dataDirectory="data"
>>>>> brokerName="activemq" xmlns="http://activemq.apache.org/schema/core">
>>>>> <networkConnectors>
>>>>> <networkConnector name="NoB" networkTTL="1"
>>>>> uri="static://(tcp://BROKER1:61616,tcp://BROKER2:61616,tcp://BROKER3:61616)"/>
>>>>> </networkConnectors>
>>>>> <transportConnectors>
>>>>> <transportConnector uri="tcp://BROKER4:61616"/>
>>>>> </transportConnectors>
>>>>> <managementContext>
>>>>> <managementContext connectorPort="1399"
>>>>> jmxDomainName="org.apache.activemq"/>
>>>>> </managementContext>
>>>>> </broker>
>>>>> 
>>>>> Following is the illustrated topology :
>>>>>  http://old.nabble.com/file/p29107245/topology.jpg 
>>>>> 
>>>>> All our applications use Topic. They publish and subscribe messages
>>>>> using a TopicConnectionFactory defined as follow :
>>>>> failover:(tcp://BROKER1:61616?connectionTimeout=2000&soTimeout=2000&wireFormat.maxInactivityDuration=2000,tcp://BROKER2:61616?connectionTimeout=2000&soTimeout=2000&wireFormat.maxInactivityDuration=2000,tcp://BROKER3:61616?connectionTimeout=2000&soTimeout=2000&wireFormat.maxInactivityDuration=2000,tcp://BROKER4:61616?connectionTimeout=2000&soTimeout=2000&wireFormat.maxInactivityDuration=2000)?jms.useAsyncSend=true&maxReconnectDelay=2000&backup=false&useExponentialBackOff=false&maxReconnectAttempts=2
>>>>> 
>>>>> The application causing troubles subscribes bytes messages on Topic A,
>>>>> performs a dedicated processing internally and publishes object
>>>>> messages on Topic B. The bytes messages posted on Topic A are created
>>>>> by 6 or 7 publishers. Object messages published on Topic B are also
>>>>> received by different consumers. All messages used for now are NON
>>>>> PERSISTENT message and all subscritions are non durable with
>>>>> AUTO-ACKNOWLEDGE mode.
>>>>> 
>>>>> After 3 or 4 days of normal work, the transport thread called
>>>>> "ActiveMQ Transport: tcp://BROKER1/10.160.14.31:61616" starts been
>>>>> recreated and connected on another broker element. More precisely,
>>>>> during 3 days, the application used the BROKER1 for publish/subscribe
>>>>> and at a given time, the transport thread get recreated passing
>>>>> randomly from one element to all other BROKER element every 5 minutes
>>>>> (more or less). Remark : it never came back connected on BROKER1.
>>>>> After couple of switching, I see our application unable retaining a
>>>>> large amount of incoming messages which cannot be treated in a timely
>>>>> maner. If we don't do anything, the application will fails in
>>>>> JavaHeapSpace.
>>>>> 
>>>>> Could it be possible to get a kind of duplicated message flooding at
>>>>> the time the transport get reconnected ? Is our BROKER configuration
>>>>> suitable (topology and network connector definition) ? Does someone
>>>>> see this problem already ?
>>>>> 
>>>>> I will really appreciate any clue, ideas or recommandation.
>>>>> Tks in advance and thanks for all the great job you do.
>>>>> denis
>>>>> 
>>>>> 
>>>> 
>>>> 
>>> 
>>> 
>> 
>> 
> 
> 

-- 
View this message in context: http://old.nabble.com/Strange-behavior-using-failover-and-network-of-broker-tp29107245p29170020.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message