activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chirag Pujara <chiragpuj...@gmail.com>
Subject Re: Activemq froze - and msgs dont get processed
Date Mon, 12 May 2014 15:46:31 GMT
I keep seing this behavior time to time. I have to cycle instance.

broker logs are filled with info logs:

INFO | The connection to 'tcp://10.190.133.15:37168' is taking a long time
to shutdown. | org.apache.activemq.broker.TransportConnection | ActiveMQ
Transport: tcp:///10.190.133.15:55356@61616

any suggestions.



On Wed, Apr 16, 2014 at 10:14 AM, Chirag Pujara <chiragpujara@gmail.com>wrote:

> I checked logs. there are bunch of Info messages about conection taking
> too long to shutdown.
>
> 2014-04-15 10:51:00,668 | INFO  | The connection to 'tcp://
> 10.190.133.15:37168' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> tcp:///10.190.133.15:55356@61616
> 2014-04-15 10:51:00,669 | INFO  | The connection to 'tcp://
> 10.190.133.14:58333' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> tcp:///10.190.133.14:48314@61616
> 2014-04-15 10:51:00,740 | INFO  | The connection to 'tcp://
> 10.190.133.15:54436' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> tcp:///10.190.133.15:55568@61616
> 2014-04-15 10:51:00,741 | INFO  | The connection to 'tcp://
> 10.190.133.15:37172' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> tcp:///10.190.133.15:55575@61616
> 2014-04-15 10:51:00,989 | INFO  | The connection to 'tcp://
> 10.190.133.14:58360' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> tcp:///10.190.133.14:48072@61616
> 2014-04-15 10:51:00,989 | INFO  | The connection to 'tcp://
> 10.190.133.14:58370' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> tcp:///10.190.133.14:48290@61616
> 2014-04-15 10:51:00,994 | INFO  | The connection to 'tcp://
> 10.190.133.14:47258' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> tcp:///10.190.133.14:48296@61616
> 2014-04-15 10:51:01,755 | INFO  | The connection to 'tcp://
> 10.190.133.14:47191' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> tcp:///10.190.133.14:47985@61616
> 2014-04-15 10:51:01,935 | INFO  | The connection to 'tcp://
> 10.190.133.14:46963' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> tcp:///10.190.133.14:48308@61616
> 2014-04-15 10:51:02,362 | INFO  | The connection to 'tcp://
> 10.190.133.15:54359' is taking a long time to shutdown. |
> org.apache.activemq.broker.TransportConnection | ActiveMQ Transport:
> tcp:///10.190.133.15:55261@61616
>
> I also checked logs around time when stop processing messages. I saw some
> warning messages.
>
> 2014-04-14 18:08:35,782 | WARN  | Transport Connection to: tcp://10.184.132.34:56916
failed: java.net.SocketException: Connection reset | org.apache.activemq.broker.TransportConnection.Transport
| ActiveMQ Transport: tcp:///10.184.132.34:56916@61616
>
> 2014-04-14 18:52:54,576 | WARN  | Transport Connection to: tcp://10.190.225.39:48556
failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport |
ActiveMQ Transport: tcp:///10.190.225.39:48556@61616
>
> 2014-04-14 19:43:35,398 | WARN  | Transport Connection to: tcp://10.190.225.39:49153
failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport |
ActiveMQ Transport: tcp:///10.190.225.39:49153@61616
>
> 2014-04-14 19:43:46,813 | WARN  | Transport Connection to: tcp://10.190.133.15:37178
failed: java.net.SocketException: Broken pipe | org.apache.activemq.broker.TransportConnection.Transport
| ActiveMQ InactivityMonitor Worker
>
> 2014-04-14 19:51:36,954 | WARN  | Transport Connection to: tcp://10.190.225.56:51240
failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport |
ActiveMQ Transport: tcp:///10.190.225.56:51240@61616
>
> 2014-04-14 20:00:27,651 | WARN  | Transport Connection to: tcp://10.190.225.72:53818
failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport |
ActiveMQ Transport: tcp:///10.190.225.72:53818@61616
>
> 2014-04-14 20:00:28,096 | WARN  | Transport Connection to: tcp://10.190.225.72:53676
failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport |
ActiveMQ Transport: tcp:///10.190.225.72:53676@61616
>
> 2014-04-14 20:00:28,097 | WARN  | Transport Connection to: tcp://10.190.225.72:53780
failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport |
ActiveMQ Transport: tcp:///10.190.225.72:53780@61616
>
> 2014-04-14 20:00:28,097 | WARN  | Transport Connection to: tcp://10.190.225.72:53718
failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport |
ActiveMQ Transport: tcp:///10.190.225.72:53718@61616
>
> 2014-04-14 20:00:28,098 | WARN  | Transport Connection to: tcp://10.190.225.72:53766
failed: java.io.EOFException | org.apache.activemq.broker.TransportConnection.Transport |
ActiveMQ Transport: tcp:///10.190.225.72:53766@61616
>
>
> We have zabbix monitor for activemq. I didnt see anything alarming for memory usage or
store usage.
>
> Enqueue and dequeue count were almost same most of time. Total message count per minute
was almost zero all the time.
>
>
> I found this similar post and there was bug related to issue. but it was fixed in 5.6
release. Does it mean it was not fixed or reintroduced due to other code change.
>
> Post: http://activemq.2283324.n4.nabble.com/Connection-taking-long-time-to-shutdown-td3717825.html
>
> Bug: https://issues.apache.org/jira/browse/AMQ-3435
>
>
> any suggestions.
>
>
> thanks,
>
> chirag
>
>
>
> On Tue, Apr 15, 2014 at 9:06 PM, artnaseef <art@artnaseef.com> wrote:
>
>> The most common cause of message flow stopping is slow consumption
>> leading to
>> exhaustion of the broker's resources.
>>
>> Things to check:
>> - Broker memory usage
>> - Broker store usage
>> - ActiveMQ log file (check for producer flow control or resource usage
>> messages)
>>
>> BTW, it is critical with any ActiveMQ installation to ensure consumption
>> >=
>> production always.  Short bursts do happen and can be handled, but
>> ultimately, you really never want to be in a situation in which
>> consumption
>> is less than production for a long time.  And, when it is known that a
>> burst
>> of production will outpace consumption, it's important to understand the
>> numbers and sizes of messages that will be in the backlog to ensure the
>> broker's memory and store settings can handle it.
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Activemq-froze-and-msgs-dont-get-processed-tp4680313p4680318.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message