activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gary Tully <gary.tu...@gmail.com>
Subject Re: Broker Hangs after some time - or does nothing.
Date Wed, 19 Feb 2014 21:27:44 GMT
have you peeked at https://issues.apache.org/jira/browse/AMQ-3451

On 19 February 2014 15:47, Chirag Pujara <chiragpujara@gmail.com> wrote:
> Update:
>
> I think I found problem why I see this behavior. We have bunch wars running
> on tomcat and sending messages to broker. We have uptime deployment process
> for all the wars. So when I undeploy war I think activemq connection
> threads dont shut down properly and thats why I see exceptions like this in
> broker logs:
>
> 2014-02-18 19:54:20,664 | WARN  | Transport Connection to: tcp://
> 10.190.225.56:39399 failed: java.io.EOFException |
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> Transport: tcp:///10.190.225.56:39399@61618
> 2014-02-18 19:54:20,664 | WARN  | Transport Connection to: tcp://
> 10.190.225.56:39412 failed: java.net.SocketException: Connection reset |
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> Transport: tcp:///10.190.225.56:39412@61618
> 2014-02-18 19:54:20,664 | WARN  | Transport Connection to: tcp://
> 10.190.225.56:39409 failed: java.io.EOFException |
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> Transport: tcp:///10.190.225.56:39409@61618
> 2014-02-18 19:55:42,270 | WARN  | Transport Connection to: tcp://
> 10.190.133.15:33382 failed:
> org.apache.activemq.transport.InactivityIOException: Channel was inactive
> for too (>1000) long: tcp://10.190.133.15:33382 |
> org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
> InactivityMonitor Worker
>
> I have tried tcpProperty: *daemon=true* on service side and on client side.
> But if I understand right it helps cleaning up threads when JVM shuts down.
> In my case we dont stop tomcat. All our deployment happens in uptime.
>
> Anyother suggestion that I can try?
>
> thanks,
> chirag
>
>
> On Tue, Feb 11, 2014 at 4:46 PM, Chirag Pujara <chiragpujara@gmail.com>wrote:
>
>> I tried again and this time I didnt send bunch of messages at same time. I
>> noticed that some message gets processed right away but some messges stays
>> on queue and doesnt get processed. If send few more messges let say 50
>> messages and it processes 40 messages and 10 msgs left on queue. I can send
>> more messages but not all get processed.
>>
>> I am attaching config file. I have enabled producerFlowControl and removed
>> slowConsumerStrategy. I am using activemq-5.10-SNAPSHOT.
>>
>>
>>
>> On Mon, Feb 10, 2014 at 9:04 PM, Chirag Pujara <chiragpujara@gmail.com>wrote:
>>
>>> i said broker hangs bcs if i tried to send more messages or messages on
>>> queue doent get processed. i cannot even create queue or send message using
>>> webconsole.
>>>
>>> will try again and rhis time will moniter memory.
>>>
>>> On Feb 10, 2014 6:22 PM, "artnaseef" <art@artnaseef.com> wrote:<
>>>>
>>>> Perhaps I misunderstood "Broker Hangs".  It helps to have more detailed
>>>> symptoms.
>>>>
>>>> The messages in the DLQ are failing at the consumer - either transactions
>>>> rolling back, or otherwise failing before commit, or CLIENT_ACKNOWLEDGE
>>>> client calling Session.recover().
>>>>
>>>> See here for more details:
>>>> http://activemq.apache.org/message-redelivery-and-dlq-handling.html
>>>>
>>>> You are right about the webconsole -- it will most likely stop
>>>> functioning
>>>> once the broker runs out of memory.
>>>>
>>>> >
>>>> >
>>>> > I have producerFlowControl to false. I seperated my producer and
>>>> > consumers.
>>>> > But still I was able to reproduce issue. For some time bunch of
>>>> messages
>>>> > started pile up in queue. Then they started to endup in DLQ. In DLQ
I
>>>> saw
>>>> > most of them have these property:
>>>> >
>>>> > dlqDeliveryFailureCause: java.lang.Throwable: Exceeded redelivery
>>>> policy
>>>> > limit:RedeliveryPolicy {destination = null, collisionAvoidanceFactor
=
>>>> > 0.15, maximumRedeliveries = 6, maximumRedeliveryDelay = -1,
>>>> > initialRedeliveryDelay = 1000, useCollisionAvoidance = false,
>>>> > useExponentialBackOff = false, backOffMultiplier = 5.0,
>>>> redeliveryDelay =
>>>> > 1000}, cause:null
>>>> >
>>>> > Does this thing has anything to do with the issue?
>>>> >
>>>> > I was not monitoring heap size when it got stuck. that will be next
>>>> thing
>>>> > to do. But I can hit webconsole from that activemq instance. So if it
>>>> had
>>>> > memory issues how come console is still working? isn't it same process?
>>>> >
>>>> > thanks,
>>>> > chirag
>>>> >
>>>> >
>>>> > On Sat, Feb 8, 2014 at 12:36 AM, artnaseef <art@artnaseef.com>
wrote:
>>>> >
>>>> >> How about a stack trace on the client?  Can you look for the consumer
>>>> >> threads
>>>> >> and see what they are doing?
>>>> >>
>>>> >> Note that with producers and consumers on the same connection, it's
>>>> >> possible
>>>> >> to reach a deadlock if producer-flow-control kicks in because the
>>>> entire
>>>> >> connection is blocked, not just the one producer.  There are two
ways
>>>> >> this
>>>> >> can happen.  First, if the client produces and consumes the same
>>>> >> destination.  Second, if the client is producing one destination
and
>>>> >> consuming a second destination and another client is consuming the
>>>> first
>>>> >> and
>>>> >> producing to the second.  Actually, more complicated possibilities
>>>> exist
>>>> >> as
>>>> >> well.
>>>> >>
>>>> >> Your best bet is to never consume and produce on the same connection
>>>> so
>>>> >> that
>>>> >> consumption never blocks due to producer-flow-control.  In that
>>>> >> scenario,
>>>> >> deadlocks won't happen because consumers can always consume.
>>>> >>
>>>> >> Back to stack traces - if the consuming threads are blocked waiting
to
>>>> >> obtain a lock while producing threads hold that lock, that could
>>>> >> indicate
>>>> >> the problem.
>>>> >>
>>>> >> Anyway, slow consumption is quite often the cause of broker hangs,
so
>>>> >> looking for slow consumption and understanding speeding consumption
>>>> (or
>>>> >> slowing production) is important.  If the broker JVM becomes
>>>> completely
>>>> >> unresponsive, try connecting jconsole or visualvm before the broker
>>>> >> hangs
>>>> >> and then watch memory.  If the JVM is running out of Heap or Permgen
>>>> >> space,
>>>> >> that would explain the hung broker.
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> View this message in context:
>>>> >>
>>>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677656.html
>>>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > If you reply to this email, your message will be added to the
>>>> discussion
>>>> > below:
>>>> >
>>>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677744.html
>>>> > To start a new topic under ActiveMQ - User, email
>>>> > ml-node+s2283324n2341805h3@n4.nabble.com
>>>> > To unsubscribe from ActiveMQ - User, visit
>>>> >
>>>> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ==
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677751.html
>>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>>
>>>
>>



-- 
http://redhat.com
http://blog.garytully.com

Mime
View raw message