activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chirag Pujara <chiragpuj...@gmail.com>
Subject Re: Broker Hangs after some time - or does nothing.
Date Wed, 19 Feb 2014 15:47:48 GMT
Update:

I think I found problem why I see this behavior. We have bunch wars running
on tomcat and sending messages to broker. We have uptime deployment process
for all the wars. So when I undeploy war I think activemq connection
threads dont shut down properly and thats why I see exceptions like this in
broker logs:

2014-02-18 19:54:20,664 | WARN  | Transport Connection to: tcp://
10.190.225.56:39399 failed: java.io.EOFException |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
Transport: tcp:///10.190.225.56:39399@61618
2014-02-18 19:54:20,664 | WARN  | Transport Connection to: tcp://
10.190.225.56:39412 failed: java.net.SocketException: Connection reset |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
Transport: tcp:///10.190.225.56:39412@61618
2014-02-18 19:54:20,664 | WARN  | Transport Connection to: tcp://
10.190.225.56:39409 failed: java.io.EOFException |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
Transport: tcp:///10.190.225.56:39409@61618
2014-02-18 19:55:42,270 | WARN  | Transport Connection to: tcp://
10.190.133.15:33382 failed:
org.apache.activemq.transport.InactivityIOException: Channel was inactive
for too (>1000) long: tcp://10.190.133.15:33382 |
org.apache.activemq.broker.TransportConnection.Transport | ActiveMQ
InactivityMonitor Worker

I have tried tcpProperty: *daemon=true* on service side and on client side.
But if I understand right it helps cleaning up threads when JVM shuts down.
In my case we dont stop tomcat. All our deployment happens in uptime.

Anyother suggestion that I can try?

thanks,
chirag


On Tue, Feb 11, 2014 at 4:46 PM, Chirag Pujara <chiragpujara@gmail.com>wrote:

> I tried again and this time I didnt send bunch of messages at same time. I
> noticed that some message gets processed right away but some messges stays
> on queue and doesnt get processed. If send few more messges let say 50
> messages and it processes 40 messages and 10 msgs left on queue. I can send
> more messages but not all get processed.
>
> I am attaching config file. I have enabled producerFlowControl and removed
> slowConsumerStrategy. I am using activemq-5.10-SNAPSHOT.
>
>
>
> On Mon, Feb 10, 2014 at 9:04 PM, Chirag Pujara <chiragpujara@gmail.com>wrote:
>
>> i said broker hangs bcs if i tried to send more messages or messages on
>> queue doent get processed. i cannot even create queue or send message using
>> webconsole.
>>
>> will try again and rhis time will moniter memory.
>>
>> On Feb 10, 2014 6:22 PM, "artnaseef" <art@artnaseef.com> wrote:<
>>>
>>> Perhaps I misunderstood "Broker Hangs".  It helps to have more detailed
>>> symptoms.
>>>
>>> The messages in the DLQ are failing at the consumer - either transactions
>>> rolling back, or otherwise failing before commit, or CLIENT_ACKNOWLEDGE
>>> client calling Session.recover().
>>>
>>> See here for more details:
>>> http://activemq.apache.org/message-redelivery-and-dlq-handling.html
>>>
>>> You are right about the webconsole -- it will most likely stop
>>> functioning
>>> once the broker runs out of memory.
>>>
>>> >
>>> >
>>> > I have producerFlowControl to false. I seperated my producer and
>>> > consumers.
>>> > But still I was able to reproduce issue. For some time bunch of
>>> messages
>>> > started pile up in queue. Then they started to endup in DLQ. In DLQ I
>>> saw
>>> > most of them have these property:
>>> >
>>> > dlqDeliveryFailureCause: java.lang.Throwable: Exceeded redelivery
>>> policy
>>> > limit:RedeliveryPolicy {destination = null, collisionAvoidanceFactor =
>>> > 0.15, maximumRedeliveries = 6, maximumRedeliveryDelay = -1,
>>> > initialRedeliveryDelay = 1000, useCollisionAvoidance = false,
>>> > useExponentialBackOff = false, backOffMultiplier = 5.0,
>>> redeliveryDelay =
>>> > 1000}, cause:null
>>> >
>>> > Does this thing has anything to do with the issue?
>>> >
>>> > I was not monitoring heap size when it got stuck. that will be next
>>> thing
>>> > to do. But I can hit webconsole from that activemq instance. So if it
>>> had
>>> > memory issues how come console is still working? isn't it same process?
>>> >
>>> > thanks,
>>> > chirag
>>> >
>>> >
>>> > On Sat, Feb 8, 2014 at 12:36 AM, artnaseef <art@artnaseef.com> wrote:
>>> >
>>> >> How about a stack trace on the client?  Can you look for the consumer
>>> >> threads
>>> >> and see what they are doing?
>>> >>
>>> >> Note that with producers and consumers on the same connection, it's
>>> >> possible
>>> >> to reach a deadlock if producer-flow-control kicks in because the
>>> entire
>>> >> connection is blocked, not just the one producer.  There are two ways
>>> >> this
>>> >> can happen.  First, if the client produces and consumes the same
>>> >> destination.  Second, if the client is producing one destination and
>>> >> consuming a second destination and another client is consuming the
>>> first
>>> >> and
>>> >> producing to the second.  Actually, more complicated possibilities
>>> exist
>>> >> as
>>> >> well.
>>> >>
>>> >> Your best bet is to never consume and produce on the same connection
>>> so
>>> >> that
>>> >> consumption never blocks due to producer-flow-control.  In that
>>> >> scenario,
>>> >> deadlocks won't happen because consumers can always consume.
>>> >>
>>> >> Back to stack traces - if the consuming threads are blocked waiting
to
>>> >> obtain a lock while producing threads hold that lock, that could
>>> >> indicate
>>> >> the problem.
>>> >>
>>> >> Anyway, slow consumption is quite often the cause of broker hangs, so
>>> >> looking for slow consumption and understanding speeding consumption
>>> (or
>>> >> slowing production) is important.  If the broker JVM becomes
>>> completely
>>> >> unresponsive, try connecting jconsole or visualvm before the broker
>>> >> hangs
>>> >> and then watch memory.  If the JVM is running out of Heap or Permgen
>>> >> space,
>>> >> that would explain the hung broker.
>>> >>
>>> >>
>>> >>
>>> >> --
>>> >> View this message in context:
>>> >>
>>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677656.html
>>> >> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>> >>
>>> >
>>> >
>>> >
>>> >
>>> > _______________________________________________
>>> > If you reply to this email, your message will be added to the
>>> discussion
>>> > below:
>>> >
>>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677744.html
>>> > To start a new topic under ActiveMQ - User, email
>>> > ml-node+s2283324n2341805h3@n4.nabble.com
>>> > To unsubscribe from ActiveMQ - User, visit
>>> >
>>> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ==
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677751.html
>>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message