activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From artnaseef <...@artnaseef.com>
Subject Re: Broker Hangs after some time - or does nothing.
Date Tue, 11 Feb 2014 00:21:46 GMT
Perhaps I misunderstood "Broker Hangs".  It helps to have more detailed
symptoms.

The messages in the DLQ are failing at the consumer - either transactions
rolling back, or otherwise failing before commit, or CLIENT_ACKNOWLEDGE
client calling Session.recover().

See here for more details:
http://activemq.apache.org/message-redelivery-and-dlq-handling.html

You are right about the webconsole -- it will most likely stop functioning
once the broker runs out of memory.

>
>
> I have producerFlowControl to false. I seperated my producer and
> consumers.
> But still I was able to reproduce issue. For some time bunch of messages
> started pile up in queue. Then they started to endup in DLQ. In DLQ I saw
> most of them have these property:
>
> dlqDeliveryFailureCause: java.lang.Throwable: Exceeded redelivery policy
> limit:RedeliveryPolicy {destination = null, collisionAvoidanceFactor =
> 0.15, maximumRedeliveries = 6, maximumRedeliveryDelay = -1,
> initialRedeliveryDelay = 1000, useCollisionAvoidance = false,
> useExponentialBackOff = false, backOffMultiplier = 5.0, redeliveryDelay =
> 1000}, cause:null
>
> Does this thing has anything to do with the issue?
>
> I was not monitoring heap size when it got stuck. that will be next thing
> to do. But I can hit webconsole from that activemq instance. So if it had
> memory issues how come console is still working? isn't it same process?
>
> thanks,
> chirag
>
>
> On Sat, Feb 8, 2014 at 12:36 AM, artnaseef <art@artnaseef.com> wrote:
>
>> How about a stack trace on the client?  Can you look for the consumer
>> threads
>> and see what they are doing?
>>
>> Note that with producers and consumers on the same connection, it's
>> possible
>> to reach a deadlock if producer-flow-control kicks in because the entire
>> connection is blocked, not just the one producer.  There are two ways
>> this
>> can happen.  First, if the client produces and consumes the same
>> destination.  Second, if the client is producing one destination and
>> consuming a second destination and another client is consuming the first
>> and
>> producing to the second.  Actually, more complicated possibilities exist
>> as
>> well.
>>
>> Your best bet is to never consume and produce on the same connection so
>> that
>> consumption never blocks due to producer-flow-control.  In that
>> scenario,
>> deadlocks won't happen because consumers can always consume.
>>
>> Back to stack traces - if the consuming threads are blocked waiting to
>> obtain a lock while producing threads hold that lock, that could
>> indicate
>> the problem.
>>
>> Anyway, slow consumption is quite often the cause of broker hangs, so
>> looking for slow consumption and understanding speeding consumption (or
>> slowing production) is important.  If the broker JVM becomes completely
>> unresponsive, try connecting jconsole or visualvm before the broker
>> hangs
>> and then watch memory.  If the JVM is running out of Heap or Permgen
>> space,
>> that would explain the hung broker.
>>
>>
>>
>> --
>> View this message in context:
>> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677656.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>
>
>
>
> _______________________________________________
> If you reply to this email, your message will be added to the discussion
> below:
> http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677744.html
> To start a new topic under ActiveMQ - User, email
> ml-node+s2283324n2341805h3@n4.nabble.com
> To unsubscribe from ActiveMQ - User, visit
> http://activemq.2283324.n4.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2341805&code=YXJ0QGFydG5hc2VlZi5jb218MjM0MTgwNXwtMjA1NDcyNjY5MQ==






--
View this message in context: http://activemq.2283324.n4.nabble.com/Broker-Hangs-after-some-time-or-does-nothing-tp4677506p4677751.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message