activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefanic <snico...@movingintelligence.com>
Subject Re: Consumers hanging on a queue although there are messages in it
Date Fri, 02 Feb 2018 14:57:15 GMT
I will try to remember doing so, are you interested in the CPU sampling
snapshot? And if so over what amount of time?

(I'm not very experienced with profiling/sampling, only really done heapdump
and threaddump analysis)


Tim Bain wrote
> Would you be able to use JVisualVM to perform CPU sampling while this is
> occurring, to find out what those threads are doing? **WARNING** Do NOT do
> CPU *profiling*, which would slow your broker significantly; sampling is
> lightweight and generally held to be safe to do against an operational
> process, but profiling is very heavyweight. Make sure you're on the right
> tab in JVisualVM.
> 
> Tim
> 
> On Feb 2, 2018 12:05 AM, "Stefanic" &lt;

> snicodem@

> &gt; wrote:
> 
>> Hi Tim,
>>
>> There are no selectors, so there are no messages left behind.
>> We have seen this three times now in production and every time when the
>> queue is empty (after about 30 minutes of refreshing) the problem goes
>> away.
>>
>> Threads behavior is rather strange, when the blocking issue is on-going
>> we
>> see 1 thread constantly running (100%) in visualvm, and all other threads
>> (31 in our case) almost seem synchronized on the same object because all
>> of
>> them start running at the same time and go into timed_waiting state at
>> the
>> same time.
>> That results in 33% running of all other threads and that is not enough
>> to
>> keep our queue empty.
>>
>> Reproducing this behavior is nearly impossible for us, it occurs randomly
>> within about a week.
>>
>> Here are the things we tried so far:
>> - Downgraded ActiveMQ broker from 5.15.2 to our previous version 5.11.1
>> but
>> the problem occurred again on that version so we assume it is not the
>> broker/server
>> - Yesterday we downgraded both ActiveMQ and Camel client libraries:
>> ActiveMQ
>> down to 5.14.5 and Camel from 2.20.1 down to 2.19.4 (I would have rather
>> only downgraded 1 at a time but it's our production environment so cannot
>> play around too much)
>>
>> I will reply here if we experience the blocked state again.
>>
>>
>>
>> --
>> Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-
>> f2341805.html
>>





--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html

Mime
View raw message