activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Netlancer <netlan...@mobigraph.co>
Subject Re: ActiveMQ JVM crashes when high number of concurrent connections get terminated
Date Tue, 03 Jun 2014 03:54:37 GMT
Hi,

  Please, can you let me know what is the effect of making 
SelectorManager ThreadPoolExecutor as bounded ?

Thanks,
nl

On 02.06.2014 09.26, Netlancer wrote:
> Hi,
>
>  yes. we increased the heap size to 8G in a machine which has 16GB 
> RAM.  the limits.conf is configured with 300000 as soft/hard limit and 
> file-max is the same as hard limit. I have attached the hs_err file .
>
> Thanks,
>
> On 02.06.2014 08.52, Gaurav Sharma wrote:
>> If possible, pls provide some more info to help diagnose this. What's 
>> the output of /etc/security/limits.conf and file-max value in 
>> /etc/sysctl.conf?
>>
>> Is 5G the max you can allocate to amq's jvm?
>>
>> Sent from my iPhone
>>
>> On Jun 1, 2014, at 20:06, Netlancer <netlancer@mobigraph.co> wrote:
>>
>> Hi,
>>
>> We have activemq running behind an LB.  we have close to 55K 
>> connections to activemq. during cases of more than 1000 connections 
>> gets terminated concurrently, we observe a huge spike in the number 
>> of threads created.following which the jvm crashes. on further 
>> investigation, these threads are pertained to executors
>>
>> SelectorManager - defaultExecutor - labelled with "ActiveMQ NIO 
>> Worker" - the executor has maximum thread pool size of Integer.MAX_VALUE
>> TaskRunnerFactory - defaultExecutor - labelled as "ActiveMQ Task" - 
>> this executor as well has Integer.MAX_VALUE as the max pool size.
>> BrokerService.getTaskRunnerFactory - labelled as "ActiveMQ 
>> BrokerService[%s] Task-" - this executor is also created with 
>> Integer.MAX_VALUE
>>
>> the configured persistence store is leveldb and dedicatedtaskrunner 
>> is disabled. The JVM is running with 8GB RAM with the max heap at 5G.
>>
>> following snippet from the hs_pid crash file
>>
>> Java Threads: ( => current thread )
>>   0x00007f1e42863800 JavaThread "ActiveMQ NIO Worker 9053" daemon 
>> [_thread_blocked, id=10690, 
>> stack(0x00007f1d6fa53000,0x00007f1d6fa8c000)]
>>   0x00007f1ea40dc000 JavaThread "ActiveMQ NIO Worker 9047" daemon 
>> [_thread_blocked, id=10689, 
>> stack(0x00007f1d6fa8c000,0x00007f1d6fac5000)]
>>   0x00007f1eac116800 JavaThread "ActiveMQ NIO Worker 9054" daemon 
>> [_thread_blocked, id=10688, 
>> stack(0x00007f1d6fac5000,0x00007f1d6fafe000)]
>>   0x00007f1df43f5800 JavaThread "ActiveMQ NIO Worker 9046" daemon 
>> [_thread_blocked, id=10687, 
>> stack(0x00007f1d6fafe000,0x00007f1d6fb37000)]
>>   0x00007f1e42870000 JavaThread "ActiveMQ NIO Worker 9056" daemon 
>> [_thread_blocked, id=10686, 
>> stack(0x00007f1d6fb37000,0x00007f1d6fb70000)]
>>   0x00007f1ef6054800 JavaThread "ActiveMQ NIO Worker 9107" daemon 
>> [_thread_blocked, id=10685, 
>> stack(0x00007f1d6fb70000,0x00007f1d6fba9000)]
>>   0x00007f1ef46af800 JavaThread "ActiveMQ NIO Worker 9059" daemon 
>> [_thread_blocked, id=10684, 
>> stack(0x00007f1d6fba9000,0x00007f1d6fbe2000)]
>>   0x00007f1e240cd000 JavaThread "ActiveMQ NIO Worker 9041" daemon 
>> [_thread_blocked, id=10683, 
>> stack(0x00007f1d6fbe2000,0x00007f1d6fc1b000)]
>>   0x00007f1e0016a800 JavaThread "ActiveMQ NIO Worker 9060" daemon 
>> [_thread_blocked, id=10682, 
>> stack(0x00007f1d6fc1b000,0x00007f1d6fc54000)]
>>   0x00007f1e5019d800 JavaThread "ActiveMQ NIO Worker 9061" daemon 
>> [_thread_blocked, id=10681, 
>> stack(0x00007f1d6fc54000,0x00007f1d6fc8d000)]
>>   0x00007f1e8006f000 JavaThread "ActiveMQ NIO Worker 9039" daemon 
>> [_thread_blocked, id=10680, 
>> stack(0x00007f1d6fc8d000,0x00007f1d6fcc6000)]
>>   0x00007f1e4c7df800 JavaThread "ActiveMQ NIO Worker 9101" daemon 
>> [_thread_blocked, id=10679, 
>> stack(0x00007f1d6fcc6000,0x00007f1d6fcff000)]
>>
>>   0x00007f1da6794800 JavaThread "ActiveMQ BrokerService[MQMaster] 
>> Task-1781" daemon [_thread_blocked, id=8302, 
>> stack(0x00007f1db5a51000,0x00007f1db5a8a000)]
>>   0x00007f1da6791000 JavaThread "ActiveMQ BrokerService[MQMaster] 
>> Task-1780" daemon [_thread_blocked, id=8298, 
>> stack(0x00007f1dc6d4c000,0x00007f1dc6d85000)]
>>   0x00007f1da6793000 JavaThread "ActiveMQ BrokerService[MQMaster] 
>> Task-1779" daemon [_thread_blocked, id=8296, 
>> stack(0x00007f1dc6e30000,0x00007f1dc6e69000)]
>>   0x00007f1da6790000 JavaThread "ActiveMQ BrokerService[MQMaster] 
>> Task-1778" daemon [_thread_blocked, id=8293, 
>> stack(0x00007f1db6903000,0x00007f1db693c000)]
>>   0x00007f1da678f000 JavaThread "ActiveMQ BrokerService[MQMaster] 
>> Task-1777" daemon [_thread_blocked, id=8292, 
>> stack(0x00007f1db9dc8000,0x00007f1db9e01000)]
>>
>> Please, can someone provide inputs on why this is happening.
>>
>> Thanks
>> nl
>>
>>
>>
>>
>>
>



Mime
View raw message