activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Netlancer <>
Subject ActiveMQ JVM crashes when high number of concurrent connections get terminated
Date Mon, 02 Jun 2014 03:06:40 GMT

  We have activemq running behind an LB.  we have close to 55K 
connections to activemq. during cases of more than 1000 connections gets 
terminated concurrently, we observe a huge spike in the number of 
threads created.following which the jvm crashes. on further 
investigation, these threads are pertained to executors

SelectorManager - defaultExecutor - labelled with "ActiveMQ NIO Worker" 
- the executor has maximum thread pool size of Integer.MAX_VALUE
TaskRunnerFactory - defaultExecutor - labelled as "ActiveMQ Task" - this 
executor as well has Integer.MAX_VALUE as the max pool size.
BrokerService.getTaskRunnerFactory - labelled as "ActiveMQ 
BrokerService[%s] Task-" - this executor is also created with 

the configured persistence store is leveldb and dedicatedtaskrunner is 
disabled. The JVM is running with 8GB RAM with the max heap at 5G.

following snippet from the hs_pid crash file

Java Threads: ( => current thread )
   0x00007f1e42863800 JavaThread "ActiveMQ NIO Worker 9053" daemon 
[_thread_blocked, id=10690, stack(0x00007f1d6fa53000,0x00007f1d6fa8c000)]
   0x00007f1ea40dc000 JavaThread "ActiveMQ NIO Worker 9047" daemon 
[_thread_blocked, id=10689, stack(0x00007f1d6fa8c000,0x00007f1d6fac5000)]
   0x00007f1eac116800 JavaThread "ActiveMQ NIO Worker 9054" daemon 
[_thread_blocked, id=10688, stack(0x00007f1d6fac5000,0x00007f1d6fafe000)]
   0x00007f1df43f5800 JavaThread "ActiveMQ NIO Worker 9046" daemon 
[_thread_blocked, id=10687, stack(0x00007f1d6fafe000,0x00007f1d6fb37000)]
   0x00007f1e42870000 JavaThread "ActiveMQ NIO Worker 9056" daemon 
[_thread_blocked, id=10686, stack(0x00007f1d6fb37000,0x00007f1d6fb70000)]
   0x00007f1ef6054800 JavaThread "ActiveMQ NIO Worker 9107" daemon 
[_thread_blocked, id=10685, stack(0x00007f1d6fb70000,0x00007f1d6fba9000)]
   0x00007f1ef46af800 JavaThread "ActiveMQ NIO Worker 9059" daemon 
[_thread_blocked, id=10684, stack(0x00007f1d6fba9000,0x00007f1d6fbe2000)]
   0x00007f1e240cd000 JavaThread "ActiveMQ NIO Worker 9041" daemon 
[_thread_blocked, id=10683, stack(0x00007f1d6fbe2000,0x00007f1d6fc1b000)]
   0x00007f1e0016a800 JavaThread "ActiveMQ NIO Worker 9060" daemon 
[_thread_blocked, id=10682, stack(0x00007f1d6fc1b000,0x00007f1d6fc54000)]
   0x00007f1e5019d800 JavaThread "ActiveMQ NIO Worker 9061" daemon 
[_thread_blocked, id=10681, stack(0x00007f1d6fc54000,0x00007f1d6fc8d000)]
   0x00007f1e8006f000 JavaThread "ActiveMQ NIO Worker 9039" daemon 
[_thread_blocked, id=10680, stack(0x00007f1d6fc8d000,0x00007f1d6fcc6000)]
   0x00007f1e4c7df800 JavaThread "ActiveMQ NIO Worker 9101" daemon 
[_thread_blocked, id=10679, stack(0x00007f1d6fcc6000,0x00007f1d6fcff000)]

   0x00007f1da6794800 JavaThread "ActiveMQ BrokerService[MQMaster] 
Task-1781" daemon [_thread_blocked, id=8302, 
   0x00007f1da6791000 JavaThread "ActiveMQ BrokerService[MQMaster] 
Task-1780" daemon [_thread_blocked, id=8298, 
   0x00007f1da6793000 JavaThread "ActiveMQ BrokerService[MQMaster] 
Task-1779" daemon [_thread_blocked, id=8296, 
   0x00007f1da6790000 JavaThread "ActiveMQ BrokerService[MQMaster] 
Task-1778" daemon [_thread_blocked, id=8293, 
   0x00007f1da678f000 JavaThread "ActiveMQ BrokerService[MQMaster] 
Task-1777" daemon [_thread_blocked, id=8292, 

Please, can someone provide inputs on why this is happening.


View raw message