activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benjamin Huang (JIRA)" <>
Subject [jira] [Comment Edited] (AMQ-5486) Thread synchronization overhead is unexpectedly high
Date Wed, 24 Dec 2014 03:28:13 GMT


Benjamin  Huang edited comment on AMQ-5486 at 12/24/14 3:27 AM:

one of my idea is to make thread pool behavior tunable, please refer to attachment 'tunable_threadpool_options.patch'
and avoid worker thread from being created too actively ( use CallerRunsPolicy as RejectionHandler)

was (Author: benjaminhuang):
one of my idea is to make thread pool behavior tunable, please refer to attachment

> Thread synchronization overhead is unexpectedly high
> ----------------------------------------------------
>                 Key: AMQ-5486
>                 URL:
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.9.1
>         Environment: UbuntuServer 12.04 x86_64, Linux Kernel 3.20.23, OpenJDK 1.7.0_51
64-Bit Server VM, Xms 2G, Xms 8G, CPU: E5-2620 v2 X 2, 64 GB RAM
>            Reporter: Benjamin  Huang
>         Attachments: Method_Statistics.html,, Monitor_Usage_Statistics.html,
monitor_usage_stats.png, nio_worker_blocked_1.png, nio_worker_blocked_2.png, nio_worker_blocked_frequently.png,
> There's about 20 topics with virtual topic enabled, hundreds of comsumers/producers connected
to MQ on NIO transport connector. During the run there're about 12000 msg flow in per second,
not a very high rate, but ActiveMQ consumes a lot of CPU resource (about 600%~1000%). To find
out what's the most CPU consuming code path, I use JProfiler to dig into the process. 
> Among all the NIO worker threads, most of them were frequently blocked and did a little
job between the 'unblocked' time. While they're expected spend most of their time slices on
waiting for work item and processed them.
> !nio_worker_blocked_frequently.png!
> After reviewing the monitor usage history and stats, I think these NIO workers were competing
fiercely with each other on executing a synchronized method (DestinationMap::get), which is
also the most hot spot in the program . I also notice that the caller AbstractRegion::getDestinations
acquires a read lock before calling it, so I guess this could be a left out, read lock is
the actual lock type required here.
> !monitor_usage_stats.png!
> !nio_worker_blocked_1.png!
> !nio_worker_blocked_2.png!
> It's too difficult for me to list all critical sections between NIO workers, or between
NIO workers and BrokerService which adds up to the overall synchronize overhead. So I attach
the relevant info, with the hope of finding a complete solution to this.

This message was sent by Atlassian JIRA

View raw message