activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Benjamin Huang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMQ-5486) Thread synchronization overhead is unexpectedly high
Date Fri, 12 Dec 2014 08:25:13 GMT

     [ https://issues.apache.org/jira/browse/AMQ-5486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Benjamin  Huang updated AMQ-5486:
---------------------------------
    Attachment: nio_worker_blocked_frequently.png
                nio_worker_blocked_2.png
                nio_worker_blocked_1.png

> Thread synchronization overhead is unexpectedly high
> ----------------------------------------------------
>
>                 Key: AMQ-5486
>                 URL: https://issues.apache.org/jira/browse/AMQ-5486
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Broker
>    Affects Versions: 5.9.1
>         Environment: UbuntuServer 12.04 x86_64, Linux Kernel 3.20.23, OpenJDK 1.7.0_51
64-Bit Server VM, Xms 2G, Xms 8G, CPU: E5-2620 v2 X 2, 64 GB RAM
>            Reporter: Benjamin  Huang
>         Attachments: nio_worker_blocked_1.png, nio_worker_blocked_2.png, nio_worker_blocked_frequently.png
>
>
> There's about 20 topics with virtual topic enabled, hundreds of comsumers/producers connected
to MQ on NIO transport connector. During the run there're about 12000 msg flow in per second,
not a very high rate, but ActiveMQ consumes a lot of CPU resource (about 600%~1000%). To find
out what's the most CPU consuming code path, I use JProfiler to dig into the process. 
> Among all the NIO worker threads, most of them were frequently blocked and did a little
job between the 'unblocked' time. While they're expected spend most of their time slices on
waiting for work item and processed them.
> !nio_worker_blocked_frequently.png!
> After reviewing the monitor usage history and stats, I think these NIO workers were competing
fiercely with each other on executing a synchronized method (DestinationMap::get), which is
also the most hot spot in the program . I also notice that the caller AbstractRegion::getDestinations
acquires a read lock before calling it, so I guess this could be a left out, read lock is
the actual lock type required here.
> !monitor_usage_stats.png!
> !nio_worker_blocked_1.png!
> !nio_worker_blocked_2.png!
> As a rookie to Java, it's too difficult for me to list all critical sections between
NIO workers, or between NIO workers and BrokerService which adds up to the overall synchronize
overhead. So I attach the relevant info, with the hope of finding a feasible solution to this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message