activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Matan Zruya (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (AMQ-3885) ActiveMQ java client doesn't scale to thousands of queues
Date Sun, 17 Jun 2012 14:47:42 GMT

     [ https://issues.apache.org/jira/browse/AMQ-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Matan Zruya updated AMQ-3885:
-----------------------------

    Description: 
The ActiveMQ broker scales to tens of thousands of queues easily when using -Dorg.apache.activemq.UseDedicatedTaskRunner=false
(false by default).

A problem actually arises in the java client side, when a client is listening to X queues
using 1 connection and Y sessions per queue, using a JMS message listener, X * Y threads will
be created, when X * Y is not bounded,

This is because each ActiveMQConnection object holds a TaskRunnerFactory which in turn has
a ThreadPoolExecutor, the max pool size of the executor is defined to be Integer.MAX_VALUE,
with the combination of a SynchronousQueue it creates as many threads as it pleases.

the behavior of the executor is dependent on min pool size, core pool size, queue impl and
rejection policy 

a few options to handle this :
1. the default rejection policy throws an exception which is not handled,
ThreadPoolExecutor.CallerRunsPolicy sounds better then the default

2. Change the queue impl to LinkedBlockingQueue and change the core pool size to a bounded
value

3. Provide the means to supply an external executor to be shared in the app

  was:
The ActiveMQ broker scales to tens of thousands of queues easily when using -Dorg.apache.activemq.UseDedicatedTaskRunner=false
(false by default).

A problem actually arises in the java client side, when a client is listening to X queues
using 1 connection and Y sessions per queue, using a JMS message listener, X * Y threads will
be created, when X * Y is not bounded,

This is because each ActiveMQConnection object holds a TaskRunnerFactory which in turn has
a ThreadPoolExecutor, the max pool size of the executor is defined to be Integer.MAX_VALUE,
with the combination of a SynchronousQueue it creates as many threads as it pleases.

Shouldn't this executor be bounded with a reasonable value ? (~200 or a constructor passed
value) and have the rest of the tasks queued,
where if more concurrency is necessary, the work could be distributed across more connections...

The ideal solution would be if multiple ActiveMQConnection objects could share a common TaskExecutor.

But i feel like the minimum would be limiting the executor so the OS doesn't halt completely
because of context switches.


    
> ActiveMQ java client doesn't scale to thousands of queues
> ---------------------------------------------------------
>
>                 Key: AMQ-3885
>                 URL: https://issues.apache.org/jira/browse/AMQ-3885
>             Project: ActiveMQ
>          Issue Type: Bug
>          Components: Connector
>    Affects Versions: 5.6.0
>         Environment: core i7 laptop running win7 64bit with 8gb of ram
>            Reporter: Matan Zruya
>              Labels: ActiveMQConnection, TaskRunnerFactory, client, java
>
> The ActiveMQ broker scales to tens of thousands of queues easily when using -Dorg.apache.activemq.UseDedicatedTaskRunner=false
(false by default).
> A problem actually arises in the java client side, when a client is listening to X queues
using 1 connection and Y sessions per queue, using a JMS message listener, X * Y threads will
be created, when X * Y is not bounded,
> This is because each ActiveMQConnection object holds a TaskRunnerFactory which in turn
has a ThreadPoolExecutor, the max pool size of the executor is defined to be Integer.MAX_VALUE,
with the combination of a SynchronousQueue it creates as many threads as it pleases.
> the behavior of the executor is dependent on min pool size, core pool size, queue impl
and rejection policy 
> a few options to handle this :
> 1. the default rejection policy throws an exception which is not handled,
> ThreadPoolExecutor.CallerRunsPolicy sounds better then the default
> 2. Change the queue impl to LinkedBlockingQueue and change the core pool size to a bounded
value
> 3. Provide the means to supply an external executor to be shared in the app

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message