ambari-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jayush Luniya <jlun...@hortonworks.com>
Subject Re: ThreadPoolExecutor being used incorrectly
Date Thu, 14 Jan 2016 18:54:38 GMT
Sounds good Srimanth.
Thanks
Jayush

On 1/14/16, 10:40 AM, "Srimanth Gunturi" <sgunturi@hortonworks.com> wrote:

>Hi Jayush,
>It is very nice that you have noticed the same issue as well and have a
>solution - I will look into it.
>
>I have a working patch myself where I use a custom rejection handler that
>relies on a secondary queue (unbounded), and made the Executor's queue
>bounded (like Michael Harp mentioned).
>
>I will give both a spin and write down in JIRA.
>Regards,
>Srimanth
>
>
>
>________________________________________
>From: Jayush Luniya <jluniya@hortonworks.com>
>Sent: Thursday, January 14, 2016 10:32 AM
>To: dev@ambari.apache.org
>Subject: Re: ThreadPoolExecutor being used incorrectly
>
>Hi Srimanth,
>
>I recently noticed this behavior too. I filed an external JIRA with a
>prototype to address this issue.
>https://issues.apache.org/jira/browse/AMBARI-14671
>
>
>Regards
>Jayush
>
>On 1/14/16, 10:01 AM, "Srimanth Gunturi" <sgunturi@hortonworks.com> wrote:
>
>>Hello,
>>Wanted to write this down and have discussion about
>>'java.util.concurrent.ThreadPoolExecutor' being used incorrectly in
>>Ambari.
>>
>>Almost all usages call the below constructor:
>>
>>public ThreadPoolExecutor(
>>                              int corePoolSize,
>>                              int maximumPoolSize,
>>                              long keepAliveTime,
>>                              TimeUnit unit,
>>                              BlockingQueue<Runnable> workQueue)
>>
>>Where typical values used are:
>>
>>new ThreadPoolExecutor(
>>            20,
>>            100,
>>            30000L,
>>            TimeUnit.MILLISECONDS,
>>            new LinkedBlockingQueue<Runnable>()) // Unbounded queue
>>
>>The weird thing about 'ThreadPoolExecutor' is that the number of threads
>>increases from the 'corePoolSize' (20) to the 'maximumPoolSize' (100)
>>ONLY when the BlockingQueue is full. Whenever we use an unbounded
>>blocking queue, the number of threads never goes past the 'corePoolSize'.
>>
>>The javadoc states: "If there are more than corePoolSize but less than
>>maximumPoolSize threads running, a new thread will be created only if the
>>queue is full"
>>
>>So our thinking that the Executor will automatically increase the number
>>of threads upto 'maximumPoolSize' and store overflowing requests into the
>>queue (unbounded) is incorrect. Due to our usage, the number of threads
>>never increases beyond 'corePoolSize'.
>>
>>I do not understand the reason why it is implemented this way, but we run
>>into performance issues by getting stuck with 'corePoolSize' thread
>>count. I am looking into a fix for using ThreadPoolExecutor where the
>>number of threads increases upto 'maximumPoolSize'.
>>
>>Regards,
>>Srimanth?
>>
>
>
>


Mime
View raw message