hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6265) Make ContainerLauncherImpl.INITIAL_POOL_SIZE configurable to better control to launch/kill containers
Date Sat, 14 Mar 2015 14:37:43 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14361809#comment-14361809

Hudson commented on MAPREDUCE-6265:

FAILURE: Integrated in Hadoop-Hdfs-trunk #2064 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2064/])
MAPREDUCE-6265. Make ContainerLauncherImpl.INITIAL_POOL_SIZE configurable to better control
to launch/kill containers. Contributed by Zhihai Xu (ozawa: rev 9d38520c8e42530a817a7f69c9aa73a9ad40639c)
* hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
* hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java
* hadoop-mapreduce-project/CHANGES.txt

> Make ContainerLauncherImpl.INITIAL_POOL_SIZE configurable to better control to launch/kill
> -----------------------------------------------------------------------------------------------------
>                 Key: MAPREDUCE-6265
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv2
>            Reporter: zhihai xu
>            Assignee: zhihai xu
>             Fix For: 2.7.0
>         Attachments: MAPREDUCE-6265.000.patch, MAPREDUCE-6265.001.patch, MAPREDUCE-6265.002.patch
> make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread
pool size to launch/kill containers
> Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at 
> {code}
>   protected static final int INITIAL_POOL_SIZE = 10;
> {code}
> We should make it configurable because the thread pool size will be decided by INITIAL_POOL_SIZE,
limitOnPoolSize and number of node used by the AM.
> Since we already made limitOnPoolSize configurable, it make senses to also make INITIAL_POOL_SIZE
configurable to better manage the thread pool size.
> We saw some issue due to the small thread pool size when some node is down. The recovery
from a shutdown node take very long time due to all the ContainerLauncher threads are blocked
by IPC client connection to the shutdown node.

This message was sent by Atlassian JIRA

View raw message