hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-6265) Make ContainerLauncherImpl.INITIAL_POOL_SIZE configurable to better control to launch/kill containers
Date Sat, 14 Mar 2015 15:01:43 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-6265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14361823#comment-14361823
] 

Hudson commented on MAPREDUCE-6265:
-----------------------------------

FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #123 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/123/])
MAPREDUCE-6265. Make ContainerLauncherImpl.INITIAL_POOL_SIZE configurable to better control
to launch/kill containers. Contributed by Zhihai Xu (ozawa: rev 9d38520c8e42530a817a7f69c9aa73a9ad40639c)
* hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
* hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/launcher/TestContainerLauncher.java
* hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* hadoop-mapreduce-project/CHANGES.txt


> Make ContainerLauncherImpl.INITIAL_POOL_SIZE configurable to better control to launch/kill
containers
> -----------------------------------------------------------------------------------------------------
>
>                 Key: MAPREDUCE-6265
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6265
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mrv2
>            Reporter: zhihai xu
>            Assignee: zhihai xu
>             Fix For: 2.7.0
>
>         Attachments: MAPREDUCE-6265.000.patch, MAPREDUCE-6265.001.patch, MAPREDUCE-6265.002.patch
>
>
> make INITIAL_POOL_SIZE in ContainerLauncherImpl configurable to better control the thread
pool size to launch/kill containers
> Currently INITIAL_POOL_SIZE in ContainerLauncherImpl is hard-coded at 
> {code}
>   protected static final int INITIAL_POOL_SIZE = 10;
> {code}
> We should make it configurable because the thread pool size will be decided by INITIAL_POOL_SIZE,
limitOnPoolSize and number of node used by the AM.
> Since we already made limitOnPoolSize configurable, it make senses to also make INITIAL_POOL_SIZE
configurable to better manage the thread pool size.
> We saw some issue due to the small thread pool size when some node is down. The recovery
from a shutdown node take very long time due to all the ContainerLauncher threads are blocked
by IPC client connection to the shutdown node.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message