hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Naganarasimha G R (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-3571) AM does not re-blacklist NMs after ignoring-blacklist event happens?
Date Sun, 15 Jan 2017 12:20:26 GMT

    [ https://issues.apache.org/jira/browse/YARN-3571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15823114#comment-15823114
] 

Naganarasimha G R commented on YARN-3571:
-----------------------------------------

Seems to be MapReduce related, and seems to exist in current version too,  We can move this
to MapREDUCE project ?

> AM does not re-blacklist NMs after ignoring-blacklist event happens?
> --------------------------------------------------------------------
>
>                 Key: YARN-3571
>                 URL: https://issues.apache.org/jira/browse/YARN-3571
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager, resourcemanager
>    Affects Versions: 2.5.1
>            Reporter: Hao Zhu
>
> Detailed analysis are in item "3 Will AM re-blacklist NMs after ignoring-blacklist event
happens?" of below link:
> http://www.openkb.info/2015/05/when-will-application-master-blacklist.html
> The current behavior is : if that Node Manager has ever been blacklisted before, then
it will not be blacklisted again after ignore-blacklist happens; Else, it will be blacklisted.
> However I think the right behavior should be : AM can re-blacklist NMs even after ignoring-blacklist
happens once.
>  The code logic is in function containerFailedOnHost(String hostName) of RMContainerRequestor.java:
> {code}
>   protected void containerFailedOnHost(String hostName) {
>     if (!nodeBlacklistingEnabled) {
>       return;
>     }
>     if (blacklistedNodes.contains(hostName)) {
>       if (LOG.isDebugEnabled()) {
>         LOG.debug("Host " + hostName + " is already blacklisted.");
>       }
>       return; //already blacklisted
> {code}
> The reason of above behavior is in above item 2: when ignoring-blacklist happens, it
only ask RM to clear "blacklistAdditions", however it dose not clear the "blacklistedNodes"
variable.
> This behavior may cause the whole job/application to fail if the previous blacklisted
NM was released after ignoring-blacklist event happens.
> Imagine a serial murder is released from prison just because the prison is 33% full,
and horribly he/she will never be put in prison again. Only new murder will be put in prison.
> Example to prove:
> Test 1:
> One node(h4) has issue, other 3 nodes are healthy.
> The job failed with below AM logs:
> {code}
> [root@h1 container_1430425729977_0006_01_000001]# egrep -i 'failures on node|blacklist|FATAL'
syslog
> 2015-05-02 18:38:41,246 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
nodeBlacklistingEnabled:true
> 2015-05-02 18:38:41,246 INFO [main] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
blacklistDisablePercent is 1
> 2015-05-02 18:39:07,249 FATAL [IPC Server handler 3 on 41696] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
Task: attempt_1430425729977_0006_m_000002_0 - exited : java.io.IOException: Spill failed
> 2015-05-02 18:39:07,297 INFO [Thread-49] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
1 failures on node h4.poc.com
> 2015-05-02 18:39:07,950 FATAL [IPC Server handler 16 on 41696] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
Task: attempt_1430425729977_0006_m_000008_0 - exited : java.io.IOException: Spill failed
> 2015-05-02 18:39:07,954 INFO [Thread-49] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
2 failures on node h4.poc.com
> 2015-05-02 18:39:08,148 FATAL [IPC Server handler 17 on 41696] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
Task: attempt_1430425729977_0006_m_000007_0 - exited : java.io.IOException: Spill failed
> 2015-05-02 18:39:08,152 INFO [Thread-49] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
3 failures on node h4.poc.com
> 2015-05-02 18:39:08,152 INFO [Thread-49] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
Blacklisted host h4.poc.com
> 2015-05-02 18:39:08,561 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
Update the blacklist for application_1430425729977_0006: blacklistAdditions=1 blacklistRemovals=0
> 2015-05-02 18:39:08,561 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
Ignore blacklisting set to true. Known: 4, Blacklisted: 1, 25%
> 2015-05-02 18:39:09,563 INFO [RMCommunicator Allocator] org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
Update the blacklist for application_1430425729977_0006: blacklistAdditions=0 blacklistRemovals=1
> 2015-05-02 18:39:32,912 FATAL [IPC Server handler 19 on 41696] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
Task: attempt_1430425729977_0006_m_000002_1 - exited : java.io.IOException: Spill failed
> 2015-05-02 18:39:35,076 FATAL [IPC Server handler 1 on 41696] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
Task: attempt_1430425729977_0006_m_000009_0 - exited : java.io.IOException: Spill failed
> 2015-05-02 18:39:35,133 FATAL [IPC Server handler 5 on 41696] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
Task: attempt_1430425729977_0006_m_000008_1 - exited : java.io.IOException: Spill failed
> 2015-05-02 18:39:57,308 FATAL [IPC Server handler 17 on 41696] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
Task: attempt_1430425729977_0006_m_000002_2 - exited : java.io.IOException: Spill failed
> 2015-05-02 18:40:00,174 FATAL [IPC Server handler 10 on 41696] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
Task: attempt_1430425729977_0006_m_000009_1 - exited : java.io.IOException: Spill failed
> 2015-05-02 18:40:00,227 FATAL [IPC Server handler 12 on 41696] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
Task: attempt_1430425729977_0006_m_000007_1 - exited : java.io.IOException: Spill failed
> 2015-05-02 18:40:22,905 FATAL [IPC Server handler 3 on 41696] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
Task: attempt_1430425729977_0006_m_000018_0 - exited : java.io.IOException: Spill failed
> 2015-05-02 18:40:24,413 FATAL [IPC Server handler 19 on 41696] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
Task: attempt_1430425729977_0006_m_000009_2 - exited : java.io.IOException: Spill failed
> 2015-05-02 18:40:26,086 FATAL [IPC Server handler 16 on 41696] org.apache.hadoop.mapred.TaskAttemptListenerImpl:
Task: attempt_1430425729977_0006_m_000002_3 - exited : java.io.IOException: Spill failed
> {code}
> From above logs, we can see the node h4 got blacklisted after 3 task failures.
> Immediately after that, the igoring-blacklist event happened.
> Then node h4 will never be blacklisted again.
> When task 1430425729977_0006_m_000002 failed for 4 times, the whole job failed



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message