hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-1800) using map output fetch failures to blacklist nodes is problematic
Date Wed, 19 May 2010 21:41:55 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-1800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12869360#action_12869360
] 

Joydeep Sen Sarma commented on MAPREDUCE-1800:
----------------------------------------------

the problem is that the current heuristics also cause bad behavior when uplinks/core-switches
degrade.

i agree that the case of a single node that is not able to send map outputs is something that
hadoop should detect/correct automatically - but i don't think the current heuristic (by itself)
is a good one because of the previous point. 

i don't have a good alternative solution/proposals. a few thoughts pop to mind:
- separate blacklisting of TTs due to map/reduce task failures from blacklisting due to map-output
fetch failures. the thresholds and policies required seem different.
- if the scope of the fault is nic/port/process/os problems affecting a 'single' node - then
we should only take into map-fetch failures that happen within the same rack. (ie. assign
blame to a TT only if other TTs within the same rack cannot communicate to it)
- blame should be laid by a multitude of different hosts. It's no good if 4 reducers on TT1
cannot get map outputs from TT2 and this results in blacklisting of TT2. It's possible that
TT1 itself has a bad port/nic. 

(just thinking aloud, i don't have a careful understanding of the code beyond what's been
relayed to me by others :-)).

> using map output fetch failures to blacklist nodes is problematic
> -----------------------------------------------------------------
>
>                 Key: MAPREDUCE-1800
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-1800
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>            Reporter: Joydeep Sen Sarma
>
> If a mapper and a reducer cannot communicate, then either party could be at fault. The
current hadoop protocol allows reducers to declare nodes running the mapper as being at fault.
When sufficient number of reducers do so - then the map node can be blacklisted. 
> In cases where networking problems cause substantial degradation in communication across
sets of nodes - then large number of nodes can become blacklisted as a result of this protocol.
The blacklisting is often wrong (reducers on the smaller side of the network partition can
collectively cause nodes on the larger network partitioned to be blacklisted) and counterproductive
(rerunning maps puts further load on the (already) maxed out network links).
> We should revisit how we can better identify nodes with genuine network problems (and
what role, if any, map-output fetch failures have in this).

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message