hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Junping Du (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1773) ShuffleHeader should have a format that can inform about errors
Date Tue, 14 Jun 2016 16:49:57 GMT

    [ https://issues.apache.org/jira/browse/YARN-1773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329865#comment-15329865
] 

Junping Du commented on YARN-1773:
----------------------------------

Hi [~varun_saxena], this JIRA has pending for a while. Do you still have plan to fix it?

> ShuffleHeader should have a format that can inform about errors
> ---------------------------------------------------------------
>
>                 Key: YARN-1773
>                 URL: https://issues.apache.org/jira/browse/YARN-1773
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 2.3.0, 2.4.0
>            Reporter: Bikas Saha
>            Assignee: Varun Saxena
>            Priority: Critical
>
> Currently, the ShuffleHeader (which is a Writable) simply tries to read the successful
header (mapid, reduceid etc). If there is an error then the input will have an error message
instead of (mapid, reducedid etc). Thus parsing the ShuffleHeader fails and since we dont
know where the error message ends, we cannot consume the remaining input stream which may
have good data from the remaining map outputs. Being able to encode the error in the ShuffleHeader
will let us parse out the error correctly and move on to the remaining data.
> The shuffle handler response should say which maps are in error and which are fine, what
the error was for the erroneous maps. These will help report diagnostics for easier upstream
reporting.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message