hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kuhu Shukla (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (YARN-7244) ShuffleHandler is not aware of disks that are added
Date Wed, 27 Sep 2017 12:07:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16181059#comment-16181059
] 

Kuhu Shukla edited comment on YARN-7244 at 9/27/17 12:06 PM:
-------------------------------------------------------------

Thank you [~bibinchundatt] for the review comments!
bq. Better to check directory exists first if we are not concerned of permission . thoughts?
Makes sense to me.
bq. Testcase is successful even if YARN_SHUFFLE_BAD_DIRS_FILTER_ENABLED set to false.
That is expected. This config essentially decides if we should stick to existing behavior
(value=true) of removing directories if they are bad or not (value=false).


was (Author: kshukla):
bq. Better to check directory exists first if we are not concerned of permission . thoughts?
Makes sense to me.
bq. Testcase is successful even if YARN_SHUFFLE_BAD_DIRS_FILTER_ENABLED set to false.
That is expected. This config essentially decides if we should stick to existing behavior
(value=true) of removing directories if they are bad or not (value=false).

> ShuffleHandler is not aware of disks that are added
> ---------------------------------------------------
>
>                 Key: YARN-7244
>                 URL: https://issues.apache.org/jira/browse/YARN-7244
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Kuhu Shukla
>            Assignee: Kuhu Shukla
>         Attachments: YARN-7244.001.patch, YARN-7244.002.patch
>
>
> The ShuffleHandler permanently remembers the list of "good" disks on NM startup. If disks
later are added to the node then map tasks will start using them but the ShuffleHandler will
not be aware of them. The end result is that the data cannot be shuffled from the node leading
to fetch failures and re-runs of the map tasks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message