hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kuhu Shukla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7244) ShuffleHandler is not aware of disks that are added
Date Wed, 27 Sep 2017 12:07:00 GMT

    [ https://issues.apache.org/jira/browse/YARN-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16182450#comment-16182450
] 

Kuhu Shukla commented on YARN-7244:
-----------------------------------

Thank you [~sunilg] for the review comments! 
bq. We could push this config name to LocalDirAllocator and then read from NM end
I am not sure how we can initialize the config specifically for LocalDirAllocator (may be
add a constructor?) and what would reading it from NM end mean. Agreed that code separation
is important here. May be not having this as a config might help?
bq. Do you think, we can improve this to skip as default behavior itself?
I did not fully get what you have in mind here. Could you help me understand and elaborate
a bit. Thanks a lot!

> ShuffleHandler is not aware of disks that are added
> ---------------------------------------------------
>
>                 Key: YARN-7244
>                 URL: https://issues.apache.org/jira/browse/YARN-7244
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Kuhu Shukla
>            Assignee: Kuhu Shukla
>         Attachments: YARN-7244.001.patch, YARN-7244.002.patch
>
>
> The ShuffleHandler permanently remembers the list of "good" disks on NM startup. If disks
later are added to the node then map tasks will start using them but the ShuffleHandler will
not be aware of them. The end result is that the data cannot be shuffled from the node leading
to fetch failures and re-runs of the map tasks.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message