hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kuhu Shukla (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-7244) ShuffleHandler is not aware of disks that are added
Date Thu, 28 Sep 2017 14:23:01 GMT

    [ https://issues.apache.org/jira/browse/YARN-7244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16184239#comment-16184239

Kuhu Shukla commented on YARN-7244:

Thank you [~jlowe], [~sunilg] for the review/comments.
bq. We could make a pull API where the aux service can essentially directly call the NM's
LocalDirHandlerService for getting a path to read or a path to write, then the aux service
doesn't even have to manage the directories itself if all it cares about is finding a place
to write or read.

A pull model where the Shuffle handler /aux service does not maintain valid dirs state would
be my preference but the other pull approach would work too. I will start reworking the patch
in the meantime and will finalize based on what we decide. Appreciate your thoughts.

> ShuffleHandler is not aware of disks that are added
> ---------------------------------------------------
>                 Key: YARN-7244
>                 URL: https://issues.apache.org/jira/browse/YARN-7244
>             Project: Hadoop YARN
>          Issue Type: Bug
>            Reporter: Kuhu Shukla
>            Assignee: Kuhu Shukla
>         Attachments: YARN-7244.001.patch, YARN-7244.002.patch
> The ShuffleHandler permanently remembers the list of "good" disks on NM startup. If disks
later are added to the node then map tasks will start using them but the ShuffleHandler will
not be aware of them. The end result is that the data cannot be shuffled from the node leading
to fetch failures and re-runs of the map tasks.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org

View raw message