Mailing-List: contact yarn-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: yarn-issues@hadoop.apache.org
Date: Tue, 30 Sep 2014 20:24:36 +0000 (UTC)
From: "Varun Vasudev (JIRA)" <jira@apache.org>
To: yarn-issues@hadoop.apache.org
Message-ID: <JIRA.12532140.1321876387000.158153.1412108676103@Atlassian.JIRA>
In-Reply-To: <JIRA.12532140.1321876387000@Atlassian.JIRA>
References: <JIRA.12532140.1321876387000@Atlassian.JIRA>
 <JIRA.12532140.1321876387187@arcas>
Subject: [jira] [Updated] (YARN-90) NodeManager should identify failed disks
 becoming good back again
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/YARN-90?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Varun Vasudev updated YARN-90:
------------------------------
    Attachment: apache-yarn-90.7.patch

Uploaded a new patch to address the comments by Jason.

{quote}

bq.    I've changed it to "Disk(s) health report: ". My only concern with this is that there might be scripts looking for the "Disk(s) failed" log line for monitoring. What do you think?

If that's true then the code should bother to do a diff between the old disk list and the new one, logging which disks turned bad using the "Disk(s) failed" line and which disks became healthy with some other log message.
{quote}

Fixed. We now have two log messages - one indicating when disks go bad and one when disks get marked as good.

{quote}
bq.    Directories are only cleaned up during startup. The code tests for existence of the directories and the correct permissions. This does mean that container directories left behind for any reason won't get cleaned up unit the NodeManager is restarted. Is that ok?

This could still be problematic for the NM work-preserving restart case, as we could try to delete an entire disk tree with active containers on it due to a hiccup when the NM restarts. I think a better approach is a periodic cleanup scan that looks for directories under yarn-local and yarn-logs that shouldn't be there. This could be part of the health check scan or done separately. That way we don't have to wait for a disk to turn good or bad to catch leaked entities on the disk due to some hiccup. Sorta like an fsck for the NM state on disk. That is best done as a separate JIRA, as I think this functionality is still an incremental improvement without it.
{quote}

The current code will only cleanup if the NM recovery can't be carried out.
{noformat}
      if (!stateStore.canRecover()) {
        cleanUpLocalDirs(lfs, delService);
        initializeLocalDirs(lfs);
        initializeLogDirs(lfs);
      }
{noformat}
Will that handle the case you mentioned?

bq. checkDirs unnecessarily calls union(errorDirs, fullDirs) twice.

Fixed.

bq. isDiskFreeSpaceOverLimt is now named backwards, as the code returns true if the free space is under the limit.

Fixed.
bq. getLocalDirsForCleanup and getLogDirsForCleanup should have javadoc comments like the other methods.

Fixed.

{quote}
Nit: The union utility function doesn't technically perform a union but rather a concatenation, and it'd be a little clearer if the name reflected that. Also the function should leverage the fact that it knows how big the ArrayList will be after the operations and give it the appropriate hint to its constructor to avoid reallocations.
{quote}

Fixed.

> NodeManager should identify failed disks becoming good back again
> -----------------------------------------------------------------
>
>                 Key: YARN-90
>                 URL: https://issues.apache.org/jira/browse/YARN-90
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager
>            Reporter: Ravi Gummadi
>            Assignee: Varun Vasudev
>         Attachments: YARN-90.1.patch, YARN-90.patch, YARN-90.patch, YARN-90.patch, YARN-90.patch, apache-yarn-90.0.patch, apache-yarn-90.1.patch, apache-yarn-90.2.patch, apache-yarn-90.3.patch, apache-yarn-90.4.patch, apache-yarn-90.5.patch, apache-yarn-90.6.patch, apache-yarn-90.7.patch
>
>
> MAPREDUCE-3121 makes NodeManager identify disk failures. But once a disk goes down, it is marked as failed forever. To reuse that disk (after it becomes good), NodeManager needs restart. This JIRA is to improve NodeManager to reuse good disks(which could be bad some time back).


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)