hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Varun Vasudev (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-1781) NM should allow users to specify max disk utilization for local disks
Date Fri, 07 Mar 2014 10:35:48 GMT

    [ https://issues.apache.org/jira/browse/YARN-1781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13923760#comment-13923760
] 

Varun Vasudev commented on YARN-1781:
-------------------------------------

{quote}
Styling

    Some of the patch has non-standard formatting. Please fix them. For e.g., we say

    if (condition) {
      //
    } else {
      //
    }
{quote}

My apologies for this. Fixed.

{quote}
YarnConfiguration

    Create a prefix for .disk-health-checker. and use it for all Disk-check related configs.
    max-space-utilization-perc -> max-disk-utilization-per-disk-percentage?
    min-free-space-mb -> min-free-space-per-disk-percentage?
{quote}

Fixed.(I'm presuming "min-free-space-per-disk-percentage" meant "min-free-space-per-disk-mb")

{quote}
DirectoryCollection

    checkDiskPercentageLimit() -> isDiskUsageUnderPercentageLimit() and checkDiskSpaceLimit()
-> isDiskFreeSpaceWithinLimit()
    Please replace 'perc/Perc' etc with the more verbose 'percentage/Percentage' everywhere
{quote}
Fixed

{quote}
LocalDirsHandlerService

    maxUsedSpacePerc -> maxUsableSpacePercentagePerDisk and minFreeSpaceMB -> minFreeSpacePerDiskMB
{quote}

Fixed

{quote}
TestDirectoryCollection

    The test using testDir.getTotalSpace() can be flaky depending on what other things are
running on the testing machine. Can we just mock the disk-usage here?
    testCutoffSetters -> testDiskLimitsCutoffSetters,
{quote}

getTotalSpace() should be consistent no? According to the Java docs, it returns the size of
the partition which should be the same every run. The idea of the test is to simulate a case
where we don't allow any of the disk to be used.

> NM should allow users to specify max disk utilization for local disks
> ---------------------------------------------------------------------
>
>                 Key: YARN-1781
>                 URL: https://issues.apache.org/jira/browse/YARN-1781
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager
>            Reporter: Varun Vasudev
>            Assignee: Varun Vasudev
>         Attachments: apache-yarn-1781.0.patch, apache-yarn-1781.1.patch, apache-yarn-1781.2.patch,
apache-yarn-1781.3.patch
>
>
> This is related to YARN-257(it's probably a sub task?). Currently, the NM does not detect
full disks and allows full disks to be used by containers leading to repeated failures. YARN-257
deals with graceful handling of full disks. This ticket is only about detection of full disks
by the disk health checkers.
> The NM should allow users to set a maximum disk utilization for local disks and mark
disks as bad once they exceed that utilization. At the very least, the NM should at least
detect full disks.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message