hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bikas Saha (JIRA)" <j...@apache.org>
Subject [jira] [Created] (YARN-1324) NodeManager should assign 1 local directory directory to a container
Date Fri, 18 Oct 2013 22:25:42 GMT
Bikas Saha created YARN-1324:

             Summary: NodeManager should assign 1 local directory directory to a container
                 Key: YARN-1324
                 URL: https://issues.apache.org/jira/browse/YARN-1324
             Project: Hadoop YARN
          Issue Type: Improvement
    Affects Versions: 2.2.0
            Reporter: Bikas Saha

Currently, for every container, the NM creates a directory on every disk and expects the container-task
to choose 1 of them and load balance the use of the disks across all containers. 
1) This may have worked fine in the MR world where MR tasks would randomly choose dirs but
in general we cannot expect every app/task writer to understand these nuances and randomly
pick disks. So we could end up overloading the first disk if most people decide to use the
first disk.
2) This makes a number of NM operations to scan every disk (thus randomizing that disk) to
locate the dir which the task has actually chosen to use for its files. Makes all these operations
expensive for the NM as well as disruptive for users of disks that did not have the real task
working dirs.

I propose that NM should up-front decide the disk it is assigning to tasks. It could choose
to do so randomly or weighted-randomly by looking at space and load on each disk. So it could
do a better job of load balancing. Then, it would associate the chosen working directory with
the container context so that subsequent operations on the NM can directly seek to the correct
location instead of having to seek on every disk.

This message was sent by Atlassian JIRA

View raw message