hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Joydeep Sen Sarma (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-2677) separate space reservation for hdfs blocks and intermediate storage
Date Tue, 22 Jan 2008 17:46:34 GMT
separate space reservation for hdfs blocks and intermediate storage

                 Key: HADOOP-2677
                 URL: https://issues.apache.org/jira/browse/HADOOP-2677
             Project: Hadoop
          Issue Type: Improvement
          Components: dfs
    Affects Versions: 0.14.4
            Reporter: Joydeep Sen Sarma
            Priority: Critical

both dfs client buffering (and i imagine map-reduce intermediate data) and datanode try to
honor the same space reservation (dfs.du.reserved). But this is problematic because once hdfs/data-node
fill up a node - there's no space left for map-reduce computations.

ideally - hdfs should be allowed to consume upto some watermark (say 60%) and then dfs buffering/intermediate
storage should be allowed to consume space upto some higher watermark (say 90%). this way
the node will always remain usable.

we are hitting this problem in a cluster where a few nodes have lower amount of space. while
the cluster overall has space left, these nodes are hitting their space limits. but now tasks
scheduled on these nodes fail because dfs client does not find space to buffer to. there's
no workaround really i can think of.

another option would be to globally allocate hdfs blocks based on space availability (keep
all nodes at the same space utilization % approx.).

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message