hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Harsh J (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-329) separate space reservation for hdfs blocks and intermediate storage
Date Wed, 25 Jan 2012 04:34:40 GMT

    [ https://issues.apache.org/jira/browse/HDFS-329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192856#comment-13192856
] 

Harsh J commented on HDFS-329:
------------------------------

Agree, given reserved space config, which should be set to the minimum free space you want
your shared MR scratch dirs to have, this should not be an issue -- coupled with periodic
balancing.
                
> separate space reservation for hdfs blocks and intermediate storage
> -------------------------------------------------------------------
>
>                 Key: HDFS-329
>                 URL: https://issues.apache.org/jira/browse/HDFS-329
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Joydeep Sen Sarma
>            Priority: Critical
>
> both dfs client buffering (and i imagine map-reduce intermediate data) and datanode try
to honor the same space reservation (dfs.du.reserved). But this is problematic because once
hdfs/data-node fill up a node - there's no space left for map-reduce computations.
> ideally - hdfs should be allowed to consume upto some watermark (say 60%) and then dfs
buffering/intermediate storage should be allowed to consume space upto some higher watermark
(say 90%). this way the node will always remain usable.
> we are hitting this problem in a cluster where a few nodes have lower amount of space.
while the cluster overall has space left, these nodes are hitting their space limits. but
now tasks scheduled on these nodes fail because dfs client does not find space to buffer to.
there's no workaround really i can think of.
> another option would be to globally allocate hdfs blocks based on space availability
(keep all nodes at the same space utilization % approx.).

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message