hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yoram Arnon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-296) Do not assign blocks to a datanode with < x mb free
Date Wed, 14 Jun 2006 16:44:30 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-296?page=comments#action_12416226 ] 

Yoram Arnon commented on HADOOP-296:
------------------------------------

managing individual configuration files on each node is exceedingly difficult.
when I make a config change, say, when a new config variable is added, it's usually for the
entire cluster(s); I make the change on one node, then distribute to all the other nodes.
That would override individual changes made to those files. editing each individual file separately
is clearly unmanageable, so for larger cluster it's really important to have a *very* small
number of configurations.

As a rule, less configuration is better. In this particular case, improving the allocation
strategy to do a better job automatically, possibly using a single configured strategy variable
but better without, would be far better than individual configuration per node.
At the end of the day, you're going to select values for these two new config items based
on some logic, using real world numbers as a guide. Why not just implement that logic in the
code?

> Do not assign blocks to a datanode with < x mb free
> ---------------------------------------------------
>
>          Key: HADOOP-296
>          URL: http://issues.apache.org/jira/browse/HADOOP-296
>      Project: Hadoop
>         Type: New Feature

>   Components: dfs
>     Versions: 0.3.2
>     Reporter: Johan Oskarson
>  Attachments: minspace.patch, minspacev2.patch
>
> We're running a smallish cluster with very different machines, some with only 60 gb harddrives
> This creates a problem when inserting files into the dfs, these machines run out of space
quickly and then they cannot run any map reduce operations
> A solution would be to not assign any new blocks once the space is below a certain user
configurable threshold
> This free space could then be used by the map reduce operations instead (if that's on
the same disk)

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira


Mime
View raw message