hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Milind Bhandarkar (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-64) DataNode should be capable of managing multiple volumes
Date Mon, 07 Aug 2006 19:29:15 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-64?page=comments#action_12426300 ] 
            
Milind Bhandarkar commented on HADOOP-64:
-----------------------------------------

I have profiled the namenode, and most action in namenode happens as a response to heartbeat
and blockreport messages. Therefore I think it is important to enable the datanode to handle
multiple volumes. this also relates to hadoop-50, which needs handling multiple directories.
The scheme I have in mind is for datanode to load-balance among volumes (that corrspond to
multiple disks) and then within a volume, block-placement will be done within multiple directories
according to block-id. I am currently preparing a proposal on this issue.

> DataNode should be capable of managing multiple volumes
> -------------------------------------------------------
>
>                 Key: HADOOP-64
>                 URL: http://issues.apache.org/jira/browse/HADOOP-64
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.2.0
>            Reporter: Sameer Paranjpye
>         Assigned To: Konstantin Shvachko
>            Priority: Minor
>             Fix For: 0.6.0
>
>
> The dfs Datanode can only store data on a single filesystem volume. When a node runs
its disks JBOD this means running a Datanode per disk on the machine. While the scheme works
reasonably well on small clusters, on larger installations (several 100 nodes) it implies
a very large number of Datanodes with associated management overhead in the Namenode.
> The Datanod should be enhanced to be able to handle multiple volumes on a single machine.

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message