hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-64) DataNode should be capable of managing multiple volumes
Date Thu, 23 Mar 2006 00:14:31 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-64?page=comments#action_12371494 ] 

Doug Cutting commented on HADOOP-64:

I am not convinced, w/o benchmarks, that this is required.  Multiple datanodes in the same
JVM (as currently implemented) share a single TCP connection to the namenode.  However each
currently sends separate heartbeats to the namenode.  Thus the primary impact of the proposed
change would be that these heartbeats could be combined into a single RPC.  The processing
on the server would be the same, only spread over fewer RPC calls.  So this change is primarily
warranted if heartbeat RPC overhead dominates namenode performance.  Even if that's proven
to be the case, then we can achieve a similar effect much more simply by increasing the heartbeat

> DataNode should be capable of managing multiple volumes
> -------------------------------------------------------
>          Key: HADOOP-64
>          URL: http://issues.apache.org/jira/browse/HADOOP-64
>      Project: Hadoop
>         Type: Improvement
>   Components: dfs
>     Versions: 0.2
>     Reporter: Sameer Paranjpye
>     Assignee: Konstantin Shvachko
>     Priority: Minor
>      Fix For: 0.2

> The dfs Datanode can only store data on a single filesystem volume. When a node runs
its disks JBOD this means running a Datanode per disk on the machine. While the scheme works
reasonably well on small clusters, on larger installations (several 100 nodes) it implies
a very large number of Datanodes with associated management overhead in the Namenode.
> The Datanod should be enhanced to be able to handle multiple volumes on a single machine.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators:
For more information on JIRA, see:

View raw message