hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2121) Add ability to create block replicas during off-switch reads
Date Sun, 03 Jul 2011 02:34:22 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13059156#comment-13059156
] 

Allen Wittenauer commented on HDFS-2121:
----------------------------------------

Doesn't this assume that the process reading the block is long-lasting and/or slow to read
the entire block?  That doesn't seem to jive with the vast majority of HDFS use cases.  Moving
the block would need to be worth the network overhead incurred.

This goes against the "move the code, not the data" mantra that Hadoop is based off.  

Hotspots are likely better taken care of by increasing replication rather than just moving
them.

> Add ability to create block replicas during off-switch reads
> ------------------------------------------------------------
>
>                 Key: HDFS-2121
>                 URL: https://issues.apache.org/jira/browse/HDFS-2121
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Arun C Murthy
>
> Idea: when a DFSClient reads an off-switch replica, we could tee an extra, temporal replica
to the local DN if the NN thinks it makes sense. 
> Then we could delete the ones which were part of the original 2 on the rack to rebalance
or just rely on the balancer... all this would ensure better spread across the cluster. This
would also help with hotspots on datasets.
> Thoughts?

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message