hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron T. Myers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
Date Thu, 28 Feb 2013 17:39:15 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13589696#comment-13589696
] 

Aaron T. Myers commented on HDFS-2576:
--------------------------------------

bq. For now if the balancer (which is run manually for HDFS) is too problematic, we can change
the balancer implementation so that it doesn't move blocks belonging to certain directories
(the information of blocks to paths seems to be there).

This is a good idea independent of this work, and is tracked by HDFS-4420.

I think I agree with all the points that Jon has made. A persistent block affinity group seems
like the best solution to me, but the patch as it's been so far described seems to provide
a good incremental improvement and won't preclude future improvements to the feature.
                
> Namenode should have a favored nodes hint to enable clients to have control over block
placement.
> -------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-2576
>                 URL: https://issues.apache.org/jira/browse/HDFS-2576
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Pritam Damania
>         Attachments: hdfs-2576-1.txt
>
>
> Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes
to place the blocks for a file for higher level of locality. For this purpose there is a need
of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations
where the client wants to put each block. The proposed solution is a favored nodes parameter
in the addBlock() method and in the create() file method to enable the clients to give the
hints to the NameNode about the locations of each replica of the block. Note that this would
be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide
whether it can respect the hints or not.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message