hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jonathan Hsieh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
Date Wed, 27 Feb 2013 23:45:16 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13588917#comment-13588917

Jonathan Hsieh commented on HDFS-2576:

bq. Sorry, I don't understand. What do you mean by this?

I believe DD is referring to discussion on the related HBASE-4755 jira.

I think we should define what the goal API is for this in HDFS, and for that we probably need
more HBase context.  The HDFS API's we choose here may limit the possibilities for HBase,
or require exposing even more more HDFS internals.

The DD's current approach seems like a reasonable first step -- HBase hints HDFS, but in the
longer run the balancers at the different levels may "fight".

For the HBase use case it seems that the two extreme design points are:  

* HBase is oblivious of physical placement but has the ability to asks HDFS for the "next
most local" node (so that HBase can reassign the region assoiciated with the files to that
machine).  Maybe HBase provides HDFS a block placement policy plug-in that provides mechanism
to get the "next most local" api.  (encapsulation approach)
* HDFS provides a raw block placement api, and HBase completely manages block placement with
an balancing/block interrogation in HDFS.  We may need some override to prevent the HDFS block
balancer from doing anything on particular files/subdirs.  (end-to-end principle approach)

Do you guys have opinions on which direction we want to head? (DD, ATM?)
> Namenode should have a favored nodes hint to enable clients to have control over block
> -------------------------------------------------------------------------------------------------
>                 Key: HDFS-2576
>                 URL: https://issues.apache.org/jira/browse/HDFS-2576
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Pritam Damania
>         Attachments: hdfs-2576-1.txt
> Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes
to place the blocks for a file for higher level of locality. For this purpose there is a need
of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations
where the client wants to put each block. The proposed solution is a favored nodes parameter
in the addBlock() method and in the create() file method to enable the clients to give the
hints to the NameNode about the locations of each replica of the block. Note that this would
be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide
whether it can respect the hints or not.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message