hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
Date Wed, 27 Feb 2013 19:39:12 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13588674#comment-13588674
] 

stack commented on HDFS-2576:
-----------------------------

bq. I can't see how we can easily handle this problem.

I am not sure we can; we cannot fix fellas resolve for them.  It would be good though if you
could do something to make it so when the problem is present, it is noted somehow.  A WARN
log saying I sent you a list of favored nodes and you ignored it would probably get pretty
annoying pretty quickly but maybe in a trace-level log adding to an existing log whether or
not favored nodes is being respected would help us know if this feature is working or not
(or say, adding to 'LOG.debug(src + ": masked=" + masked);' if favoredNodes are being asked
for) or in the NN, it could note a DN is asking us to use favored nodes not part of the cluster?
 For example when you do this:

+      // If the current cluster doesn't contain the node, fallback to
+      // something machine local and then rack local.

Do you think there could be a log here -- its pretty odd being asked use a node not part of
the cluster?

On comment, if a new patch, would suggest an edit to make your intent more clear.

Patch still good by me.  Good on you Devaraj
                
> Namenode should have a favored nodes hint to enable clients to have control over block
placement.
> -------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-2576
>                 URL: https://issues.apache.org/jira/browse/HDFS-2576
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Pritam Damania
>         Attachments: hdfs-2576-1.txt
>
>
> Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes
to place the blocks for a file for higher level of locality. For this purpose there is a need
of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations
where the client wants to put each block. The proposed solution is a favored nodes parameter
in the addBlock() method and in the create() file method to enable the clients to give the
hints to the NameNode about the locations of each replica of the block. Note that this would
be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide
whether it can respect the hints or not.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message