hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2576) Namenode should have a favored nodes hint to enable clients to have control over block placement.
Date Thu, 28 Feb 2013 05:51:19 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13589228#comment-13589228
] 

Uma Maheswara Rao G commented on HDFS-2576:
-------------------------------------------

{quote}
>From a design point of view would it be possible to just add an attribute or flag on hdfs
files or directories that specify an block affinity group? This would seem cheaper than an
alternative that specifies specific favored dn's lists for block replicas. This would seem
more robust than something specified only at file creation time, and more managable in the
long term if data nodes membership changes over time.
{quote}
+1
Also looks like this idea is closure to http://www.vldb.org/pvldb/vol4/p575-eltabakh.pdf (cohadoop)

Anyway HDFS take care of keeping first block local to the node.(is that the case like one
RS create the files, another RS will work on them? not sure is there a case ). Once HDFS choosen
the targets for first block, subsequent block can use the same location as previous block?
I am not sure what is the Strict requirement of passing node addresses from client. Even if
further files also wanted to choose the same targets, we can pass same group id then the blocks
for that file also will be placed in same nodes.  This looks to be generic idea. Persisting
thism will help HDFS balancers to replicate if DN alone goes down for the replication. I might
miss lot of history in this discussion, please correct me if I am wrong.
Also, most of the required new target choosing mechanism should go as new pluggable policy.[may
be required api can be added to policy interface to fit this requirement]

                
> Namenode should have a favored nodes hint to enable clients to have control over block
placement.
> -------------------------------------------------------------------------------------------------
>
>                 Key: HDFS-2576
>                 URL: https://issues.apache.org/jira/browse/HDFS-2576
>             Project: Hadoop HDFS
>          Issue Type: New Feature
>            Reporter: Pritam Damania
>         Attachments: hdfs-2576-1.txt
>
>
> Sometimes Clients like HBase are required to dynamically compute the datanodes it wishes
to place the blocks for a file for higher level of locality. For this purpose there is a need
of a way to give the Namenode a hint in terms of a favoredNodes parameter about the locations
where the client wants to put each block. The proposed solution is a favored nodes parameter
in the addBlock() method and in the create() file method to enable the clients to give the
hints to the NameNode about the locations of each replica of the block. Note that this would
be just a hint and finally the NameNode would look at disk usage, datanode load etc. and decide
whether it can respect the hints or not.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message