hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-3799) Design a pluggable interface to place replicas of blocks in HDFS
Date Mon, 11 May 2009 13:57:45 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-3799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708035#action_12708035

dhruba borthakur commented on HADOOP-3799:

I would like to figure out how to expose the filename in the API. It can be easily done when
this API is invoked at the time when a new block is being allocated to a file. However,  when
the replicator takes a under-replicated block and tries to get a new replica location for
that block, it is costly to the find the filename.

The filename can be deduced by traversing the INode-tree that is maintained by the namenode.
But it is a costly operation beccause one has to traverse the entire branch starting from
the specified INOde to the root. One option is to pass in the INode into the block-placement
algorithm. f the algorithm needs the complete path name of the file in question, then it has
to do the costly operation of generating the full path name of the file. However, this makes
the interface kind-of less elegant. I am still debating how to do it right. 

> Design a pluggable interface to place replicas of blocks in HDFS
> ----------------------------------------------------------------
>                 Key: HADOOP-3799
>                 URL: https://issues.apache.org/jira/browse/HADOOP-3799
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>         Attachments: BlockPlacementPluggable.txt
> The current HDFS code typically places one replica on local rack, the second replica
on remote random rack and the third replica on a random node of that remote rack. This algorithm
is baked in the NameNode's code. It would be nice to make the block placement algorithm a
pluggable interface. This will allow experimentation of different placement algorithms based
on workloads, availability guarantees and failure models.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message