hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5168) BlockPlacementPolicy does not work for cross node group dependencies
Date Mon, 31 Mar 2014 11:57:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13955116#comment-13955116
] 

Hadoop QA commented on HDFS-5168:
---------------------------------

{color:red}-1 overall{color}.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12637455/HDFS-5168.patch
  against trunk revision .

    {color:green}+1 @author{color}.  The patch does not contain any @author tags.

    {color:green}+1 tests included{color}.  The patch appears to include 10 new or modified
test files.

      {color:red}-1 javac{color}.  The applied patch generated 1486 javac compiler warnings
(more than the trunk's current 1484 warnings).

    {color:green}+1 javadoc{color}.  There were no new javadoc warning messages.

    {color:green}+1 eclipse:eclipse{color}.  The patch built with eclipse:eclipse.

    {color:green}+1 findbugs{color}.  The patch does not introduce any new Findbugs (version
1.3.9) warnings.

    {color:green}+1 release audit{color}.  The applied patch does not increase the total number
of release audit warnings.

    {color:red}-1 core tests{color}.  The patch failed these unit tests in hadoop-common-project/hadoop-common
hadoop-hdfs-project/hadoop-hdfs hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common:

                  org.apache.hadoop.hdfs.server.namenode.TestFsck

    {color:green}+1 contrib tests{color}.  The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6562//testReport/
Javac warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/6562//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6562//console

This message is automatically generated.

> BlockPlacementPolicy does not work for cross node group dependencies
> --------------------------------------------------------------------
>
>                 Key: HDFS-5168
>                 URL: https://issues.apache.org/jira/browse/HDFS-5168
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Nikola Vujic
>            Assignee: Nikola Vujic
>            Priority: Critical
>         Attachments: HDFS-5168.patch
>
>
> Block placement policies do not work for cross rack/node group dependencies. In reality
this is needed when compute servers and storage fall in two independent fault domains, then
both BlockPlacementPolicyDefault and BlockPlacementPolicyWithNodeGroup are not able to provide
proper block placement.
> Let's suppose that we have Hadoop cluster with one rack with two servers, and we run
2 VMs per server. Node group topology for this cluster would be:
>  server1-vm1 -> /d1/r1/n1
>  server1-vm2 -> /d1/r1/n1
>  server2-vm1 -> /d1/r1/n2
>  server2-vm2 -> /d1/r1/n2
> This is working fine as long as server and storage fall into the same fault domain but
if storage is in a different fault domain from the server, we will not be able to handle that.
For example, if storage of server1-vm1 is in the same fault domain as storage of server2-vm1,
then we must not place two replicas on these two nodes although they are in different node
groups.
> Two possible approaches:
> - One approach would be to define cross rack/node group dependencies and to use them
when excluding nodes from the search space. This looks as the cleanest way to fix this as
it requires minor changes in the BlockPlacementPolicy classes.
> - Other approach would be to allow nodes to fall in more than one node group. When we
chose a node to hold a replica we have to exclude from the search space all nodes from the
node groups where the chosen node belongs. This approach may require major changes in the
NetworkTopology.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message