hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brahma Reddy Battula (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-8647) Abstract BlockManager's rack policy into BlockPlacementPolicy
Date Mon, 19 Oct 2015 17:38:05 GMT

    [ https://issues.apache.org/jira/browse/HDFS-8647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14963662#comment-14963662
] 

Brahma Reddy Battula commented on HDFS-8647:
--------------------------------------------

[~mingma] 
{quote}BlockManager should swap addedNode and delNodeHint.,{quote}
yes you are correct..

{quote}After the fix, the only failed unit test left is {{TestBalancer#testBalancerWithPinnedBlocks}}.
Can you please investigate if it is related to the patch?{quote}
.{code}cluster = new MiniDFSCluster.Builder(conf).numDataNodes(capacities.length)
  .hosts(new String[]{"localhost", "localhost"})
  .racks(new String[]{"rack0", "rack1"}).simulatedCapacities(capacities).build(){code}

2 DNs are started with "rack1". Ideally we should not create 2 DNs with the same hostname.And
Pinning depends on favoredNodes.DFSClient#create(..) only uses host:port, if favoredNodes
is created by {{new InetSocketAddress(ip, port)}}

DFSClient will attempt a reverse lookup locally to get host:port, instead of sending ip:port
directly to NameNode.
.
MiniDFSCluster use fake hostname "host1.foo.com" to start DataNodes.DFSClient doesn't use
StaticMapping. So if DFSClient do reverse lookup, "127.0.0.1:8020" becomes "localhost:8020".

{quote}{{public}} can be removed from {{public DatanodeStorageInfo chooseReplicaToDelete}}.{quote}

it's overidden in test {{TestDnfencing#chooseReplicaToDelete}} .hence I make this method visible
for test.

Remaning changes are done and uploaded the patch....Hope I address all of your comments..

> Abstract BlockManager's rack policy into BlockPlacementPolicy
> -------------------------------------------------------------
>
>                 Key: HDFS-8647
>                 URL: https://issues.apache.org/jira/browse/HDFS-8647
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Ming Ma
>            Assignee: Brahma Reddy Battula
>         Attachments: HDFS-8647-001.patch, HDFS-8647-002.patch, HDFS-8647-003.patch, HDFS-8647-004.patch,
HDFS-8647-004.patch, HDFS-8647-005.patch, HDFS-8647-006.patch, HDFS-8647-007.patch
>
>
> Sometimes we want to have namenode use alternative block placement policy such as upgrade
domains in HDFS-7541.
> BlockManager has built-in assumption about rack policy in functions such as useDelHint,
blockHasEnoughRacks. That means when we have new block placement policy, we need to modify
BlockManager to account for the new policy. Ideally BlockManager should ask BlockPlacementPolicy
object instead. That will allow us to provide new BlockPlacementPolicy without changing BlockManager.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message