hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jing Zhao (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly
Date Tue, 29 Sep 2015 21:57:06 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14935973#comment-14935973
] 

Jing Zhao commented on HDFS-1172:
---------------------------------

# It is not necessary to call {{numNodes}} again. We can directly use {{numNodes}}.
{code}
     int numNodes = curBlock.numNodes();
     ......
+    DatanodeStorageInfo[] expectedStorages =
+        curBlock.getUnderConstructionFeature().getExpectedStorageLocations();
+    if (curBlock.numNodes() < expectedStorages.length) {
{code}

# We'd better place the new "adding block to pending replica queue" logic only in {{checkReplication}}.
Several reasons for this: 
#* {{completeBlock}} is also called by {{forceCompleteBlock}}, which is invoked when loading
edits. At this time we should not update pending replication queue since the NN is just being
started.
#* {{completeBlock}} can often be called when NN has only received 1 block_received msg, updating
pending replication queue at this time means later when further IBRs (incremental block reports)
come we need to remove these DN from pending queue again.
#* Semantically updating pending queue is more closely coupled with updating neededReplication
queue.
# Instead of making changes to {{PendingBlockInfo}}'s constructor, when updating the pending
replication queue, you can prepare all the corresponding {{DatanodeDescriptor}} in an array
first, and call {{pendingReplications.increment}} only once.
# Do we need to call {{computeAllPendingWork}} in {{TestReplication#pendingReplicationCount}}?
# Let's add a maximum retry count or total waiting time for {{waitForNoPendingReplication}}.


> Blocks in newly completed files are considered under-replicated too quickly
> ---------------------------------------------------------------------------
>
>                 Key: HDFS-1172
>                 URL: https://issues.apache.org/jira/browse/HDFS-1172
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>            Assignee: Masatake Iwasaki
>         Attachments: HDFS-1172-150907.patch, HDFS-1172.008.patch, HDFS-1172.009.patch,
HDFS-1172.010.patch, HDFS-1172.patch, hdfs-1172.txt, hdfs-1172.txt, replicateBlocksFUC.patch,
replicateBlocksFUC1.patch, replicateBlocksFUC1.patch
>
>
> I've seen this for a long time, and imagine it's a known issue, but couldn't find an
existing JIRA. It often happens that we see the NN schedule replication on the last block
of files very quickly after they're completed, before the other DNs in the pipeline have a
chance to report the new block. This results in a lot of extra replication work on the cluster,
as we replicate the block and then end up with multiple excess replicas which are very quickly
deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message