hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Masatake Iwasaki (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly
Date Thu, 08 Oct 2015 16:27:28 GMT

     [ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Masatake Iwasaki updated HDFS-1172:
-----------------------------------
    Attachment: HDFS-1172.012.patch

I update the patch.
* addressed the failure of TestRecoverStripedFile: fixed to avoid updating pendingReplications
if file is striped.
* added calling to {{DataNodeTestUtils#triggerHeartbeat}} in order to make sure {{TestReplication#testNoExtraReplicationWhenBlockReceivedIsLate}}
fails without the fix of BlockManager.
* fixed checkstyle warning except for file length.
* fixed whitespace error.
* release audit is not related to the fix.
* failure of TestBlockReport and TestCheckpoint is not related to the code path of the patch.
I could not reproduce the failure on my env.

> Blocks in newly completed files are considered under-replicated too quickly
> ---------------------------------------------------------------------------
>
>                 Key: HDFS-1172
>                 URL: https://issues.apache.org/jira/browse/HDFS-1172
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>            Assignee: Masatake Iwasaki
>         Attachments: HDFS-1172-150907.patch, HDFS-1172.008.patch, HDFS-1172.009.patch,
HDFS-1172.010.patch, HDFS-1172.011.patch, HDFS-1172.012.patch, HDFS-1172.patch, hdfs-1172.txt,
hdfs-1172.txt, replicateBlocksFUC.patch, replicateBlocksFUC1.patch, replicateBlocksFUC1.patch
>
>
> I've seen this for a long time, and imagine it's a known issue, but couldn't find an
existing JIRA. It often happens that we see the NN schedule replication on the last block
of files very quickly after they're completed, before the other DNs in the pipeline have a
chance to report the new block. This results in a lot of extra replication work on the cluster,
as we replicate the block and then end up with multiple excess replicas which are very quickly
deleted.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message