hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravi Prakash (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9205) Do not schedule corrupt blocks for replication
Date Wed, 24 Aug 2016 23:01:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15435895#comment-15435895

Ravi Prakash commented on HDFS-9205:

Thanks for the change Nicholas! Should this line be modified? https://github.com/apache/hadoop/blob/a1f3293762dddb0ca953d1145f5b53d9086b25b8/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/LowRedundancyBlocks.java#L62

I think most often this queue had missing blocks, so it didn't really make sense to re-replicate
missing blocks anyway. We should be careful about removing this queue though, because its
where the [count of missing blocks is taken from|https://github.com/apache/hadoop/blob/a1f3293762dddb0ca953d1145f5b53d9086b25b8/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java#L4112]

> Do not schedule corrupt blocks for replication
> ----------------------------------------------
>                 Key: HDFS-9205
>                 URL: https://issues.apache.org/jira/browse/HDFS-9205
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>            Reporter: Tsz Wo Nicholas Sze
>            Assignee: Tsz Wo Nicholas Sze
>            Priority: Minor
>             Fix For: 2.8.0
>         Attachments: h9205_20151007.patch, h9205_20151007b.patch, h9205_20151008.patch,
h9205_20151009.patch, h9205_20151009b.patch, h9205_20151013.patch, h9205_20151015.patch
> Corrupted blocks by definition are blocks cannot be read. As a consequence, they cannot
be replicated.  In UnderReplicatedBlocks, there is a queue for QUEUE_WITH_CORRUPT_BLOCKS and
chooseUnderReplicatedBlocks may choose blocks from it.  It seems that scheduling corrupted
block for replication is wasting resource and potentially slow down replication for the higher
priority blocks.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message