hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uma Maheswara Rao G (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-1765) Block Replication should respect under-replication block priority
Date Thu, 24 Nov 2011 18:46:40 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13156865#comment-13156865

Uma Maheswara Rao G commented on HDFS-1765:

One scenario here:
 Accidentally user configured single rack so, all the block goes to under replicated.
after some time, high priority blocks got added to queues. 
While processing it is trying to process only previos underreplicated blocks, even though
there are high priority blocks added.

Looking at this part, single replecatioIndex maintained for all the priority queues. For suppose
if it needs to process 100blocks in this iteration, it may not be able to find the targets
because there is only single rack available. 
so, the replecationIndex will not be decremented. In Next iteration it will try to skip the
number of replecationIndex blocks. Lets say 50 blocks added to high priority queue now. ReplIndex
was already 100. So, it will skip first 100 blcoks and start picking the remaining. Due to
this, high peiority blocks are not getting chance to replicate quickly.

> Block Replication should respect under-replication block priority
> -----------------------------------------------------------------
>                 Key: HDFS-1765
>                 URL: https://issues.apache.org/jira/browse/HDFS-1765
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.23.0
>            Reporter: Hairong Kuang
>            Assignee: Hairong Kuang
>             Fix For: 0.24.0
>         Attachments: underReplicatedQueue.pdf
> Currently under-replicated blocks are assigned different priorities depending on how
many replicas a block has. However the replication monitor works on blocks in a round-robin
fashion. So the newly added high priority blocks won't get replicated until all low-priority
blocks are done. One example is that on decommissioning datanode WebUI we often observe that
"blocks with only decommissioning replicas" do not get scheduled to replicate before other
blocks, so risking data availability if the node is shutdown for repair before decommission

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message