hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hairong Kuang (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-940) pendingReplications of FSNamesystem is not informative
Date Tue, 27 Feb 2007 06:54:05 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12476080
] 

Hairong Kuang commented on HADOOP-940:
--------------------------------------

This patch sets the time out period to be 10 minutes and the pending replication monitor thread
scans the pending replication queue for time out blocks once every 1 minute. Because the chance
that a block does not get replicated is rare, I feel that the monitor thread scans too frequently.
Does it make more sense to set the time out period to be 5 minutes and the scanning frequency
to be once every 5 minutes?

Each entry in the pendingReplication queue is a block with only one timer. When a replica
is placed, the timer associated with the block is reset. This may cause that the re-replication
of a replica gets delayed. For example, if a replica's replication request gets lost but another
replica of the same block is placed right before its timer gets expired, the lost request
won't be detected until twice of the timeout period is passed.

> pendingReplications of FSNamesystem is not informative
> ------------------------------------------------------
>
>                 Key: HADOOP-940
>                 URL: https://issues.apache.org/jira/browse/HADOOP-940
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.10.1
>            Reporter: Hairong Kuang
>         Attachments: pendingReplication.patch
>
>
> Currently when a neededReplication block is scheduled to be replicated, it is put to
the pendingReplications queue. When it is no longer under replicated, it is pulled out of
the pendingReplications queue. But the queue does not provide any information like how many
targets have been choosen or who those targets are. PendingReplications are not used when
deciding if a block is under replication. This may cause a block to be over replications or
inaccurate estimate of its replication priority.
> For example, when a block has 1 replicas but it's replication factor is 2, a data node
is choosen to replicate this block and the block is put in the pendingReplications queue.
If the block's replication factor is changed to be 3 before the block replication notification,
which is the next block report, comes in, the block will be put into neededReplictions queue
again under the assumption that it needs to choose 2 targets instead of 1. So the block will
end up with 4 replicas.
> I propose that we change pendingReplications to be a map from a block to the choosen
data nodes. Data nodes in both pendingReplications and blockMap are used when deciding the
total number of replicas that a block has. When the name node is notified that the block is
replicated in a choosen data node, the data node is moved from pendingReplications to blockMap.
> Each choosen target is also associated with a timer indicating how long it expects to
receive the block replication notification. PendingReplications queue needs to be periodically
scanned to remove those data nodes whose timer is expired.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message