Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Message-ID: <16797168.43511274820574799.JavaMail.jira@thor>
Date: Tue, 25 May 2010 16:49:34 -0400 (EDT)
From: "dhruba borthakur (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Subject: [jira] Commented: (HDFS-1172) Blocks in newly completed files are
 considered under-replicated too quickly
In-Reply-To: <30139643.33841274509276732.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12871353#action_12871353 ] 

dhruba borthakur commented on HDFS-1172:
----------------------------------------

> UnderReplicatedBlocks could be augmented to carry a dontProcessUntil timestamp. 

To expand on this idea, we can delay replication of a block until a few seconds (configurable) after the modification time of the file. That could avoid storing an additional timestamp in UnderReplicatedBlocks. 

> Blocks in newly completed files are considered under-replicated too quickly
> ---------------------------------------------------------------------------
>
>                 Key: HDFS-1172
>                 URL: https://issues.apache.org/jira/browse/HDFS-1172
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>
> I've seen this for a long time, and imagine it's a known issue, but couldn't find an existing JIRA. It often happens that we see the NN schedule replication on the last block of files very quickly after they're completed, before the other DNs in the pipeline have a chance to report the new block. This results in a lot of extra replication work on the cluster, as we replicate the block and then end up with multiple excess replicas which are very quickly deleted.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.