Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Message-ID: <24958369.33861274509696244.JavaMail.jira@thor>
Date: Sat, 22 May 2010 02:28:16 -0400 (EDT)
From: "Todd Lipcon (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Subject: [jira] Commented: (HDFS-1172) Blocks in newly completed files are
 considered under-replicated too quickly
In-Reply-To: <30139643.33841274509276732.JavaMail.jira@thor>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12870278#action_12870278 ] 

Todd Lipcon commented on HDFS-1172:
-----------------------------------

Particular sequence of events:

# client finishes writing to block with 3 replicas
# first DN happens to heartbeat, so addStoredBlock is called in the NN
# client calls completeFile, which calls checkReplicationFactor(newFile) when finalizing the INode
# NN adds block to pending replication
# Replication monitor runs and schedules two replications
# second and third pipeline DNs send their addStoredBlock notifications with their heartbeats
# Replications finish, and the new replicas report the new blocks as well
# NN notices the excess replicas and schedules deletion

This doesn't cause major issues, but we do end up wasting a fair amount of disk and network resources.

The question is why sometimes the immediate heartbeat on blockReceived doesn't trigger as it's supposed to.

> Blocks in newly completed files are considered under-replicated too quickly
> ---------------------------------------------------------------------------
>
>                 Key: HDFS-1172
>                 URL: https://issues.apache.org/jira/browse/HDFS-1172
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>
> I've seen this for a long time, and imagine it's a known issue, but couldn't find an existing JIRA. It often happens that we see the NN schedule replication on the last block of files very quickly after they're completed, before the other DNs in the pipeline have a chance to report the new block. This results in a lot of extra replication work on the cluster, as we replicate the block and then end up with multiple excess replicas which are very quickly deleted.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.