hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly
Date Sat, 22 May 2010 06:28:16 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12870278#action_12870278
] 

Todd Lipcon commented on HDFS-1172:
-----------------------------------

Particular sequence of events:

# client finishes writing to block with 3 replicas
# first DN happens to heartbeat, so addStoredBlock is called in the NN
# client calls completeFile, which calls checkReplicationFactor(newFile) when finalizing the
INode
# NN adds block to pending replication
# Replication monitor runs and schedules two replications
# second and third pipeline DNs send their addStoredBlock notifications with their heartbeats
# Replications finish, and the new replicas report the new blocks as well
# NN notices the excess replicas and schedules deletion

This doesn't cause major issues, but we do end up wasting a fair amount of disk and network
resources.

The question is why sometimes the immediate heartbeat on blockReceived doesn't trigger as
it's supposed to.

> Blocks in newly completed files are considered under-replicated too quickly
> ---------------------------------------------------------------------------
>
>                 Key: HDFS-1172
>                 URL: https://issues.apache.org/jira/browse/HDFS-1172
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
>
> I've seen this for a long time, and imagine it's a known issue, but couldn't find an
existing JIRA. It often happens that we see the NN schedule replication on the last block
of files very quickly after they're completed, before the other DNs in the pipeline have a
chance to report the new block. This results in a lot of extra replication work on the cluster,
as we replicate the block and then end up with multiple excess replicas which are very quickly
deleted.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message