hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Carey (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly
Date Tue, 25 May 2010 22:05:34 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12871385#action_12871385

Scott Carey commented on HDFS-1172:

bq. # Scott's solution of making the primary DN send the blockReceived on account of all DNs
would work, but sounds complicated, expecially in the failure cases (eg what if the primary
DN fails just before sending the RPC? Do we lose all the replicas? No good!)

Yeah, its complicated.  To simplify failure scenarios, leave the rest to be similar to the
current state -- the next regularly scheduled ping from a DN will provide the new block information,
but the primary DN will still do its best to send all the block data it can gather so that
the initial registration is as complete as possible.  Perhaps the NN treats this extra information
as provisional, until it gets a ping from the other DN's to confirm.  

Functionally, this won't differ much from Dhruba's proposition, and is more complicated.

> Blocks in newly completed files are considered under-replicated too quickly
> ---------------------------------------------------------------------------
>                 Key: HDFS-1172
>                 URL: https://issues.apache.org/jira/browse/HDFS-1172
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
> I've seen this for a long time, and imagine it's a known issue, but couldn't find an
existing JIRA. It often happens that we see the NN schedule replication on the last block
of files very quickly after they're completed, before the other DNs in the pipeline have a
chance to report the new block. This results in a lot of extra replication work on the cluster,
as we replicate the block and then end up with multiple excess replicas which are very quickly

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message