hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Scott Carey (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1172) Blocks in newly completed files are considered under-replicated too quickly
Date Mon, 24 May 2010 18:10:23 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12870749#action_12870749

Scott Carey commented on HDFS-1172:

Perhaps when the write pipeline completes, it should pass back the block information so that
the initial commit to the NN can atomically add all the blocks.


DN's in pipe are DN1, DN2, DN3.

A block is being written, the client writes to DN1, which writes to DN2, which writes to DN3.
 When DN3 completes, it notifies DN2 and provides its block replica information.  When DN2
completes and has DN3's response, it passes its information, along with DN3's, to DN1.  When
DN1 completes, and has DN2's information along with DN3's, it reports to the NN the information
about all 3 replicas, and lastly returns to the original client.

This will have a few benefits:

Fewer RPC's to the NN, and therefore less NN load.
Atomic visibility of all replicas to the NN and clients.

> Blocks in newly completed files are considered under-replicated too quickly
> ---------------------------------------------------------------------------
>                 Key: HDFS-1172
>                 URL: https://issues.apache.org/jira/browse/HDFS-1172
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: name-node
>    Affects Versions: 0.21.0
>            Reporter: Todd Lipcon
> I've seen this for a long time, and imagine it's a known issue, but couldn't find an
existing JIRA. It often happens that we see the NN schedule replication on the last block
of files very quickly after they're completed, before the other DNs in the pipeline have a
chance to report the new block. This results in a lot of extra replication work on the cluster,
as we replicate the block and then end up with multiple excess replicas which are very quickly

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message