hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Buddy (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-5434) Write resiliency for replica count 1
Date Mon, 28 Oct 2013 13:53:31 GMT
Buddy created HDFS-5434:

             Summary: Write resiliency for replica count 1
                 Key: HDFS-5434
                 URL: https://issues.apache.org/jira/browse/HDFS-5434
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: namenode
    Affects Versions: 2.2.0
            Reporter: Buddy
            Priority: Minor

If a file has a replica count of one, the HDFS client is exposed to write failures if the
data node fails during a write. With a pipeline of size of one, no recovery is possible if
the sole data node dies.

A simple fix is to force a minimum pipeline size of 2, while leaving the replication count
as 1. The implementation for this is fairly non-invasive.

Although the replica count is one, the block will be written to two data nodes instead of
one. If one of the data nodes fails during the write, normal pipeline recovery will ensure
that the write succeeds to the other data node.

The existing code in the name node will prune the extra replica when it receives the block
received reports for the finalized block from both data nodes. This results in the intended
replica count of one for the block.

This behavior should be controlled by a configuration option such as dfs.namenode.minPipelineSize.

This behavior can be implemented in FSNameSystem.getAdditionalBlock by ensuring that the pipeline
size passed to BlockPlacementPolicy.chooseTarget in the replication parameter is at least:

        max(replication, ${dfs.namenode.minPipelineSize})

This message was sent by Atlassian JIRA

View raw message