hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6325) Append should fail if the last block has insufficient number of replicas
Date Mon, 12 May 2014 00:56:17 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6325?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13994737#comment-13994737

Konstantin Shvachko commented on HDFS-6325:

Here are some thoughts after experimenting and discussions.

# We should still allow executing append if the number of live DNs on the cluster is less
than file replication factor. This is to support cases with small clusters or replication
being set intentionally high.
# We should probably go for option (1) with min-replication. This has an advantage that there
will be no change in semantics with current implementation. The disadvantage is that once
file is opened for append the ongoing replications of the last block will fail, because of
the genStamp change.
# In the end our main goal is to make append fail earlier to prevent the file from being under
construction while nobody can use it, which happens when the last  block has 0 replicas.

> Append should fail if the last block has insufficient number of replicas
> ------------------------------------------------------------------------
>                 Key: HDFS-6325
>                 URL: https://issues.apache.org/jira/browse/HDFS-6325
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 2.2.0
>            Reporter: Konstantin Shvachko
>            Assignee: Keith Pak
>         Attachments: HDFS-6325.patch, HDFS-6325_test.patch, appendTest.patch
> Currently append() succeeds on a file with the last block that has no replicas. But the
subsequent updatePipeline() fails as there are no replicas with the exception "Unable to retrieve
blocks locations for last block". This leaves the file unclosed, and others can not do anything
with it until its lease expires.
> The solution is to check replicas of the last block on the NameNode and fail during append()
rather than during updatePipeline().
> How many replicas should be present before NN allows to append? I see two options:
> # min-replication: allow append if the last block is minimally replicated (1 by default)
> # full-replication: allow append if the last block is fully replicated (3 by default)

This message was sent by Atlassian JIRA

View raw message