hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-898) Sequential generation of block ids
Date Wed, 20 Jan 2010 23:08:55 GMT

    [ https://issues.apache.org/jira/browse/HDFS-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12803044#action_12803044

Konstantin Shvachko commented on HDFS-898:

 (1) I have a block (id = 1001, gs = 7777), which belongs to an existing file, replication
factor = 3. I have two replicas of this block (id = 1001, gs = 7775) and (id = 1001, gs =
7770). This makes the block corrupt, and we don't remove replicas of a corrupt block. Then
a third data-node starts, which was down for long time, it has replica (id = 1001, gs = 7),
which at some point belonged to a different file, which was removed from the system since
then. So I have three replicas, and the name-node does not have info to determine which replica
is stale, and which one belonged to the removed file.
Does that make sense?

(2) Yes, all old blocks have gs=0. So if a have an old block with three valid replicas, and
a DN with a prehistoric replica comes up, then you will not be able to distinguish between
them. I mean that introduction of generation stamps does not solve the prehistoric block problem
for blocks that existed before blocks had generation stamps.

> Sequential generation of block ids
> ----------------------------------
>                 Key: HDFS-898
>                 URL: https://issues.apache.org/jira/browse/HDFS-898
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: name-node
>    Affects Versions: 0.20.1
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.22.0
> This is a proposal to replace random generation of block ids with a sequential generator
in order to avoid block id reuse in the future.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message