hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zhe Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7969) Erasure coding: lease recovery for striped block groups
Date Mon, 23 Mar 2015 23:39:52 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14376918#comment-14376918
] 

Zhe Zhang commented on HDFS-7969:
---------------------------------

In the latest [design doc | https://issues.apache.org/jira/secure/attachment/12697210/HDFSErasureCodingDesign-20150206.pdf],
[~szetszwo] has a good summary of handling the generation stamp of a striped block group.
This JIRA aims to implement lease recovery for striped block groups. Other scenarios related
to GS (append, truncate, failures in writing) will be handled in separate JIRAs.

Below is the process of a lease recovery (borrowed from a [blog post | http://blog.cloudera.com/blog/2015/02/understanding-hdfs-recovery-processes-part-1/]
by [~yzhangal]).
# Get the DataNodes which contain the last block of f.
# Assign one of the DataNodes as the primary DataNode p.
# p obtains a new generation stamp from the NameNode.
# p gets the block info from each DataNode.
# p computes the minimum block length.
# p updates the DataNodes, which have a valid generation  stamp, with the new generation stamp
and the minimum block length.
# p acknowledges the NameNode the update results.
# NameNode updates the BlockInfo.
# NameNode remove f’s lease (other writer can now obtain the lease for writing to f).
# NameNode commit changes to edit log.

The main updates should be the following:
bq. Assign one of the DataNodes as the primary DataNode p.
We _might_ need a different algorithm for selecting primary DN. 
bq. p computes the minimum block length.
This needs to be updated with striping logic.

> Erasure coding: lease recovery for striped block groups
> -------------------------------------------------------
>
>                 Key: HDFS-7969
>                 URL: https://issues.apache.org/jira/browse/HDFS-7969
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Zhe Zhang
>            Assignee: Zhe Zhang
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message