hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "dhruba borthakur (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-2656) Support for upgrading existing cluster to facilitate appends to HDFS files
Date Sat, 08 Mar 2008 00:19:46 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-2656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12576454#action_12576454
] 

dhruba borthakur commented on HADOOP-2656:
------------------------------------------

The Datanode needs to store a block generation stamp for each block. The original idea was
to store the block generation stamp inside the meta file of each block. One major disadvantage
of this approach is that generation of a block report requires that each meta file be opened
and the generation stamp read from it. With about 50K blocks per datanode and a seek time
of 10 ms, this might require 300 seconds. This means that the time to restart a cluster cannot
be lesser than this 300 seconds.

A few other alternatives:
1. Encode the generation stamp into the name of the metafile. Each metafile will look like
blkxxxxxx.genstamp.meta. The block file will remain the same.

2. Encode the generation stamp into the name of the block file. Each block file will be of
the form blkxxxxxx.genstamp. The metafile will remain the same.

3. Encode the generation stamp into the name of a new zero-size file named blkxxxxx.genstamp.
The block file and the metadata file will remain the same.

4. A completely separate file (one per datanode) that records the metadata of all blocks in
the datanode.

I propose that we implement option 1.

> Support for upgrading existing cluster to facilitate appends to HDFS files
> --------------------------------------------------------------------------
>
>                 Key: HADOOP-2656
>                 URL: https://issues.apache.org/jira/browse/HADOOP-2656
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: dfs
>            Reporter: dhruba borthakur
>            Assignee: dhruba borthakur
>
> HADOOP-1700 describes the design for supporting appends to HDFS files. This design requires
a distributed-upgrade to existing cluster installations. The design specifies that the DataNode
persist the 8-byte BlockGenerationStamp in the block metadata file. The upgrade code will
introduce this new field in the block metadata file and initialize this value to 0.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message