Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hdfs-issues@hadoop.apache.org
Date: Sat, 10 Dec 2011 01:46:40 +0000 (UTC)
From: "Aaron T. Myers (Updated) (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: 
 <591168073.61081.1323481600102.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <41277235.23317.1322602060004.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Updated] (HDFS-2602) Standby needs to maintain BlockInfo
 while following edits
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


     [ https://issues.apache.org/jira/browse/HDFS-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Aaron T. Myers updated HDFS-2602:
---------------------------------

    Attachment: HDFS-2602.patch

Here's a patch which includes all of the changes from the most recent patch posted on HDFS-1108. Additionally, this patch modifies FSEditLogLoader to do the following:

* Separate the handling of OP_ADD and OP_CLOSE.
* More clearly separate the file create vs. block add parts of OP_ADD.
* Adds a new method, updateBlocks(), which updates the in-memory data structures of the NN on OP_CLOSE or OP_ADD in the case of adding a block.

This patch also cleans up some of the file creation code, in particular that OP_ADD is no longer used to create directories, in lieu of OP_MKDIR.

With this patch, TestStandbyIsHot passes.

It wouldn't be difficult to separate the two cases of OP_ADD (file create and block addition) into separate opcodes, along the lines of OP_CREATE and OP_ADD_BLOCK, but there seems to be little benefit in doing so, except slight code clarity improvement. If people feel strongly I'd be happy to implement that.
                
> Standby needs to maintain BlockInfo while following edits
> ---------------------------------------------------------
>
>                 Key: HDFS-2602
>                 URL: https://issues.apache.org/jira/browse/HDFS-2602
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ha
>    Affects Versions: HA branch (HDFS-1623)
>            Reporter: Todd Lipcon
>            Assignee: Aaron T. Myers
>            Priority: Critical
>         Attachments: HDFS-2602.patch
>
>
> As described in HDFS-1975:
> When we close a file, or add another block to a file, we write OP_CLOSE or OP_ADD in the txn log. FSEditLogLoader, when it sees these types of transactions, creates new BlockInfo objects for all of the blocks listed in the transaction. These new BlockInfos have no block locations associated. So, when we close a file, the SBNN loses its block locations info for that file and is no longer "hot".
> I have an ugly hack which copies over the old BlockInfos from the existing INode, but I'm not convinced it's the right way. It might be cleaner to add new opcode types like OP_ADD_ADDITIONAL_BLOCK, and actually treat OP_CLOSE as just a finalization of INodeFileUnderConstruction to INodeFile, rather than replacing block info at all.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira