Return-Path: 
 <hdfs-issues-return-76454-apmail-hadoop-hdfs-issues-archive=hadoop.apache.org@hadoop.apache.org>
X-Original-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org
Delivered-To: apmail-hadoop-hdfs-issues-archive@minotaur.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id 20C8D10CB5
	for <apmail-hadoop-hdfs-issues-archive@minotaur.apache.org>;
 Tue, 21 Jan 2014 18:06:31 +0000 (UTC)
Received: (qmail 79618 invoked by uid 500); 21 Jan 2014 18:06:25 -0000
Delivered-To: apmail-hadoop-hdfs-issues-archive@hadoop.apache.org
Received: (qmail 79586 invoked by uid 500); 21 Jan 2014 18:06:24 -0000
Mailing-List: contact hdfs-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hdfs-issues-help@hadoop.apache.org>
List-Unsubscribe: <mailto:hdfs-issues-unsubscribe@hadoop.apache.org>
List-Post: <mailto:hdfs-issues@hadoop.apache.org>
List-Id: <hdfs-issues.hadoop.apache.org>
Reply-To: hdfs-issues@hadoop.apache.org
Delivered-To: mailing list hdfs-issues@hadoop.apache.org
Received: (qmail 79575 invoked by uid 99); 21 Jan 2014 18:06:24 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 21 Jan 2014 18:06:24 +0000
Date: Tue, 21 Jan 2014 18:06:24 +0000 (UTC)
From: "Suresh Srinivas (JIRA)" <jira@apache.org>
To: hdfs-issues@hadoop.apache.org
Message-ID: <JIRA.12686884.1388532040456.40831.1390327584701@arcas>
In-Reply-To: <JIRA.12686884.1388532040456@arcas>
References: <JIRA.12686884.1388532040456@arcas>
Subject: [jira] [Commented] (HDFS-5709) Improve upgrade with existing files
 and directories named ".snapshot"
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/HDFS-5709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13877671#comment-13877671 ] 

Suresh Srinivas commented on HDFS-5709:
---------------------------------------

bq.could you lay out your alternative proposal for a conf option? I could rename the conf and make it so it takes a delimited set of kv pairs, e.g. ".snapshot=.user-snapshot,.some-new-reserved=.renamed-new-reserved", but I felt that was kind of ugly. I wanted full sub rather than prefix here for flexibility.
This is not what I have in mind. /<path>/<reserved_file> could be renamed to /<path>/<reserved_file>+<configured_rename_suffix>. The user finds all the renamed files (from the log) and renames them once the system comes up, if necessary. In fact coming to think of it, I think the suffix should probably be not configurable either. The system can choose some convention where rename suffix could be - ".<layout_version>.reserved_renamed_after_ugprade". The user must run -upgrade with a new option that allows renaming of reserved file.

The advantages of this are:
# Post upgrade at any time (until user renames the file), all the renamed files can be found
# The probability of conflict of file names during upgrade is lower
# Unnecessary configuration changes are avoided

bq. With just a prefix mutation, I could easily imagine having to run some operation after the NN had started up to then find all of the renamed paths and again rename them to some other name with the prefix removed. That's a pain that we shouldn't put our users through.
I do not understand what the pain is. It is just renaming the files. Also what if a user wants to rename .snapshot in one directory to x and .snapshot in another directory to y, based on the context of how the file is being used.

bq.  Empirically we rarely add reserved names (only two in the lifetime of HDFS so far) and I don't anticipate adding many more
If we looked at this same question last year, we would have said we will never have reserved names in HDFS at all.  I think there are some that I have plans of adding. I want to add some directories in the future for storing file system specific information. This is the way I envision moving fsimage and possibly finalized editlog segments into HDFS itself.

bq. ... but rather to do as Andrew suggests and add a conf option per reserved name, and add the code necessary for each one as it comes up....
I think adding one configuration for each reserved name does not seem right. In fact, if possible we should avoid configuration changes at all for renames that are required *one time* during upgrade to a version that adds reserved file names.




> Improve upgrade with existing files and directories named ".snapshot"
> ---------------------------------------------------------------------
>
>                 Key: HDFS-5709
>                 URL: https://issues.apache.org/jira/browse/HDFS-5709
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 3.0.0, 2.2.0
>            Reporter: Andrew Wang
>            Assignee: Andrew Wang
>              Labels: snapshots, upgrade
>         Attachments: hdfs-5709-1.patch, hdfs-5709-2.patch, hdfs-5709-3.patch, hdfs-5709-4.patch, hdfs-5709-5.patch
>
>
> Right now in trunk, upgrade fails messily if the old fsimage or edits refer to a directory named ".snapshot". We should at least print a better error message (which I believe was the original intention in HDFS-4666), and [~atm] proposed automatically renaming these files and directories.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)