hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron T. Myers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5709) Improve upgrade with existing files and directories named ".snapshot"
Date Tue, 21 Jan 2014 19:01:20 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13877737#comment-13877737

Aaron T. Myers commented on HDFS-5709:

bq. The user must run -upgrade with a new option that allows renaming of reserved file.

I don't think we should be introducing another startup option for this, mostly because I can't
imagine a user _not_ wanting to automatically rename files that use names that are now reserved.
The current process of having to downgrade the software, start the cluster, rename all the
files (not all of which may have been found during the attempted upgrade!) is so onerous that
we really should just do it for users.

bq. I do not understand what the pain is. It is just renaming the files. 

The pain is in the difficulty of automating this. A tool would have to either do the equivalent
of {{`find / -name ...'}} on the file system, which in a very large HDFS instance might be
untenable, or parse the NN startup logs which is an error-prone thing to try to do.

How about the following proposal, which hopefully appeases everyone:

* We do renames of reserved names by default, for the reasons stated above.
* By default we do as Suresh suggested and rename all reserved file names to "<original
file name>.<layout_version>.reserved_renamed_after_ugprade"
* Optionally, one can set a single configuration parameter as Andrew proposed ('e.g. ".snapshot=.user-snapshot,.some-new-reserved=.renamed-new-reserved"').
If on renaming a reserved file name it's in this configured list, we use that for the substitution.
If it's not present in this configured list, then we do the default as Suresh suggested.

> Improve upgrade with existing files and directories named ".snapshot"
> ---------------------------------------------------------------------
>                 Key: HDFS-5709
>                 URL: https://issues.apache.org/jira/browse/HDFS-5709
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: namenode
>    Affects Versions: 3.0.0, 2.2.0
>            Reporter: Andrew Wang
>            Assignee: Andrew Wang
>              Labels: snapshots, upgrade
>         Attachments: hdfs-5709-1.patch, hdfs-5709-2.patch, hdfs-5709-3.patch, hdfs-5709-4.patch,
> Right now in trunk, upgrade fails messily if the old fsimage or edits refer to a directory
named ".snapshot". We should at least print a better error message (which I believe was the
original intention in HDFS-4666), and [~atm] proposed automatically renaming these files and

This message was sent by Atlassian JIRA

View raw message