hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Aaron T. Myers (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5223) Allow edit log/fsimage format changes without changing layout version
Date Wed, 18 Sep 2013 18:29:57 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13771059#comment-13771059

Aaron T. Myers commented on HDFS-5223:

I was chatting about this informally with [~tlipcon] a day or two ago, and we came up with
the following two alternative implementations:

# Introduce a new separate "NN metadata version" number which is decoupled from the existing
"layout version". We will allow the NN to start up if its NN metadata version number is higher
than what's in the fsimage/edit log headers without requiring the '-upgrade' flag. From now
on the addition of new edit log opcodes would increment the NN metadata version, and we would
require that changes made to the format of existing fsimage/edit log entries be done in a
backward compatible fashion. We would freeze the existing "layout version" number and from
now on only increment this in the case of more fundamental NN metadata version changes.
# Introduce a set of NN metadata format feature flags which can be enabled or disabled by
the admin at runtime. These feature flags could be enabled/disabled entirely independently,
so we would move away from a strictly-increasing NN metadata version number. The fsimage and
edit log header would be changed to enumerate which of these features were enabled. We will
allow the NN to start up only if its software supports the full set of features identified
in the fsimage/edit log headers.

I'd love to solicit others thoughts/feedback on these proposals, or suggest an alternative
if you have one.
> Allow edit log/fsimage format changes without changing layout version
> ---------------------------------------------------------------------
>                 Key: HDFS-5223
>                 URL: https://issues.apache.org/jira/browse/HDFS-5223
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 2.1.1-beta
>            Reporter: Aaron T. Myers
> Currently all HDFS on-disk formats are version by the single layout version. This means
that even for changes which might be backward compatible, like the addition of a new edit
log op code, we must go through the full `namenode -upgrade' process which requires coordination
with DNs, etc. HDFS should support a lighter weight alternative.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

View raw message