hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5223) Allow edit log/fsimage format changes without changing layout version
Date Wed, 15 Jan 2014 21:39:22 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13872634#comment-13872634

Todd Lipcon commented on HDFS-5223:

Just skimmed really quick. I noticed that this doesn't separate flags into "compatible" ones
and "incompatible ones" per the discussion above. Implementations like ext3 and nilfs2 do
this so that it's possible to introduce features and label them as backward-compatible or
at least readonly-back-compatible. 

As an example of a readonly-back-compatible flag, consider the case of adding new ops like
"Add cache pool" and "remove cache pool". An old NN could easily start up and simply ignore
these opcodes that it doesn't understand (once we have protobuf-ified). Another example would
be adding a new field to the inode structure such as "preferred storage class". An old NN
could simply ignore the new fields in read-only mode, or drop them relatively safely in a
downgrade scenario. On the other hand, a feature such as compression, or adding OP_ADD_BLOCK
instead of OP_UPDATE_BLOCK would not be ro-compatible since an old NN wouldn't be able to
reconstruct the user data.

I think it would be short-sighted of us to not include a similar functionality in our flags,
even if this initial patch doesn't handle the two types of flags differently.

> Allow edit log/fsimage format changes without changing layout version
> ---------------------------------------------------------------------
>                 Key: HDFS-5223
>                 URL: https://issues.apache.org/jira/browse/HDFS-5223
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 2.1.1-beta
>            Reporter: Aaron T. Myers
>            Assignee: Colin Patrick McCabe
>         Attachments: HDFS-5223.004.patch
> Currently all HDFS on-disk formats are version by the single layout version. This means
that even for changes which might be backward compatible, like the addition of a new edit
log op code, we must go through the full `namenode -upgrade' process which requires coordination
with DNs, etc. HDFS should support a lighter weight alternative.

This message was sent by Atlassian JIRA

View raw message