hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Radia (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-5223) Allow edit log/fsimage format changes without changing layout version
Date Tue, 07 Apr 2015 18:10:13 GMT

    [ https://issues.apache.org/jira/browse/HDFS-5223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14483654#comment-14483654

Sanjay Radia commented on HDFS-5223:

For the edits one could require that in order to downgrade you must do a "save-image" and
then delete the null edits-log. We would then limit our solution to the image. For the image
we could do the following
* Add a *second* layout version field (call it "compatible-layout-version") that indicates
which version can safely read the image without data-loss. A NN that starts up will compare
this field with its current layout version and then proceed as long as the edits is null.
** The ACL example (see Jira description) will state that the previous version  can safely
read the image without data loss. Of course newly created ACLs would be lost.
** Truncate example is tricky: one can safely downgrade if the truncate operation was not
used. We could add code to not allow such new features till finalize is done.  This is somewhat
analogous to what ext3 was trying to do with  its superblock feature flags (see Todd's comment
above); what I am proposing is slightly different since it limits such features till upgrade
is finalized while ext3's approach is more general in that you can downgrade at anytime as
long as you have used the feature.  Alternatively, we could simply not support downgrade for
such a feature and simply mark the compatible-layout-version accordingly.

> Allow edit log/fsimage format changes without changing layout version
> ---------------------------------------------------------------------
>                 Key: HDFS-5223
>                 URL: https://issues.apache.org/jira/browse/HDFS-5223
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 2.1.1-beta
>            Reporter: Aaron T. Myers
>            Assignee: Colin Patrick McCabe
>         Attachments: HDFS-5223.004.patch
> Currently all HDFS on-disk formats are version by the single layout version. This means
that even for changes which might be backward compatible, like the addition of a new edit
log op code, we must go through the full `namenode -upgrade' process which requires coordination
with DNs, etc. HDFS should support a lighter weight alternative.
> Copied description from HDFS-8075 which is a duplicate and now closed.
> Background
> * HDFS image layout was changed to use Protobufs to allow easier forward and backward
> * Hdfs has a layout version which is changed on each change (even if it an  optional
protobuf field was added).
> * Hadoop supports two ways of going back during an upgrade:
> **  downgrade: go back to old binary version but use existing image/edits so that newly
created files are not lost
> ** rollback: go back to "checkpoint" created before upgrade was started - hence newly
created files are lost.
> Layout needs to be revisited if we want to support downgrade is some circumstances which
we dont today. Here are use cases:
> * Some changes can support downgrade even though they was a change in layout since there
is not real data loss but only loss of new functionality. E.g. when we added ACLs one could
have downgraded - there is no data loss but you will lose the newly created ACLs. That is
acceptable for a user since one does not expect to retain the newly added ACLs in an old version.
> * Some changes may lead to data-loss if the functionality was used. For example, the
recent truncate will cause data loss if the functionality was actually used. Now one can tell
admins NOT use such new such new features till the upgrade is finalized in which case one
could potentially support downgrade.
> * A fairly fundamental change to layout where a downgrade is not possible but a rollback
is. Say we change the layout completely from protobuf to something else. Another example is
when HDFS moves to support partial namespace in memory - they is likely to be a fairly fundamental
change in layout.

This message was sent by Atlassian JIRA

View raw message