hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Radia (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDFS-8075) Revist layout version
Date Tue, 07 Apr 2015 17:34:12 GMT
Sanjay Radia created HDFS-8075:

             Summary: Revist layout version
                 Key: HDFS-8075
                 URL: https://issues.apache.org/jira/browse/HDFS-8075
             Project: Hadoop HDFS
          Issue Type: Bug
          Components: HDFS
    Affects Versions: 2.6.0
            Reporter: Sanjay Radia

* HDFS image layout was changed to use Protobufs to allow easier forward and backward compatibility.
* Hdfs has a layout version which is changed on each change (even if it an  optional protobuf
field was added).
* Hadoop supports two ways of going back during an upgrade:
**  downgrade: go back to old binary version but use existing image/edits so that newly created
files are not lost
** rollback: go back to "checkpoint" created before upgrade was started - hence newly created
files are lost.

Layout needs to be revisited if we want to support downgrade is some circumstances which we
dont today. Here are use cases:
* Some changes can support downgrade even though they was a change in layout since there is
not real data loss but only loss of new functionality. E.g. when we added ACLs one could have
downgraded - there is no data loss but you will lose the newly created ACLs. That is acceptable
for a user since one does not expect to retain the newly added ACLs in an old version.
* Some changes may lead to data-loss if the functionality was used. For example, the recent
truncate will cause data loss if the functionality was actually used. Now one can tell admins
NOT use such new such new features till the upgrade is finalized in which case one could potentially
support downgrade.
* A fairly fundamental change to layout where a downgrade is not possible but a rollback is.
Say we change the layout completely from protobuf to something else. Another example is when
HDFS moves to support partial namespace in memory - they is likely to be a fairly fundamental
change in layout.

This message was sent by Atlassian JIRA

View raw message