hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (HADOOP-1286) Distributed cluster upgrade
Date Tue, 03 Jul 2007 00:43:04 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-1286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12500275
] 

Konstantin Shvachko edited comment on HADOOP-1286 at 7/2/07 5:41 PM:
---------------------------------------------------------------------

Some more ideas on the distributed upgrade.

- Upgradeable interface
    All upgrade objects should implement this interface. It includes the following methods
(see attached file for details):
    getName() returns component name that is to be upgraded by this object
    getDescription() description of the object for displaying
    versionFrom() which version does the object upgrade
    versionTo() which version does the object upgrade to
    static newInstance() creates new instance of this object
    getUpgradeStatus() - returns a value in [0,100]
    UpgradeCommand processUpgradeCommand(UpgradeCommand comm)

- Upgrade object is a class that implements Upgradeable interface
   Naming conventions for the upgrade objects
    Upgrade_<version>_<component name>
    E.g. Upgrade_5_NameNode

- UpgradeObjectCollection
A class that statically lists versions and their upgrade objects.
One can get a sequence of upgrade objects given the component name and two versions.
    <componentName, verStart, verEnd>  ==>  <UpgradeObject(verStart), ...., UpgradeObject(verEnd)>

- UpgradeManager class should detect which upgrade objects are required to perform the upgrade,
and applies the upgrades in the correct sequence.

- UpgradeCommand
ClientProtocol and DatanodeProtocol should be extended with one extra call
UpgradeCommand processUpgradeCommand(UpgradeCommand comm)
This command calls a respective procedure on current upgrade object.
The object itself should recognize the actions to be performed and perform them.

The attached file contains the Upgradeable interface and the UpgradeCommand base class, 
which in real life should of course be in separate java files. 



 was:
Some more ideas on the distributed upgrade.

- Upgradeable interface
    All upgrade objects should implement this interface. It includes the following methods
(see attached file for details):
    getName() returns component name that is to be upgraded by this object
    getDescription() description of the object for displaying
    versionFrom() which version does the object upgrade
    versionTo() which version does the object upgrade to
    static newInstance() creates new instance of this object
    getUpgradeStatus() - returns a value in [0,100]
    UpgradeCommand processUpgradeCommand(UpgradeCommand comm)

- Upgrade object is a class that implements Upgradeable interface
   Naming conventions for the upgrade objects
    Upgrade<component name>From<version from>To<version to>
    E.g. UpgradeNameNodeFrom6To7

- UpgradeObjectCollection
A class that statically lists versions and their upgrade objects.
One can get a sequence of upgrade objects given the component name and two versions.
    <componentName, verStart, verEnd>  ==>  <UpgradeObject(verStart), ...., UpgradeObject(verEnd)>

- UpgradeManager class should detect which upgrade objects are required to perform the upgrade,
and applies the upgrades in the correct sequence.

- UpgradeCommand
ClientProtocol and DatanodeProtocol should be extended with one extra call
UpgradeCommand processUpgradeCommand(UpgradeCommand comm)
This command calls a respective procedure on current upgrade object.
The object itself should recognize the actions to be performed and perform them.

The attached file contains the Upgradeable interface and the UpgradeCommand base class, 
which in real life should of course be in separate java files. 


> Distributed cluster upgrade
> ---------------------------
>
>                 Key: HADOOP-1286
>                 URL: https://issues.apache.org/jira/browse/HADOOP-1286
>             Project: Hadoop
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.13.0
>            Reporter: Konstantin Shvachko
>            Assignee: Konstantin Shvachko
>             Fix For: 0.14.0
>
>         Attachments: DistUpgradeFramework3.patch, Upgradeable.java
>
>
> Some data layout changes in HDFS require more than just a version upgrade introduced
in HADOOP-702,
> because the cluster can function properly only when all components have upgraded, and
the components
> need to communicate to each other and exchange data before they can perform the upgrade.
> The CRC upgrade discussed in HADOOP-1134 is one of such examples. Future enhancements
like
> implementation of appends can change block meta-data and may require distributed upgrades.
> Distributed upgrade (DU) starts with a version upgrade (VU) so that at any time one could
rollback
> all changes and start over.
> When VU is finished the name-node enters safe mode and persistently records that the
DU have been started.
> It will also need to write a record when DU is finished. This is necessary to report
unfinished upgrades in case
> of failure or for monitoring.
> The actual upgrade code from version vO to vN should be implemented in a separate UpgradeObject
class,
> which implements interface Upgradeable.
> We create a new UpgradeObject for each pair of versions vO to vN that require a DU.
> We keep a (hard coded) table that determines which UpgradeObject(s) are applicable for
the version pairs.
> Something like:
> || Old version || New version || class names ||
> | vO1 | vN1 | NameUpgradeObject1, DataUpgradeObject1 |
> | vO2 | vN2 | NameUpgradeObject2, DataUpgradeObject2 |
> where vO1 < vN1 < vO2 < vN2 ...
> Now, if we need to upgrade from version version vX to version vY, we look for all pairs
<vOi, vNi>
> in the table such that vX < vOi < vNi < vY and perform corresponding DUs one
after another as they appear in the table.
> Each DU can and most probably should contain multiple UpgradeObjects.
> I'd define one object for the name-node and one for the data-nodes.
> The upgrade objects (in the same row) can communicate to each other either via existing
protocols or using
> temporary protocols defined exclusively for this particular upgrade objects.
> I envision that some DUs will need to use old  (vO) protocols to exchange the pre-upgrade
data,
> and new (vN) protocols to reoport the upgraded data.
> UpgradeObjects should be able to bypass safe mode restrictions, be able to +modify+ name-node
data.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message