hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dinesh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-9096) Issue in Rollback (after rolling upgrade) from hadoop 2.7.1 to 2.4.0
Date Wed, 19 Oct 2016 08:54:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15588098#comment-15588098
] 

Dinesh commented on HDFS-9096:
------------------------------

Hi [~kihwal],

*With journal nodes and zookeeper instances running, changed value from -63 to -57. Rollback
failed with the following exception*
|16/10/19 13:43:19 WARN namenode.FSNamesystem: Encountered exception loading fsimage
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage
directory \Metadata\data\dfs\namenode. Reported: -57. Expecting = -63.
        at org.apache.hadoop.hdfs.server.namenode.NNStorage.readProperties(NNStorage.java:633)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:339)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:215)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:584)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:644)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:811)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:795)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)|

*Changed version back to -63 and executed -rollingUpgrade rollback command. Now faced the
following exception*
|16/10/19 13:43:43 ERROR namenode.FSImage: Failed to load image from FSImageFile(file=\Metadata\data\dfs\namenode\current\fsimage_rollback_0000000000000004951,
cpktTxId=0000000000000004951)
java.io.IOException: Image version -57 is not equal to the software version -63
        at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal
(FSImageFormatProtobuf.java:196)
        at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:179)
        at org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:226)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:957)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:941)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:740)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:676)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:584)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:644)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:811)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:795)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/10/19 13:43:43 WARN namenode.FSNamesystem: Encountered exception loading fsimage
java.io.IOException: Failed to load an FSImage file!
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:687)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:584)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:644)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:811)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:795)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/10/19 13:43:43 INFO mortbay.log: Stopped|

> Issue in Rollback (after rolling upgrade) from hadoop 2.7.1 to 2.4.0
> --------------------------------------------------------------------
>
>                 Key: HDFS-9096
>                 URL: https://issues.apache.org/jira/browse/HDFS-9096
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: rolling upgrades
>    Affects Versions: 2.4.0
>            Reporter: Harpreet Kaur
>
> I tried to do rolling upgrade from hadoop 2.4.0 to hadoop 2.7.1. As per http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#dfsadmin_-rollingUpgrade
one can rollback to previous release provided the finalise step is not done. I upgraded the
setup but didnot finalise the upgrade and tried to rollback HDFS to 2.4.0
> I tried the following steps
>   1.  Shutdown all NNs and DNs.
>   2.  Restore the pre-upgrade release in all machines.
>   3.  Start NN1 as Active with the "-rollingUpgrade rollback<http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#namenode_-rollingUpgrade>"
option.
> I am getting the following error after 3rd step
> 15/09/01 17:53:35 INFO namenode.AclConfigFlag: ACLs enabled? false
> 15/09/01 17:53:35 INFO common.Storage: Lock on <<NameNode dir>>/in_use.lock
acquired by nodename 12152@VM-2
> 15/09/01 17:53:35 WARN namenode.FSNamesystem: Encountered exception loading fsimage
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of
storage directory /data/yarn/namenode. Reported: -63. Expecting = -56.
> at org.apache.hadoop.hdfs.server.common.StorageInfo.setLayoutVersion(StorageInfo.java:178)
> at org.apache.hadoop.hdfs.server.common.StorageInfo.setFieldsFromProperties(StorageInfo.java:131)
> at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:608)
> at org.apache.hadoop.hdfs.server.common.StorageInfo.readProperties(StorageInfo.java:228)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:309)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:882)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:639)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:455)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:511)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:670)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:655)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1304)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1370)
> 15/09/01 17:53:35 INFO mortbay.log: Stopped SelectChannelConnector@0.0.0.0:50070
> 15/09/01 17:53:35 INFO impl.MetricsSystemImpl: Stopping NameNode metrics system...
> 15/09/01 17:53:35 INFO impl.MetricsSystemImpl: NameNode metrics system stopped.
> 15/09/01 17:53:35 INFO impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
> 15/09/01 17:53:35 FATAL namenode.NameNode: Exception in namenode join
> From rolling upgrade documentation it can be inferred that rolling upgrade is supported
Hadoop 2.4.0 onwards but rollingUpgrade rollback to Hadoop 2.4.0 seems to be broken in Hadoop
2.4.0. It throws above mentioned error.
> Are there any other steps to perform rollback (from rolling upgrade) or is it not supported
to rollback to Hadoop 2.4.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message