hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Dinesh (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-9096) Issue in Rollback (after rolling upgrade) from hadoop 2.7.1 to 2.4.0
Date Tue, 18 Oct 2016 13:35:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-9096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15475894#comment-15475894
] 

Dinesh edited comment on HDFS-9096 at 10/18/16 1:35 PM:
--------------------------------------------------------

Facing same issue when I Rollback (after rolling upgrade) from Hadoop 2.7.2 to 2.5.2; Could
any one please tell, is this a known bug?

Based on below logs, please suggest can we consider this issue as a new bug,

My Name node log details:

|C:\SDK\Hadoop\bin>hdfs namenode -rollingUpgrade rollback
16/10/18 18:58:25 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = myserver/192.168.10.1
STARTUP_MSG:   args = [-rollingUpgrade, rollback]
STARTUP_MSG:   version = 2.5.2
STARTUP_MSG:   classpath = ...
STARTUP_MSG:   build = Unknown -r Unknown; compiled by 'Dinesh' on 2016-01-14T11:05Z
STARTUP_MSG:   java = 1.7.0_51
************************************************************/
16/10/18 18:58:25 INFO namenode.NameNode: createNameNode [-rollingUpgrade, rollback]
16/10/18 18:58:25 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
16/10/18 18:58:25 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
16/10/18 18:58:25 INFO impl.MetricsSystemImpl: NameNode metrics system started
16/10/18 18:58:25 INFO namenode.NameNode: fs.defaultFS is hdfs://hacluster
16/10/18 18:58:25 INFO namenode.NameNode: Clients are to use hacluster to access this namenode/service.
16/10/18 18:58:26 INFO hdfs.DFSUtil: Starting web server as: ${dfs.web.authentication.kerberos.principal}
16/10/18 18:58:26 INFO hdfs.DFSUtil: Starting Web-server for hdfs at: http://myserver.root.Dinesh.lan:50070
16/10/18 18:58:26 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log)
via org.mortbay.log.Slf4jLog
16/10/18 18:58:26 INFO http.HttpRequestLog: Http request log for http.requests.namenode is
not defined
16/10/18 18:58:26 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
16/10/18 18:58:26 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
to context hdfs
16/10/18 18:58:26 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
to context static
16/10/18 18:58:26 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
to context logs
16/10/18 18:58:26 INFO http.HttpServer2: Added filter 'org.apache.hadoop.hdfs.web.AuthFilter'
(class=org.apache.hadoop.hdfs.web.AuthFilter)
16/10/18 18:58:26 INFO http.HttpServer2: addJerseyResourcePackage: packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
pathSpec=/webhdfs/v1/*
16/10/18 18:58:26 INFO http.HttpServer2: Jetty bound to port 50070
16/10/18 18:58:26 INFO mortbay.log: jetty-6.1.26
16/10/18 18:58:26 WARN server.AuthenticationFilter: 'signature.secret' configuration not set,
using a random value as secret
16/10/18 18:58:26 INFO mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@myserver.root.Dinesh.lan:50070
16/10/18 18:58:26 WARN namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir)
configured. Beware of data loss due to lack of redundant storage directories!
16/10/18 18:58:26 INFO namenode.FSNamesystem: fsLock is fair:true
16/10/18 18:58:26 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
16/10/18 18:58:26 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
16/10/18 18:58:26 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec
is set to 000:00:00:00.000
16/10/18 18:58:26 INFO blockmanagement.BlockManager: The block deletion will start around
2016 Oct 18 18:58:26
16/10/18 18:58:26 INFO util.GSet: Computing capacity for map BlocksMap
16/10/18 18:58:26 INFO util.GSet: VM type       = 64-bit
16/10/18 18:58:26 INFO util.GSet: 2.0% max memory 910.5 MB = 18.2 MB
16/10/18 18:58:26 INFO util.GSet: capacity      = 2^21 = 2097152 entries
16/10/18 18:58:26 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
16/10/18 18:58:26 INFO blockmanagement.BlockManager: defaultReplication         = 3
16/10/18 18:58:26 INFO blockmanagement.BlockManager: maxReplication             = 512
16/10/18 18:58:26 INFO blockmanagement.BlockManager: minReplication             = 1
16/10/18 18:58:26 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
16/10/18 18:58:26 INFO blockmanagement.BlockManager: shouldCheckForEnoughRacks  = false
16/10/18 18:58:26 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
16/10/18 18:58:26 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
16/10/18 18:58:26 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
16/10/18 18:58:26 INFO namenode.FSNamesystem: fsOwner             = SYSTEM (auth:SIMPLE)
16/10/18 18:58:26 INFO namenode.FSNamesystem: supergroup          = Users
16/10/18 18:58:26 INFO namenode.FSNamesystem: isPermissionEnabled = false
16/10/18 18:58:26 INFO namenode.FSNamesystem: Determined nameservice ID: hacluster
16/10/18 18:58:26 INFO namenode.FSNamesystem: HA Enabled: true
16/10/18 18:58:26 INFO namenode.FSNamesystem: Append Enabled: true
16/10/18 18:58:26 INFO util.GSet: Computing capacity for map INodeMap
16/10/18 18:58:26 INFO util.GSet: VM type       = 64-bit
16/10/18 18:58:26 INFO util.GSet: 1.0% max memory 910.5 MB = 9.1 MB
16/10/18 18:58:26 INFO util.GSet: capacity      = 2^20 = 1048576 entries
16/10/18 18:58:26 INFO namenode.NameNode: Caching file names occuring more than 10 times
16/10/18 18:58:26 INFO util.GSet: Computing capacity for map cachedBlocks
16/10/18 18:58:26 INFO util.GSet: VM type       = 64-bit
16/10/18 18:58:26 INFO util.GSet: 0.25% max memory 910.5 MB = 2.3 MB
16/10/18 18:58:26 INFO util.GSet: capacity      = 2^18 = 262144 entries
16/10/18 18:58:26 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
16/10/18 18:58:26 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
16/10/18 18:58:26 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
16/10/18 18:58:26 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
16/10/18 18:58:26 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and
retry cache entry expiry time is 600000 millis
16/10/18 18:58:26 INFO util.GSet: Computing capacity for map NameNodeRetryCache
16/10/18 18:58:26 INFO util.GSet: VM type       = 64-bit
16/10/18 18:58:26 INFO util.GSet: 0.029999999329447746% max memory 910.5 MB = 279.7 KB
16/10/18 18:58:26 INFO util.GSet: capacity      = 2^15 = 32768 entries
16/10/18 18:58:26 INFO namenode.NNConf: ACLs enabled? false
16/10/18 18:58:26 INFO namenode.NNConf: XAttrs enabled? true
16/10/18 18:58:26 INFO namenode.NNConf: Maximum size of an xattr: 16384
16/10/18 18:58:26 INFO common.Storage: Lock on \Metadata\data\dfs\namenode\in_use.lock acquired
by nodename 8804@myserver
16/10/18 18:58:26 WARN namenode.FSNamesystem: Encountered exception loading fsimage
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage
directory C:\Metadata\data\dfs\namenode. Reported: -63. Expecting = -57.
        at org.apache.hadoop.hdfs.server.common.StorageInfo.setLayoutVersion(StorageInfo.java:178)
        at org.apache.hadoop.hdfs.server.common.StorageInfo.setFieldsFromProperties(StorageInfo.java:131)
        at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:608)
        at org.apache.hadoop.hdfs.server.common.StorageInfo.readProperties(StorageInfo.java:228)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:323)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:955)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:700)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:529)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:585)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:751)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:735)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1407)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1473)
16/10/18 18:58:26 INFO mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@myserver.root.Dinesh.lan:50070
16/10/18 18:58:26 INFO impl.MetricsSystemImpl: Stopping NameNode metrics system...
16/10/18 18:58:26 INFO impl.MetricsSystemImpl: NameNode metrics system stopped.
16/10/18 18:58:26 INFO impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
16/10/18 18:58:26 FATAL namenode.NameNode: Exception in namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage
directory C:\Metadata\data\dfs\namenode. Reported: -63. Expecting = -57.
        at org.apache.hadoop.hdfs.server.common.StorageInfo.setLayoutVersion(StorageInfo.java:178)
        at org.apache.hadoop.hdfs.server.common.StorageInfo.setFieldsFromProperties(StorageInfo.java:131)
        at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:608)
        at org.apache.hadoop.hdfs.server.common.StorageInfo.readProperties(StorageInfo.java:228)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:323)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:955)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:700)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:529)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:585)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:751)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:735)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1407)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1473)
16/10/18 18:58:26 INFO util.ExitUtil: Exiting with status 1
16/10/18 18:58:26 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at myserver/192.168.10.1
*************************************************************|


was (Author: dineshkumarp):
Facing same issue when I Rollback (after rolling upgrade) from Hadoop 2.7.2 to 2.5.2; Could
any one please tell, is this a known bug?

Based on below logs, please suggest can we consider this issue as a new bug,

My Name node log details:

|C:\Hadoop\bin>hdfs namenode -rollingUpgrade rollback
16/10/18 15:54:47 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = myservernode/192.168.1.21
STARTUP_MSG:   args = [-rollingUpgrade, rollback]
STARTUP_MSG:   version = 2.7.2
STARTUP_MSG:   classpath = ....
16/10/18 15:54:47 INFO namenode.NameNode: createNameNode [-rollingUpgrade, rollback]
16/10/18 15:54:48 INFO impl.MetricsConfig: loaded properties from hadoop-metrics2.properties
16/10/18 15:54:48 INFO impl.MetricsSystemImpl: Scheduled snapshot period at 10 second(s).
16/10/18 15:54:48 INFO impl.MetricsSystemImpl: NameNode metrics system started
16/10/18 15:54:48 INFO namenode.NameNode: fs.defaultFS is hdfs://hacluster
16/10/18 15:54:48 INFO namenode.NameNode: Clients are to use hacluster to access this namenode/service.
16/10/18 15:54:48 INFO hdfs.DFSUtil: Starting Web-server for hdfs at: http://myservernode.root.server.lan:50070
16/10/18 15:54:48 INFO mortbay.log: Logging to org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log)
via org.mortbay.log.Slf4jLog
16/10/18 15:54:48 INFO server.AuthenticationFilter: Unable to initialize FileSignerSecretProvider,
falling back to use random secrets.
16/10/18 15:54:48 INFO http.HttpRequestLog: Http request log for http.requests.namenode is
not defined
16/10/18 15:54:48 INFO http.HttpServer2: Added global filter 'safety' (class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
16/10/18 15:54:48 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
to context hdfs
16/10/18 15:54:48 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
to context logs
16/10/18 15:54:48 INFO http.HttpServer2: Added filter static_user_filter (class=org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter)
to context static
16/10/18 15:54:48 INFO http.HttpServer2: Added filter 'org.apache.hadoop.hdfs.web.AuthFilter'
(class=org.apache.hadoop.hdfs.web.AuthFilter)
16/10/18 15:54:48 INFO http.HttpServer2: addJerseyResourcePackage: packageName=org.apache.hadoop.hdfs.server.namenode.web.resources;org.apache.hadoop.hdfs.web.resources,
pathSpec=/webhdfs/v1/*
16/10/18 15:54:48 INFO http.HttpServer2: Jetty bound to port 50070
16/10/18 15:54:48 INFO mortbay.log: jetty-6.1.26
16/10/18 15:54:48 INFO mortbay.log: Started HttpServer2$SelectChannelConnectorWithSafeStartup@myservernode.root.server.lan:50070
16/10/18 15:54:48 WARN namenode.FSNamesystem: Only one image storage directory (dfs.namenode.name.dir)
configured. Beware of data loss due to lack of redundant storage directories!
16/10/18 15:54:48 INFO namenode.FSNamesystem: No KeyProvider found.
16/10/18 15:54:48 INFO namenode.FSNamesystem: fsLock is fair:true
16/10/18 15:54:48 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
16/10/18 15:54:48 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
16/10/18 15:54:48 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec
is set to 000:00:00:00.000
16/10/18 15:54:48 INFO blockmanagement.BlockManager: The block deletion will start around
2016 Oct 18 15:54:48
16/10/18 15:54:48 INFO util.GSet: Computing capacity for map BlocksMap
16/10/18 15:54:48 INFO util.GSet: VM type       = 64-bit
16/10/18 15:54:48 INFO util.GSet: 2.0% max memory 455 MB = 9.1 MB
16/10/18 15:54:48 INFO util.GSet: capacity      = 2^20 = 1048576 entries
16/10/18 15:54:48 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
16/10/18 15:54:48 INFO blockmanagement.BlockManager: defaultReplication         = 3
16/10/18 15:54:48 INFO blockmanagement.BlockManager: maxReplication             = 512
16/10/18 15:54:48 INFO blockmanagement.BlockManager: minReplication             = 1
16/10/18 15:54:48 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
16/10/18 15:54:48 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
16/10/18 15:54:48 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
16/10/18 15:54:48 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
16/10/18 15:54:48 INFO namenode.FSNamesystem: fsOwner             = SYSTEM (auth:SIMPLE)
16/10/18 15:54:48 INFO namenode.FSNamesystem: supergroup          = supergroup
16/10/18 15:54:48 INFO namenode.FSNamesystem: isPermissionEnabled = false
16/10/18 15:54:48 INFO namenode.FSNamesystem: Determined nameservice ID: hacluster
16/10/18 15:54:48 INFO namenode.FSNamesystem: HA Enabled: true
16/10/18 15:54:48 INFO namenode.FSNamesystem: Append Enabled: true
16/10/18 15:54:48 INFO util.GSet: Computing capacity for map INodeMap
16/10/18 15:54:48 INFO util.GSet: VM type       = 64-bit
16/10/18 15:54:48 INFO util.GSet: 1.0% max memory 455 MB = 4.5 MB
16/10/18 15:54:48 INFO util.GSet: capacity      = 2^19 = 524288 entries
16/10/18 15:54:48 INFO namenode.FSDirectory: ACLs enabled? false
16/10/18 15:54:48 INFO namenode.FSDirectory: XAttrs enabled? true
16/10/18 15:54:48 INFO namenode.FSDirectory: Maximum size of an xattr: 16384
16/10/18 15:54:48 INFO namenode.NameNode: Caching file names occuring more than 10 times
16/10/18 15:54:48 INFO util.GSet: Computing capacity for map cachedBlocks
16/10/18 15:54:48 INFO util.GSet: VM type       = 64-bit
16/10/18 15:54:48 INFO util.GSet: 0.25% max memory 455 MB = 1.1 MB
16/10/18 15:54:48 INFO util.GSet: capacity      = 2^17 = 131072 entries
16/10/18 15:54:48 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
16/10/18 15:54:48 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
16/10/18 15:54:48 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
16/10/18 15:54:48 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets
= 10
16/10/18 15:54:48 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
16/10/18 15:54:48 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes =
1,5,25
16/10/18 15:54:48 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
16/10/18 15:54:48 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and
retry cache entry expiry time is 600000 millis
16/10/18 15:54:48 INFO util.GSet: Computing capacity for map NameNodeRetryCache
16/10/18 15:54:48 INFO util.GSet: VM type       = 64-bit
16/10/18 15:54:48 INFO util.GSet: 0.029999999329447746% max memory 455 MB = 139.8 KB
16/10/18 15:54:48 INFO util.GSet: capacity      = 2^14 = 16384 entries
16/10/18 15:54:48 INFO common.Storage: Lock on \Metadata\data\dfs\namenode\in_use.lock acquired
by nodename 13228@myservernode
16/10/18 15:54:49 INFO client.QuorumJournalManager: Starting recovery process for unclosed
journal segments...
16/10/18 15:54:49 INFO client.QuorumJournalManager: Successfully started new epoch 6
16/10/18 15:54:49 INFO client.QuorumJournalManager: Beginning recovery of unclosed segment
starting at txid 5879
16/10/18 15:54:49 INFO client.QuorumJournalManager: Recovery prepare phase complete. Responses:
192.168.1.22:8485: segmentState { startTxId: 5879 endTxId: 5942 isInProgress: true } lastWriterEpoch:
5 lastCommittedTxId: 5941
192.168.1.23:8485: segmentState { startTxId: 5879 endTxId: 5942 isInProgress: true } lastWriterEpoch:
5 lastCommittedTxId: 5941
16/10/18 15:54:49 INFO client.QuorumJournalManager: Using longest log: 172.16.100.93:8485=segmentState
{
  startTxId: 5879
  endTxId: 5942
  isInProgress: true
}
lastWriterEpoch: 5
lastCommittedTxId: 5941
16/10/18 15:54:49 INFO namenode.FileJournalManager: Recovering unfinalized segments in \Metadata\data\dfs\namenode\current
16/10/18 15:54:49 INFO namenode.FileJournalManager: Finalizing edits file \Metadata\data\dfs\namenode\current\edits_inprogress_0000000000000005718
-> \Metadata\data\dfs\namenode\current\edits_0000000000000005718-0000000000000005718
16/10/18 15:54:50 ERROR namenode.FSImage: Failed to load image from FSImageFile(file=\Metadata\data\dfs\namenode\current\fsimage_rollback_0000000000000005713,
cpktTxId=0000000000000005713)
java.io.IOException: Image version -57 is not equal to the software version -63
        at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.loadInternal(FSImageFormatProtobuf.java:196)
        at org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Loader.load(FSImageFormatProtobuf.java:179)
        at org.apache.hadoop.hdfs.server.namenode.FSImageFormat$LoaderDelegator.load(FSImageFormat.java:226)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:957)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:941)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImageFile(FSImage.java:740)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:676)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:584)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:644)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:811)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:795)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/10/18 15:54:50 WARN namenode.FSNamesystem: Encountered exception loading fsimage
java.io.IOException: Failed to load an FSImage file!
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:687)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:584)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:644)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:811)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:795)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/10/18 15:54:50 INFO mortbay.log: Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@myservernode.root.server.lan:50070
16/10/18 15:54:50 INFO impl.MetricsSystemImpl: Stopping NameNode metrics system...
16/10/18 15:54:50 INFO impl.MetricsSystemImpl: NameNode metrics system stopped.
16/10/18 15:54:50 INFO impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
16/10/18 15:54:50 ERROR namenode.NameNode: Failed to start namenode.
java.io.IOException: Failed to load an FSImage file!
        at org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:687)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:294)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:975)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:681)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:584)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:644)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:811)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:795)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1488)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1554)
16/10/18 15:54:50 INFO util.ExitUtil: Exiting with status 1
16/10/18 15:54:50 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at myservernode/192.168.1.21
************************************************************/|

> Issue in Rollback (after rolling upgrade) from hadoop 2.7.1 to 2.4.0
> --------------------------------------------------------------------
>
>                 Key: HDFS-9096
>                 URL: https://issues.apache.org/jira/browse/HDFS-9096
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: rolling upgrades
>    Affects Versions: 2.4.0
>            Reporter: Harpreet Kaur
>
> I tried to do rolling upgrade from hadoop 2.4.0 to hadoop 2.7.1. As per http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#dfsadmin_-rollingUpgrade
one can rollback to previous release provided the finalise step is not done. I upgraded the
setup but didnot finalise the upgrade and tried to rollback HDFS to 2.4.0
> I tried the following steps
>   1.  Shutdown all NNs and DNs.
>   2.  Restore the pre-upgrade release in all machines.
>   3.  Start NN1 as Active with the "-rollingUpgrade rollback<http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsRollingUpgrade.html#namenode_-rollingUpgrade>"
option.
> I am getting the following error after 3rd step
> 15/09/01 17:53:35 INFO namenode.AclConfigFlag: ACLs enabled? false
> 15/09/01 17:53:35 INFO common.Storage: Lock on <<NameNode dir>>/in_use.lock
acquired by nodename 12152@VM-2
> 15/09/01 17:53:35 WARN namenode.FSNamesystem: Encountered exception loading fsimage
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of
storage directory /data/yarn/namenode. Reported: -63. Expecting = -56.
> at org.apache.hadoop.hdfs.server.common.StorageInfo.setLayoutVersion(StorageInfo.java:178)
> at org.apache.hadoop.hdfs.server.common.StorageInfo.setFieldsFromProperties(StorageInfo.java:131)
> at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:608)
> at org.apache.hadoop.hdfs.server.common.StorageInfo.readProperties(StorageInfo.java:228)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:309)
> at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:882)
> at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:639)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:455)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:511)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:670)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:655)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1304)
> at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1370)
> 15/09/01 17:53:35 INFO mortbay.log: Stopped SelectChannelConnector@0.0.0.0:50070
> 15/09/01 17:53:35 INFO impl.MetricsSystemImpl: Stopping NameNode metrics system...
> 15/09/01 17:53:35 INFO impl.MetricsSystemImpl: NameNode metrics system stopped.
> 15/09/01 17:53:35 INFO impl.MetricsSystemImpl: NameNode metrics system shutdown complete.
> 15/09/01 17:53:35 FATAL namenode.NameNode: Exception in namenode join
> From rolling upgrade documentation it can be inferred that rolling upgrade is supported
Hadoop 2.4.0 onwards but rollingUpgrade rollback to Hadoop 2.4.0 seems to be broken in Hadoop
2.4.0. It throws above mentioned error.
> Are there any other steps to perform rollback (from rolling upgrade) or is it not supported
to rollback to Hadoop 2.4.0.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message