hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Nirmal Kumar <nirmal.ku...@impetus.co.in>
Subject RE: Any reference for upgrade hadoop from 1.x to 2.2
Date Wed, 04 Dec 2013 14:11:08 GMT
Thanks Adam,

I am upgrading from *Apache Hadoop-1.2.0* to *Apache Hadoop-2.0.6-alpha*

I am getting the same exception when using the command: ./hadoop-daemon.sh start namenode
-upgrade

2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
            = cloud (auth:SIMPLE)
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
         = supergroup
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled
= true
2013-12-05 00:56:42,312 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
false
2013-12-05 00:56:42,317 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
true
2013-12-05 00:56:42,784 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file
names occuring more than 10 times
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct
= 0.9990000128746033
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes
= 0
2013-12-05 00:56:42,789 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension
    = 30000
2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock
acquired by nodename 31742@Impetus-942.impetus.co.in
2013-12-05 00:56:42,911 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system stopped.
2013-12-05 00:56:42,912 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system shutdown complete.
2013-12-05 00:56:42,913 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage
directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
        at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
        at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
        at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
        at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-05 00:56:42,922 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106
************************************************************/

I also referred https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
Upgrading from older release to 0.23 and configuring federation
Older releases supported a single Namenode. Here are the steps enable federation:
Step 1: Upgrade the cluster to newer release. During upgrade you can provide a ClusterID as
follows:
> $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR  -upgrade -clusterId
<cluster_ID>
If ClusterID is not provided, it is auto generated.
But Getting:
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
 -upgrade -clusterId testclusterid1111
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
 -upgrade -clusterId
Error: Could not find or load main class start
[cloud@Impetus-942 hadoop-2.0.6-alpha]$

I have the following environment variables set :

YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin

Regards,
-Nirmal

From: Adam Kawa [mailto:kawa.adam@gmail.com]
Sent: Tuesday, December 03, 2013 11:58 PM
To: user@hadoop.apache.org
Cc: rdyer@iastate.edu
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2

@Nirmal,

And later, you need to make a decision to finalize the upgrade or rollback.

2013/12/3 Adam Kawa <kawa.adam@gmail.com<mailto:kawa.adam@gmail.com>>
@Nirmal,

You need to run NameNode with upgrade option e.g.
$ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade

2013/12/3 Nirmal Kumar <nirmal.kumar@impetus.co.in<mailto:nirmal.kumar@impetus.co.in>>
Hi All,

I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single
node environment.

I did the following:

*         Installed Apache Hadoop-1.2.0

*         Ran word count sample MR jobs. The jobs executed successfully.

*         I stop all the services in Apache Hadoop-1.2.0 and then was able to start all services
again.

*         The previous submitted jobs are visible after the stop/start in the job tracker
url.

Next I installed Apache Hadoop-2.0.6-alpha alongside.
I used the SAME data directory locations that were in Apache Hadoop-1.2.0 in the configuration
files namely:
core-site.xml
----------------
$hadoop.tmp.dir                                              /home/cloud/hadoop_migration/hadoop-data/tempdir

hdfs-site.xml
-----------------
$dfs.data.dir                                                      /home/cloud/hadoop_migration/hadoop-data/data
$dfs.name.dir                                                    /home/cloud/hadoop_migration/hadoop-data/name

I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha installation I am getting
the error:

2013-12-03 18:28:23,941 INFO org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties
from hadoop-metrics2.properties
2013-12-03 18:28:24,080 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled
snapshot period at 10 second(s).
2013-12-03 18:28:24,081 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system started
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
image storage directory (dfs.namenode.name.dir) configured. Beware of dataloss due to lack
of redundant storage directories!
2013-12-03 18:28:24,576 WARN org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one
namespace edits storage directory (dfs.namenode.edits.dir) configured. Beware of dataloss
due to lack of redundant storage directories!
2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader: Refreshing hosts (include/exclude)
list
2013-12-03 18:28:24,749 INFO org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
dfs.block.invalidate.limit=1000
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: dfs.block.access.token.enable=false
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: defaultReplication
        = 1
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplication
            = 512
2013-12-03 18:28:24,762 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: minReplication
            = 1
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: maxReplicationStreams
     = 2
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: shouldCheckForEnoughRacks
 = false
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: replicationRecheckInterval
= 3000
2013-12-03 18:28:24,763 INFO org.apache.hadoop.hdfs.server.blockmanagement.BlockManager: encryptDataTransfer
       = false
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner
            = cloud (auth:SIMPLE)
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup
         = supergroup
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled
= true
2013-12-03 18:28:24,771 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled:
false
2013-12-03 18:28:24,776 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled:
true
2013-12-03 18:28:25,230 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file
names occuring more than 10 times
2013-12-03 18:28:25,243 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct
= 0.9990000128746033
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes
= 0
2013-12-03 18:28:25,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem: dfs.namenode.safemode.extension
    = 30000
2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage: Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock
acquired by nodename 21371@Impetus-942.impetus.co.in<mailto:21371@Impetus-942.impetus.co.in>
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
metrics system...
2013-12-03 18:28:25,462 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system stopped.
2013-12-03 18:28:25,473 INFO org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics
system shutdown complete.
2013-12-03 18:28:25,474 FATAL org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in
namenode join
org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected version of storage
directory /home/cloud/hadoop_migration/hadoop-data/name. Reported: -41. Expecting = -40.
        at org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
        at org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
        at org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
        at org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
        at org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
        at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
        at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with status 1
2013-12-03 18:28:25,481 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at Impetus-942.impetus.co.in/192.168.41.106<http://Impetus-942.impetus.co.in/192.168.41.106>
************************************************************/

Independently both the installations(Apache Hadoop-1.2.0 and Apache Hadoop-2.0.6-alpha) are
working for me. I am able to run the MR jobs on both the installations independently though.
But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha.

Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha?

Thanks,
-Nirmal

From: Nirmal Kumar
Sent: Wednesday, November 27, 2013 2:56 PM
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>; rdyer@iastate.edu<mailto:rdyer@iastate.edu>
Subject: RE: Any reference for upgrade hadoop from 1.x to 2.2

Hello Sandy,

The post was useful and gave an insight of the migration.

I am doing a test migration from Apache Hadoop-1.2.0 to Apache Hadoop-2.0.6-alpha on a single
node environment.
I am having the Apache Hadoop-1.2.0 up and running.

Can you please let me know the steps that one should follow for the migration?
I am thinking of doing something like:

*         Install Apache Hadoop-2.0.6-alpha alongside the existing Apache Hadoop-1.2.0

*         Use the same HDFS locations

*         Change the various required configuration files

*         Stop Apache Hadoop-1.2.0  and start Apache Hadoop-2.0.6-alpha

*         Verify all the services are running

*         Test via mapreduce (test MRv1 and MRv2 examples)

*         Check Web UI Console and verify the MRv1 and MRv2 jobs

These above steps needs to be performed on all the nodes in a cluster environment.

The translation table mapping old configuration to new would be definitely *very* useful.

Also the existing Hadoop ecosystem components needs to be considered:

*         Hive Scripts

*         Pig Scripts

*         Oozie Workflows
Their compatibility and version support would need to be checked.

Also thinking of any risks like Data Loss, others that one should keep in mind.

Also I found: http://strataconf.com/strata2014/public/schedule/detail/32247

Thanks,
-Nirmal

From: Robert Dyer [mailto:psybers@gmail.com]
Sent: Friday, November 22, 2013 9:08 PM
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2

Thanks Sandy! These seem helpful!

"MapReduce cluster configuration options have been split into YARN configuration options,
which go in yarn-site.xml; and MapReduce configuration options, which go in mapred-site.xml.
Many have been given new names to reflect the shift. ... We'll follow up with a full translation
table in a future post."

This type of translation table mapping old configuration to new would be *very* useful!

- Robert
On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sandy.ryza@cloudera.com<mailto:sandy.ryza@cloudera.com>>
wrote:
For MapReduce and YARN, we recently published a couple blog posts on migrating:
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/

hope that helps,
Sandy

On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <nirmal.kumar@impetus.co.in<mailto:nirmal.kumar@impetus.co.in>>
wrote:
Hi All,

I am also looking into migrating\upgrading from Apache Hadoop 1.x to Apache Hadoop 2.x.
I didn't find any doc\guide\blogs for the same.
Although there are guides\docs for the CDH and HDP migration\upgradation from Hadoop 1.x to
Hadoop 2.x
Would referring those be of some use?

I am looking for similar guides\docs for Apache Hadoop 1.x to Apache Hadoop 2.x.

I found something on slideshare though. Not sure how much useful that is going to be. I still
need to verify that.
http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade

Any suggestions\comments will be of great help.

Thanks,
-Nirmal

From: Jilal Oussama [mailto:jilal.oussama@gmail.com<mailto:jilal.oussama@gmail.com>]
Sent: Friday, November 08, 2013 9:13 PM
To: user@hadoop.apache.org<mailto:user@hadoop.apache.org>
Subject: Re: Any reference for upgrade hadoop from 1.x to 2.2

I am looking for the same thing if anyone can point us to a good direction please.
Thank you.

(Currently running Hadoop 1.2.1)

2013/11/1 YouPeng Yang <yypvsxf19870706@gmail.com<mailto:yypvsxf19870706@gmail.com>>
Hi users

   Are there any reference docs to introduce how to upgrade hadoop from 1.x to 2.2.

Regards


________________________________






NOTE: This message may contain information that is confidential, proprietary, privileged or
otherwise protected by law. The message is intended solely for the named addressee. If received
in error, please destroy and notify the sender. Any use of this email is prohibited when received
in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors, virus, interception
or interference.

________________________________






NOTE: This message may contain information that is confidential, proprietary, privileged or
otherwise protected by law. The message is intended solely for the named addressee. If received
in error, please destroy and notify the sender. Any use of this email is prohibited when received
in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors, virus, interception
or interference.

________________________________






NOTE: This message may contain information that is confidential, proprietary, privileged or
otherwise protected by law. The message is intended solely for the named addressee. If received
in error, please destroy and notify the sender. Any use of this email is prohibited when received
in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors, virus, interception
or interference.



________________________________






NOTE: This message may contain information that is confidential, proprietary, privileged or
otherwise protected by law. The message is intended solely for the named addressee. If received
in error, please destroy and notify the sender. Any use of this email is prohibited when received
in error. Impetus does not represent, warrant and/or guarantee, that the integrity of this
communication has been maintained nor that the communication is free of errors, virus, interception
or interference.

Mime
View raw message