hadoop-mapreduce-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sandy Ryza <sandy.r...@cloudera.com>
Subject Re: Any reference for upgrade hadoop from 1.x to 2.2
Date Thu, 05 Dec 2013 17:07:43 GMT
Unfortunately there is no way to see MR1 jobs in the MR2 job history.

-Sandy


On Thu, Dec 5, 2013 at 3:47 AM, Nirmal Kumar <nirmal.kumar@impetus.co.in>wrote:

>  Hi Adam,
>
>
>
> *Apache Hadoop-2.0.6-alpha *has the following issue.
>
>
>
> This issue got fixed in 2.1.0-beta<https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>
>
>
>
> 1.       Hadoop HDFS <https://issues.apache.org/jira/browse/HDFS>
>
> 2.       HDFS-4917 <https://issues.apache.org/jira/browse/HDFS-4917>
>
> *Start-dfs.sh cannot pass the parameters correctly*
>
>
>
>
> https://issues.apache.org/jira/browse/HDFS-4917?jql=project%20%3D%20HDFS%20AND%20text%20~%20upgrade
>
>
>
> I setup *Apache Hadoop **2.1.0-beta
> <https://issues.apache.org/jira/browse/HDFS/fixforversion/12324031>* and
> then were able to run the commands :
>
> ./hadoop-daemon.sh start namenode -upgrade
>
> ./hdfs dfsadmin -finalizeUpgrade
>
>
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner             =
> cloud (auth:SIMPLE)
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup          =
> supergroup
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-05 21:16:44,412 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-05 21:16:44,426 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map INodeMap
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: VM type       =
> 32-bit
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: 1.0% max memory
> = 889 MB
>
> 2013-12-05 21:16:44,908 INFO org.apache.hadoop.util.GSet: capacity      =
> 2^21 = 2097152 entries
>
> 2013-12-05 21:16:44,923 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-05 21:16:44,930 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension     = 30000
>
> 2013-12-05 21:16:44,931 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache on
> namenode is enabled
>
> 2013-12-05 21:16:44,932 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Retry cache will use
> 0.03 of total heap and retry cache entry expiry time is 600000 millis
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: Computing
> capacity for map Namenode Retry Cache
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: VM type       =
> 32-bit
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet:
> 0.029999999329447746% max memory = 889 MB
>
> 2013-12-05 21:16:44,947 INFO org.apache.hadoop.util.GSet: capacity      =
> 2^16 = 65536 entries
>
> 2013-12-05 21:16:45,038 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 8695@Impetus-942.impetus.co.in
>
> 2013-12-05 21:16:45,128 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Using clusterid: CID-4ece2cb2-6159-4836-a428-4f0e324dab13
>
> 2013-12-05 21:16:45,145 INFO
> org.apache.hadoop.hdfs.server.namenode.FileJournalManager: Recovering
> unfinalized segments in
> /home/cloud/hadoop_migration/hadoop-data/name/current
>
> 2013-12-05 21:16:45,166 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrading to sequential
> block IDs. Generation stamp for new blocks set to 1099511628823
>
> 2013-12-05 21:16:45,169 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Loading image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage using no
> compression
>
> 2013-12-05 21:16:45,169 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files = 45
>
> 2013-12-05 21:16:45,203 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Number of files under
> construction = 0
>
> 2013-12-05 21:16:45,204 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage of size 4975
> bytes loaded in 0 seconds.
>
> 2013-12-05 21:16:45,204 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Loaded image for txid 0
> from /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage
>
> 2013-12-05 21:16:45,211 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Reading
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits expecting start
> txid #1
>
> 2013-12-05 21:16:45,211 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Start loading edits file
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits
>
> 2013-12-05 21:16:45,232 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Edits file
> /home/cloud/hadoop_migration/hadoop-data/name/current/edits of size 4 edits
> # 0 loaded in 0 seconds
>
> 2013-12-05 21:16:45,233 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Starting upgrade of image
> directory /home/cloud/hadoop_migration/hadoop-data/name.
>
>    old LV = -41; old CTime = 0.
>
>    new LV = -47; new CTime = 1386258405233
>
> 2013-12-05 21:16:45,241 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Saving image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000
> using no compression
>
> 2013-12-05 21:16:45,321 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Image file
> /home/cloud/hadoop_migration/hadoop-data/name/current/fsimage.ckpt_0000000000000000000
> of size 4923 bytes saved in 0 seconds.
>
> 2013-12-05 21:16:45,365 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector:
> No version file in /home/cloud/hadoop_migration/hadoop-data/name
>
> *2013-12-05 21:16:45,421 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Upgrade of
> /home/cloud/hadoop_migration/hadoop-data/name is complete.*
>
> 2013-12-05 21:16:45,422 INFO
> org.apache.hadoop.hdfs.server.namenode.FSEditLog: Starting log segment at 1
>
> 2013-12-05 21:16:45,741 INFO
> org.apache.hadoop.hdfs.server.namenode.NameCache: initialized with 0
> entries 0 lookups
>
> 2013-12-05 21:16:45,741 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Finished loading
> FSImage in 791 msecs
>
> 2013-12-05 21:16:46,079 INFO org.apache.hadoop.ipc.Server: Starting Socket
> Reader #1 for port 54310
>
> 2013-12-05 21:16:46,113 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Registered
> FSNamesystemState MBean
>
> 2013-12-05 21:16:46,126 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
> construction: 0
>
> 2013-12-05 21:16:46,126 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Number of blocks under
> construction: 0
>
> 2013-12-05 21:16:46,127 INFO org.apache.hadoop.hdfs.StateChange: STATE*
> Safe mode ON.
>
> The reported blocks 0 needs additional 15 blocks to reach the threshold
> 0.9990 of total blocks 15.
>
> Safe mode will be turned off automatically
>
> 2013-12-05 21:16:46,167 INFO org.apache.hadoop.ipc.Server: IPC Server
> Responder: starting
>
> 2013-12-05 21:16:46,176 INFO org.apache.hadoop.ipc.Server: IPC Server
> listener on 54310: starting
>
> 2013-12-05 21:16:46,177 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: NameNode RPC up at:
> localhost/127.0.0.1:54310
>
> 2013-12-05 21:16:46,177 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Starting services
> required for active state
>
> 2013-12-05 21:23:08,461 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Finalizing upgrade for
> storage directory /home/cloud/hadoop_migration/hadoop-data/name.
>
>    cur LV = -47; cur CTime = 1386258405233
>
> *2013-12-05 21:23:08,461 INFO
> org.apache.hadoop.hdfs.server.namenode.FSImage: Finalize upgrade for
> /home/cloud/hadoop_migration/hadoop-data/name is complete.*
>
>
>
> I can now see the existing files in the HDFS that were used by earlier Map
> Reduce jobs(input\output files) using *Apache Hadoop-1.2.0*
>   However, I cannot see the history of those Map Reduce jobs through
> MapReduce JobHistory Server.
>
> Is there some way in which I can see the history of those Map Reduce jobs
> as well?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, December 04, 2013 7:41 PM
> *To:* user@hadoop.apache.org
> *Cc:* rdyer@iastate.edu
>
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Adam,
>
>
>
> I am upgrading from **Apache Hadoop-1.2.0** to **Apache
> Hadoop-2.0.6-alpha**
>
>
>
> I am getting the same exception when using the command: ./hadoop-daemon.sh
> start namenode -upgrade
>
>
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner             =
> cloud (auth:SIMPLE)
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup          =
> supergroup
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-05 00:56:42,312 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-05 00:56:42,317 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-05 00:56:42,784 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-05 00:56:42,789 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension     = 30000
>
> 2013-12-05 00:56:42,840 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 31742@Impetus-942.impetus.co.in
>
> 2013-12-05 00:56:42,911 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-05 00:56:42,912 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-05 00:56:42,912 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-05 00:56:42,913 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
>         at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
>         at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
>         at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-05 00:56:42,918 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-05 00:56:42,922 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> I also referred
> https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/Federation.html
>
> *Upgrading from older release to 0.23 and configuring federation*
>
> Older releases supported a single Namenode. Here are the steps enable
> federation:
>
> Step 1: Upgrade the cluster to newer release. During upgrade you can
> provide a ClusterID as follows:
>
> > $HADOOP_PREFIX_HOME/bin/hdfs start namenode --config $HADOOP_CONF_DIR
> -upgrade -clusterId <cluster_ID>
>
> If ClusterID is not provided, it is auto generated.
>
> But Getting:
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config
> /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop  -upgrade
> -clusterId testclusterid1111
>
> Error: Could not find or load main class start
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$ bin/hdfs start namenode --config
> /home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop  -upgrade
> -clusterId
>
> Error: Could not find or load main class start
>
> [cloud@Impetus-942 hadoop-2.0.6-alpha]$
>
>
>
> I have the following environment variables set :
>
>
>
> YARN_CLASSPATH=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin/yarn
>
> HADOOP_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_PREFIX=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_HDFS_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_COMMON_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> HADOOP_YARN_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
> JAVA_HOME=/usr/lib/jvm/jdk1.7.0_45
>
> HADOOP_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
>
> YARN_CONF_DIR=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/etc/hadoop
>
> HADOOP_MAPRED_HOME=/home/cloud/hadoop_migration/hadoop-2.0.6-alpha
>
>
> PATH=/usr/lib64/qt-3.3/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/cloud/bin:/usr/lib/jvm/jdk1.7.0_45/bin:/home/cloud/hadoop_migration/hadoop-2.0.6-alpha/bin:/home/cloud/Manisha/maven/apache-maven-3.1.1/bin
>
>
>
> Regards,
>
> -Nirmal
>
>
>
> *From:* Adam Kawa [mailto:kawa.adam@gmail.com <kawa.adam@gmail.com>]
> *Sent:* Tuesday, December 03, 2013 11:58 PM
> *To:* user@hadoop.apache.org
> *Cc:* rdyer@iastate.edu
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> @Nirmal,
>
>
>
> And later, you need to make a decision to finalize the upgrade or rollback.
>
>
>
> 2013/12/3 Adam Kawa <kawa.adam@gmail.com>
>
>  @Nirmal,
>
>
>
> You need to run NameNode with upgrade option e.g.
>
> $ /usr/lib/hadoop/sbin/hadoop-daemon.sh start namenode -upgrade
>
>
>
> 2013/12/3 Nirmal Kumar <nirmal.kumar@impetus.co.in>
>
>  Hi All,
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
>
>
> I did the following:
>
> ·         Installed Apache Hadoop-1.2.0
>
> ·         Ran word count sample MR jobs. The jobs executed successfully.
>
> ·         I stop all the services in Apache Hadoop-1.2.0 and then was
> able to start all services again.
>
> ·         The previous submitted jobs are visible after the stop/start in
> the job tracker url.
>
>
>
> Next I installed Apache Hadoop-2.0.6-alpha alongside.
>
> I used the SAME data directory locations that were in Apache Hadoop-1.2.0
> in the configuration files namely:
>
> core-site.xml
>
> ----------------
>
> $hadoop.tmp.dir
> /home/cloud/hadoop_migration/hadoop-data/tempdir
>
>
>
> hdfs-site.xml
>
> -----------------
>
> $dfs.data.dir
> /home/cloud/hadoop_migration/hadoop-data/data
>
> $dfs.name.dir
> /home/cloud/hadoop_migration/hadoop-data/name
>
>
>
> I am UNABLE to start the NameNode from Apache Hadoop-2.0.6-alpha
> installation I am getting the error:
>
>
>
> 2013-12-03 18:28:23,941 INFO
> org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
> hadoop-metrics2.properties
>
> 2013-12-03 18:28:24,080 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
> period at 10 second(s).
>
> 2013-12-03 18:28:24,081 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> started
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one image storage
> directory (dfs.namenode.name.dir) configured. Beware of dataloss due to
> lack of redundant storage directories!
>
> 2013-12-03 18:28:24,576 WARN
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Only one namespace
> edits storage directory (dfs.namenode.edits.dir) configured. Beware of
> dataloss due to lack of redundant storage directories!
>
> 2013-12-03 18:28:24,744 INFO org.apache.hadoop.util.HostsFileReader:
> Refreshing hosts (include/exclude) list
>
> 2013-12-03 18:28:24,749 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager:
> dfs.block.invalidate.limit=1000
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> dfs.block.access.token.enable=false
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> defaultReplication         = 1
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplication             = 512
>
> 2013-12-03 18:28:24,762 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> minReplication             = 1
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> maxReplicationStreams      = 2
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> shouldCheckForEnoughRacks  = false
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> replicationRecheckInterval = 3000
>
> 2013-12-03 18:28:24,763 INFO
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager:
> encryptDataTransfer        = false
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: fsOwner             =
> cloud (auth:SIMPLE)
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: supergroup          =
> supergroup
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: isPermissionEnabled =
> true
>
> 2013-12-03 18:28:24,771 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: HA Enabled: false
>
> 2013-12-03 18:28:24,776 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem: Append Enabled: true
>
> 2013-12-03 18:28:25,230 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: Caching file names
> occuring more than 10 times
>
> 2013-12-03 18:28:25,243 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.threshold-pct = 0.9990000128746033
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.min.datanodes = 0
>
> 2013-12-03 18:28:25,244 INFO
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem:
> dfs.namenode.safemode.extension     = 30000
>
> 2013-12-03 18:28:25,288 INFO org.apache.hadoop.hdfs.server.common.Storage:
> Lock on /home/cloud/hadoop_migration/hadoop-data/name/in_use.lock acquired
> by nodename 21371@Impetus-942.impetus.co.in
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Stopping NameNode
> metrics system...
>
> 2013-12-03 18:28:25,462 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> stopped.
>
> 2013-12-03 18:28:25,473 INFO
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: NameNode metrics system
> shutdown complete.
>
> 2013-12-03 18:28:25,474 FATAL
> org.apache.hadoop.hdfs.server.namenode.NameNode: Exception in namenode join
>
> org.apache.hadoop.hdfs.server.common.IncorrectVersionException: Unexpected
> version of storage directory /home/cloud/hadoop_migration/hadoop-data/name.
> Reported: -41. Expecting = -40.
>
>         at
> org.apache.hadoop.hdfs.server.common.Storage.setLayoutVersion(Storage.java:1079)
>
>         at
> org.apache.hadoop.hdfs.server.common.Storage.setFieldsFromProperties(Storage.java:887)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.setFieldsFromProperties(NNStorage.java:583)
>
>         at
> org.apache.hadoop.hdfs.server.common.Storage.readProperties(Storage.java:918)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:304)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:200)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:627)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:469)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:403)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:437)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:609)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:594)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1169)
>
>         at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1235)
>
> 2013-12-03 18:28:25,479 INFO org.apache.hadoop.util.ExitUtil: Exiting with
> status 1
>
> 2013-12-03 18:28:25,481 INFO
> org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
>
> /************************************************************
>
> SHUTDOWN_MSG: Shutting down NameNode at
> Impetus-942.impetus.co.in/192.168.41.106
>
> ************************************************************/
>
>
>
> Independently both the installations(Apache Hadoop-1.2.0 and Apache
> Hadoop-2.0.6-alpha) are working for me. I am able to run the MR jobs on
> both the installations independently though.
>
> But I aim to migrate the data and jobs submitted from Apache Hadoop-1.2.0
> to Apache Hadoop-2.0.6-alpha.
>
>
>
> Is there any HDFS compatibility issues from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha?
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Nirmal Kumar
> *Sent:* Wednesday, November 27, 2013 2:56 PM
> *To:* user@hadoop.apache.org; rdyer@iastate.edu
> *Subject:* RE: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Hello Sandy,
>
>
>
> The post was useful and gave an insight of the migration.
>
>
>
> I am doing a test migration from Apache Hadoop-1.2.0 to Apache
> Hadoop-2.0.6-alpha on a single node environment.
>
> I am having the Apache Hadoop-1.2.0 up and running.
>
>
>
> Can you please let me know the steps that one should follow for the
> migration?
>
> I am thinking of doing something like:
>
> ·         Install Apache Hadoop-2.0.6-alpha alongside the existing Apache
> Hadoop-1.2.0
>
> ·         Use the same HDFS locations
>
> ·         Change the various required configuration files
>
> ·         Stop Apache Hadoop-1.2.0  and start Apache Hadoop-2.0.6-alpha
>
> ·         Verify all the services are running
>
> ·         Test via mapreduce (test MRv1 and MRv2 examples)
>
> ·         Check Web UI Console and verify the MRv1 and MRv2 jobs
>
>
>
> These above steps needs to be performed on all the nodes in a cluster
> environment.
>
>
>
> The translation table mapping old configuration to new would be definitely
> *very* useful.
>
>
>
> Also the existing Hadoop ecosystem components needs to be considered:
>
> ·         Hive Scripts
>
> ·         Pig Scripts
>
> ·         Oozie Workflows
>
> Their compatibility and version support would need to be checked.
>
>
>
> Also thinking of any risks like Data Loss, others that one should keep in
> mind.
>
>
>
> Also I found:
> http://strataconf.com/strata2014/public/schedule/detail/32247
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Robert Dyer [mailto:psybers@gmail.com <psybers@gmail.com>]
> *Sent:* Friday, November 22, 2013 9:08 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> Thanks Sandy! These seem helpful!
>
>
>
> "MapReduce cluster configuration options have been split into YARN
> configuration options, which go in yarn-site.xml; and MapReduce
> configuration options, which go in mapred-site.xml. Many have been given
> new names to reflect the shift. ... *We’ll follow up with a full
> translation table in a future post.*"
>
>
> This type of translation table mapping old configuration to new would be
> *very* useful!
>
>
>
> - Robert
>
> On Fri, Nov 22, 2013 at 2:15 AM, Sandy Ryza <sandy.ryza@cloudera.com>
> wrote:
>
>  For MapReduce and YARN, we recently published a couple blog posts on
> migrating:
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-users/
>
>
> http://blog.cloudera.com/blog/2013/11/migrating-to-mapreduce-2-on-yarn-for-operators/
>
>
>
> hope that helps,
>
> Sandy
>
>
>
> On Fri, Nov 22, 2013 at 3:03 AM, Nirmal Kumar <nirmal.kumar@impetus.co.in>
> wrote:
>
>  Hi All,
>
>
>
> I am also looking into migrating\upgrading from Apache Hadoop 1.x to
> Apache Hadoop 2.x.
>
> I didn’t find any doc\guide\blogs for the same.
>
> Although there are guides\docs for the CDH and HDP migration\upgradation
> from Hadoop 1.x to Hadoop 2.x
>
> Would referring those be of some use?
>
>
>
> I am looking for similar guides\docs for Apache Hadoop 1.x to Apache
> Hadoop 2.x.
>
>
>
> I found something on slideshare though. Not sure how much useful that is
> going to be. I still need to verify that.
>
> http://www.slideshare.net/mikejf12/an-example-apache-hadoop-yarn-upgrade
>
>
>
> Any suggestions\comments will be of great help.
>
>
>
> Thanks,
>
> -Nirmal
>
>
>
> *From:* Jilal Oussama [mailto:jilal.oussama@gmail.com]
> *Sent:* Friday, November 08, 2013 9:13 PM
> *To:* user@hadoop.apache.org
> *Subject:* Re: Any reference for upgrade hadoop from 1.x to 2.2
>
>
>
> I am looking for the same thing if anyone can point us to a good direction
> please.
>
> Thank you.
>
> (Currently running Hadoop 1.2.1)
>
>
>
> 2013/11/1 YouPeng Yang <yypvsxf19870706@gmail.com>
>
>   Hi users
>
>    Are there any reference docs to introduce how to upgrade hadoop from
> 1.x to 2.2.
>
>
>
> Regards
>
>
>
>
>  ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
>  ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
>  ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
>
>
>
>
>
>  ------------------------------
>
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>
> ------------------------------
>
>
>
>
>
>
> NOTE: This message may contain information that is confidential,
> proprietary, privileged or otherwise protected by law. The message is
> intended solely for the named addressee. If received in error, please
> destroy and notify the sender. Any use of this email is prohibited when
> received in error. Impetus does not represent, warrant and/or guarantee,
> that the integrity of this communication has been maintained nor that the
> communication is free of errors, virus, interception or interference.
>

Mime
View raw message