hadoop-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From mg <userformailingli...@gmail.com>
Subject Re: Namenode fails to start after upgrade from CDH 4.0.1 to 4.1.2
Date Tue, 13 Nov 2012 13:18:50 GMT
Meanwhile we found that the seen_txid files are empty in 4 of 5 
replicated namenode directories.

The edits_inprogress_... files are identical in all 5 dirs with the tx 
id from the one non-empty seen_txid file.

The fsimage files are identical, too.

Otherwise there are differences between every 2 of the 5 dirs as what 
the edits_ files are concerned.

Is it safe to copy the one non-empty seen_txid file over into the other 
4 nn directories?

Cheers,
Martin

On 13.11.2012 12:03, mg wrote:
> Hi,
>
> we just upgraded a cluster from CDH 4.0.1 to 4.1.2 on a number of nodes
> running on Ubuntu 12.04 (Precise).
>
> We first upgraded Cloudera Manager (now 4.1.0), then ran apt-get
> dist-upgrade on all nodes, started CM, checked and updated the
> configuration and attempted to start the cluster.
>
> However, the HDFS NameNode fails to start with the exception appended
> below.
>
> There is sufficient space on all partitions. We do not bind against
> wildcard addresses (at least not yet).
>
> Any ideas? Stacktrace follows.
>
> Cheers,
> Martin
>
> FATAL    org.apache.hadoop.hdfs.server.namenode.NameNode
> Exception in namenode join
> java.lang.NumberFormatException: null
>      at java.lang.Long.parseLong(Long.java:375)
>      at java.lang.Long.valueOf(Long.java:525)
>      at
> org.apache.hadoop.hdfs.util.PersistentLongFile.readFile(PersistentLongFile.java:93)
>
>      at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.readTransactionIdFile(NNStorage.java:425)
>
>      at
> org.apache.hadoop.hdfs.server.namenode.FSImageTransactionalStorageInspector.inspectDirectory(FSImageTransactionalStorageInspector.java:71)
>
>      at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.inspectStorageDirs(NNStorage.java:1039)
>
>      at
> org.apache.hadoop.hdfs.server.namenode.NNStorage.readAndInspectDirs(NNStorage.java:1093)
>
>      at
> org.apache.hadoop.hdfs.server.namenode.FSImage.loadFSImage(FSImage.java:598)
>
>      at
> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:267)
>
>      at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:534)
>
>      at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:424)
>
>      at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:386)
>
>      at
> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:398)
>
>      at
> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:432)
>
>      at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:608)
>      at
> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:589)
>      at
> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1140)
>
>      at
> org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:1204)

Mime
View raw message